20.04 on RPi CM4: Rebuild kernel with PREEMPT_RT patch?

[High Noon]

So I have Ubuntu 20.04 64-bit on a Raspberry Pi Compute Module 4
(CM4).

I need to rebuild the kernel with a PREEMPT_RT patch from [1].

In fact, I built a kernel but something is going wrong: At boot, the
new kernel is unable to detect the CM4's eMMC and gets stuck in an
endless retry loop.

I don't know if the problem is in the kernel itself or the many files
in /boot or /boot/firmware.

This is what I did so far:

The stock kernel is:
$ uname -a
Linux ubuntu 5.4.0-1042-raspi #46-Ubuntu SMP PREEMPT Fri Jul 30 00:35:40 UTC 2021 aarch64 aarch64 aarch64 GNU/Linux

When building a PREEMPT_RT kernel, the kernel source version and
PREEMPT_RT patch version must match. I selected kernel and patch
version 5.4.138; see [2].

I built the new kernel on the RPi CM4:

$ sudo apt-get build-dep linux
$ sudo apt-get install build-essential git libncurses-dev flex bison openssl libssl-dev dkms libelf-dev libudev-dev libpci-dev libiberty-dev autoconf fakeroot
$ mkdir ~/kernel
$ cd ~/kernel
$ wget https://cdn.kernel.org/pub/linux/kernel/v5.x/linux-5.4.138.tar.gz
$ tar -xzf linux-5.4.138.tar.gz
$ mv linux-5.4.138 linux-5.4.138-rt62
$ wget http://cdn.kernel.org/pub/linux/kernel/projects/rt/5.4/patch-5.4.138-rt62.patch.gz
$ gunzip patch-5.4.138-rt62.patch.gz
$ cd linux-5.4.138-rt62
$ patch -p1 < ../patch-5.4.138-rt62.patch
$ cp /boot/config-5.4.0-1042-raspi .config
$ yes '' | make oldconfig
$ make menuconfig

In Kconfig (make menuconfig) I enabled these options:
CONFIG_PREEMPT_RT
CONFIG_HIGH_RES_TIMERS
CONFIG_NO_HZ_FULL
CONFIG_HZ_1000
CPU_FREQ_DEFAULT_GOV_PERFORMANCE

And I replaced the contents of CONFIG_SYSTEM_TRUSTED_KEYS with an
empty string (it was "debian/canonical-certs.pem").

$ time make -j `nproc` deb-pkg

This produced four .deb files:

linux-image-5.4.138-rt62_5.4.138-rt62-1_arm64.deb
linux-image-5.4.138-rt62_5.4.138-rt62-1_arm64-dbg.deb
linux-headers-5.4.138-rt62_5.4.138-rt62-1_arm64.deb
linux-libc-dev_5.4.138-rt62-1_arm64.deb

I moved the -dbg one elsewhere and installed the remaining .deb files:

$ sudo dpkg -i *.deb

Now I encountered these problems:

(1) linux-image-5.4.138-rt62_5.4.138-rt62-1_arm64.deb failed to
install completely with an error about post-install processing. Sorry
I no longer have the exact error string.

(2) Although dpkg copied the kernel and initrd image to /boot, it did
not update the /boot/vmlinuz and /boot/initrd.img symlinks to these.

So I updated the symlinks manually and tried to reboot, but it booted
the original kernel, not the one I built.

Upon investigation, it appears that uboot doesn't care about the files
and symlinks in /boot. Rather, it loads a separate copy at
/boot/firmware/vmlinuz and /boot/firmware/initrd.img. So I manually
copied the newly built kernel and image here and tried booting. Then
I ran into problem #3:

(3) It now boots the newly built kernel, but the kernel gets stuck in
an endless loop looking for the eMMC. Yes, my CM4 has a eMMC.

I'll get you the exact error text if it's important, but I'm wondering
if I messed up something much more fundamental.

I will say that I referred to many tutorials, blogs, StackOverflow
questions, etc., to figure out how to get this far, but unfortunately
the steps and solutions I found seem not applicable to this system, or
I'm missing a crucial step somewhere.

Some of the sites I referred are listed below in [3] through [8].

Thoughts, help, advice are appreciated.

Warm regards,
high12noon blog

References:

[1] https://wiki.linuxfoundation.org/realtime/preempt_rt_versions

[2] Announcement of the 5.4.138-rt62 patch:
http://lkml.iu.edu/hypermail/linux/kernel/2108.1/00980.html

[3] https://www.raspberrypi.org/documentation/computers/linux_kernel.html

[4] https://chenna.me/blog/2020/02/23/how-to-setup-preempt-rt-on-ubuntu-18-04/
(yes I know it's for 18.04)

[5] http://robskelly.com/2020/10/14/raspberry-pi-4-with-64-bit-os-and-preempt_rt/
(this one seems wrong; the `uname -a` output shown at the end shows
the regular PREEMPT kernel is running, not PREEMPT_RT)

[6] https://www.fdmobileinventions.com/blog/2019/01/18/cross-compiling-the-linux-kernel-for-raspberry-pi/
(this one is for 32-bit kernels; I used the correct toolchain for 64-bit)

[7] https://www.raspberrypi.org/documentation/computers/linux_kernel.html#building-the-kernel
(the install instructions seem off: it seems Raspberry Pi OS's /boot
directory layout is very different than Ubuntu's)

[8] https://stackoverflow.com/questions/64893460/cross-compiling-kernel-for-raspberry-pi-4
(Used for reference but didn't end up cross-compiling)