Reducing initramfs size and speed up the generation

[Benjamin Drung]

On Sat, 2023-07-08 at 03:49 +0200, Benjamin Drung wrote:
> On Sat, 2023-07-08 at 01:25 +0100, Dimitri John Ledkov wrote:
> > On Sat, 8 Jul 2023 at 01:19, Benjamin Drung <bdrung at ubuntu.com> wrote:
> > > 
> > > Hi all,
> > > 
> > > a year ago we changed the default compression and level for the
> > > initramfs to zstd -1. This fixed the very slow creation times on
> > > development boards (see bug #1958148), but that leads to bigger
> > > initramfs sizes that triggered other bugs (like bug #1842320).
> > > Big initramfs sizes can also fill up small sized /boot partitions easily
> > > (grooming the 850 initramfs-tools bugs revealed several such reports).
> > > 
> > > Using xz -9 would give very good compression, but it takes very long
> > > (especially on slow development boards) and a lot of memory (good luck
> > > on Raspberry Pis with small memory like Pi Zeros).
> > > 
> > > I propose following approach to address the drawback: Create cpio
> > > archives (compressed with xz -9) for the kernel modules and firmware
> > > files when building the kernel/firmware Debian package. Then ship those
> > > cpio archives in the package (or in a separate binary package). Then the
> > > CPU load it put on the builders. The cpio archives would contain the
> > > modules for MODULES=most.
> > > 
> > > mkinitramfs will then look for those cpio archives and uses those in
> > > case they are present. Such a initramfs would look like this:
> > > 
> > > * AMD/Intel microcode cpio archive (on amd64)
> > > * main cpio archive compressed with zstd -1
> > > * kernel modules from the Debian package compressed with xz -9
> > > * firmware files from the Debian package compressed with xz -9
> > > 
> > 
> > Majority of our instances boot without initrd, and there too they
> > don't load most of the modules.
> > Creating xz -9 compressed archive of all modules, still pays the
> > penalty to decompress most of them, and then not modprobe them.
> > I was hoping to achieve a similar in spirit approach, but didn't quite
> > have the time to implement is:
> > 
> > 1) change linux-modules and linux-firmware to ship .ko.zst
> > firmware.bin.zst compressed with zstd -19 at .deb build time
> > 2) this saves install size of the packages, with only slightly
> > increased download size
> > 3) modify initramfs-tools to include compressed files into a separate
> > initrd, which is not compressed (i.e. exclude .zst files from the
> > default main compressed cpio archive, and append them in the second
> > main cpio archive that is uncompressed)
> > 4) this should achieve quick initrd creation, which will be smaller in
> > size that current status, and will boot faster as it will only
> > decompress modules/firmware it actually needs at boot
> > 
> > For experimentation locally, you can recompress .ko with zstd in place
> > in /lib/modules/; and rerun depmod. To then test initramfs-tools
> > changes that skip over .zst compressed files and add them as is in an
> > uncompressed appended cpio.
> 
> That is a very good idea. I created a draft for point 3 in [2]. It moves
> the compressed files into a separate directory and creates a separate
> cpio archive for that directory without compressing it:
> 
> * AMD/Intel microcode cpio archive (on amd64)
> * main cpio archive (compressed)
> * compressed kernel modules / firmware (not compressed)
> 
> Sadly this does not work (yet). cpio complains with "premature end of
> archive" when looking at it and the kernel fails to extract the last
> cpio part. I am heading to bed now leaving that bug for another day.
> 
> [2] https://code.launchpad.net/~bdrung/ubuntu/+source/initramfs-tools/+git/initramfs-tools/+ref/ubuntu/compressed

Okay. It works now. The not-compressed cpio archive must not be the last
one. So the order is now:

* AMD/Intel microcode cpio archive (on amd64)
* compressed kernel modules / firmware (not compressed)
* main cpio archive (compressed)

I'll really stop now. For a first comparison, the firmware files need to
be converted correctly. There are symlinks in /lib/firmware. So running
following was not correct/enough:

find /lib/firmware -name '*.bin' | while read -r fw; do
    sudo zstd -19 -z -o "${fw}.zst" "$fw"
    sudo rm "$fw"
done

If you want to help, hand me a correct conversion script.

-- 
Benjamin Drung
Debian & Ubuntu Developer