Reducing initramfs size and speed up the generation

[Michael Hudson-Doyle]

On Tue, 11 Jul 2023 at 11:17, Benjamin Drung <bdrung at ubuntu.com> wrote:

> On Sun, 2023-07-09 at 15:29 -0700, Steve Langasek wrote:
> > On Sun, Jul 09, 2023 at 04:28:42AM +0200, Heinrich Schuchardt wrote:
> > > > > Benjamin Drung <bdrung at ubuntu.com> schrieb am Sa., 8. Juli 2023,
> 02:19:
> >
> > > > > > Hi all,
> >
> > > > > > a year ago we changed the default compression and level for the
> > > > > > initramfs to zstd -1. This fixed the very slow creation times on
> > > > > > development boards (see bug #1958148), but that leads to bigger
> > > > > > initramfs sizes that triggered other bugs (like bug #1842320).
> > > > > > Big initramfs sizes can also fill up small sized /boot
> partitions easily
> > > > > > (grooming the 850 initramfs-tools bugs revealed several such
> reports).
> >
> > > > > > Using xz -9 would give very good compression, but it takes very
> long
> > > > > > (especially on slow development boards) and a lot of memory
> (good luck
> > > > > > on Raspberry Pis with small memory like Pi Zeros).
> >
> > > > > > I propose following approach to address the drawback: Create cpio
> > > > > > archives (compressed with xz -9) for the kernel modules and
> firmware
> > > > > > files when building the kernel/firmware Debian package. Then
> ship those
> > > > > > cpio archives in the package (or in a separate binary package).
> Then the
> > > > > > CPU load it put on the builders. The cpio archives would contain
> the
> > > > > > modules for MODULES=most.
> >
> > > > > > mkinitramfs will then look for those cpio archives and uses
> those in
> > > > > > case they are present. Such a initramfs would look like this:
> >
> > > > > > * AMD/Intel microcode cpio archive (on amd64)
> > > > > > * main cpio archive compressed with zstd -1
> > > > > > * kernel modules from the Debian package compressed with xz -9
> > > > > > * firmware files from the Debian package compressed with xz -9
> >
> > > > > > After working on initramfs-tools as part my day job, my fingers
> were
> > > > > > itching and I had to create a quick and dirty draft in my free
> night
> > > > > > time. You can find the result of the last two hours in [1]. This
> draft
> > > > > > has a mkinitramfs-kernel script that creates a cpio archive
> containing
> > > > > > the kernel modules and firmware (that needs to be split later
> on).
> >
> > > > > > The lunar test result on my AMD Ryzen 7 5700G look promising:
> Building
> > > > > > 6.2.0-24-generic-modules-most.cpio.xz takes around 90 seconds
> and is
> > > > > > 54.9 MiB in size. Creating the initramfs speeds up from around
> 8.7
> > > > > > seconds to 3.5 seconds (saves 60 %). The size reduces from 133.1
> MiB to
> > > > > > 80.7 MiB (saves 39.4 %). So the boot needs 52.4 MiB less, but
> > > > > > /lib/modules need 54.9 MiB for the cpio archive.
> >
> > > > > > The drawback is that building the kernel would take longer, the
> package
> > > > > > takes more space on the archive and mirrors, and downloading
> them could
> > > > > > take longer on slow connections.
> >
> > > > > > Implementing my proposal would be relative easy for
> initramfs-tools, but
> > > > > > would mean some work for the kernel team.
> >
> > > > > > What do you think?
> >
> > > Will the user still be able to add further modules and will machine
> specific
> > > configuration files (e.g. for booting from iSCSI) still be included
> into the
> > > initrd?
> >
> > I think a robust implementation of this on the initramfs-tools side looks
> > like:
> >
> >  - identify all the contents that belong in the initramfs
> >  - among those contents, find all zstd-compressed files, if any, and
> store
> >    them in an uncompressed initramfs
> >  - put the rest of the contents in a compressed initramfs
> >
> > This would be compatible with kernel packages whether they are changed to
> > ship zstd-compressed modules or not and allow for a smooth transition.
>
> This is exactly what I tried to implement over the weekend. The result
> looked promising. The creation time and initramfs size decreased by a
> few percent. Booting that initramfs failed and today I figured out why.
> Now I have a new draft version that works, but is slower to compress,
> because it moves the compressed files around after running depmod.
> Fiddling with the list of files passed to cpio should give back the
> speed, but that is a task for later.
>
> So without further ado here is the result so far:
>

These look like excellent results, and this approach makes a lot of sense
to me. I've wanted to see this improved for ages, so I'm happy to see
someone actually taking steps to make this happen! Spending builder time on
maximum compression vs doing it on every Ubuntu system in existence is
surely better for everyone.

Would the idea be that linux-modules-$VER-generic would contain only the
compressed modules?

I was wondering if it make sense to construct a zstd dictionary for
compressing kernel modules but I didn't realize they need to be available
at decompression time, I'm not sure the kernel would support that.

Cheers,
mwh
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ubuntu.com/archives/ubuntu-devel/attachments/20230711/c1e2f7c9/attachment-0001.html>