Reducing initramfs size and speed up the generation

[Dimitri John Ledkov]

On Fri, 21 Jul 2023 at 17:59, Steve Langasek <steve.langasek at ubuntu.com> wrote:
>
> On Tue, Jul 11, 2023 at 01:28:28AM +0200, Benjamin Drung wrote:
> > > > Okay. It works now. The not-compressed cpio archive must not be the last
> > > > one. So the order is now:
> > > >
> > > > * AMD/Intel microcode cpio archive (on amd64)
> > > > * compressed kernel modules / firmware (not compressed)
> > > > * main cpio archive (compressed)
> > > >
> > > > I'll really stop now. For a first comparison, the firmware files need to
> > > > be converted correctly. There are symlinks in /lib/firmware. So running
> > > > following was not correct/enough:
> > > >
> > > > find /lib/firmware -name '*.bin' | while read -r fw; do
> > > >     sudo zstd -19 -z -o "${fw}.zst" "$fw"
> > > >     sudo rm "$fw"
> > > > done
> > > >
> > > > If you want to help, hand me a correct conversion script.
>
> > > Some filenames in /lib/firmware contain spaces. There are many more
> > > files than .bin ones. A number of the files are readmes.
>
> > > Recent changes in linux-firmware.git add "install-xz" and "install-zstd"
> > > targets to make than will do what you want I guess. I haven't checked
> > > that this was actually merged; it was discussed at least on 2023-03-01
> > > on the mailing-list. It's probably the best path forward in any case.
>
> > > In case it isn't merged, you can have a look at
> > > https://lore.kernel.org/linux-firmware/20230301-fixes-and-compression-v2-0-e2b71974e842@gmail.com/T/
>
> > I agree. The conversion script was just for a quick way to test. The
> > clean approach would be to patch linux-firmware to produce two
> > additional binary packages: linux-firmware-zst and linux-firmware-xz.
>
> We need to find an upgrade path here that doesn't involve having to have
> multiple linux-firmware binary packages with different compression types.
> Ending up with three copies of the firmware in /lib on disk because of
> different dependencies is not an improvement!
>
> I understand the reason for being concerned about keeping uncompressed
> firmware available is that not all kernels have support for compressed
> firmware.  However we should work out a path that lets us switch to
> compressed firmware on releases where we know it's supported.  What kernel
> version introduced support for this?
>

XZ compressed firmware load was added in v5.3 and is turned on in the
focal GA v5.4 kernel.

ZSTD compressed firmware load was added in v5.19 and is available in
Jammy HWE kernels, kinetic and later.

initramfs-tools in Jammy has unoptimized support for xz compressed firmware.

Thus it is safe for linux-firmware to switch to ZSTD compressed
firmware straight away in mantic.

If it is a concern that v5.15 jammy kernel may potentially be used
after partial / incomplete upgrade to Mantic, we can opt into using XZ
compressed firmware or I can backport ZSTD firmware support to jammy
GA kernel - it is a trivial patch (given that ZSTD itself is otherwise
available and used by the jammy kernel).

-- 
okurrr,

Dimitri