Revisiting default initramfs compression

Wed Mar 9 08:46:19 UTC 2022

On Wed, Mar 09, 2022 at 02:10:57PM +1300, Michael Hudson-Doyle wrote:
> On Thu, 9 Dec 2021 at 06:13, Julian Andres Klode <julian.klode at canonical.com>
> wrote:
> 
> > Hi all,
> >
> > some time ago, the default compressor for initramfs was changed
> > from lz4 -9 to zstd -19. This caused significant problems:
> >
> 
> Exactly three months later... we still haven't taken any action on this.
> Time to do something!
> 
> I have a few questions below but tl;dr: unless there are immediate
> objections, I'm going to make a change to initramfs-tools to allow the
> compression level to be configured and set the default to 12 for zstd.

So xypron had a patch to change the default level to 9 for sponsoring
out for a couple of months now (no idea how that level came up).

We pushed back on that as it does not account for low-memory systems
which we need to take care of as well.

We then postponed any implementation to after a discussion in
Frankfurt.

I think the summary from the Frankfurt discussion was:

- lz4 -1 is the right choice for low-memory systems
- if you have more memory, zstd -1 becomes the best choice
- pigz is outperforming both a bunch of times

But that's really for waveform to share.

As a rule we said that we want compression to not take more than
10% of available memory, so that we don't eat into whatever
applications you run too much.

We also need to compare sizes to what we have now and adjust the
size of /boot accordingly.

> > - it is very slow
> > - it uses a lot of memory
> >
> > The former is a problem for everyone, the latter means that
> > zstd just crashes on a Pi Zero.
> >
> > This is an analysis of what we have in terms of time spent,
> > memory spent, and file size achieved, and where we can
> > go from here.
> >
> > # Comparison of different compression levels
> >
> > ## Desktop (ThinkPad T480s, jammy)
> >
> > level    usertime   elapsed memory fileSize
> > lz4         9.65s    11.09s    12M      64M
> > -1          5.69s         6.99s    24M      57M
> > -6         12.59s     8.58s    99M      47M
> > -12        19.85s    10.82s   249M      41M
> > -19        71.29s    26.95s   519M      35M
> >
> > -> I believe that somewhere around -12 is a decent
> >    compromise between size and speed.
> >
> 
> I would agree that it's certainly better than -19.
> 
> 
> > ## Pi 4 (arm64, focal)
> >
> > Times have been measured for mkinitramfs only. A full
> > update-initramfs call spends much more time copying
> > some firmware bits to boot partition with flash-kernel
> >
> > level    usertime   elapsed memory fileSize
> > lz4        21.10s    64.85s    21M      29M
> > -1         13.73s    44.55s    21M      27M
> > -6         26.07s    49.09s    91M      24M
> > -12        48.18s    54.67s   203M      22M
> > -19       130.07s    92.80s   350M      20M
> >
> > -> 6 is essentially free if the Pi 4 is idle. Nice.
> > -> -6 is still 20% of total RAM of a Pi 0
> >
> 
> Are people really going to run an arm64 userland on a Pi 0?
> 
> Any "real" solution for pi 0 has to involve doing at least the bulk of the
> compression not on the pi, there's no real way around that. Which is
> something we should do, but realistically it's not happening for jammy.

You are right that armhf might be more useful there.
> 
> 
> > -> There's no meaningful difference between -6 and -12
> >    in terms of time elapsed. -6 uses 116% CPU, -12 uses
> >    145% CPU.
> >
> > ## Adaptive compression
> >
> > zstd also supports adaptive compression, compressing as hard as
> > it can while not impacting I/O speed. So hardware with slow I/O
> > like a Pi would compress harder to avoid idling.
> >
> > This is somewhat suboptimal with recent update-initramfs though,
> > as it first writes the cpio archive to the disk and then compresses
> > it rather than doing it in a pipe where that would be more
> > meaningful.
> >
> > Question: Does zstd --adapt adapt to memory available?
> >
> 
> While attractive, this does feel a bit risky. We want to be able to make
> reasonable predictions about the size of the initrd.
> 
> 
> > # Way(s) forward
> >
> > To remedy the issue the proposal is to build with
> >
> > - zstd -1 on hardware with 512 MB or less memory
> > - zstd between -1 and -19 on other hardware
> > - zstd -19 during image building
> >
> 
> I think this broadly makes sense. I'll notice that currently
> initramfs-tools doesn't allow tuning the compression level at all :/
> Probably fairly routine to add support for that though.
> 
> 
> > Finding the right level between -1 and -19 is hard. The more
> > cores you have, the less penalty you pay for higher level.
> >
> 
> You suggested -12 above. How about we try that to start with?
> 
> Do you have an idea how to detect "512 MB or less memory"?

Parse /proc/meminfo / free output.
>
> > Going for adaptive compression would remove the guess work, but
> > will result in larger images on faster machines. Maybe that's
> > fine, though - they probably have more space on /boot anyway?
> >
> > If we want to aim for 5% of total memory, we should probably
> > aim for something like:
> >
> > -1  on <= 512MB
> > -6  on <= 2 GB (or --adapt=min=1,max=6)
> > -12 on the rest (or --adapt=min=12)
> >
> > It's clear that in all cases, zstd -1 is at least better than the
> > lz4 -9 we used before; both in terms of space used, and time spent.
> >
> 
> Yeah that's interesting.

During the discussion in Frankfurt, it was argued some
architectures/boards really need lz4 -1 (and that lz4 -9 makes
no sense whatsoever).

-- 
debian developer - deb.li/jak | jak-linux.org - free software dev
ubuntu core developer                              i speak de, en