Revisiting default initramfs compression

Thu Dec 9 03:47:13 UTC 2021

On Wed, 8 Dec 2021, 17:13 Julian Andres Klode, <julian.klode at canonical.com>
wrote:

> Hi all,
>
> some time ago, the default compressor for initramfs was changed
> from lz4 -9 to zstd -19. This caused significant problems:
>
> - it is very slow
> - it uses a lot of memory
>
> The former is a problem for everyone, the latter means that
> zstd just crashes on a Pi Zero.
>

When zstd -19 was chosen it was for the fastest bootspeed.

Overall at all compression levels zstd decompression CPU time is faster
than bootloader IO time. Meaning any gains in compression ratio improve
boot speed, even marginal between the highest levels. (One may argue that
marginal speed up will never be recouped because one will never reboot
enough times, but so far we have been putting high value on fast cold boot
speed).

Compressed initrd that contains compressed kernel modules and compressed
firmware is larger in size, leading to slower boot speed than current
implementation of compressing cpio of uncompressed files. (Nested
compression is bad). Thus in jammy initramfs-tools was updated to
decompress firmware & modules if they are stored compressed on disk prior
to including them in the initrd.

Failure to create new initrd is obviously critical and must be avoided.
Thus if zstd -19 is going to fail, it should fallback to something else.

Are you observing "it is very slow" from the point of view of stable
release or devel release?

In devel series one may be installing updates every 6 hours, with each one
of them triggering initramfs update, which gets overridden without ever
booting the initrd that was generated. Whereas stable updates do not
usually trigger initramfs updates as often, meaning it happens upon
security & kernel updates circa once every three weeks.

Do we want to slow down stable series boot speed, due to desire to shorten
the time to apply updates?

In the past we have also discussed forking initramfs update into a systemd
unit which one can limit resources too. Especially since it may not be
critical to complete that as part of the apt transaction, or need to
complete it successfully at all if previous initrd for a given kernel abi
exists or when instance boots without initrd anyway.

The installation of updates should be routine and a background task, rather
than something we optimise at the expense of reboot & boot performance.

We'd need to look at the whole cycle again of we want to optimize for
"discover, install, and reboot into all updates", which so far has not been
the overall KPI goal.

This is an analysis of what we have in terms of time spent,
> memory spent, and file size achieved, and where we can
> go from here.
>
> # Comparison of different compression levels
>
> ## Desktop (ThinkPad T480s, jammy)
>
> level    usertime   elapsed memory fileSize
> lz4         9.65s    11.09s    12M      64M
> -1          5.69s         6.99s    24M      57M
> -6         12.59s     8.58s    99M      47M
> -12        19.85s    10.82s   249M      41M
> -19        71.29s    26.95s   519M      35M
>
> -> I believe that somewhere around -12 is a decent
>    compromise between size and speed.
>
> ## Pi 4 (arm64, focal)
>
> Times have been measured for mkinitramfs only. A full
> update-initramfs call spends much more time copying
> some firmware bits to boot partition with flash-kernel
>
> level    usertime   elapsed memory fileSize
> lz4        21.10s    64.85s    21M      29M
> -1         13.73s    44.55s    21M      27M
> -6         26.07s    49.09s    91M      24M
> -12        48.18s    54.67s   203M      22M
> -19       130.07s    92.80s   350M      20M
>
> -> 6 is essentially free if the Pi 4 is idle. Nice.
> -> -6 is still 20% of total RAM of a Pi 0
> -> There's no meaningful difference between -6 and -12
>    in terms of time elapsed. -6 uses 116% CPU, -12 uses
>    145% CPU.
>
> ## Adaptive compression
>
> zstd also supports adaptive compression, compressing as hard as
> it can while not impacting I/O speed. So hardware with slow I/O
> like a Pi would compress harder to avoid idling.
>
> This is somewhat suboptimal with recent update-initramfs though,
> as it first writes the cpio archive to the disk and then compresses
> it rather than doing it in a pipe where that would be more
> meaningful.
>
> Question: Does zstd --adapt adapt to memory available?
>
> # Way(s) forward
>
> To remedy the issue the proposal is to build with
>
> - zstd -1 on hardware with 512 MB or less memory
> - zstd between -1 and -19 on other hardware
> - zstd -19 during image building
>
> Finding the right level between -1 and -19 is hard. The more
> cores you have, the less penalty you pay for higher level.
>
> Going for adaptive compression would remove the guess work, but
> will result in larger images on faster machines. Maybe that's
> fine, though - they probably have more space on /boot anyway?
>
> If we want to aim for 5% of total memory, we should probably
> aim for something like:
>
> -1  on <= 512MB
> -6  on <= 2 GB (or --adapt=min=1,max=6)
> -12 on the rest (or --adapt=min=12)
>
> It's clear that in all cases, zstd -1 is at least better than the
> lz4 -9 we used before; both in terms of space used, and time spent.
>
> # Concerns
>
> Lowering the compression level will reduce the boot speed by fractions
> of a second on hardware with fast I/O.
>
> --
> debian developer - deb.li/jak | jak-linux.org - free software dev
> ubuntu core developer                              i speak de, en
>
> --
> ubuntu-devel mailing list
> ubuntu-devel at lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ubuntu.com/archives/ubuntu-devel/attachments/20211209/4bfc29a9/attachment.html>