/boot disk partition size

Mon Feb 14 21:20:36 UTC 2022

Hi all,

I thought I'd provide some context on this, reiterating what I wrote on
Friday.

Summary: We would like to see some discussion about increasing the
/boot partition for automatic partitioning on LVM/Encrypted systems to
2GB for 22.04. We believe this would be a relatively inexpensive
change, and would be much better than the 1.5GB set to be a part of
20.04.4.

Background:

I'm part of the team that makes the Kubuntu Focus line of laptop
computers. On encrypted LVM machines, using the default disk setup, we
have found the /boot partition is way too small.  We discovered that
the /boot partition gets overfilled while apt is installing updates.
Part of this problem is that Kubuntu uses Discover to do updating (not
update-manager like other flavors excluding Ubuntu Studio which also
uses Discover), which uses PackageKit as a backend.

PackageKit has a bug in which it individually installs packages when
updating, causing those packages to be marked as manually installed and
therefore not letting 'apt autoremove' do its job. As you can imagine,
this breaks the old kernel removal process that update-manager does on
other flavors. This was recently fixed upstream[1] in PackageKit and
hopefully will release in time to beat the Feature Freeze / Debian
Import Freeze. 

We have created a script to mitigate this that automatically cleans
kernels. We also increased the /boot partition size on our OEM-
installed systems. We want to move our findings upstream to the OS.
With the default 768MB /boot partition, sometimes even then three
kernels would be installed and overfill with a new kernel update before
it could be cleaned, especially if the user has more than one kernel
flavor. A use case for two kernel flavors would be using 'lowlatency'
for lower hardware latency, and using 'generic' for power-saving
attributes when compared to 'lowlatency'. This would be especially
advantageous on desktop-replacement-style laptop computers, such as the
Kubuntu Focus M2[2].

Bear in mind also that the Nvidia drivers make the initrd for any given
kernel significantly larger than it would be otherwise. With this in
mind, we filed LP: #1960089[3], not knowing that Brian Murray had
already patched and uploaded a larger boot partition prior [4]. Seeing
Brian's patch, we don't believe it still makes the /boot partition
large enough, especially considering the Nvidia drivers.

>From LP: #1960089 (sizes here reported in MiB and GiB):

   Summary:

   We propose to increase the LVM /boot partition to 2.0 GB. This provides
   the space needed so advanced users can use best practice to manage up
   to 3 kernel flavors. The current /boot partition on 20.04 and 22.04 is
   limited to just 705M, which allows only 3 concurrent kernels before
   filling and sometimes locking the system (each image set takes 180M
   total; 4 x 180 = 720M > 705M).

   Reasoning:

   Best practice recommends users keep at least two version of each kernel
   flavor in the /boot directory. If a user has 3 kernel flavors installed
   (e.g. oem, generic-hwe, and lowlatency-hwe), then one needs to reserve
   room for 2 x 3 = 6 kernels.
   The system needs the headroom of at least two additional kernels during
   any automated clean-up process due to package removal scheduling. I
   propose to also reserve room for 2 additional kernels as a safety
   measure. Thus the total recommend available space should accommodate 10
   kernels.

   Each kernel file set takes up 180M in the /boot partition when used
   with Nvidia driver modules. These files include initrd.img, system.map,
   and vmlinuz. With future kernel and module growth, this may surpass
   200M soon. Therefore, we suggest planning for 200M for each kernel.

   We therefore request a total LVM /boot partition size of 10 image x
   200M = 2.0 GB.

   Other Considerations:

   When unattended-upgrades works correctly (which does not yet employ
   best practice), we have seen users with just a single kernel flavor
   over-fill their /boot partitions. This is because unattended-upgrades
   can retain up to 4 kernels, while the /boot partition is only large
   enough for 3. I am currently working with others to improve the
   unattended-upgrades algorithm to use best practice.

   The installer could allow users to resize the /boot partition during
   installation. In this case, we highly recommend a 2.0 GB default for
   the reasons outlined above.

After discovering LP: #1959971, Mike commented:

   Hi Łukasz and Brian: I have been doing quite a bit of research in this
   area recently, and also am advocating to get kernel management and
   cleanup improved, especially for users who must have separate /boot
   partitions. This means professional users who are required to have full
   disk encryption according to company IT policies.

   Using best practice to manage 3 kernel packages (e.g. oem, lowlatency-
   hwe, generic-hwe) results in the need to maintain 6 kernel file sets
   (latest and penultimate version for each kernel). Because of the way
   kernel cleaning is scheduled, additional space is also required for at
   least 2 additional images. You can see the entire reasoning and details
   at LP: #1960089.

   Our conclusion is that 2.0 GB is the preferred target /boot partition
   size for professional systems. This is reinforced by my research which
   shows many pros recommend this same size when discussing partitioning
   (Just one
   example: https://adventures-in-tech.blogspot.com/2018/10/encrypted-ubuntu-installation-with.html
   )

   Given disk space these days, this seems like a small price to pay to
   support best practice and (hopefully) a future unattended upgrades
   algorithm that honors them as well.

I happened to see some discussion in #ubuntu-meeting by the Foundations
Team about this, mentioning Mike's calculations are off. We were told
to respond via this list, at which point Mike wrote the following
(which got moderated as he's not an Ubuntu Developer per se): 

On Thu, 2022-02-10 at 18:38 -0800, Michael Mikowski wrote:
> Hi Everyone:
> Members of the ubuntu-meeting asked me to clarify the issue with
> /boot. You can see the bugs filed at
> https://launchpad.net/bugs/1959971 and
> https://launchpad.net/bugs/1960089.
> Personal Background:
> I currently lead the Kubuntu Focus engineering efforts, and have been
> a professional product engineer specializing in mission-critical
> HP/HA/HV systems for decades. You can see a overview of my experience
> at https://michaelmikowski.com.
> 
> My Assessments:
> 
> 0. This is a low-cost, high-return proposal. People have pushed back
> against increasing the /boot partition, but I find most of the
> reasoning does not consider the true impact of a small /boot
> partition to the complete product.
> 
> 1. The cost to provide the space needed is minimal for almost all FDE
> systems. So why not just do it? Of course, there is more work to be
> done for the rest of the product (guide users on kernel selection;
> improve the kernel cleaning logic), but this is an important
> component that is required in any complete solution. It would provide
> substantial relief to many users immediately.
> 
> 2. An overfull /boot partition is catastrophic for many users. Many
> cannot recover their system because they don't have a rescue disk or
> knowledge of ext4, chroot, LUKS, lvm, cryptesetup, package
> management, and more. These people either reload the OS or give up.
> 
> 3. This happens frequently, even when everything works as designed,
> and even when just using a single kernel flavor. While on IRC
> yesterday discussing this, 3 unsolicited individuals stepped in to
> comment about how this is needed. And those are advanced developers
> who know better. Search for 'ubuntu full /boot partition' and check
> out how frequently this continues to occur. 
> 
> 4. There are many use cases where multiple kernel flavors will occur,
> such as when the users switches from HWE to OEM to address hardware
> issues, or they install lowlatency for studio work. When this
> happens, the current size boot partition is highly likely to fill.
> This can corrupt the system and require a rescue disk for recovery.
> 
> Check out the DFMEA which is used to assess the severity of a failure
> mode: 
> https://bugs.launchpad.net/ubuntu/+source/partman-auto/+bug/1959971/comments/7
> 
> Finally, I assure you the calculations for kernel boot-file-set sizes
> (180M with Nvidia GPU) are correct for 20.04, and will be correct for
> 22.04 unless the compression technique has been changed. It seems
> perfectly reasonable to expect this size to grow to 200M or more over
> the next 2 years. Also, the headroom + safety factor specified (10
> kernel file-sets total) is reasonable when you consider that systems
> that reboot less frequently will accumulate kernels and that a single
> upgrade can install multiple additional kernel images.
> 
> Sincerely, Mike
> 

Łukasz Zemczak responded to LP: #1959971 today following a detailed
analysis Mike did in Comment #7.

   Thank you for the detailed assessment! For 20.04 (at least for 20.04.4)
   we will only bump the partition sizes slightly, as mentioned in the
   current description (and in the current SRU). We need to thread
   carefully as it's a point-release of an already released series. As for
   22.04, we are certainly open to discussion and your findings will be
   very helpful to make the best decision.
   Let's see how it goes.

We are in 100% agreement with Łukasz on this and would like to see some
discussion on this increase in /boot partition size. 

Thanks in advance,
Erich

[1] https://github.com/PackageKit/PackageKit/issues/450
[2] https://kfocus.org/spec/spec-m2.html
[3] https://launchpad.net/bugs/1960089
[4] https://launchpad.net/bugs/1959971

--
Erich Eickmeyer
Project Leader, Ubuntu Studio
Member, Ubuntu Community Council
UX & Packaging Director, Kubuntu Focus
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ubuntu.com/archives/ubuntu-devel/attachments/20220214/8dcd9d0d/attachment-0001.html>