[Bug 2090972] Re: /boot intermittently fails to mount on boot

Tue Dec 17 15:30:50 UTC 2024

Hi Matthew and Timo,
Thanks for getting the Noble and Oracular proposed packages posted so promptly.  I'll go ahead and kick off test runs on these this morning and can follow up by Friday morning (Pacific Time) about whether they have passed.  If you don't think that's long enough, I can let them run over the holiday break and can follow up on January, 3rd.  I've previously been able to reproduce the problem on Noble within 4 hours on a synthetic test.

Thanks, and hope you have a relaxing holiday.

-K

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to util-linux in Ubuntu.
https://bugs.launchpad.net/bugs/2090972

Title:
  /boot intermittently fails to mount on boot

Status in util-linux package in Ubuntu:
  Fix Released
Status in util-linux source package in Noble:
  Fix Committed
Status in util-linux source package in Oracular:
  Fix Committed
Status in util-linux source package in Plucky:
  Fix Released

Bug description:
  [Impact]

  Starting on Noble, we see /boot fail to mount in approximately one out
  of every two thousand boots.  The error looks like this:

     Found device dev-disk-by\x2dlabel-BOOT.device - QEMU NVMe Ctrl BOOT.
     Starting systemd-fsck at dev-disk-by… Check on /dev/disk/by-label/BOOT...
     Checking in progress on 1 disk (0.0% complete)
     Checking in progress on 0 disks (100.0% complete)
     Finished msystemd-fsck at dev-disk-by… Check on /dev/disk/by-label/BOOT.
     Mounting boot.mount - /boot...
     [    3.051612] /dev/disk/by-label/BOOT: Can't lookup blockdev
     FAILED Failed to mount boot.mount - /boot.
     See 'systemctl status boot.mount' for details.

  This has resulted in a number of different failure modes for our
  users.

  Anything that needs to interact with /boot during provisioning will
  fail.  This is usually something running update-grub or similar.

  If we manage to succeed in booting, this can cause subsequent kernel
  updates or tools that install kernel modules to fail, because
  update-grub fails.

  We've _also_ seen this manifest on the root filesystem.  In that case,
  the boot succeeded, but the by-label links remain absent.  When this
  occurs, we find installing kernel packages fails because mkinitramfs
  can't locate the root disk by label.

  [ Testcase ]

  It's a vexing problem, and so to reproduce we ran cloud images in a boot
  loop until we could reliably reproduce the problem.  Unfortunately, we
  weren't able to work out anything that made this happen faster, so it's
  been a bit slow coming.

  It turns out the problem here is that libblkid recently added support to
  compute the checksum of the superblocks on ext4 filesystems, and Noble
  is the first release to include a version of util-linux new enough to
  have this feature.  When libblkid determines an ext4 superblock's
  checksum is corrupt, it refuses to identify the device as having a
  filesystem, which leads to the removeal of the uuid and by-label fields.
  systemd-udevd then removes these symlinks.  This is where it all goes
  wrong.

  From our debug traces, it's possible to see this clearly:

  (udev-worker)[208]: nvme0n1p16: Probe /dev/nvme0n1p16 with raid and
  offset=0

  systemd-udevd[208]: 208: libblkid: LOWPROBE: [36] ext4dev:
  systemd-udevd[208]: 208: libblkid:   BUFFER:         reuse: off=1024 len=1024 (for off=1024 len=1024)
  systemd-udevd[208]: 208: libblkid: LOWPROBE:         magic sboff=56, kboff=1
  systemd-udevd[208]: 208: libblkid: LOWPROBE:         call probefunc()
  systemd-udevd[208]: 208: libblkid:   BUFFER:         reuse: off=1024 len=1024 (for off=1024 len=1024)
  systemd-udevd[208]: incorrect checksum for type ext4dev, got D919EB5600000000, expected A47F6CF000000000
  systemd-udevd[208]: 208: libblkid: LOWPROBE: [37] ext4:
  systemd-udevd[208]: 208: libblkid:   BUFFER:         reuse: off=1024 len=1024 (for off=1024 len=1024)
  systemd-udevd[208]: 208: libblkid: LOWPROBE:         magic sboff=56, kboff=1
  systemd-udevd[208]: 208: libblkid: LOWPROBE:         call probefunc()
  systemd-udevd[208]: 208: libblkid:   BUFFER:         reuse: off=1024 len=1024 (for off=1024 len=1024)
  systemd-udevd[208]: incorrect checksum for type ext4, got D919EB5600000000, expected A47F6CF000000000

  (udev-worker)[208]: nvme0n1p16: Removing/updating old device symlink '/dev/disk/by-uuid/ce209fd3-a875-4607-9089-35b9de605bd0', which is no longer belonging to this device.
  (udev-worker)[208]: nvme0n1p16: No reference left for '/dev/disk/by-uuid/ce209fd3-a875-4607-9089-35b9de605bd0', removing
  (udev-worker)[208]: nvme0n1p16: Removing/updating old device symlink '/dev/disk/by-label/BOOT', which is no longer belonging to this device.
  (udev-worker)[208]: nvme0n1p16: No reference left for '/dev/disk/by-label/BOOT', removing
  (udev-worker)[208]: nvme0n1p16: Successfully created symlink '/dev/block/259:4' to '/dev/nvme0n1p16'
  (udev-worker)[208]: nvme0n1p16: sd-device: Created db file '/run/udev/data/b259:4' for '/devices/pci0000:00/0000:00:01.0/nvme/nvme0/nvme0n1/nvme0n1p16'
  (udev-worker)[208]: nvme0n1p16: Adding watch on '/dev/nvme0n1p16'
  (udev-worker)[208]: nvme0n1p16: Device processed (SEQNUM=1630, ACTION=change)
  (udev-worker)[208]: nvme0n1p16: sd-device-monitor(worker): Passed 1315 byte to netlink monitor.

  We've also been running with a version of this patch backported to
  2.39.3-9ubuntu6.1 in our production environment for a couple of months.
  It's completely eliminated this problem for us.  We were observing this
  about once an hour and have had no recurrence since rolling out the fix.

  There are test packages available in the following ppa:

  https://launchpad.net/~mruffell/+archive/ubuntu/lp2090972-updates

  [Where problems could occur]

  We are changing how superblocks are read off of filesystems at initial device
  probe time. Luckily, the initial read of the superblock is unchanged from what
  happens now. Only if that read fails to compute the correct checksum, then we
  read the superblock from the underlying disk with O_DIRECT, and hopefully
  compute the correct checksum. If that yeilds an mismatch, then we raise the
  incorrect checksum error as usual.

  Since this just adds a quick re-read and recompute of the checksum, if a race
  between memory and the underlying disk would occur, the recheck would take
  minimal amounts of time and would not be noticeable during instance boot time.

  If a regression were to occur, it would affect checksum computation, which could
  cause the disk to be declared corrupted and not containing a valid filesystem,
  which would cause boot to fail, and would have a large impact to users.

  [Other Info]

  Fortunately, the fix here is straight-forward and is similar to what we
  did for resize2fs: use O_DIRECT when reading the superblock.  We've
  already sent a patch upstream and gotten it accepted there:

  commit 483c9f38e377ff0b009f546a2c4ee91a1d61588c
  From: Krister Johansen <kjlx at templeofstupid.com>
  Date: Mon, 18 Nov 2024 12:35:22 -0800
  Subject: libblkid: fix spurious ext superblock checksum mismatches
  Link: https://github.com/util-linux/util-linux/commit/483c9f38e377ff0b009f546a2c4ee91a1d61588c

  We had a discussion with Ted Ts'o about this as well, and he had some
  ideas for future improvements, but nothing that we're implementing in
  this fix:

  https://lore.kernel.org/util-
  linux/6d16e6d83ab48d2ea4402db17c9c0ed5514933a7.1731961869.git.kjlx at templeofstupid.com/T/#m55eb5087639dcfcfd5708144b1b48caf0cf762b8

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/util-linux/+bug/2090972/+subscriptions