[Bug 1854981] Re: system doesn't properly boot as expected if /usr is on its own LV
Launchpad Bug Tracker
1854981 at bugs.launchpad.net
Thu Dec 12 14:34:50 UTC 2019
This bug was fixed in the package lvm2 - 2.02.176-4.1ubuntu3.18.04.2
---------------
lvm2 (2.02.176-4.1ubuntu3.18.04.2) bionic; urgency=medium
* d/p/fix-auto-activation-at-boot.patch: (LP: #1854981)
Allow LV auto-activation (e.g. /usr on it's separate LV)
-- Eric Desrochers <eric.desrochers at canonical.com> Thu, 05 Dec 2019
15:15:56 +0000
** Changed in: lvm2 (Ubuntu Bionic)
Status: Fix Committed => Fix Released
** Changed in: lvm2 (Ubuntu Disco)
Status: Fix Committed => Fix Released
--
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to initramfs-tools in Ubuntu.
https://bugs.launchpad.net/bugs/1854981
Title:
system doesn't properly boot as expected if /usr is on its own LV
Status in initramfs-tools package in Ubuntu:
Won't Fix
Status in lvm2 package in Ubuntu:
Fix Released
Status in initramfs-tools source package in Xenial:
Won't Fix
Status in lvm2 source package in Xenial:
Fix Released
Status in initramfs-tools source package in Bionic:
Won't Fix
Status in lvm2 source package in Bionic:
Fix Released
Status in initramfs-tools source package in Disco:
Won't Fix
Status in lvm2 source package in Disco:
Fix Released
Bug description:
[Impact]
In summary, the problem is a LVM 'auto-activation' issue. Basically the LVM activation doesn't occurs during the boot process (except for LVM root partition) and since /usr is an important component before the 'pivot'[0], the init inside the initramfs space tries to mount /usr. If /usr is a LVM volume then it will try to mount it while its backend device is not activated yet (due to the actual auto-activation issue), thus the mounting will fails and bring the user to the initramfs prompt for debugging.
Workaround:
At the initramfs prompt: lvm vgchange -ay and then 'ctrl-d' to resume the boot process.
[Test Case]
- Install Bionic or Disco
- Create a VG with /usr on its on LV
- Complete the installation
- At reboot the system will be stuck at mounting /usr file system
Extra notes:
Since this problem is ONLY related to LVM activation. If one create a non-LVM /usr separate partition, everything will work as expected.
Other separate LVM (e.g. /home, /var/, /srv) if on its own separate
LVM volume will not be activate either, but since they don't need to
be mounted in the initramfs space, it's not a problem, they can be
activated later on (after the pivot) by systemd.
So the only combination possible is when /usr is on its separate LV
only.
[Regression potential]
I don't foresee any regression, the fix will instruct udev rule to
pvscan activate volume based on their major/minor number if LVM is
scanned.
+RUN+="(LVM_EXEC)/lvm pvscan --cache --activate ay --major $major
--minor $minor", ENV{LVM_SCANNED}="1"
The current situation doesn't auto-activate at all.
One downside, for users who will see this fixed, will still experience
the situation in standard ISO, since I'm afraid Disco won't produces
new ISO. (it is not recommended to do new disco installs anyway since
it's going EOL shortly) One would need to rely on the mini.iso for
fetching the latest lvm2 package in the archive.
IIRC Bionic will have a new point release somewhere in Feb 2020 but
until then one would need to rely on the mini.iso for fetching the
latest lvm2 package in the archive as well.
or use the workaround in the [Impact] section and then apt-get dist-
upgrade in order to get the latest lvm2.
[Other information]
This problem only exhibit in Bionic and Disco.
Xenial and Eoan and late didn't exhibit the situation.
https://www.freedesktop.org/wiki/Software/systemd/TheCaseForTheUsrMerge/
https://wiki.debian.org/UsrMerge
[Original Description]
Only the lv for root volume get activated, because of the grub
parameter that specifies/enforce it "root=/dev/mapper/ubuntu-vg-ubuntu
--lv"
http://man7.org/linux/man-pages/man7/bootparam.7.html
'root=...'
This argument tells the kernel what device is to be used as
the root filesystem while booting.
If one add a separate LV for /usr, the system will go straight to
initramfs prompt failling to mount /usr.
At initramfs prompt, we notice that 'lv-usr' status is 'NOT
available'.
Performing 'lvm vgchange -ay' at initramfs prompt workaround the
problem and allow one to successfully boot.
Adding 'debug' parameter, we clearly we see /root being detected and mounted:
initramfs.debug:
=> + mount -r -t ext4 /dev/mapper/ubuntu--vg-ubuntu--lv /root
=> + mountroot_status=0
+ [ ]
+ log_end_msg
+ _log_msg done.\n
+ [ n = y ]
+ printf done.\n
done.
+ read_fstab_entry /usr
+ found=1
+ [ -f /root/etc/fstab ]
+ read MNT_FSNAME MNT_DIR MNT_TYPE MNT_OPTS MNT_FREQ MNT_PASS MNT_JUNK
+ [ / = /usr ]
+ read MNT_FSNAME MNT_DIR MNT_TYPE MNT_OPTS MNT_FREQ MNT_PASS MNT_JUNK
+ [ /usr = /usr ]
+ [ -n ]
+ found=0
+ break 2
+ return 0
+ log_begin_msg Mounting /usr file system
+ _log_msg Begin: Mounting /usr file system ...
then the code read /etc/fstab and specifically search for /usr (most
likely because of the /usr binary merged) and try to mount if if
found.
initramfs-tools:init
271 maybe_break mount
272 log_begin_msg "Mounting root file system"
273 # Always load local and nfs (since these might be needed for /etc or
274 # /usr, irrespective of the boot script used to mount the rootfs).
275 . /scripts/local
276 . /scripts/nfs
277 . /scripts/${BOOT}
278 parse_numeric "${ROOT}"
279 maybe_break mountroot
280 mount_top
281 mount_premount
282 mountroot
283 log_end_msg
284
=> 285 if read_fstab_entry /usr; then
=> 286 log_begin_msg "Mounting /usr file system"
=> 287 mountfs /usr
=> 288 log_end_msg
=> 289 fi
In this case, /usr is present in /etc/fstab but the logical volume is
not available, so it is mounting a filesystem without his backend
device, thus goes straight to the initramfs prompt because /usr
couldn't be mounted.
It clearly seems to be an 'auto-activation' issue at boot.
For other such as /home, /var, it's not a big deal cause they can be
activated later on in the process and they are (I haven't check but I
guess systemd or systemd unit/generator is taking care of it at some
point), but /usr is a important piece to have mounted before the pivot
since it contains most of the crucial binary, especially that nowadays
/sbin, /bin, and /lib are pointing to /usr:
lrwxrwxrwx 1 root root 10 Aug 26 13:51 libx32 -> usr/libx32
lrwxrwxrwx 1 root root 9 Aug 26 13:51 lib64 -> usr/lib64
lrwxrwxrwx 1 root root 9 Aug 26 13:51 lib32 -> usr/lib32
lrwxrwxrwx 1 root root 7 Aug 26 13:51 lib -> usr/lib
lrwxrwxrwx 1 root root 7 Aug 26 13:51 bin -> usr/bin
lrwxrwxrwx 1 root root 8 Aug 26 13:51 sbin -> usr/sbin
NOTE:
* That doesn't affect /usr found in /etc/fstab on its separate
partition when no LVM involve (e.g. /dev/vdb /usr ext4 ....). It only
cause issue when /usr is in a LVM context.
* I was able to reproduce on Bionic and Disco so far.
Eoan doesn't seem to exhibit the situation so far in my testing.
* While certain release such as Bionic, Xenial doesn't come implicitly
with the /usr merge approach. One can install package 'usrmerge' and
convert their system /usr merged. I don't think removing the
read_fstable_entry for /usr is an option here, as some user could
potentially decide to convert their system with 'usrmerge' pkg.
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/initramfs-tools/+bug/1854981/+subscriptions
More information about the foundations-bugs
mailing list