[Bug 1460790] Re: 14.10 to 15.04 upgraded system with separate /usr fails to boot

[Robert Hardy]

Please note attempting to mitigate this by switching back to upstart
failed as well.

Doing so did make it so the system would boot past mounting /usr however
for no obvious reason the system stopped automatically mounting the /var
partition.

With no network services running, the system would stop the boot process
on a screen and complain /var hadn't mounted and prompt me to press m to
enter emergency mode. Once I did that and logged in, /var was in fact
not mounted and a mount -av would immediately mount it and then exiting
emergency mode would cause a normal boot to complete. This should be a
blocking issue for 16.04 as it will cause a lot of grief to end users
when servers fail during upgrades. If I didn't have an IPMI remote
console device I would have been really stuck.

To be clear the only fix that worked was connecting using an IPMI remote console device, booting my server from a rescue iso, taring up each of my partitions with tar cpf and dumping them on a disk outside my raid array. I then wiped all the partitions off my main raid array and switched to a configurations where I had 1 primary boot, 1 primary swap and a root and home partition in a LVM physical volume. I then formated boot ext2, root btrfs and home xfs and extracted all the tars. I then follow a process similar to the one below to reinstall grub: https://help.ubuntu.com/community/Grub2/Installing#via_ChRoot
At the end of the process, inside the chroot, I also did an: update-initramfs -u

The result was a working system but was way way more effort than should
have been required.

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to ubuntu-release-upgrader in
Ubuntu.
https://bugs.launchpad.net/bugs/1460790

Title:
  14.10 to  15.04 upgraded system with separate /usr fails to boot

Status in ubuntu-release-upgrader package in Ubuntu:
  New

Bug description:
  One of my servers has a separate partition for /usr partition.

  I recently tried to upgrade my server from 14.04 to 14.10 and then 15.04.
  The first upgrade went well. The 2nd from 14.10 to 15.04 "succeeded" with no obvious issues until you reboot. Unfortunately what actually happened was a total fail. The result was a system which simply cannot boot.

  Basically when you attempt to reboot you end up at an emergency maintenance prompt early in the boot process.
  After a lot of digging one of the messages in the logs gives:
  systemd[1]: /usr appears to be on its own filesystem and is not already mounted. This is not a supported setup. Some things will probably break (sometimes even silently) in mysterious ways. Consult http://freedesktop.org/wiki/Software/systemd/separate-usr-is-broken

  This leaves users with two options:
  1. Connect to IPMI. Boot with a rescue CD and repartition and move software around so you no longer have a separate /usr. This is incredibly PAINFUL!
  2. Switch back to upstart see: https://wiki.ubuntu.com/SystemdForUpstartUsers#Permanent_switch_back_to_upstart

  In my opinion do-release-upgrade should do one of:
  1. Check if /usr is on a separate partition before starting the actual upgrade. Display a big warning to user before proceeding.
  2. Check if /usr is on a separate partition and automatically switch to upstart if it is.

  I will file a separate bug in an attempt to get systemd fixed.

  ProblemType: Bug
  DistroRelease: Ubuntu 15.04
  Package: ubuntu-release-upgrader-core 1:15.04.14
  ProcVersionSignature: Ubuntu 3.19.0-18.18-generic 3.19.6
  Uname: Linux 3.19.0-18-generic x86_64
  ApportVersion: 2.17.2-0ubuntu1.1
  Architecture: amd64
  CrashDB: ubuntu
  Date: Mon Jun  1 14:55:57 2015
  InstallationDate: Installed on 2010-05-23 (1835 days ago)
  InstallationMedia: Ubuntu-Server 10.04 LTS "Lucid Lynx" - Release amd64 (20100427)
  PackageArchitecture: all
  ProcEnviron:
   TERM=xterm
   PATH=(custom, no user)
   XDG_RUNTIME_DIR=<set>
   LANG=en_CA.UTF-8
   SHELL=/bin/tcsh
  SourcePackage: ubuntu-release-upgrader
  UpgradeStatus: Upgraded to vivid on 2015-06-01 (0 days ago)

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/ubuntu-release-upgrader/+bug/1460790/+subscriptions