[Bug 1889556] Re: grub-install failure does not fail package upgrade (and does not roll back to matching modules)

[Patrick Gagnon-Renaud]

We had this problem yesterday on only one of two Ubuntu 18.04 instance
on AWS. We had to rollback to a previous snapshot to make the instance
bootable again.

So we did a quick test with these two AMI:

ubuntu/images/hvm-ssd/ubuntu-bionic-18.04-amd64-server-20200408 - ami-0edd51cc29813e254
ubuntu/images/hvm-ssd/ubuntu-bionic-18.04-amd64-server-20190722.1 - ami-0d0eaed20348a3389

We deployed one fresh instance for each of those AMI and ran the same
update process that we usually do (we the apt module from ansible to
perform the updates). After the reboot, both instances had the issue
(being stuck at "error: symbol `grub_calloc' not found").

Then, we did the same thing, but ran the apt upgrade manually on both
instances and selected the default option on every prompt that shows up.
After the reboot, both instances were normally accessible, without any
sign of the issue.

Hope that this information can be useful!

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to grub2 in Ubuntu.
https://bugs.launchpad.net/bugs/1889556

Title:
  grub-install failure does not fail package upgrade (and does not roll
  back to matching modules)

Status in grub2 package in Ubuntu:
  Confirmed
Status in grub2 source package in Xenial:
  Confirmed
Status in grub2 source package in Bionic:
  Confirmed
Status in grub2 source package in Focal:
  Confirmed
Status in grub2 source package in Groovy:
  Confirmed

Bug description:
  Failure to install new grub core to the specified device does not
  correctly prevent upgrade to incompatible modules (LP: #1889509)

  $ sudo debconf-get-selections |grep sda
  grub-pc	grub-pc/install_devices_disks_changed	multiselect	/dev/sda
  grub-pc	grub-pc/install_devices	multiselect	/dev/sda

  $ mount|grep nvme
  /dev/nvme0n1p1 on / type ext4 (rw,relatime,discard,data=ordered)

  $ ls /dev/sda
  ls: cannot access '/dev/sda': No such file or directory

  $ sudo env DEBIAN_FRONTEND=noninteractive apt full-upgrade -y
  ...
  Get:10 http://us-west-2.ec2.archive.ubuntu.com/ubuntu xenial-updates/main amd64 grub-pc-bin amd64 2.02~beta2-36ubuntu3.26 [891 kB]
  ...
  Installing for i386-pc platform.
  grub-install: error: cannot find a GRUB drive for /dev/sda.  Check your device.map.
  Generating grub configuration file ...
  Found linux image: /boot/vmlinuz-4.4.0-1111-aws
  Found linux image: /boot/vmlinuz-4.4.0-1109-aws
  Found initrd image: /boot/initrd.img-4.4.0-1109-aws
  done
  ...

  # update-grub failed during the install but the return code is 0
  $ echo $?
  0

  # The package is installed without apparent error, but the instance will fail to reboot (LP: #1889509)
  $ dpkg -l|grep grub-pc
  ii  grub-pc                          2.02~beta2-36ubuntu3.26                    amd64        GRand Unified Bootloader, version 2 (PC/BIOS version)
  ii  grub-pc-bin                      2.02~beta2-36ubuntu3.26                    amd64        GRand Unified Bootloader, version 2 (PC/BIOS binaries)

  # If I reboot it will fail to boot:
  Booting from Hard Disk 0...
  error: symbol `grub_calloc' not found.
  Entering rescue mode...
  grub rescue> _

  ---

  Xenial in AWS (us-west-2 ami-060d1be0dd4526759 built on 20200611)
  The  debconf for grub was not set to the correct device when cloud-init first ran (LP: #1877491) or when the fix for that was applied (LP: #1889555)
  The fact that grub-install fails during the upgrade but does not fail the package install (and cause a rollback) means that how we have a mismatch between grub core and modules which breaks boot (LP: #1889509).

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/grub2/+bug/1889556/+subscriptions