[Bug 1642298] Re: Grub package upgrades overwrites NVRAM, causing MAAS boot order to be overwritten.

[Rod Smith]

Andres, it's perfectly possible to redeploy a system after it's been set
to boot from the local disk by either adjusting the boot order on the
node or by deleting GRUB from the local disk. I don't recall which of
those I did in the procedure noted in my comment #2, but it was likely
one of those things; I simply didn't mention it; probably I thought it
was obvious that I was working around the problem.

I'm not familiar with freeipmi-tools; however, I've just tested the
following with ipmitool, and neither forced a server to PXE-boot:

ipmitool -H 10.20.30.13 -I lanplus -U user -P pass chassis bootparam set force_pxe
ipmitool -H 10.20.30.13 -I lanplus -U pass -P pass chassis bootdev pxe

Both times, ipmitool reported that the node was set to PXE-boot, but in
both cases, the boot order reported by efibootmgr did not change, and
when I rebooted, the node booted from the first item on that list (that
is, not PXE-booting). I verified the boot order both by watching the
boot process on the terminal and noting no PXE-boot attempt and by
verifying via efibootmgr, once the system was up, that the BootCurrent
variable was set to the "ubuntu" entry, not to a PXE-boot entry. I did
this testing on betelnut, a Dell PowerEdge T110 in Lexington.

As to what GRUB should be doing on installation, that's different for
MAAS vs. a non-network install (say, your laptop). EFI is explicitly
designed to support multiple boot loaders on a disk, so an OS that
installs in a non-PXE-boot way *MUST* register itself with the firmware,
and in practice, OSes normally set their boot loaders as the default.
Thus, changes to the boot order on OS installation are normal and
expected behavior in the EFI world. MAAS is the special case here; if a
MAAS-deployed OS should continue to boot with the help of the MAAS
server, then the OS should either not register itself or register itself
but ensure that the PXE-boot option(s) come earlier in the boot list
than the local boot loader.

Note that, back in February, Dann Frazier submitted a change to GRUB
and/or curtin that fixed this bug by causing the behavior you describe
as optimal -- that is, the GRUB package no longer touched the EFI boot
order when it was installed from the network, and it kept a debconf
variable to ensure that future GRUB updates didn't cause problems (those
updates, really, are the problem). (I don't see Dann's changes linked
from this bug report, but maybe I'm missing something.) That, however,
caused nodes to fail to boot if the MAAS server became inaccessible,
since then there was no fallback to boot from GRUB on the hard disk. I
believe somebody filed a bug on that, but I don't have a reference. Dann
noted in comment #21 to this bug report that a GRUB SRU was imminent
that would likely cause a regression on THIS bug, and in comment #23, I
confirmed that this was the case.

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to grub2 in Ubuntu.
https://bugs.launchpad.net/bugs/1642298

Title:
  Grub package upgrades overwrites NVRAM, causing MAAS boot order to be
  overwritten.

Status in curtin:
  Confirmed
Status in MAAS:
  Confirmed
Status in grub2 package in Ubuntu:
  Fix Released
Status in grub2 source package in Trusty:
  Triaged
Status in grub2 source package in Xenial:
  Triaged
Status in grub2 source package in Yakkety:
  Triaged

Bug description:
  [Impact]
  Typically when you install Ubuntu on an EFI system, it installs a new default EFI boot entry that makes the system reboot directly into the OS. During MAAS installs, curtin is careful to disable that behavior. MAAS requires the default boot entry to remain PXE, so that it can direct the system to boot from disk or network as necessary. curtin does this by passing --no-nvram to grub-install when installing the bootloader.

  *Update*: newer curtin releases actually allow the creation of a new
  boot entry, but updates the boot menu to make PXE the default. That
  change is orthogonal to this bug.

  ***However***, this doesn't stop a new default boot entry from being
  added after deploy. If the user installs a grub package update or
  manually runs 'grub-install', booting from disk will become the
  default, and MAAS will lose control of the system.

  [Proposed Solution (er... glorified workaround)]
  The GRUB package in zesty now has support for setting the --no-nvram flag *persistently*. This is implemented via a debconf template (grub2/update_nvram). If curtin sets this flag to "false" during install, post-deploy grub updates will also pass the --no-nvram flag when running grub-install.

  This isn't a perfect solution - users can still call grub-install
  manually and omit this flag.

  [Test Case]
   - MAAS deploy an EFI system.
   - After deploy, login and run 'sudo apt --reinstall install grub-efi-$(dpkg --print-architecture)
   - Reboot and observe that the system does not PXE boot.

  [Regression Risk]
   - The GRUB implementation does not change the defaults of the package. The user would need to opt-in to the "grub2/update_nvram=false". This option is also only presented to users who specifically request a low debconf priority (e.g. expert mode installs).
   - XXX curtin risk XXX

To manage notifications about this bug go to:
https://bugs.launchpad.net/curtin/+bug/1642298/+subscriptions