[Bug 1960758] Re: UEFI libvirt servers can't boot on Ubuntu 20.04 hypervisors with Ussuri/Victoria

[Andreas Hasenack]

Hello Mauricio, or anyone else affected,

Accepted nova into focal-proposed. The package will build now and be
available at
https://launchpad.net/ubuntu/+source/nova/2:21.2.4-0ubuntu2.6 in a few
hours, and then in the -proposed repository.

Please help us by testing this new package.  See
https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how
to enable and use -proposed.  Your feedback will aid us getting this
update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug,
mentioning the version of the package you tested, what testing has been
performed on the package and change the tag from verification-needed-
focal to verification-done-focal. If it does not fix the bug for you,
please add a comment stating that, and change the tag to verification-
failed-focal. In either case, without details of your testing we will
not be able to proceed.

Further information regarding the verification process can be found at
https://wiki.ubuntu.com/QATeam/PerformingSRUVerification .  Thank you in
advance for helping!

N.B. The updated package will be released to -updates after the bug(s)
fixed by this package have been verified and the package has been in
-proposed for a minimum of 7 days.

** Changed in: nova (Ubuntu Focal)
       Status: In Progress => Fix Committed

** Tags added: verification-needed verification-needed-focal

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to Ubuntu Cloud Archive.
https://bugs.launchpad.net/bugs/1960758

Title:
  UEFI libvirt servers can't boot on Ubuntu 20.04 hypervisors with
  Ussuri/Victoria

Status in Ubuntu Cloud Archive:
  Invalid
Status in Ubuntu Cloud Archive ussuri series:
  Triaged
Status in Ubuntu Cloud Archive victoria series:
  Fix Committed
Status in OpenStack Compute (nova):
  Invalid
Status in OpenStack Compute (nova) ussuri series:
  Invalid
Status in OpenStack Compute (nova) victoria series:
  Invalid
Status in nova package in Ubuntu:
  Invalid
Status in nova source package in Focal:
  Fix Committed

Bug description:
  Impact:
  ===

  Currently, setting `hw_firwmare_type=uefi` may create
  _unbootable_ servers on 20.04 hypervisors with Ussuri
  and Victoria (Wallaby and later are OK).

  We should not use the Secure Boot firmware on the 'pc'
  machine type, as 'q35' is _required_ by OVMF firmware
  if SMM feature is built (usually the case, to actually
  secure the SB feature).
  [See comment #6 for research and #7 for test evidence.]

  We should not use the Secure Boot firmware on the 'q35'
  machine type _either_, as it might not work regardless,
  since other libvirt XML options such as SMM and S3/S4
  disable may be needed for Secure Boot to work, but are
  _not_ configured by Openstack Ussuri (no SB support).

  
  Approach:
  ===

  Considering how long Focal/Ussuri have been out there
  (and maybe worked with UEFI enabled for some cases?)
  add a config option to _opt-in_ to actually supported
  UEFI loaders for nova/libvirt.

  This seems to benefit downstream/Ubuntu more (although
  other distros might be affected) add the config option
  "ubuntu_libvirt_uefi_loader_path" (disabled by default)
  in the DEFAULT libvirt config section (so it can be set
  in nova-compute charm's 'config-flags' option).

  
  Test Plan:
  ===

  $ openstack image set --property hw_firmware_type=uefi $IMAGE
  $ openstack server create --image $IMAGE --flavor $FLAVOR --network $NETWORK uefi-server

  (with patched packages:)
  Set `ubuntu_libvirt_uefi_loader_path = true` in `[DEFAULT]` in /etc/nova/nova.conf
  (eg `juju config nova-compute config-flags='ubuntu_libvirt_uefi_loader_path=true'`)
  $ openstack server stop uefi-server
  $ openstack server start uefi-server

  - Expected Result:

  The server's libvirt XML uses UEFI _without_ Secure Boot.

          <loader readonly='yes'
  type='pflash'>/usr/share/OVMF/OVMF_CODE.fd</loader>

  The guest boots, and console log confirms UEFI mode:

          $ openstack console log show srv | grep -i -e efi -e bios
          ...
          Creating boot entry "Boot0003" with label "ubuntu" for file "\EFI\ubuntu\shimx64.efi"
          ...
          [    0.000000] efi: EFI v2.70 by EDK II
          [    0.000000] efi:  SMBIOS=0x7fbcd000  ACPI=0x7fbfa000  ACPI
          2.0=0x7fbfa014  MEMATTR=0x7eb30018
          [    0.000000] SMBIOS 2.8 present.
          [    0.000000] DMI: OpenStack Foundation OpenStack Nova, BIOS 0.0.0 02/06/2015
          ...

  - Actual Result:

  The server's libvirt XML uses UEFI _with_ Secure Boot.

          <loader readonly='yes'
  type='pflash'>/usr/share/OVMF/OVMF_CODE.secboot.fd</loader>

  The guest doesn't boot; empty console log; qemu-kvm looping at 100%
  CPU.

          $ openstack console log show srv | grep -i -e efi -e bios
          $ openstack console log show srv | wc -l
          0

          $ juju run --app nova-compute 'top -b -d1 -n5 | grep qemu'
            67205 libvirt+  ... 100.0   1.4   1:18.35 qemu-sy+
            67205 libvirt+  ... 100.0   1.4   1:19.36 qemu-sy+
            67205 libvirt+  ...  99.0   1.4   1:20.36 qemu-sy+
            67205 libvirt+  ... 101.0   1.4   1:21.37 qemu-sy+
            67205 libvirt+  ... 100.0   1.4   1:22.38 qemu-sy+

  
  Where problems could occur:
  ===

  The changes are opt-in with `ubuntu_libvirt_uefi_loader_path=true`,
  so users are not affected by default.

  Theoretically, regressions would more likely manifest and be contained
  in nova's libvirt driver, when `hw_firwmare_type=uefi` (not by default).

  The expected symptoms of regressions are boot failures (server starts
  from openstack perspective, but doesn't boot to the operating system).

  
  Other Info:
  ===

  - Hypervisor running Ubuntu 20.04 LTS (Focal)
  - Nova packages from Ussuri (Ubuntu Archive) or Victoria (Cloud Archive).

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1960758/+subscriptions