[Bug 1960758] Re: UEFI libvirt servers can't boot on Ubuntu 20.04 hypervisors with Ussuri/Victoria

Wed Aug 23 18:36:35 UTC 2023

For non-juju installations of nova-common (which ships
/etc/nova/nova.conf), if there was a local change in nova.conf, this
update will trigger a dpkg conf prompt:

 ==> Modified (by you or by a script) since installation.
 ==> Package distributor has shipped an updated version.
   What would you like to do about it ?  Your options are:
    Y or I  : install the package maintainer's version
    N or O  : keep your currently-installed version
      D     : show the differences between the versions
      Z     : start a shell to examine the situation
 The default action is to keep your current version.
*** nova.conf (Y/I/N/O/D/Z) [default=N] ? d

--- /etc/nova/nova.conf 2023-08-23 18:33:23.107167993 +0000
+++ /etc/nova/nova.conf.dpkg-new        2023-07-25 20:34:00.000000000 +0000
@@ -6,7 +6,6 @@
 #
 # From nova.conf
 #
-# andreas was here
 
 #
 # Availability zone for internal services. For more information, refer to the
@@ -659,6 +658,18 @@
 # mc - Memcache ServiceGroup driver
 #servicegroup_driver = db
 
+# DEPRECATED:
+# Determine UEFI loader path list for images with hw_firmware_type=uefi. For
+# more information, refer to the documentation. (boolean value)
+# This option is deprecated for removal since 21.2.4.
+# Its value may be silently ignored in the future.
+# Reason:
+# This option was added to work around issues with booting UEFI instances
+# in Ussuri and Victoria (LP: #1960758) if the Secure Boot loader is used
+# (happens by default on Ubuntu 20.04). This option does not exist nor is
+# needed in Wallaby and later, which actually support Secure Boot.
+#ubuntu_libvirt_uefi_loader_path = false
+
 #
 # From oslo.log
 #


Usually this is not desirable, specially if a new default value won't change. Is there another way to fix this? Is there a precedent for such a change in the history of updates of this stack of packages? Or is it very unlikely that nova.conf will have been changed locally?
\

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to Ubuntu Cloud Archive.
https://bugs.launchpad.net/bugs/1960758

Title:
  UEFI libvirt servers can't boot on Ubuntu 20.04 hypervisors with
  Ussuri/Victoria

Status in Ubuntu Cloud Archive:
  Invalid
Status in Ubuntu Cloud Archive ussuri series:
  Triaged
Status in Ubuntu Cloud Archive victoria series:
  Triaged
Status in OpenStack Compute (nova):
  Invalid
Status in OpenStack Compute (nova) ussuri series:
  Invalid
Status in OpenStack Compute (nova) victoria series:
  Invalid
Status in nova package in Ubuntu:
  Invalid
Status in nova source package in Focal:
  In Progress

Bug description:
  Impact:
  ===

  Currently, setting `hw_firwmare_type=uefi` may create
  _unbootable_ servers on 20.04 hypervisors with Ussuri
  and Victoria (Wallaby and later are OK).

  We should not use the Secure Boot firmware on the 'pc'
  machine type, as 'q35' is _required_ by OVMF firmware
  if SMM feature is built (usually the case, to actually
  secure the SB feature).
  [See comment #6 for research and #7 for test evidence.]

  We should not use the Secure Boot firmware on the 'q35'
  machine type _either_, as it might not work regardless,
  since other libvirt XML options such as SMM and S3/S4
  disable may be needed for Secure Boot to work, but are
  _not_ configured by Openstack Ussuri (no SB support).

  
  Approach:
  ===

  Considering how long Focal/Ussuri have been out there
  (and maybe worked with UEFI enabled for some cases?)
  add a config option to _opt-in_ to actually supported
  UEFI loaders for nova/libvirt.

  This seems to benefit downstream/Ubuntu more (although
  other distros might be affected) add the config option
  "ubuntu_libvirt_uefi_loader_path" (disabled by default)
  in the DEFAULT libvirt config section (so it can be set
  in nova-compute charm's 'config-flags' option).

  
  Test Plan:
  ===

  $ openstack image set --property hw_firmware_type=uefi $IMAGE
  $ openstack server create --image $IMAGE --flavor $FLAVOR --network $NETWORK uefi-server

  (with patched packages:)
  Set `ubuntu_libvirt_uefi_loader_path = true` in `[DEFAULT]` in /etc/nova/nova.conf
  (eg `juju config nova-compute config-flags='ubuntu_libvirt_uefi_loader_path=true'`)
  $ openstack server stop uefi-server
  $ openstack server start uefi-server

  - Expected Result:

  The server's libvirt XML uses UEFI _without_ Secure Boot.

          <loader readonly='yes'
  type='pflash'>/usr/share/OVMF/OVMF_CODE.fd</loader>

  The guest boots, and console log confirms UEFI mode:

          $ openstack console log show srv | grep -i -e efi -e bios
          ...
          Creating boot entry "Boot0003" with label "ubuntu" for file "\EFI\ubuntu\shimx64.efi"
          ...
          [    0.000000] efi: EFI v2.70 by EDK II
          [    0.000000] efi:  SMBIOS=0x7fbcd000  ACPI=0x7fbfa000  ACPI
          2.0=0x7fbfa014  MEMATTR=0x7eb30018
          [    0.000000] SMBIOS 2.8 present.
          [    0.000000] DMI: OpenStack Foundation OpenStack Nova, BIOS 0.0.0 02/06/2015
          ...

  - Actual Result:

  The server's libvirt XML uses UEFI _with_ Secure Boot.

          <loader readonly='yes'
  type='pflash'>/usr/share/OVMF/OVMF_CODE.secboot.fd</loader>

  The guest doesn't boot; empty console log; qemu-kvm looping at 100%
  CPU.

          $ openstack console log show srv | grep -i -e efi -e bios
          $ openstack console log show srv | wc -l
          0

          $ juju run --app nova-compute 'top -b -d1 -n5 | grep qemu'
            67205 libvirt+  ... 100.0   1.4   1:18.35 qemu-sy+
            67205 libvirt+  ... 100.0   1.4   1:19.36 qemu-sy+
            67205 libvirt+  ...  99.0   1.4   1:20.36 qemu-sy+
            67205 libvirt+  ... 101.0   1.4   1:21.37 qemu-sy+
            67205 libvirt+  ... 100.0   1.4   1:22.38 qemu-sy+

  
  Where problems could occur:
  ===

  The changes are opt-in with `ubuntu_libvirt_uefi_loader_path=true`,
  so users are not affected by default.

  Theoretically, regressions would more likely manifest and be contained
  in nova's libvirt driver, when `hw_firwmare_type=uefi` (not by default).

  The expected symptoms of regressions are boot failures (server starts
  from openstack perspective, but doesn't boot to the operating system).

  
  Other Info:
  ===

  - Hypervisor running Ubuntu 20.04 LTS (Focal)
  - Nova packages from Ussuri (Ubuntu Archive) or Victoria (Cloud Archive).

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1960758/+subscriptions


