[Bug 1960758] Re: UEFI libvirt servers can't boot on Ubuntu 20.04 hypervisors with Ussuri/Victoria

Wed Aug 23 21:31:57 UTC 2023

Hi Andreas,

Thanks for reviewing!

Summary: among all points you raised, the least-negative answer/option
seems to be, indeed, this is another case of changes to nova.conf (i.e.,
there are precedents; 6 versions of the file in the 21 package versions
since focal-release).

...

Details:

> For non-juju installations of nova-common (which ships
/etc/nova/nova.conf), if there was a local change in nova.conf, this
update will trigger a dpkg conf prompt:

Nice catch; thanks for the careful review, as always!

> Usually this is not desirable, specially if a new default value won't
change. [snip]

Agreed, it's not desirable.

> [snip] Is there another way to fix this?

- If 'this' is nova.conf, probably not in an elegant way, as it's
automatically generated at build-time (so it's likely a hack in
`debian/rules`).

- Now, if 'this' is the technical approach, probably not, because an
opt-in is probably the best compromise between stability (no behavior
change/less regression risk) and functionality (fix), in this case.

> [snip] Is there a precedent for such a change in the history of
updates of this stack of packages? [snip]

There _are_ 5 precedents of nova.conf changes since the version in
focal-release (total of 6 unique md5sums; see shell commands below).

> [snip] Or is it very unlikely that nova.conf will have been changed
locally?

It's very _likely_ that nova.conf will have been changed.
The nova docs for installation [1, 2] change nova.conf immediately after package install:

  1. Install the packages:
  2. Edit the /etc/nova/nova.conf file and complete the following actions:

Thanks again,
Mauricio

...

$ curl -s 'https://launchpad.net/ubuntu/focal/+source/nova' \
  | grep -o '/ubuntu/+source/nova/[^"]\+' | sort -Vu \
  | sed -n '/2:21.0.0~b3~git2020041013.57ff308d6d-0ubuntu2/,$ p' \
  | cut -d/ -f5- | cut -d: -f2- \
  | while read VERSION; do \
      wget "https://launchpad.net/ubuntu/+archive/primary/+files/nova-common_${VERSION}_all.deb"; \
    done

$ for deb in *.deb; do dir=${deb%_*}; dpkg-deb -x $deb $dir; done

$ md5sum nova-common_*/etc/nova/nova.conf | sort
0c12bc3f6253cb10b1e25fc43bec7edb  nova-common_21.0.0-0ubuntu0.20.04.1/etc/nova/nova.conf
0c12bc3f6253cb10b1e25fc43bec7edb  nova-common_21.0.0-0ubuntu0.20.04.2/etc/nova/nova.conf
0c12bc3f6253cb10b1e25fc43bec7edb  nova-common_21.0.0-0ubuntu0.20.04.3/etc/nova/nova.conf
34e79bb12168a44ce1bd8d824a309788  nova-common_21.1.0-0ubuntu1/etc/nova/nova.conf
34e79bb12168a44ce1bd8d824a309788  nova-common_21.1.0-0ubuntu2/etc/nova/nova.conf
34e79bb12168a44ce1bd8d824a309788  nova-common_21.1.1-0ubuntu1/etc/nova/nova.conf
34e79bb12168a44ce1bd8d824a309788  nova-common_21.1.1-0ubuntu2/etc/nova/nova.conf
5b99773742952b3633791b641a36aa5e  nova-common_21.2.2-0ubuntu1/etc/nova/nova.conf
5b99773742952b3633791b641a36aa5e  nova-common_21.2.3-0ubuntu1/etc/nova/nova.conf
5b99773742952b3633791b641a36aa5e  nova-common_21.2.4-0ubuntu1/etc/nova/nova.conf
5b99773742952b3633791b641a36aa5e  nova-common_21.2.4-0ubuntu2/etc/nova/nova.conf
79b5bb5ba349227ec756340b673b8e4c  nova-common_21.2.4-0ubuntu2.1/etc/nova/nova.conf
79b5bb5ba349227ec756340b673b8e4c  nova-common_21.2.4-0ubuntu2.2/etc/nova/nova.conf
79b5bb5ba349227ec756340b673b8e4c  nova-common_21.2.4-0ubuntu2.3/etc/nova/nova.conf
79b5bb5ba349227ec756340b673b8e4c  nova-common_21.2.4-0ubuntu2.4/etc/nova/nova.conf
79b5bb5ba349227ec756340b673b8e4c  nova-common_21.2.4-0ubuntu2.5/etc/nova/nova.conf
b89558162c52dcf9c4d3eda3f9764664  nova-common_21.0.0~b3~git2020041013.57ff308d6d-0ubuntu2/etc/nova/nova.conf
dd548356b4cc6bd3de2e3bd493fb8f2f  nova-common_21.1.2-0ubuntu1/etc/nova/nova.conf
dd548356b4cc6bd3de2e3bd493fb8f2f  nova-common_21.1.2-0ubuntu2/etc/nova/nova.conf
dd548356b4cc6bd3de2e3bd493fb8f2f  nova-common_21.2.0-0ubuntu1/etc/nova/nova.conf
dd548356b4cc6bd3de2e3bd493fb8f2f  nova-common_21.2.1-0ubuntu1/etc/nova/nova.conf

$ md5sum nova-common_*/etc/nova/nova.conf | sort | wc -l
21

$ md5sum nova-common_*/etc/nova/nova.conf | sort -k1,1 -u | wc -l
6

[1] https://docs.openstack.org/nova/ussuri/install/compute-install-ubuntu.html#install-and-configure-components
[2] https://docs.openstack.org/nova/ussuri/install/controller-install-ubuntu.html#install-and-configure-components

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to Ubuntu Cloud Archive.
https://bugs.launchpad.net/bugs/1960758

Title:
  UEFI libvirt servers can't boot on Ubuntu 20.04 hypervisors with
  Ussuri/Victoria

Status in Ubuntu Cloud Archive:
  Invalid
Status in Ubuntu Cloud Archive ussuri series:
  Triaged
Status in Ubuntu Cloud Archive victoria series:
  Triaged
Status in OpenStack Compute (nova):
  Invalid
Status in OpenStack Compute (nova) ussuri series:
  Invalid
Status in OpenStack Compute (nova) victoria series:
  Invalid
Status in nova package in Ubuntu:
  Invalid
Status in nova source package in Focal:
  In Progress

Bug description:
  Impact:
  ===

  Currently, setting `hw_firwmare_type=uefi` may create
  _unbootable_ servers on 20.04 hypervisors with Ussuri
  and Victoria (Wallaby and later are OK).

  We should not use the Secure Boot firmware on the 'pc'
  machine type, as 'q35' is _required_ by OVMF firmware
  if SMM feature is built (usually the case, to actually
  secure the SB feature).
  [See comment #6 for research and #7 for test evidence.]

  We should not use the Secure Boot firmware on the 'q35'
  machine type _either_, as it might not work regardless,
  since other libvirt XML options such as SMM and S3/S4
  disable may be needed for Secure Boot to work, but are
  _not_ configured by Openstack Ussuri (no SB support).

  Approach:
  ===

  Considering how long Focal/Ussuri have been out there
  (and maybe worked with UEFI enabled for some cases?)
  add a config option to _opt-in_ to actually supported
  UEFI loaders for nova/libvirt.

  This seems to benefit downstream/Ubuntu more (although
  other distros might be affected) add the config option
  "ubuntu_libvirt_uefi_loader_path" (disabled by default)
  in the DEFAULT libvirt config section (so it can be set
  in nova-compute charm's 'config-flags' option).

  Test Plan:
  ===

  $ openstack image set --property hw_firmware_type=uefi $IMAGE
  $ openstack server create --image $IMAGE --flavor $FLAVOR --network $NETWORK uefi-server

  (with patched packages:)
  Set `ubuntu_libvirt_uefi_loader_path = true` in `[DEFAULT]` in /etc/nova/nova.conf
  (eg `juju config nova-compute config-flags='ubuntu_libvirt_uefi_loader_path=true'`)
  $ openstack server stop uefi-server
  $ openstack server start uefi-server

  - Expected Result:

  The server's libvirt XML uses UEFI _without_ Secure Boot.

          <loader readonly='yes'
  type='pflash'>/usr/share/OVMF/OVMF_CODE.fd</loader>

  The guest boots, and console log confirms UEFI mode:

          $ openstack console log show srv | grep -i -e efi -e bios
          ...
          Creating boot entry "Boot0003" with label "ubuntu" for file "\EFI\ubuntu\shimx64.efi"
          ...
          [    0.000000] efi: EFI v2.70 by EDK II
          [    0.000000] efi:  SMBIOS=0x7fbcd000  ACPI=0x7fbfa000  ACPI
          2.0=0x7fbfa014  MEMATTR=0x7eb30018
          [    0.000000] SMBIOS 2.8 present.
          [    0.000000] DMI: OpenStack Foundation OpenStack Nova, BIOS 0.0.0 02/06/2015
          ...

  - Actual Result:

  The server's libvirt XML uses UEFI _with_ Secure Boot.

          <loader readonly='yes'
  type='pflash'>/usr/share/OVMF/OVMF_CODE.secboot.fd</loader>

  The guest doesn't boot; empty console log; qemu-kvm looping at 100%
  CPU.

          $ openstack console log show srv | grep -i -e efi -e bios
          $ openstack console log show srv | wc -l
          0

          $ juju run --app nova-compute 'top -b -d1 -n5 | grep qemu'
            67205 libvirt+  ... 100.0   1.4   1:18.35 qemu-sy+
            67205 libvirt+  ... 100.0   1.4   1:19.36 qemu-sy+
            67205 libvirt+  ...  99.0   1.4   1:20.36 qemu-sy+
            67205 libvirt+  ... 101.0   1.4   1:21.37 qemu-sy+
            67205 libvirt+  ... 100.0   1.4   1:22.38 qemu-sy+

  Where problems could occur:
  ===

  The changes are opt-in with `ubuntu_libvirt_uefi_loader_path=true`,
  so users are not affected by default.

  Theoretically, regressions would more likely manifest and be contained
  in nova's libvirt driver, when `hw_firwmare_type=uefi` (not by default).

  The expected symptoms of regressions are boot failures (server starts
  from openstack perspective, but doesn't boot to the operating system).

  Other Info:
  ===

  - Hypervisor running Ubuntu 20.04 LTS (Focal)
  - Nova packages from Ussuri (Ubuntu Archive) or Victoria (Cloud Archive).

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1960758/+subscriptions