[Bug 1960758] Re: UEFI libvirt servers can't boot on Ubuntu 20.04 hypervisors with Ussuri/Victoria

Fri Aug 25 14:38:16 UTC 2023

Hi Andreas o/

Thanks for the discussion! Addressing your points:

> When I see a new config file option being added already in a deprecated
> state, I have to wonder if it's the right fix: [...]

The usage of a deprecated config option is arbitrary (it may be not used),
but the semantics are actually correct [1] for the scenario of this issue
(as the config option is not available in future releases):

  class oslo_config.cfg.Opt(...)

    deprecated_for_removal – indicates whether this opt is planned for removal in a future release
    deprecated_reason – indicates why this opt is planned for removal in a future release. [...]
    deprecated_since – indicates which release this opt was deprecated in. [...]

> If we need a workaround, maybe it shouldn't be in the config file like
> that. It's opt-in, the default value is such that it doesn't change
> behavior, sounds like the config file is just a means for documentation.

I can appreciate your opinion (and have a similar take on this, in general),
however, the package/project goes with all options being in the config file
(with docs), including a dedicated 'workarounds' section, with many opt-ins:

  $ grep -e '^\[' -e '^#[^ ]' nova.conf
  ...
  [workarounds]
  #disable_rootwrap = false
  #disable_libvirt_livesnapshot = false
  #handle_virt_lifecycle_events = true
  #disable_group_policy_check_upcall = false
  #enable_numa_live_migration = false
  #ensure_libvirt_rbd_instance_dir_cleanup = false
  #disable_fallback_pcpu_query = false
  #never_download_image_if_on_rbd = false
  #disable_native_luksv1 = false
  #rbd_volume_local_attach = false
  #reserve_disk_resource_for_image_cache = false
  ...

(There are technical reasons this workaround is in the defaults section,
instead of workarounds: for integration with the juju charm, see MR [2].)

> If this is indeed the only way to address this, could it not be in the
> default config file, and perhaps just be a "hidden" option that will be
> honored in this version? Its documentation could be elsewhere.

Although there are technical means to achieve that suggestion (eg, d/rules
sed, or non-standard usage of the config option library in the source code):
- that does not seem to be what the users of this package expect; and
- hiding it could hinder someone affected to find out the solution needed.

So, I'd suggest that we continue with that the package/project already
use.

...

Hopefully the points above address your questions and make the upload be
considered for SRU acceptance again, as-is.

Summary:
- there have been other config file changes in the past (~25% of uploads
after focal-release, as in #21);
- the package/project already describes all config options in nova.conf
(with documentation), including workarounds and deprecations, which are
correct/in-line with the release-scope of the introduced config option.

Thanks again,
Mauricio

[1] https://docs.openstack.org/oslo.config/ussuri/reference/api/oslo_config.html#oslo_config.cfg.ConfigOpts
[2] https://code.launchpad.net/~mfo/ubuntu/+source/nova/+git/nova/+merge/447799/comments/1198731

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to Ubuntu Cloud Archive.
https://bugs.launchpad.net/bugs/1960758

Title:
  UEFI libvirt servers can't boot on Ubuntu 20.04 hypervisors with
  Ussuri/Victoria

Status in Ubuntu Cloud Archive:
  Invalid
Status in Ubuntu Cloud Archive ussuri series:
  Triaged
Status in Ubuntu Cloud Archive victoria series:
  Fix Committed
Status in OpenStack Compute (nova):
  Invalid
Status in OpenStack Compute (nova) ussuri series:
  Invalid
Status in OpenStack Compute (nova) victoria series:
  Invalid
Status in nova package in Ubuntu:
  Invalid
Status in nova source package in Focal:
  In Progress

Bug description:
  Impact:
  ===

  Currently, setting `hw_firwmare_type=uefi` may create
  _unbootable_ servers on 20.04 hypervisors with Ussuri
  and Victoria (Wallaby and later are OK).

  We should not use the Secure Boot firmware on the 'pc'
  machine type, as 'q35' is _required_ by OVMF firmware
  if SMM feature is built (usually the case, to actually
  secure the SB feature).
  [See comment #6 for research and #7 for test evidence.]

  We should not use the Secure Boot firmware on the 'q35'
  machine type _either_, as it might not work regardless,
  since other libvirt XML options such as SMM and S3/S4
  disable may be needed for Secure Boot to work, but are
  _not_ configured by Openstack Ussuri (no SB support).

  Approach:
  ===

  Considering how long Focal/Ussuri have been out there
  (and maybe worked with UEFI enabled for some cases?)
  add a config option to _opt-in_ to actually supported
  UEFI loaders for nova/libvirt.

  This seems to benefit downstream/Ubuntu more (although
  other distros might be affected) add the config option
  "ubuntu_libvirt_uefi_loader_path" (disabled by default)
  in the DEFAULT libvirt config section (so it can be set
  in nova-compute charm's 'config-flags' option).

  Test Plan:
  ===

  $ openstack image set --property hw_firmware_type=uefi $IMAGE
  $ openstack server create --image $IMAGE --flavor $FLAVOR --network $NETWORK uefi-server

  (with patched packages:)
  Set `ubuntu_libvirt_uefi_loader_path = true` in `[DEFAULT]` in /etc/nova/nova.conf
  (eg `juju config nova-compute config-flags='ubuntu_libvirt_uefi_loader_path=true'`)
  $ openstack server stop uefi-server
  $ openstack server start uefi-server

  - Expected Result:

  The server's libvirt XML uses UEFI _without_ Secure Boot.

          <loader readonly='yes'
  type='pflash'>/usr/share/OVMF/OVMF_CODE.fd</loader>

  The guest boots, and console log confirms UEFI mode:

          $ openstack console log show srv | grep -i -e efi -e bios
          ...
          Creating boot entry "Boot0003" with label "ubuntu" for file "\EFI\ubuntu\shimx64.efi"
          ...
          [    0.000000] efi: EFI v2.70 by EDK II
          [    0.000000] efi:  SMBIOS=0x7fbcd000  ACPI=0x7fbfa000  ACPI
          2.0=0x7fbfa014  MEMATTR=0x7eb30018
          [    0.000000] SMBIOS 2.8 present.
          [    0.000000] DMI: OpenStack Foundation OpenStack Nova, BIOS 0.0.0 02/06/2015
          ...

  - Actual Result:

  The server's libvirt XML uses UEFI _with_ Secure Boot.

          <loader readonly='yes'
  type='pflash'>/usr/share/OVMF/OVMF_CODE.secboot.fd</loader>

  The guest doesn't boot; empty console log; qemu-kvm looping at 100%
  CPU.

          $ openstack console log show srv | grep -i -e efi -e bios
          $ openstack console log show srv | wc -l
          0

          $ juju run --app nova-compute 'top -b -d1 -n5 | grep qemu'
            67205 libvirt+  ... 100.0   1.4   1:18.35 qemu-sy+
            67205 libvirt+  ... 100.0   1.4   1:19.36 qemu-sy+
            67205 libvirt+  ...  99.0   1.4   1:20.36 qemu-sy+
            67205 libvirt+  ... 101.0   1.4   1:21.37 qemu-sy+
            67205 libvirt+  ... 100.0   1.4   1:22.38 qemu-sy+

  Where problems could occur:
  ===

  The changes are opt-in with `ubuntu_libvirt_uefi_loader_path=true`,
  so users are not affected by default.

  Theoretically, regressions would more likely manifest and be contained
  in nova's libvirt driver, when `hw_firwmare_type=uefi` (not by default).

  The expected symptoms of regressions are boot failures (server starts
  from openstack perspective, but doesn't boot to the operating system).

  Other Info:
  ===

  - Hypervisor running Ubuntu 20.04 LTS (Focal)
  - Nova packages from Ussuri (Ubuntu Archive) or Victoria (Cloud Archive).

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1960758/+subscriptions