[Bug 1960758] Re: UEFI libvirt servers can't boot on Ubuntu 20.04 hypervisors with Ussuri/Victoria

Thu Aug 3 21:05:53 UTC 2023

Test steps/results (Upgrade: Ussuri -> Victoria)
------------------------------------------------

The 'test' server with UEFI was started on Ussuri
with the patch applied (i.e., it boots correctly;
see comment #15).

Now we upgrade from Openstack Ussuri to Victoria,
apply the patch, and restart the 'test' server
(it still boots correctly; as in comment #16).

Upgrade Reference:
	https://docs.openstack.org/project-deploy-guide/charm-deployment-guide/wallaby/upgrade-openstack.html

Commands and versions before/after:

        $ juju config rabbitmq-server source=cloud:focal-victoria

	$ juju config keystone openstack-origin=cloud:focal-victoria
		keystone                            17.0.1 ...
		keystone                            18.1.0 ...

	$ juju config cinder openstack-origin=cloud:focal-victoria
		cinder                              16.4.2 ...
		cinder                              17.4.0 ...

	$ juju config glance openstack-origin=cloud:focal-victoria
		glance                              20.2.0 ...
		glance                              21.1.0 ...

	$ juju config neutron-api openstack-origin=cloud:focal-victoria
		neutron-api                         16.4.2 ...
		neutron-api                         17.4.1 ...

	$ juju config neutron-gateway openstack-origin=cloud:focal-victoria
		neutron-gateway                     16.4.2 ...
		neutron-gateway                     17.4.1 ...

	$ juju config placement openstack-origin=cloud:focal-victoria
		placement                           3.0.1 ...
		placement                           4.0.0 ...

	$ juju config nova-cloud-controller openstack-origin=cloud:focal-victoria
		nova-cloud-controller               21.2.4 ...
		nova-cloud-controller               22.4.0 ...

	$ juju config nova-compute openstack-origin=cloud:focal-victoria
		nova-compute                        21.2.4 ...
		nova-compute                        22.4.0 ...

Verified that the config option is still in nova.conf:
	/etc/nova/nova.conf:ubuntu_libvirt_uefi_loader_path = True

Applied the changes.
Restarted nova-compute.service.

The old XML from Ussuri (VM is still running)

	$ juju ssh nova-compute/0 sudo virsh dumpxml instance-00000002 2>&1 | sed -n '/<os>/,/<\/os>/p'
	  <os>
	    <type arch='x86_64' machine='pc-i440fx-4.2'>hvm</type>
	    <loader readonly='yes' type='pflash'>/usr/share/OVMF/OVMF_CODE.fd</loader>
	    <nvram>/var/lib/libvirt/qemu/nvram/instance-00000002_VARS.fd</nvram>
	    <boot dev='hd'/>
	    <smbios mode='sysinfo'/>
	  </os>

Stop/Start the server:

	$ openstack server stop test
	$ openstack server show test | grep -e vm_state -e status -e task_state -e instance_name
	| OS-EXT-SRV-ATTR:instance_name       | instance-00000002                                        |
	| OS-EXT-STS:task_state               | None                                                     |
	| OS-EXT-STS:vm_state                 | stopped                                                  |
	| status                              | SHUTOFF                                                  |

	$ openstack server start test
	$ openstack server show test | grep -e vm_state -e status -e task_state -e instance_name
	| OS-EXT-SRV-ATTR:instance_name       | instance-00000002                                        |
	| OS-EXT-STS:task_state               | None                                                     |
	| OS-EXT-STS:vm_state                 | active                                                   |
	| status                              | ACTIVE                                                   |

The new XML from Victoria (VM restarted; loader changed as expected in
Victoria too):

	$ juju ssh nova-compute/0 sudo virsh dumpxml instance-00000002 2>&1 | sed -n '/<os>/,/<\/os>/p'
	  <os>
	    <type arch='x86_64' machine='pc-i440fx-4.2'>hvm</type>
	    <loader readonly='yes' type='pflash'>/usr/share/OVMF/OVMF_CODE.fd</loader>
	    <nvram>/var/lib/libvirt/qemu/nvram/instance-00000002_VARS.fd</nvram>
	    <boot dev='hd'/>
	    <smbios mode='sysinfo'/>
	  </os>

	$ openstack console log show test | grep -o 'test login:'
	test login:

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to nova in Ubuntu.
https://bugs.launchpad.net/bugs/1960758

Title:
  UEFI libvirt servers can't boot on Ubuntu 20.04 hypervisors with
  Ussuri/Victoria

Status in OpenStack Compute (nova):
  Invalid
Status in OpenStack Compute (nova) ussuri series:
  Invalid
Status in OpenStack Compute (nova) victoria series:
  Invalid
Status in nova package in Ubuntu:
  Invalid
Status in nova source package in Focal:
  In Progress

Bug description:
  Description:
  ===

  Currently, setting hw_firwmare_type=uefi might create
  _unbootable_ servers on 20.04 hypervisors with Ussuri
  and Victoria (Wallaby and later are OK).

  This might hit other distros w/ OVMF_CODE.secboot.fd.

  We should not use the Secure Boot firmware on the 'pc'
  machine type, as 'q35' is _required_ by OVMF firmware
  if SMM feature is built (usually the case, to actually
  secure the SB feature).
  [See comment #6 for research and #7 for test evidence.]

  Update: the 'q35' machine type may not be safe either,
  since other libvirt XML options such as SMM and S3/S4
  disable may be needed for Secure Boot to work, but are
  _not_ configured by Openstack Ussuri (no SB support).

  So, a hopefully reasonable solution (considering for
  how long Focal/Ussuri have been out there, and maybe
  work with UEFI enabled for some cases?) is an option
  to filter which UEFI loaders are used in nova/libvirt.

  Since apparently/historically this patch should be
  downstream, name it ""ubuntu_libvirt_uefi_loader_path"
  a boolean in the DEFAULT libvirt config section,
  so the nova-compute charm's 'config-flags' option
  can be used to enable it.

  Steps to Reproduce:
  ===

  $ openstack image set --property hw_firmware_type=uefi $IMAGE
  $ openstack server create --image $IMAGE --flavor $FLAVOR --network $NETWORK uefi-server

  Expected Result:
  ===

  The server's libvirt XML uses UEFI _without_ Secure Boot.

          <loader readonly='yes'
  type='pflash'>/usr/share/OVMF/OVMF_CODE.fd</loader>

  Guest boots, and console log confirms UEFI mode:

          $ openstack console log show srv | grep -i -e efi -e bios
          ...
          Creating boot entry "Boot0003" with label "ubuntu" for file "\EFI\ubuntu\shimx64.efi"
          ...
          [    0.000000] efi: EFI v2.70 by EDK II
          [    0.000000] efi:  SMBIOS=0x7fbcd000  ACPI=0x7fbfa000  ACPI
          2.0=0x7fbfa014  MEMATTR=0x7eb30018
          [    0.000000] SMBIOS 2.8 present.
          [    0.000000] DMI: OpenStack Foundation OpenStack Nova, BIOS 0.0.0 02/06/2015
          ...

  Actual Result:
  ===

  The server's libvirt XML uses UEFI _with_ Secure Boot.

          <loader readonly='yes'
  type='pflash'>/usr/share/OVMF/OVMF_CODE.secboot.fd</loader>

  Guest doesn't boot; empty console log; qemu-kvm looping at 100% CPU.

          $ openstack console log show srv | grep -i -e efi -e bios
          $ openstack console log show srv | wc -l
          0

          $ juju run --app nova-compute 'top -b -d1 -n5 | grep qemu'
            67205 libvirt+  ... 100.0   1.4   1:18.35 qemu-sy+
            67205 libvirt+  ... 100.0   1.4   1:19.36 qemu-sy+
            67205 libvirt+  ...  99.0   1.4   1:20.36 qemu-sy+
            67205 libvirt+  ... 101.0   1.4   1:21.37 qemu-sy+
            67205 libvirt+  ... 100.0   1.4   1:22.38 qemu-sy+

  Environment:
  ===

  - Hypervisor running Ubuntu 20.04 LTS (Focal)
  - Nova from Ussuri (Ubuntu Archive) or Victoria (Cloud Archive).

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1960758/+subscriptions