[Bug 1960758] Re: UEFI libvirt servers can't boot on Ubuntu 20.04 hypervisors with Ussuri/Victoria
Mauricio Faria de Oliveira
1960758 at bugs.launchpad.net
Tue Jul 25 21:02:42 UTC 2023
Test steps/results with the attached patch
------------------------------------------
Create server with BIOS firmware (it boots):
$ openstack server create --image jammy --flavor m1.small
--network private test
$ openstack console log show test | grep -o 'test login:'
test login:
$ openstack server show test | grep -e vm_state -e status -e task_state -e instance_name
| OS-EXT-SRV-ATTR:instance_name | instance-00000002 |
| OS-EXT-STS:task_state | None |
| OS-EXT-STS:vm_state | active |
| status | ACTIVE |
$ juju ssh nova-compute/0 sudo virsh dumpxml instance-00000002 | sed -n '/<os>/,/<\/os>/p'
<os>
<type arch='x86_64' machine='pc-i440fx-4.2'>hvm</type>
<boot dev='hd'/>
<smbios mode='sysinfo'/>
</os>
$ openstack server stop test
$ openstack server delete test
Create server with UEFI firmware (it does _not_ boot):
$ openstack image set jammy --property hw_firmware_type=uefi
$ openstack server create --image jammy --flavor m1.small
--network private test
$ openstack console log show test | grep -o 'test login:'
$
$ openstack server show test | grep -e vm_state -e status -e task_state -e instance_name
| OS-EXT-SRV-ATTR:instance_name | instance-00000003 |
| OS-EXT-STS:task_state | None |
| OS-EXT-STS:vm_state | active |
| status | ACTIVE |
(*) Note it uses the secure-boot loader (secboot), for which
support has not been introduced in Ussuri, but in Wallaby.
$ juju ssh nova-compute/0 sudo virsh dumpxml instance-00000003 | sed -n '/<os>/,/<\/os>/p'
<os>
<type arch='x86_64' machine='pc-i440fx-4.2'>hvm</type>
<loader readonly='yes' type='pflash'>/usr/share/OVMF/OVMF_CODE.secboot.fd</loader>
<nvram>/var/lib/libvirt/qemu/nvram/instance-00000003_VARS.fd</nvram>
<boot dev='hd'/>
<smbios mode='sysinfo'/>
</os>
The QEMU process keeps spinning at 100% CPU utilization.
$ juju ssh nova-compute/0 top -b -d1 -n5 | grep qemu
355887 libvirt+ 20 0 2482736 52228 21776 S 100.0 1.3 2:23.33 qemu-system-x86
355887 libvirt+ 20 0 2482736 52228 21776 S 100.0 1.3 2:24.34 qemu-system-x86
355887 libvirt+ 20 0 2482736 52228 21776 S 99.0 1.3 2:25.34 qemu-system-x86
355887 libvirt+ 20 0 2482736 52228 21776 S 99.0 1.3 2:26.34 qemu-system-x86
355887 libvirt+ 20 0 2482736 52228 21776 S 98.0 1.3 2:27.32 qemu-system-x86
$ openstack server stop test
$ openstack server show test | grep -e vm_state -e status -e task_state -e instance_name
| OS-EXT-SRV-ATTR:instance_name | instance-00000003 |
| OS-EXT-STS:task_state | None |
| OS-EXT-STS:vm_state | stopped |
| status | SHUTOFF |
Apply the patch, and enable the option (not enabled by default).
$ juju config nova-compute config-
flags='ubuntu_libvirt_uefi_loader_path=True'
$ juju status | grep nova-compute/0
nova-compute/0* active idle 9 10.5.1.110 Unit is ready
$ juju ssh nova-compute/0 systemctl status nova-compute.service | grep Active:
Active: active (running) ... 48s ago
$ juju ssh nova-compute/0 sudo grep ubuntu_libvirt_uefi_loader_path /etc/nova/nova.conf
ubuntu_libvirt_uefi_loader_path = True
Start server with UEFI firmware (it now _does_ boot!):
$ openstack server start test
$ openstack server show test | grep -e vm_state -e status -e task_state -e instance_name
| OS-EXT-SRV-ATTR:instance_name | instance-00000003 |
| OS-EXT-STS:task_state | None |
| OS-EXT-STS:vm_state | active |
| status | ACTIVE |
$ openstack console log show test | grep -o 'test login:'
test login:
(*) Note it now uses the regular loader (not secboot.fd).
$ juju ssh nova-compute/0 sudo virsh dumpxml instance-00000003 | sed -n '/<os>/,/<\/os>/p'
<os>
<type arch='x86_64' machine='pc-i440fx-4.2'>hvm</type>
<loader readonly='yes' type='pflash'>/usr/share/OVMF/OVMF_CODE.fd</loader>
<nvram>/var/lib/libvirt/qemu/nvram/instance-00000003_VARS.fd</nvram>
<boot dev='hd'/>
<smbios mode='sysinfo'/>
</os>
** Description changed:
Description:
===
Currently, setting hw_firwmare_type=uefi might create
_unbootable_ servers on 20.04 hypervisors with Ussuri
and Victoria (Wallaby and later are OK).
This might hit other distros w/ OVMF_CODE.secboot.fd.
We should not use the Secure Boot firmware on the 'pc'
machine type, as 'q35' is _required_ by OVMF firmware
if SMM feature is built (usually the case, to actually
- secure the SB feature).
+ secure the SB feature).
[See comment #6 for research and #7 for test evidence.]
+
+ Update: the 'q35' machine type may not be safe either,
+ since other libvirt XML options such as SMM and S3/S4
+ disable may be needed for Secure Boot to work, but are
+ _not_ configured by Openstack Ussuri (no SB support).
+
+ So, a hopefully reasonable solution (considering for
+ how long Focal/Ussuri have been out there, and maybe
+ work with UEFI enabled for some cases?) is an option
+ to filter which UEFI loaders are used in nova/libvirt.
+
+ Since apparently/historically this patch should be
+ downstream, name it ""ubuntu_libvirt_uefi_loader_path"
+ a boolean in the DEFAULT libvirt config section,
+ so the nova-compute charm's 'config-flags' option
+ can be used to enable it.
Steps to Reproduce:
===
$ openstack image set --property hw_firmware_type=uefi $IMAGE
$ openstack server create --image $IMAGE --flavor $FLAVOR --network $NETWORK uefi-server
Expected Result:
===
The server's libvirt XML uses UEFI _without_ Secure Boot.
<loader readonly='yes'
type='pflash'>/usr/share/OVMF/OVMF_CODE.fd</loader>
Guest boots, and console log confirms UEFI mode:
$ openstack console log show srv | grep -i -e efi -e bios
...
Creating boot entry "Boot0003" with label "ubuntu" for file "\EFI\ubuntu\shimx64.efi"
...
[ 0.000000] efi: EFI v2.70 by EDK II
[ 0.000000] efi: SMBIOS=0x7fbcd000 ACPI=0x7fbfa000 ACPI
2.0=0x7fbfa014 MEMATTR=0x7eb30018
[ 0.000000] SMBIOS 2.8 present.
[ 0.000000] DMI: OpenStack Foundation OpenStack Nova, BIOS 0.0.0 02/06/2015
...
Actual Result:
===
The server's libvirt XML uses UEFI _with_ Secure Boot.
<loader readonly='yes'
type='pflash'>/usr/share/OVMF/OVMF_CODE.secboot.fd</loader>
Guest doesn't boot; empty console log; qemu-kvm looping at 100% CPU.
$ openstack console log show srv | grep -i -e efi -e bios
$ openstack console log show srv | wc -l
0
$ juju run --app nova-compute 'top -b -d1 -n5 | grep qemu'
67205 libvirt+ ... 100.0 1.4 1:18.35 qemu-sy+
67205 libvirt+ ... 100.0 1.4 1:19.36 qemu-sy+
67205 libvirt+ ... 99.0 1.4 1:20.36 qemu-sy+
67205 libvirt+ ... 101.0 1.4 1:21.37 qemu-sy+
67205 libvirt+ ... 100.0 1.4 1:22.38 qemu-sy+
Environment:
===
- Hypervisor running Ubuntu 20.04 LTS (Focal)
- Nova from Ussuri (Ubuntu Archive) or Victoria (Cloud Archive).
--
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to nova in Ubuntu.
https://bugs.launchpad.net/bugs/1960758
Title:
UEFI libvirt servers can't boot on Ubuntu 20.04 hypervisors with
Ussuri/Victoria
Status in OpenStack Compute (nova):
Invalid
Status in OpenStack Compute (nova) ussuri series:
Invalid
Status in OpenStack Compute (nova) victoria series:
Invalid
Status in nova package in Ubuntu:
Invalid
Status in nova source package in Focal:
In Progress
Bug description:
Description:
===
Currently, setting hw_firwmare_type=uefi might create
_unbootable_ servers on 20.04 hypervisors with Ussuri
and Victoria (Wallaby and later are OK).
This might hit other distros w/ OVMF_CODE.secboot.fd.
We should not use the Secure Boot firmware on the 'pc'
machine type, as 'q35' is _required_ by OVMF firmware
if SMM feature is built (usually the case, to actually
secure the SB feature).
[See comment #6 for research and #7 for test evidence.]
Update: the 'q35' machine type may not be safe either,
since other libvirt XML options such as SMM and S3/S4
disable may be needed for Secure Boot to work, but are
_not_ configured by Openstack Ussuri (no SB support).
So, a hopefully reasonable solution (considering for
how long Focal/Ussuri have been out there, and maybe
work with UEFI enabled for some cases?) is an option
to filter which UEFI loaders are used in nova/libvirt.
Since apparently/historically this patch should be
downstream, name it ""ubuntu_libvirt_uefi_loader_path"
a boolean in the DEFAULT libvirt config section,
so the nova-compute charm's 'config-flags' option
can be used to enable it.
Steps to Reproduce:
===
$ openstack image set --property hw_firmware_type=uefi $IMAGE
$ openstack server create --image $IMAGE --flavor $FLAVOR --network $NETWORK uefi-server
Expected Result:
===
The server's libvirt XML uses UEFI _without_ Secure Boot.
<loader readonly='yes'
type='pflash'>/usr/share/OVMF/OVMF_CODE.fd</loader>
Guest boots, and console log confirms UEFI mode:
$ openstack console log show srv | grep -i -e efi -e bios
...
Creating boot entry "Boot0003" with label "ubuntu" for file "\EFI\ubuntu\shimx64.efi"
...
[ 0.000000] efi: EFI v2.70 by EDK II
[ 0.000000] efi: SMBIOS=0x7fbcd000 ACPI=0x7fbfa000 ACPI
2.0=0x7fbfa014 MEMATTR=0x7eb30018
[ 0.000000] SMBIOS 2.8 present.
[ 0.000000] DMI: OpenStack Foundation OpenStack Nova, BIOS 0.0.0 02/06/2015
...
Actual Result:
===
The server's libvirt XML uses UEFI _with_ Secure Boot.
<loader readonly='yes'
type='pflash'>/usr/share/OVMF/OVMF_CODE.secboot.fd</loader>
Guest doesn't boot; empty console log; qemu-kvm looping at 100% CPU.
$ openstack console log show srv | grep -i -e efi -e bios
$ openstack console log show srv | wc -l
0
$ juju run --app nova-compute 'top -b -d1 -n5 | grep qemu'
67205 libvirt+ ... 100.0 1.4 1:18.35 qemu-sy+
67205 libvirt+ ... 100.0 1.4 1:19.36 qemu-sy+
67205 libvirt+ ... 99.0 1.4 1:20.36 qemu-sy+
67205 libvirt+ ... 101.0 1.4 1:21.37 qemu-sy+
67205 libvirt+ ... 100.0 1.4 1:22.38 qemu-sy+
Environment:
===
- Hypervisor running Ubuntu 20.04 LTS (Focal)
- Nova from Ussuri (Ubuntu Archive) or Victoria (Cloud Archive).
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1960758/+subscriptions
More information about the Ubuntu-openstack-bugs
mailing list