[Bug 1978489] Re: libvirt / cgroups v2: cannot boot instance with more than 16 CPUs
Stefan Lupsa
1978489 at bugs.launchpad.net
Tue Jan 23 14:50:32 UTC 2024
Hello, I've tried testing this in the following setup: juju openstack on
focal with openstack-origin=cloud:focal-yoga on nova-compute charm with
a 12 vcpu flavor in nova.
Test cases covered: launch 12 cpu instance on jammy host, live-migrate
12 cpu instance from focal-yoga host to jammy host.
Initial nova compute host packages:
# dpkg -l | grep nova
ii nova-api-metadata 3:25.2.1-0ubuntu1~cloud0 all OpenStack Compute - metadata API frontend
ii nova-common 3:25.2.1-0ubuntu1~cloud0 all OpenStack Compute - common files
ii nova-compute 3:25.2.1-0ubuntu1~cloud0 all OpenStack Compute - compute node base
ii nova-compute-kvm 3:25.2.1-0ubuntu1~cloud0 all OpenStack Compute - compute node (KVM)
ii nova-compute-libvirt 3:25.2.1-0ubuntu1~cloud0 all OpenStack Compute - compute node libvirt support
ii python3-nova 3:25.2.1-0ubuntu1~cloud0 all OpenStack Compute Python 3 libraries
ii python3-novaclient 2:17.6.0-0ubuntu1~cloud0 all client library for OpenStack Compute API - 3.x
Upgrade 1 compute node os to jammy with do-release-upgreade, resulting in nova packages:
# dpkg -l | grep nova
ii nova-api-metadata 3:25.2.1-0ubuntu1 all OpenStack Compute - metadata API frontend
ii nova-common 3:25.2.1-0ubuntu1 all OpenStack Compute - common files
ii nova-compute 3:25.2.1-0ubuntu1 all OpenStack Compute - compute node base
ii nova-compute-kvm 3:25.2.1-0ubuntu1 all OpenStack Compute - compute node (KVM)
ii nova-compute-libvirt 3:25.2.1-0ubuntu1 all OpenStack Compute - compute node libvirt support
ii python3-nova 3:25.2.1-0ubuntu1 all OpenStack Compute Python 3 libraries
ii python3-novaclient 2:17.6.0-0ubuntu1 all client library for OpenStack Compute API - 3.x
Create and migrate instances of the test flavor to the upgraded compute
replicates the bug:
Live migration to jammy host:
2024-01-23 13:26:07.290 15908 ERROR nova.virt.libvirt.driver [-] [instance: 41b66168-6d5d-44d8-92fe-cf51dfddeac6] Live Migration failure: error from service: GDBus.Error:org.freedesktop.DBus.Error.InvalidArgs: Value specified in CPUWeight is out of range: libvirt.libvirtError: error from service: GDBus.Error:org.freedesktop.DBus.Error.InvalidArgs: Value specified in CPUWeight is out of range
Instance creation on jammy host:
2024-01-23 13:24:19.772 7695 ERROR nova.compute.manager [req-867b6b88-b27f-43b3-8c36-07fd6ab7b5c1 7c1b32195b344bd5a9bb07fa741f6595 29c6237b89424e868d1cd4fa4d9de830 - 5e9eac01f6354a47836762fcfdd3f445 5e9eac01f6354a47836762fcfdd3f445] [instance: 41b66168-6d5d-44d8-92fe-cf51dfddeac6] Failed to build and run instance: libvirt.libvirtError: error from service: GDBus.Error:org.freedesktop.DBus.Error.InvalidArgs: Value specified in CPUWeight is out of range
2024-01-23 13:24:19.772 7695 ERROR nova.compute.manager [instance: 41b66168-6d5d-44d8-92fe-cf51dfddeac6] Traceback (most recent call last):
2024-01-23 13:24:19.772 7695 ERROR nova.compute.manager [instance: 41b66168-6d5d-44d8-92fe-cf51dfddeac6] File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 2513, in _build_and_run_instance
2024-01-23 13:24:19.772 7695 ERROR nova.compute.manager [instance: 41b66168-6d5d-44d8-92fe-cf51dfddeac6] self.driver.spawn(context, instance, image_meta,
2024-01-23 13:24:19.772 7695 ERROR nova.compute.manager [instance: 41b66168-6d5d-44d8-92fe-cf51dfddeac6] File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 4334, in spawn
2024-01-23 13:24:19.772 7695 ERROR nova.compute.manager [instance: 41b66168-6d5d-44d8-92fe-cf51dfddeac6] self._create_guest_with_network(
2024-01-23 13:24:19.772 7695 ERROR nova.compute.manager [instance: 41b66168-6d5d-44d8-92fe-cf51dfddeac6] File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 7522, in _create_guest_with_network
2024-01-23 13:24:19.772 7695 ERROR nova.compute.manager [instance: 41b66168-6d5d-44d8-92fe-cf51dfddeac6] with excutils.save_and_reraise_exception():
2024-01-23 13:24:19.772 7695 ERROR nova.compute.manager [instance: 41b66168-6d5d-44d8-92fe-cf51dfddeac6] File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 227, in __exit__
2024-01-23 13:24:19.772 7695 ERROR nova.compute.manager [instance: 41b66168-6d5d-44d8-92fe-cf51dfddeac6] self.force_reraise()
2024-01-23 13:24:19.772 7695 ERROR nova.compute.manager [instance: 41b66168-6d5d-44d8-92fe-cf51dfddeac6] File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise
2024-01-23 13:24:19.772 7695 ERROR nova.compute.manager [instance: 41b66168-6d5d-44d8-92fe-cf51dfddeac6] raise self.value
2024-01-23 13:24:19.772 7695 ERROR nova.compute.manager [instance: 41b66168-6d5d-44d8-92fe-cf51dfddeac6] File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 7500, in _create_guest_with_network
2024-01-23 13:24:19.772 7695 ERROR nova.compute.manager [instance: 41b66168-6d5d-44d8-92fe-cf51dfddeac6] guest = self._create_guest(
2024-01-23 13:24:19.772 7695 ERROR nova.compute.manager [instance: 41b66168-6d5d-44d8-92fe-cf51dfddeac6] File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 7439, in _create_guest
2024-01-23 13:24:19.772 7695 ERROR nova.compute.manager [instance: 41b66168-6d5d-44d8-92fe-cf51dfddeac6] guest.launch(pause=pause)
2024-01-23 13:24:19.772 7695 ERROR nova.compute.manager [instance: 41b66168-6d5d-44d8-92fe-cf51dfddeac6] File "/usr/lib/python3/dist-packages/nova/virt/libvirt/guest.py", line 167, in launch
2024-01-23 13:24:19.772 7695 ERROR nova.compute.manager [instance: 41b66168-6d5d-44d8-92fe-cf51dfddeac6] with excutils.save_and_reraise_exception():
2024-01-23 13:24:19.772 7695 ERROR nova.compute.manager [instance: 41b66168-6d5d-44d8-92fe-cf51dfddeac6] File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 227, in __exit__
2024-01-23 13:24:19.772 7695 ERROR nova.compute.manager [instance: 41b66168-6d5d-44d8-92fe-cf51dfddeac6] self.force_reraise()
2024-01-23 13:24:19.772 7695 ERROR nova.compute.manager [instance: 41b66168-6d5d-44d8-92fe-cf51dfddeac6] File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise
2024-01-23 13:24:19.772 7695 ERROR nova.compute.manager [instance: 41b66168-6d5d-44d8-92fe-cf51dfddeac6] raise self.value
2024-01-23 13:24:19.772 7695 ERROR nova.compute.manager [instance: 41b66168-6d5d-44d8-92fe-cf51dfddeac6] File "/usr/lib/python3/dist-packages/nova/virt/libvirt/guest.py", line 165, in launch
2024-01-23 13:24:19.772 7695 ERROR nova.compute.manager [instance: 41b66168-6d5d-44d8-92fe-cf51dfddeac6] return self._domain.createWithFlags(flags)
2024-01-23 13:24:19.772 7695 ERROR nova.compute.manager [instance: 41b66168-6d5d-44d8-92fe-cf51dfddeac6] File "/usr/lib/python3/dist-packages/eventlet/tpool.py", line 193, in doit
2024-01-23 13:24:19.772 7695 ERROR nova.compute.manager [instance: 41b66168-6d5d-44d8-92fe-cf51dfddeac6] result = proxy_call(self._autowrap, f, *args, **kwargs)
2024-01-23 13:24:19.772 7695 ERROR nova.compute.manager [instance: 41b66168-6d5d-44d8-92fe-cf51dfddeac6] File "/usr/lib/python3/dist-packages/eventlet/tpool.py", line 151, in proxy_call
2024-01-23 13:24:19.772 7695 ERROR nova.compute.manager [instance: 41b66168-6d5d-44d8-92fe-cf51dfddeac6] rv = execute(f, *args, **kwargs)
2024-01-23 13:24:19.772 7695 ERROR nova.compute.manager [instance: 41b66168-6d5d-44d8-92fe-cf51dfddeac6] File "/usr/lib/python3/dist-packages/eventlet/tpool.py", line 132, in execute
2024-01-23 13:24:19.772 7695 ERROR nova.compute.manager [instance: 41b66168-6d5d-44d8-92fe-cf51dfddeac6] six.reraise(c, e, tb)
2024-01-23 13:24:19.772 7695 ERROR nova.compute.manager [instance: 41b66168-6d5d-44d8-92fe-cf51dfddeac6] File "/usr/lib/python3/dist-packages/six.py", line 719, in reraise
2024-01-23 13:24:19.772 7695 ERROR nova.compute.manager [instance: 41b66168-6d5d-44d8-92fe-cf51dfddeac6] raise value
2024-01-23 13:24:19.772 7695 ERROR nova.compute.manager [instance: 41b66168-6d5d-44d8-92fe-cf51dfddeac6] File "/usr/lib/python3/dist-packages/eventlet/tpool.py", line 86, in tworker
2024-01-23 13:24:19.772 7695 ERROR nova.compute.manager [instance: 41b66168-6d5d-44d8-92fe-cf51dfddeac6] rv = meth(*args, **kwargs)
2024-01-23 13:24:19.772 7695 ERROR nova.compute.manager [instance: 41b66168-6d5d-44d8-92fe-cf51dfddeac6] File "/usr/lib/python3/dist-packages/libvirt.py", line 1385, in createWithFlags
2024-01-23 13:24:19.772 7695 ERROR nova.compute.manager [instance: 41b66168-6d5d-44d8-92fe-cf51dfddeac6] raise libvirtError('virDomainCreateWithFlags() failed')
2024-01-23 13:24:19.772 7695 ERROR nova.compute.manager [instance: 41b66168-6d5d-44d8-92fe-cf51dfddeac6] libvirt.libvirtError: error from service: GDBus.Error:org.freedesktop.DBus.Error.InvalidArgs: Value specified in CPUWeight is out of range
2024-01-23 13:24:19.772 7695 ERROR nova.compute.manager [instance: 41b66168-6d5d-44d8-92fe-cf51dfddeac6]
Install python3-nova/jammy-proposed:
# grep proposed /etc/apt/sources.list
deb http://archive.ubuntu.com/ubuntu jammy-proposed main
# apt-get -s install python3-nova/jammy-proposed
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Selected version '3:25.2.1-0ubuntu2' (Ubuntu:22.04/jammy-proposed [all]) for 'python3-nova'
The following packages were automatically installed and are no longer required:
libpython2-stdlib libpython2.7-minimal libpython2.7-stdlib python2 python2-minimal python2.7 python2.7-minimal
Use 'sudo apt autoremove' to remove them.
The following additional packages will be installed:
nova-common nova-compute nova-compute-kvm nova-compute-libvirt
Suggested packages:
guestmount
The following packages will be REMOVED:
nova-api-metadata
The following packages will be upgraded:
nova-common nova-compute nova-compute-kvm nova-compute-libvirt python3-nova
5 upgraded, 0 newly installed, 1 to remove and 48 not upgraded.
Remv nova-api-metadata [3:25.2.1-0ubuntu1]
Inst nova-compute-kvm [3:25.2.1-0ubuntu1] (3:25.2.1-0ubuntu2 Ubuntu:22.04/jammy-proposed [all]) []
Inst nova-compute-libvirt [3:25.2.1-0ubuntu1] (3:25.2.1-0ubuntu2 Ubuntu:22.04/jammy-proposed [all]) []
Inst nova-compute [3:25.2.1-0ubuntu1] (3:25.2.1-0ubuntu2 Ubuntu:22.04/jammy-proposed [all]) []
Inst python3-nova [3:25.2.1-0ubuntu1] (3:25.2.1-0ubuntu2 Ubuntu:22.04/jammy-proposed [all]) []
Inst nova-common [3:25.2.1-0ubuntu1] (3:25.2.1-0ubuntu2 Ubuntu:22.04/jammy-proposed [all])
Conf nova-compute-kvm (3:25.2.1-0ubuntu2 Ubuntu:22.04/jammy-proposed [all])
Conf nova-compute-libvirt (3:25.2.1-0ubuntu2 Ubuntu:22.04/jammy-proposed [all])
Conf nova-compute (3:25.2.1-0ubuntu2 Ubuntu:22.04/jammy-proposed [all])
Conf python3-nova (3:25.2.1-0ubuntu2 Ubuntu:22.04/jammy-proposed [all])
Conf nova-common (3:25.2.1-0ubuntu2 Ubuntu:22.04/jammy-proposed [all])
The correct nova-api-metadata package version somehow isn't present in jammy-proposed and is removed when updating so I've installed it after python3-nova from https://launchpad.net/ubuntu/jammy/amd64/nova-api-metadata/3:25.2.1-0ubuntu2
# dpkg -l | grep nova
ii nova-api-metadata 3:25.2.1-0ubuntu2 all OpenStack Compute - metadata API frontend
ii nova-common 3:25.2.1-0ubuntu2 all OpenStack Compute - common files
ii nova-compute 3:25.2.1-0ubuntu2 all OpenStack Compute - compute node base
ii nova-compute-kvm 3:25.2.1-0ubuntu2 all OpenStack Compute - compute node (KVM)
ii nova-compute-libvirt 3:25.2.1-0ubuntu2 all OpenStack Compute - compute node libvirt support
ii python3-nova 3:25.2.1-0ubuntu2 all OpenStack Compute Python 3 libraries
ii python3-novaclient 2:17.6.0-0ubuntu1 all client library for OpenStack Compute API - 3.x
The package fixes the problem of spawning instances on the jammy host
but it does not fix the problem of nova being able to live-migrate
instances to the patched jammy host from focal even though the patched
node contains the def _update_quota_xml(instance, xml_doc) method in
/usr/lib/python3/dist-packages/nova/virt/libvirt/migration.py +75
I suspect this is because the source compute handles the migration and
the focal cloud archive package does not have the code to update the
xml.
# Source compute (cloud:focal-yoga)
# tail -0f /var/log/nova/nova-compute.log
2024-01-23 14:35:41.642 15908 WARNING nova.compute.manager [req-cc56b0d8-737c-4800-804a-c554db5a0c66 ce9ed61f54b0474db084032b60c6f9df ec12ddcbda594cc180a76c61e9587936 - e80d12f4e2814a2db31348985cc6d4d5 e80d12f4e2814a2db31348985cc6d4d5] [instance: 6284a05a-1e89-40b9-af48-284a15b905ec] Received unexpected event network-vif-plugged-8c485fe1-0de9-4f16-b450-da9737f6d373 for instance with vm_state active and task_state migrating.
2024-01-23 14:35:41.976 15908 INFO nova.compute.manager [-] [instance: 6284a05a-1e89-40b9-af48-284a15b905ec] Took 2.68 seconds for pre_live_migration on destination host sts-cpu-1-2.maas.
2024-01-23 14:35:42.514 15908 INFO nova.virt.libvirt.migration [-] [instance: 6284a05a-1e89-40b9-af48-284a15b905ec] Increasing downtime to 50 ms after 0 sec elapsed time
2024-01-23 14:35:42.629 15908 INFO nova.virt.libvirt.driver [-] [instance: 6284a05a-1e89-40b9-af48-284a15b905ec] Migration running for 0 secs, memory 100% remaining (bytes processed=0, remaining=0, total=0); disk 100% remaining (bytes processed=0, remaining=0, total=0).
2024-01-23 14:35:45.382 15908 ERROR nova.virt.libvirt.driver [-] [instance: 6284a05a-1e89-40b9-af48-284a15b905ec] Live Migration failure: error from service: GDBus.Error:org.freedesktop.DBus.Error.InvalidArgs: Value specified in CPUWeight is out of range: libvirt.libvirtError: error from service: GDBus.Error:org.freedesktop.DBus.Error.InvalidArgs: Value specified in CPUWeight is out of range
2024-01-23 14:35:45.644 15908 ERROR nova.virt.libvirt.driver [-] [instance: 6284a05a-1e89-40b9-af48-284a15b905ec] Migration operation has aborted
2024-01-23 14:35:45.799 15908 INFO nova.compute.manager [-] [instance: 6284a05a-1e89-40b9-af48-284a15b905ec] Swapping old allocation on dict_keys(['560a076f-2f24-47fa-9cb2-682abcde2d16']) held by migration b2020171-5fb6-4235-8b8f-61a70814e632 for instance
2024-01-23 14:35:47.274 15908 WARNING nova.compute.manager [req-172ac024-7513-4481-a47f-418a63612652 ce9ed61f54b0474db084032b60c6f9df ec12ddcbda594cc180a76c61e9587936 - e80d12f4e2814a2db31348985cc6d4d5 e80d12f4e2814a2db31348985cc6d4d5] [instance: 6284a05a-1e89-40b9-af48-284a15b905ec] Received unexpected event network-vif-unplugged-8c485fe1-0de9-4f16-b450-da9737f6d373 for instance with vm_state active and task_state None.
# Destination compute (jammy-proposed)
# tail -0f /var/log/nova/nova-compute.log
2024-01-23 14:35:38.463 44861 INFO nova.virt.libvirt.driver [req-f3944429-a27f-4949-8245-22e4a6e77c49 7c1b32195b344bd5a9bb07fa741f6595 29c6237b89424e868d1cd4fa4d9de830 - 5e9eac01f6354a47836762fcfdd3f445 5e9eac01f6354a47836762fcfdd3f445] Instance launched has CPU info: {"arch": "x86_64", "model": "Skylake-Server-IBRS", "vendor": "Intel", "topology": {"cells": 1, "sockets": 1, "cores": 12, "threads": 1}, "features": ["avx512cd", "mpx", "amd-ssbd", "invpcid", "popcnt", "sse", "avx512f", "vmx", "apic", "nx", "arat", "clflushopt", "sse4.1", "x2apic", "xsavec", "avx", "mca", "md-clear", "tsc_adjust", "pdcm", "pni", "avx512bw", "umip", "bmi2", "clwb", "ibpb", "ssbd", "fxsr", "mtrr", "pcid", "f16c", "ssse3", "ss", "clflush", "arch-capabilities", "sse2", "syscall", "xsaveopt", "tsc", "pse", "msr", "fsgsbase", "avx512dq", "pat", "mmx", "fma", "avx512vl", "xsaves", "pku", "smep", "erms", "hle", "ibrs", "abm", "vme", "amd-stibp", "fpu", "pdpe1gb", "lahf_lm", "sse4.2", "rdseed", "ht", "pschange-mc-no", "rsba", "movbe", "spec-ctrl", "aes", "bmi1", "mce", "xsave", "rtm", "xgetbv1", "smap", "3dnowprefetch", "pse36", "rdrand", "tsc-deadline", "cx8", "hypervisor", "avx2", "pclmuldq", "cmov", "pge", "adx", "skip-l1dfl-vmentry", "stibp", "lm", "sep", "cx16", "pae", "rdtscp", "de"]}
2024-01-23 14:35:40.179 44861 INFO os_vif [req-f3944429-a27f-4949-8245-22e4a6e77c49 7c1b32195b344bd5a9bb07fa741f6595 29c6237b89424e868d1cd4fa4d9de830 - 5e9eac01f6354a47836762fcfdd3f445 5e9eac01f6354a47836762fcfdd3f445] Successfully plugged vif VIFOpenVSwitch(active=False,address=fa:16:3e:2c:67:c0,bridge_name='br-int',has_traffic_filtering=True,id=8c485fe1-0de9-4f16-b450-da9737f6d373,network=Network(a3208873-3af8-4993-b862-74a8f8b8ca2f),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=False,vif_name='tap8c485fe1-0d')
2024-01-23 14:35:48.343 44861 INFO nova.virt.libvirt.driver [-] [instance: 6284a05a-1e89-40b9-af48-284a15b905ec] Instance destroyed successfully.
2024-01-23 14:35:48.354 44861 INFO os_vif [req-f3944429-a27f-4949-8245-22e4a6e77c49 7c1b32195b344bd5a9bb07fa741f6595 29c6237b89424e868d1cd4fa4d9de830 - 5e9eac01f6354a47836762fcfdd3f445 5e9eac01f6354a47836762fcfdd3f445] Successfully unplugged vif VIFOpenVSwitch(active=True,address=fa:16:3e:2c:67:c0,bridge_name='br-int',has_traffic_filtering=True,id=8c485fe1-0de9-4f16-b450-da9737f6d373,network=Network(a3208873-3af8-4993-b862-74a8f8b8ca2f),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=False,vif_name='tap8c485fe1-0d')
2024-01-23 14:35:48.355 44861 INFO nova.virt.libvirt.driver [req-f3944429-a27f-4949-8245-22e4a6e77c49 7c1b32195b344bd5a9bb07fa741f6595 29c6237b89424e868d1cd4fa4d9de830 - 5e9eac01f6354a47836762fcfdd3f445 5e9eac01f6354a47836762fcfdd3f445] [instance: 6284a05a-1e89-40b9-af48-284a15b905ec] Deleting instance files /var/lib/nova/instances/6284a05a-1e89-40b9-af48-284a15b905ec_del
2024-01-23 14:35:48.355 44861 INFO nova.virt.libvirt.driver [req-f3944429-a27f-4949-8245-22e4a6e77c49 7c1b32195b344bd5a9bb07fa741f6595 29c6237b89424e868d1cd4fa4d9de830 - 5e9eac01f6354a47836762fcfdd3f445 5e9eac01f6354a47836762fcfdd3f445] [instance: 6284a05a-1e89-40b9-af48-284a15b905ec] Deletion of /var/lib/nova/instances/6284a05a-1e89-40b9-af48-284a15b905ec_del complete
--
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to nova in Ubuntu.
https://bugs.launchpad.net/bugs/1978489
Title:
libvirt / cgroups v2: cannot boot instance with more than 16 CPUs
Status in OpenStack Compute (nova):
In Progress
Status in nova package in Ubuntu:
Confirmed
Status in nova source package in Jammy:
Fix Committed
Bug description:
Description
===========
Using the libvirt driver and a host OS that uses cgroups v2 (RHEL 9,
Ubuntu Jammy), an instance with more than 16 CPUs cannot be booted.
Steps to reproduce
==================
1. Boot an instance with 10 (or more) CPUs on RHEL 9 or Ubuntu Jammy
using Nova with the libvirt driver.
Expected result
===============
Instance boots.
Actual result
=============
Instance fails to boot with a 'Value specified in CPUWeight is out of
range' error.
Environment
===========
Originially report as a libvirt but in RHEL 9 [1]
Additional information
======================
This is happening because Nova defaults to 1024 * (# of CPUs) for the
value of domain/cputune/shares in the libvirt XML. This is then passed
directly by libvirt to the cgroups API, but cgroups v2 has a maximum
value of 10000. 10000 / 1024 ~= 9.76
[1] https://bugzilla.redhat.com/show_bug.cgi?id=2035518
====================================
Ubuntu SRU Details:
[Impact]
See above.
[Test Case]
See above.
[Regression Potential]
We've had this change in other jammy-based versions of the nova package for a while now, including zed, antelope, bobcat.
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1978489/+subscriptions
More information about the Ubuntu-openstack-bugs
mailing list