[Bug 1882774] Re: issues with secondary VMX execution controls

Christian Ehrhardt  1882774 at bugs.launchpad.net
Tue Jun 23 09:59:19 UTC 2020 

Prior to the fix starting a host-model guest on GCE:

ubuntu at nested-vm:~$ virsh start focal
error: Failed to start domain focal
error: internal error: process exited while connecting to monitor: 2020-06-23T09:54:47.062832Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48FH).vmx-exit-load-perf-global-ctrl [bit 12]
2020-06-23T09:54:47.062847Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(490H).vmx-entry-load-perf-global-ctrl [bit 13]
2020-06-23T09:54:47.062862Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48CH).vmx-ept-execonly [bit 0]
2020-06-23T09:54:47.062875Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48CH).vmx-eptad [bit 21]
2020-06-23T09:54:47.062891Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(491H).vmx-eptp-switching [bit 0]
2020-06-23T09:54:47.063762Z qemu-system-x86_64: error: failed to set MSR 0x48b to 0x159ff00000000
qemu-system-x86_64: /build/qemu-FC5BvZ/qemu-4.2/target/i386/kvm.c:2680: kvm_buf_set_msrs: Assertion `ret == cpu->kvm_msr_buf->nmsrs' failed.

Updating to focal-proposed worked nicely, retesting the case ...

ubuntu at nested-vm:~$ virsh start focal
Domain focal started

The guest is up and running

ubuntu at nested-vm:~$ virsh list
 Id Name State
-----------------------
 3 focal running

The guest log is now without errors:
2020-06-23 09:57:29.845+0000: starting up libvirt version: 6.0.0, package: 0ubuntu8.1 (Christian Ehrhardt <christian.ehrhardt at canonical.com> Wed, 20 May 2020 06:59:57 +0200), qemu version: 4.2.0Debian 1:4.2-3ubuntu6.3, kernel: 5.4.0-1015-gcp, hostname: nested-vm.c.prjparide.internal
LC_ALL=C \
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin \
HOME=/var/lib/libvirt/qemu/domain-3-focal \
XDG_DATA_HOME=/var/lib/libvirt/qemu/domain-3-focal/.local/share \
XDG_CACHE_HOME=/var/lib/libvirt/qemu/domain-3-focal/.cache \
XDG_CONFIG_HOME=/var/lib/libvirt/qemu/domain-3-focal/.config \
QEMU_AUDIO_DRV=spice \
/usr/bin/qemu-system-x86_64 \
-name guest=focal,debug-threads=on \
-S \
-object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-3-focal/master-key.aes \
-machine pc-q35-focal,accel=kvm,usb=off,dump-guest-core=off \
-cpu Broadwell-IBRS,vme=on,ss=on,vmx=on,f16c=on,rdrand=on,hypervisor=on,arat=on,tsc-adjust=on,umip=on,md-clear=on,stibp=on,arch-capabilities=on,ssbd=on,xsaveopt=on,abm=on,ibpb=on,amd-ssbd=on,rsba=on,skip-l1dfl-vmentry=on \
-m 512 \
-overcommit mem-lock=off \
-smp 1,sockets=1,cores=1,threads=1 \
-uuid 2e2a0717-613b-4f9c-91eb-59f93f5ecddb \
-no-user-config \
-nodefaults \
-chardev socket,id=charmonitor,fd=30,server,nowait \
-mon chardev=charmonitor,id=monitor,mode=control \
-rtc base=utc \
-no-shutdown \
-boot strict=on \
-device pcie-root-port,port=0x10,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x2 \
-device pcie-root-port,port=0x11,chassis=2,id=pci.2,bus=pcie.0,addr=0x2.0x1 \
-device pcie-root-port,port=0x12,chassis=3,id=pci.3,bus=pcie.0,addr=0x2.0x2 \
-device pcie-root-port,port=0x13,chassis=4,id=pci.4,bus=pcie.0,addr=0x2.0x3 \
-device pcie-root-port,port=0x14,chassis=5,id=pci.5,bus=pcie.0,addr=0x2.0x4 \
-device pcie-root-port,port=0x15,chassis=6,id=pci.6,bus=pcie.0,addr=0x2.0x5 \
-device pcie-root-port,port=0x16,chassis=7,id=pci.7,bus=pcie.0,addr=0x2.0x6 \
-device qemu-xhci,id=usb,bus=pci.2,addr=0x0 \
-device virtio-serial-pci,id=virtio-serial0,bus=pci.3,addr=0x0 \
-blockdev '{"driver":"file","filename":"/var/lib/uvtool/libvirt/images/x-uvt-b64-Y29tLnVidW50dS5jbG91ZC5kYWlseTpzZXJ2ZXI6MjAuMDQ6YW1kNjQgMjAyMDA2MjI=","node-name":"libvirt-3-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-3-format","read-only":true,"driver":"qcow2","file":"libvirt-3-storage","backing":null}' \
-blockdev '{"driver":"file","filename":"/var/lib/uvtool/libvirt/images/focal.qcow","node-name":"libvirt-2-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-2-format","read-only":false,"driver":"qcow2","file":"libvirt-2-storage","backing":"libvirt-3-format"}' \
-device virtio-blk-pci,scsi=off,bus=pci.4,addr=0x0,drive=libvirt-2-format,id=virtio-disk0,bootindex=1 \
-blockdev '{"driver":"file","filename":"/var/lib/uvtool/libvirt/images/focal-ds.qcow","node-name":"libvirt-1-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-1-format","read-only":false,"driver":"qcow2","file":"libvirt-1-storage","backing":null}' \
-device virtio-blk-pci,scsi=off,bus=pci.5,addr=0x0,drive=libvirt-1-format,id=virtio-disk1 \
-netdev tap,fd=32,id=hostnet0,vhost=on,vhostfd=33 \
-device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:d5:43:8f,bus=pci.1,addr=0x0 \
-chardev pty,id=charserial0 \
-device isa-serial,chardev=charserial0,id=serial0 \
-chardev socket,id=charchannel0,fd=34,server,nowait \
-device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 \
-vnc 127.0.0.1:0 \
-spice port=5901,addr=127.0.0.1,disable-ticketing,seamless-migration=on \
-device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vram64_size_mb=0,vgamem_mb=16,max_outputs=1,bus=pcie.0,addr=0x1 \
-device virtio-balloon-pci,id=balloon0,bus=pci.6,addr=0x0 \
-sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \
-msg timestamp=on
char device redirected to /dev/pts/3 (label charserial0)

Marking as verified

** Tags removed: verification-needed verification-needed-focal
** Tags added: verification-done verification-done-focal

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to Ubuntu Cloud Archive.
https://bugs.launchpad.net/bugs/1882774

Title:
  issues with secondary VMX execution controls

Status in Ubuntu Cloud Archive:
  New
Status in qemu package in Ubuntu:
  Fix Released
Status in qemu source package in Focal:
  Fix Committed

Bug description:
  [Impact]

  In qemu 4.2 was a change [1] meant to improve the handling of MSRs vs CPUID.
  It was later identified [2] as an issue and fixed.
  This has to be backported to Focal to resolve that issue on several platforms.

  An example where this occurs is:
  - Azure instances with nested virt
  - GCP instances with nested virt

  We have seen a bunch of qemu named CPU types that can expose similar behavior when used on chips that pretend to be of some type e.g. Skylake but miss some of their features to be settable.
  It isn't entirely sure thou that this will be fixed by the same - yet worth to mention.

  The impact is that qemu 4.2 as in Ubuntu 20.04 doesn't work on those
  platforms bailing out.

  [1]: https://github.com/qemu/qemu/commit/048c95163b472ed737a2f0dca4f4e23a82ac2f8a
  [2]: https://github.com/qemu/qemu/commit/4a910e1f6ab4155ec8b24c49b2585cc486916985

  [Test Case]

   * Get a GCP or Azure instance with nested virtualization enabled
   * Spawn a KVM guest on it e.g. by using uvtool-libvirt using a named type 
     matching the cpu
     e.g. if the host reports as skylake use such a type.
     You can use `virsh domcapabilities` to check what the host is
     detected as.

  [Regression Potential]

   * It is a bit hard to guess, but it should not make things worse. But if I'd expect one then the
     VMX subfeatures could change on cases not intended to. Yet we should have one of two cases:
     a) the common one is that the host can set this and has done so, it will continue as before
     b) host was unable to set these and failed, this should now work with the fix in place
     Both seem ok to me.

  [Other Info]

   * there might be a local (non cloud) way to reproduce but I don't
  know it yet

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1882774/+subscriptions

More information about the Ubuntu-openstack-bugs mailing list