[Bug 1710019] Re: support GICv3 ITS save/restore & migration

Fri Nov 17 15:45:59 UTC 2017

I've verified this for zesty, using my patched zesty kernel.
Demonstration of that follows. However, the kernel patches have not yet
been approved for zesty, and may never be. I'll hold-off on updating the
tags here until a) we have a final answer on the kernel patches or b) we
decide there is value in updating QEMU w/o the kernel side.

= Verification log =

ubuntu at dawes:~$ uname -a
Linux dawes 4.10.0-40-generic #44+gicv3sr.1-Ubuntu SMP Thu Nov 9 22:53:54 UTC 2017 aarch64 aarch64 aarch64 GNU/Linux
ubuntu at dawes:~$ cd vm-save-restore/
ubuntu at dawes:~/vm-save-restore$ ./setup.sh 
+ set -e
+ cloudrel=zesty
+ cloudimg=zesty-server-cloudimg-arm64.img
+ sudo apt-get install -y cloud-image-utils qemu-kvm qemu-utils qemu-efi libvirt-bin screen uuid-runtime
Reading package lists... Done
Building dependency tree       
Reading state information... Done
cloud-image-utils is already the newest version (0.30-0ubuntu2).
screen is already the newest version (4.5.0-5ubuntu1).
qemu-efi is already the newest version (0~20161202.7bbe0b3e-1).
uuid-runtime is already the newest version (2.29-1ubuntu2.1).
libvirt-bin is already the newest version (2.5.0-3ubuntu5.6).
qemu-kvm is already the newest version (1:2.8+dfsg-3ubuntu2.8).
qemu-utils is already the newest version (1:2.8+dfsg-3ubuntu2.8).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
+ test -f zesty-server-cloudimg-arm64.img
+ echo #!/bin/sh
+ sudo tee /etc/qemu-ifup
+ echo 
+ sudo tee -a /etc/qemu-ifup
+ echo set -e
+ sudo tee -a /etc/qemu-ifup
+ echo 
+ sudo tee -a /etc/qemu-ifup
+ echo ip link set "$1" up
+ sudo tee -a /etc/qemu-ifup
+ echo ip link set "$1" master virbr0
+ sudo tee -a /etc/qemu-ifup
ubuntu at dawes:~/vm-save-restore$ ./test.sh 
+ i=0
+ [ -f /var/log/libvirt/qemu/4273-0.log ]
+ name=4273-0
+ ./randmac.py
+ mac=00:16:3e:1c:57:55
+ uuidgen
+ uuid=1578301c-e2fe-4981-87b5-bee2c56fa14c
+ mktemp
+ xml=/tmp/tmp.D9DXzEvOHn
+ cp template.xml /tmp/tmp.D9DXzEvOHn
+ trap cleanup EXIT
+ mkdir -p vms
+ dd if=/dev/zero of=./vms/4273-0_CODE.fd bs=1M count=64
64+0 records in
64+0 records out
67108864 bytes (67 MB, 64 MiB) copied, 0.166913 s, 402 MB/s
+ dd if=/usr/share/qemu-efi/QEMU_EFI.fd of=./vms/4273-0_CODE.fd conv=notrunc
4096+0 records in
4096+0 records out
2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0602123 s, 34.8 MB/s
+ dd if=/dev/zero of=./vms/4273-0_VARS.fd bs=1M count=64
64+0 records in
64+0 records out
67108864 bytes (67 MB, 64 MiB) copied, 0.16786 s, 400 MB/s
+ cat
+ cloud-localds vms/4273-0_seed.img vms/4273-0_user-data
+ pwd
+ qemu-img create -f qcow2 -o backing_file=/home/ubuntu/vm-save-restore/zesty-server-cloudimg-arm64.img ./vms/4273-0.img
Formatting './vms/4273-0.img', fmt=qcow2 size=2361393152 backing_file=/home/ubuntu/vm-save-restore/zesty-server-cloudimg-arm64.img encryption=off cluster_size=65536 lazy_refcounts=off refcount_bits=16
+ sed -i s/\#VM\#/4273-0/ /tmp/tmp.D9DXzEvOHn
+ pwd
+ sed -i s,\#DIR\#,/home/ubuntu/vm-save-restore, /tmp/tmp.D9DXzEvOHn
+ sed -i s/\#UUID\#/1578301c-e2fe-4981-87b5-bee2c56fa14c/ /tmp/tmp.D9DXzEvOHn
+ sed -i s/\#MAC\#/00:16:3e:1c:57:55/ /tmp/tmp.D9DXzEvOHn
+ sudo virsh define /tmp/tmp.D9DXzEvOHn
Domain 4273-0 defined from /tmp/tmp.D9DXzEvOHn

+ sudo virsh start 4273-0
Domain 4273-0 started

+ sleep 60
+ sudo virsh save 4273-0 ./vms/4273-0.sav

Domain 4273-0 saved to ./vms/4273-0.sav

+ sudo virsh restore ./vms/4273-0.sav
Domain restored from ./vms/4273-0.sav

+ sleep 5
+ sudo virsh save 4273-0 ./vms/4273-0.sav

Domain 4273-0 saved to ./vms/4273-0.sav

+ sudo virsh restore ./vms/4273-0.sav
Domain restored from ./vms/4273-0.sav

+ sudo virsh save 4273-0 ./vms/4273-0.sav

Domain 4273-0 saved to ./vms/4273-0.sav

+ sudo virsh restore ./vms/4273-0.sav
Domain restored from ./vms/4273-0.sav

+ cleanup
+ sudo virsh destroy 4273-0
Domain 4273-0 destroyed

+ sudo virsh undefine 4273-0 --nvram
Domain 4273-0 has been undefined

+ rm -f ./vms/4273-0.img ./vms/4273-0_CODE.fd ./vms/4273-0_VARS.fd
/tmp/tmp.D9DXzEvOHn ./vms/4273-0.sav vms/4273-0_user-data
vms/4273-0_seed.img

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to Ubuntu Cloud Archive.
https://bugs.launchpad.net/bugs/1710019

Title:
  support GICv3 ITS save/restore & migration

Status in Ubuntu Cloud Archive:
  New
Status in linux package in Ubuntu:
  In Progress
Status in qemu package in Ubuntu:
  Fix Released
Status in linux source package in Xenial:
  Won't Fix
Status in qemu source package in Xenial:
  Won't Fix
Status in linux source package in Zesty:
  In Progress
Status in qemu source package in Zesty:
  Fix Committed
Status in linux source package in Artful:
  In Progress
Status in qemu source package in Artful:
  Fix Released

Bug description:
  [Impact]
  Virtual machines on GICv3-based ARM systems cannot be saved/restored or migrated.
  This feature was added in QEMU 2.10.

  [Test Case]
  ubuntu at grotrian:~$ sudo virsh save 7936-0 7936-0.sav

  Domain 7936-0 saved to 7936-0.sav

  ubuntu at grotrian:~$ sudo virsh restore 7396-0.sav
  error: Failed to restore domain from 7396-0.sav
  error: operation failed: job: unexpectedly failed

  ubuntu at grotrian:~$ sudo tail -3 /var/log/libvirt/qemu/7936-0.log
  2017-08-10T21:26:38.217427Z qemu-system-aarch64: State blocked by non-migratable device 'arm_gicv3_its'
  2017-08-10T21:26:38.217565Z qemu-system-aarch64: load of migration failed: Invalid argument
  2017-08-10 21:26:38.217+0000: shutting down, reason=failed

  [Regression Risk]
  The kernel changes are restricted to ARM, minimizing the regression risk on other architectures. Other than one minor offset adjustment, all patches are clean cherry-picks from upstream. Tested on Cavium ThunderX and Qualcomm Centriq.

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1710019/+subscriptions