[Bug 1718133] Re: Live block migration in Pike fails due to qemu-img

[ChristianEhrhardt]

@James, I found and fixed some of these issues, just look at "fixed in RC" on [1].
Since then all migrations I had worked, we need to find what is special in this case.

When reproducing watch out for:
- do you have the latest qemu /libvirt matching artful or is something left to be promoted to pike?
  Just report libvirt and qemu level and we can compare.
- any apparmor denials to the locking (we generate a k into the rules, but need to check)
- as usual the guest XML as we also found multi-use of an image can be an issue locking it
- as much as possible on the migration type, if you can pull out the virsh migrate (equivalent) command that would be nice
- does it use shared storage or any of the copy-storage options?

[1]: https://wiki.qemu.org/Planning/2.10

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to nova in Ubuntu.
https://bugs.launchpad.net/bugs/1718133

Title:
  Live block migration in Pike fails due to qemu-img

Status in nova package in Ubuntu:
  Confirmed
Status in qemu package in Ubuntu:
  New

Bug description:
  In Pike from Cloud Archive, Live Block Migrations fail:
  Error updating resources for node compute1.: InvalidDiskInfo: Disk info file is invalid: qemu-img failed to execute on /var/lib/nova/instances/ccca487b-d5db-4324-81fb-2665e60da038/disk : Unexpected error while running command.
  Command: /usr/bin/python -m oslo_concurrency.prlimit --as=1073741824 --cpu=30 -- env LC_ALL=C LANG=C qemu-img info /var/lib/nova/instances/ccca487b-d5db-4324-81fb-2665e60da038/disk
  Exit code: 1
  Stdout: u''
  Stderr: u'qemu-img: Could not open \'/var/lib/nova/instances/ccca487b-d5db-4324-81fb-2665e60da038/disk\': Failed to get shared "write" lock\nIs another process using the image?\n'
  2017-09-18 19:55:55.408 11607 ERROR nova.compute.manager Traceback (most recent call last):
  2017-09-18 19:55:55.408 11607 ERROR nova.compute.manager   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 6629, in update_available_resource_for_node
  2017-09-18 19:55:55.408 11607 ERROR nova.compute.manager     rt.update_available_resource(context, nodename)
  2017-09-18 19:55:55.408 11607 ERROR nova.compute.manager   File "/usr/lib/python2.7/dist-packages/nova/compute/resource_tracker.py", line 641, in update_available_resource
  2017-09-18 19:55:55.408 11607 ERROR nova.compute.manager     resources = self.driver.get_available_resource(nodename)
  2017-09-18 19:55:55.408 11607 ERROR nova.compute.manager   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 5831, in get_available_resource
  2017-09-18 19:55:55.408 11607 ERROR nova.compute.manager     disk_over_committed = self._get_disk_over_committed_size_total()
  2017-09-18 19:55:55.408 11607 ERROR nova.compute.manager   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 7330, in _get_disk_over_committed_size_total
  2017-09-18 19:55:55.408 11607 ERROR nova.compute.manager     config, block_device_info)
  2017-09-18 19:55:55.408 11607 ERROR nova.compute.manager   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 7248, in _get_instance_disk_info_from_config
  2017-09-18 19:55:55.408 11607 ERROR nova.compute.manager     backing_file = libvirt_utils.get_disk_backing_file(path)
  2017-09-18 19:55:55.408 11607 ERROR nova.compute.manager   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/utils.py", line 200, in get_disk_backing_file
  2017-09-18 19:55:55.408 11607 ERROR nova.compute.manager     backing_file = images.qemu_img_info(path, format).backing_file
  2017-09-18 19:55:55.408 11607 ERROR nova.compute.manager   File "/usr/lib/python2.7/dist-packages/nova/virt/images.py", line 72, in qemu_img_info
  2017-09-18 19:55:55.408 11607 ERROR nova.compute.manager     raise exception.InvalidDiskInfo(reason=msg)
  2017-09-18 19:55:55.408 11607 ERROR nova.compute.manager InvalidDiskInfo: Disk info file is invalid: qemu-img failed to execute on /var/lib/nova/instances/ccca487b-d5db-4324-81fb-2665e60da038/disk : Unexpected error while running command.
  2017-09-18 19:55:55.408 11607 ERROR nova.compute.manager Command: /usr/bin/python -m oslo_concurrency.prlimit --as=1073741824 --cpu=30 -- env LC_ALL=C LANG=C qemu-img info /var/lib/nova/instances/ccca487b-d5db-4324-81fb-2665e60da038/disk
  2017-09-18 19:55:55.408 11607 ERROR nova.compute.manager Exit code: 1
  2017-09-18 19:55:55.408 11607 ERROR nova.compute.manager Stdout: u''
  2017-09-18 19:55:55.408 11607 ERROR nova.compute.manager Stderr: u'qemu-img: Could not open \'/var/lib/nova/instances/ccca487b-d5db-4324-81fb-2665e60da038/disk\': Failed to get shared "write" lock\nIs another process using the image?\n'
  2017-09-18 19:55:55.408 11607 ERROR nova.compute.manager 
  2017-09-18 19:56:44.775 11607 INFO nova.compute.manager [req-b6b3e84d-88fd-4052-9e56-bc89a1739ca3 3d4ca720acb84ba19cbbc7d5042d1f56 914a8d5ddac24f1c9b74e646633bab1c - default default] [instance: ccca487b-d5db-4324-81fb-2665e60da038] Terminating instance

  Maybe it is not a nova, but qemu issue.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/nova/+bug/1718133/+subscriptions