[Bug 1906266] Re: After upgrade: "libvirt.libvirtError: Requested operation is not valid: format of backing image %s of image %s was not specified"

[Brett Milford]

Ran the test plan on focal which produced the following error:

ERROR    Requested operation is not valid: format of backing image '/var/lib/libvirt/images/base/focal-server-cloudimg-amd64' of image '/var/lib/libvirt/images/testinst/root.img' was not specified in the image metadata (See https://libvirt.org/kbase/backing_chains.html for troubleshooting)
Domain installation does not appear to have been successful.
If it was, you can restart your domain by running:
  virsh --connect qemu:///system start testinst
otherwise, please restart your installation.

Ran the test plan on focal with libvirt packages from ppa:ci-train-ppa-service/4591.
The domain booted successfully.

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to nova in Ubuntu.
https://bugs.launchpad.net/bugs/1906266

Title:
  After upgrade: "libvirt.libvirtError: Requested operation is not
  valid: format of backing image %s of image %s was not specified"

Status in OpenStack Compute (nova):
  Won't Fix
Status in libvirt package in Ubuntu:
  Fix Released
Status in nova package in Ubuntu:
  New
Status in libvirt source package in Focal:
  Triaged
Status in nova source package in Focal:
  New
Status in libvirt source package in Groovy:
  Invalid
Status in nova source package in Groovy:
  New

Bug description:
  [Impact]

   * New libvirt got more strict in regard to file format specification.
     While this is generally the right approach it causes some issues for
     upgraders that have old image chains now failing.

   * Upstream has added code to relax those checks under a set of conditions
     which will allow to go forward with stricter conditions as planned but
     at the same time not break/block upgrades.

  [Test Plan]

   * Thanks to Brett Milford for sharing his test steps for this
   
  sudo apt-get update
  sudo apt-get install libvirt-daemon-system cloud-image-utils virtinst -y

  IMG="focal-server-cloudimg-amd64.img"
  IMG_PATH="/var/lib/libvirt/images/base/$IMG"
  INSTANCE_NAME=testinst
  [ -f $IMG_PATH ] || {
  sudo mkdir -p /var/lib/libvirt/images/base
  sudo wget https://cloud-images.ubuntu.com/focal/current/focal-server-cloudimg-amd64.img \
      -O $IMG_PATH
  }
  sudo mkdir -p /var/lib/libvirt/images/$INSTANCE_NAME
  sudo qemu-img convert -O raw $IMG_PATH ${IMG_PATH%.*}
  sudo qemu-img create -f qcow2 -o backing_file=${IMG_PATH%.*} /var/lib/libvirt/images/$INSTANCE_NAME/root.img
  sudo qemu-img resize /var/lib/libvirt/images/$INSTANCE_NAME/root.img 5G

  virt-install --connect qemu:///system --name $INSTANCE_NAME --cpu host
  --os-type linux --os-variant generic --graphics vnc --console
  pty,target_type=serial --disk
  path=/var/lib/libvirt/images/$INSTANCE_NAME/root.img,bus=virtio,format=qcow2
  --network default,model=virtio --noautoconsole --vcpus 1 --memory 1024
  --import


  [Where problems could occur]

   * Of the many things that qemu/libvirt do this changes only the format
     probing. So issues (hopefully not) would be expected to appear mostly
     around complex scenarios of image files.
     We've had a look at image files and image file chains, and so far all
     were good. But there are more obscure (and not supported) cases like
     image backed by real-disk that might misbehave. But still it would
     fix Focal to be the outlier as the past was ok (didn't care) and the
     future (relaxed check) and only focal is left broken in between.

  [Other Info]

   * A lot has changes in that area, but instead of pulling in a vast set
     of changes a smaller set was identified to suite the SRU needs. It
     was so far found not found regressing anything and OTOH fixed the issue
     (tested form PPA) for affected people.

  ----

  In a site upgraded to Ussuri we are getting faults starting instances

  2020-11-30 13:41:40.586 232871 ERROR oslo_messaging.rpc.server
  libvirt.libvirtError: Requested operation is not valid: format of
  backing image '/var/lib/nova/instances/_base/xxx' of image
  '/var/lib/nova/instances/xxx' was not specified in the image metadata
  (See https://libvirt.org/kbase/backing_chains.html for
  troubleshooting)

  Bug #1864020 reports similar symptoms, where due to an upstream change
  in Libvirt v6.0.0+ images need the backing format specified.

  The fix for Bug #1864020 handles the case for new instances. However,
  for upgraded instances we're hitting the same problem, as those still
  don't have backing format specified.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1906266/+subscriptions