[Bug 1906266] Re: After upgrade: "libvirt.libvirtError: Requested operation is not valid: format of backing image %s of image %s was not specified"

[Robie Basak]

Hello Peter, or anyone else affected,

Accepted libvirt into focal-proposed. The package will build now and be
available at
https://launchpad.net/ubuntu/+source/libvirt/6.0.0-0ubuntu8.11 in a few
hours, and then in the -proposed repository.

Please help us by testing this new package.  See
https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how
to enable and use -proposed.  Your feedback will aid us getting this
update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug,
mentioning the version of the package you tested, what testing has been
performed on the package and change the tag from verification-needed-
focal to verification-done-focal. If it does not fix the bug for you,
please add a comment stating that, and change the tag to verification-
failed-focal. In either case, without details of your testing we will
not be able to proceed.

Further information regarding the verification process can be found at
https://wiki.ubuntu.com/QATeam/PerformingSRUVerification .  Thank you in
advance for helping!

N.B. The updated package will be released to -updates after the bug(s)
fixed by this package have been verified and the package has been in
-proposed for a minimum of 7 days.

** Changed in: libvirt (Ubuntu Focal)
       Status: In Progress => Fix Committed

** Tags added: verification-needed verification-needed-focal

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to nova in Ubuntu.
https://bugs.launchpad.net/bugs/1906266

Title:
  After upgrade: "libvirt.libvirtError: Requested operation is not
  valid: format of backing image %s of image %s was not specified"

Status in OpenStack Compute (nova):
  Won't Fix
Status in libvirt package in Ubuntu:
  Fix Released
Status in nova package in Ubuntu:
  New
Status in libvirt source package in Focal:
  Fix Committed
Status in nova source package in Focal:
  New
Status in libvirt source package in Groovy:
  Fix Released
Status in nova source package in Groovy:
  New

Bug description:
  [Impact]

   * New libvirt got more strict in regard to file format specification.
     While this is generally the right approach it causes some issues for
     upgraders that have old image chains now failing.

   * Upstream has added code to relax those checks under a set of conditions
     which will allow to go forward with stricter conditions as planned but
     at the same time not break/block upgrades.

  [Test Plan]

   * Thanks to Brett Milford for sharing his test steps for this
   
  sudo apt-get update
  sudo apt-get install libvirt-daemon-system cloud-image-utils virtinst -y

  IMG="focal-server-cloudimg-amd64.img"
  IMG_PATH="/var/lib/libvirt/images/base/$IMG"
  INSTANCE_NAME=testinst
  [ -f $IMG_PATH ] || {
  sudo mkdir -p /var/lib/libvirt/images/base
  sudo wget https://cloud-images.ubuntu.com/focal/current/focal-server-cloudimg-amd64.img \
      -O $IMG_PATH
  }
  sudo mkdir -p /var/lib/libvirt/images/$INSTANCE_NAME
  sudo qemu-img convert -O raw $IMG_PATH ${IMG_PATH%.*}
  sudo qemu-img create -f qcow2 -o backing_file=${IMG_PATH%.*} /var/lib/libvirt/images/$INSTANCE_NAME/root.img
  sudo qemu-img resize /var/lib/libvirt/images/$INSTANCE_NAME/root.img 5G

  virt-install --connect qemu:///system --name $INSTANCE_NAME --cpu host
  --os-type linux --os-variant generic --graphics vnc --console
  pty,target_type=serial --disk
  path=/var/lib/libvirt/images/$INSTANCE_NAME/root.img,bus=virtio,format=qcow2
  --network default,model=virtio --noautoconsole --vcpus 1 --memory 1024
  --import


  [Where problems could occur]

   * Of the many things that qemu/libvirt do this changes only the format
     probing. So issues (hopefully not) would be expected to appear mostly
     around complex scenarios of image files.
     We've had a look at image files and image file chains, and so far all
     were good. But there are more obscure (and not supported) cases like
     image backed by real-disk that might misbehave. But still it would
     fix Focal to be the outlier as the past was ok (didn't care) and the
     future (relaxed check) and only focal is left broken in between.

  [Other Info]

   * A lot has changes in that area, but instead of pulling in a vast set
     of changes a smaller set was identified to suite the SRU needs. It
     was so far found not found regressing anything and OTOH fixed the issue
     (tested form PPA) for affected people.

  ----

  In a site upgraded to Ussuri we are getting faults starting instances

  2020-11-30 13:41:40.586 232871 ERROR oslo_messaging.rpc.server
  libvirt.libvirtError: Requested operation is not valid: format of
  backing image '/var/lib/nova/instances/_base/xxx' of image
  '/var/lib/nova/instances/xxx' was not specified in the image metadata
  (See https://libvirt.org/kbase/backing_chains.html for
  troubleshooting)

  Bug #1864020 reports similar symptoms, where due to an upstream change
  in Libvirt v6.0.0+ images need the backing format specified.

  The fix for Bug #1864020 handles the case for new instances. However,
  for upgraded instances we're hitting the same problem, as those still
  don't have backing format specified.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1906266/+subscriptions