[Bug 1823445] Re: [SRU] MetadataCopyFailure: Failed to copy metadata to volume: Glance metadata cannot be updated, key signature_verified exists for volume id <volume id>
Rodrigo Barbieri
1823445 at bugs.launchpad.net
Mon Dec 9 20:29:56 UTC 2024
** Patch added: "lp1823445_jammy_bobcat.debdiff"
https://bugs.launchpad.net/cloud-archive/+bug/1823445/+attachment/5843170/+files/lp1823445_jammy_bobcat.debdiff
--
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to cinder in Ubuntu.
https://bugs.launchpad.net/bugs/1823445
Title:
[SRU] MetadataCopyFailure: Failed to copy metadata to volume: Glance
metadata cannot be updated, key signature_verified exists for volume
id <volume id>
Status in Cinder:
Fix Released
Status in Ubuntu Cloud Archive:
New
Status in Ubuntu Cloud Archive bobcat series:
New
Status in Ubuntu Cloud Archive caracal series:
New
Status in Ubuntu Cloud Archive dalmation series:
New
Status in Ubuntu Cloud Archive yoga series:
New
Status in cinder package in Ubuntu:
New
Status in cinder source package in Jammy:
New
Status in cinder source package in Noble:
New
Status in cinder source package in Oracular:
New
Bug description:
********** SRU TEMPLATE AT THE BOTTOM *********
CentOS 7.6
Kolla Ansible 7.0.2 deployment
Steps to reproduce:
1) Add "enable_force_upload = True" to [DEFAULT] section of cinder.conf to allow clone of in-use volume to an image, restarting all Cinder containers.
2) Create an image from an in-use volume: openstack image create
--disk-format qcow2 --force --volume
7b434a13-025c-4e8a-9718-bcb5d75cae18 TestImage
In our case, the VM is powered down, but the volume is attached to
this VM, so the volume status is "in-use".
3) Deploy image to new volume with: openstack volume create --image
TestImage --size 50 TestVolume
4) Status shows as "creating" for a while, then "downloading" for a
bit. Note that we are not using Ceph storage for this test. So, when
the "creating" status is shown, the image is being copied to the
compute node temporarily. The compute node has enough disk space for
this process. Then, the download occurs, where the image is
decompressed and written to the volume, after which the cinder-volume
log indicates this was successful and that the "temporary image is
deleted".
5) The metadata error occurs and the volume status is set to "error":
MetadataCopyFailure: Failed to copy metadata to volume: Glance metadata cannot be updated, key signature_verified exists for volume id 5846f529-ff3a-4d5f-bd01-0860dbb9bcf6
The respective cinder-volume log entries are at the end of this
message (no lines were removed - just spaced out for clarity) starting
at the point in time where image->volume copy process succeeded.
The volume image metadata property of the original "in-use" volume (from Step 2 above) that was imported to an image is:
| id | 7b434a13-025c-4e8a-9718-bcb5d75cae18
| volume_image_metadata | {u'checksum': u'2a291147d0e8157d0a06097f46b32791', u'min_ram': u'0', u'disk_format': u'qcow2', u'image_name': u'TestImage', u'image_id': u'9eb38b7e-35e0-4465-b074-fb2de78990dd', u'signature_verified': u'False', u'container_format': u'bare', u'min_disk': u'0', u'size': u'9635037184'}
The properties of the resulting image (from Step 2 above) are:
| id | 3b56e3c1-02c3-4713-9248-d1581f1f7221
| properties | locations='[{u'url': u'rbd://e7b5693b-800d-49fd-a483-d184e1ebb376/images/3b56e3c1-02c3-4713-9248-d1581f1f7221/snap', u'metadata': {}}]', os_hash_algo='sha512', os_hash_value='ee09c60599dbc00f44409bd7f15ae5b720bf23e8d9c48c8d19028cf947a8e10a4f3d750815e78dfe0fb15395b6d7c9a874981b62233840971073ff6268c7eda7', os_hidden='False', signature_verified='False' |
The volume image metadata property of the newly created, error'd, image is:
| id | 5846f529-ff3a-4d5f-bd01-0860dbb9bcf6 |
| volume_image_metadata | {u'signature_verified': u'False'} |
Since it appears that the signature_verified property already exists
in the target volume, I thought this might cause the error since the
source image has this property also.
So, I unset this property on the image using:
openstack image unset --property signature_verified 3b56e3c1-02c3-4713-9248-d1581f1f7221
and the process succeeded!
This seems like a bug somewhere in the metadata property cloning
process. I would expect all volume_image_metadata to be over-written
without error during a clone.
Eric
2019-04-06 00:21:28.734 137 DEBUG
cinder.volume.flows.manager.create_volume
[req-2b67aef2-a543-48b4-b5f8-b19ecef9fe0d
f6f81f8e87994ca785b746af4f39f6ee 6baee5fa402b4850a313b7d55d7b1aba -
edcea72f06da4d5a879427b35649ffa2 edcea72f06da4d5a879427b35649ffa2]
Downloaded image 3b56e3c1-02c3-4713-9248-d1581f1f7221 ((None,
[{u'url':
u'rbd://e7b5693b-800d-49fd-a483-d184e1ebb376/images/3b56e3c1-02c3-4713-9248-d1581f1f7221/snap',
u'metadata': {}}])) to volume 5846f529-ff3a-4d5f-bd01-0860dbb9bcf6
successfully. _copy_image_to_volume
/var/lib/kolla/venv/lib/python2.7/site-
packages/cinder/volume/flows/manager/create_volume.py:557
2019-04-06 00:21:29.769 137 DEBUG cinder.image.image_utils
[req-2b67aef2-a543-48b4-b5f8-b19ecef9fe0d
f6f81f8e87994ca785b746af4f39f6ee 6baee5fa402b4850a313b7d55d7b1aba -
edcea72f06da4d5a879427b35649ffa2 edcea72f06da4d5a879427b35649ffa2]
Temporary image 3b56e3c1-02c3-4713-9248-d1581f1f7221 for user
f6f81f8e87994ca785b746af4f39f6ee is deleted. fetch
/var/lib/kolla/venv/lib/python2.7/site-
packages/cinder/image/image_utils.py:824
2019-04-06 00:21:29.769 137 DEBUG
cinder.volume.flows.manager.create_volume
[req-2b67aef2-a543-48b4-b5f8-b19ecef9fe0d
f6f81f8e87994ca785b746af4f39f6ee 6baee5fa402b4850a313b7d55d7b1aba -
edcea72f06da4d5a879427b35649ffa2 edcea72f06da4d5a879427b35649ffa2]
Marking volume 5846f529-ff3a-4d5f-bd01-0860dbb9bcf6 as bootable.
_enable_bootable_flag /var/lib/kolla/venv/lib/python2.7/site-
packages/cinder/volume/flows/manager/create_volume.py:476
2019-04-06 00:21:29.775 137 DEBUG
cinder.volume.flows.manager.create_volume
[req-2b67aef2-a543-48b4-b5f8-b19ecef9fe0d
f6f81f8e87994ca785b746af4f39f6ee 6baee5fa402b4850a313b7d55d7b1aba -
edcea72f06da4d5a879427b35649ffa2 edcea72f06da4d5a879427b35649ffa2]
Copying metadata from image 3b56e3c1-02c3-4713-9248-d1581f1f7221 to
5846f529-ff3a-4d5f-bd01-0860dbb9bcf6.
_handle_bootable_volume_glance_meta
/var/lib/kolla/venv/lib/python2.7/site-
packages/cinder/volume/flows/manager/create_volume.py:432
2019-04-06 00:21:29.776 137 DEBUG
cinder.volume.flows.manager.create_volume
[req-2b67aef2-a543-48b4-b5f8-b19ecef9fe0d
f6f81f8e87994ca785b746af4f39f6ee 6baee5fa402b4850a313b7d55d7b1aba -
edcea72f06da4d5a879427b35649ffa2 edcea72f06da4d5a879427b35649ffa2]
Creating volume glance metadata for volume
5846f529-ff3a-4d5f-bd01-0860dbb9bcf6 backed by image
3b56e3c1-02c3-4713-9248-d1581f1f7221 with: {'container_format':
u'bare', 'min_ram': 0, 'disk_format': u'qcow2', 'image_name':
u'Win2019_20190405', 'image_id':
u'3b56e3c1-02c3-4713-9248-d1581f1f7221', u'signature_verified':
u'False', 'checksum': u'29eebf5c7f1bad94c751ebdb8e9d615e', 'min_disk':
0, 'size': 13862240256}. _capture_volume_image_metadata
/var/lib/kolla/venv/lib/python2.7/site-
packages/cinder/volume/flows/manager/create_volume.py:591
2019-04-06 00:21:29.792 137 ERROR
cinder.volume.flows.manager.create_volume
[req-2b67aef2-a543-48b4-b5f8-b19ecef9fe0d
f6f81f8e87994ca785b746af4f39f6ee 6baee5fa402b4850a313b7d55d7b1aba -
edcea72f06da4d5a879427b35649ffa2 edcea72f06da4d5a879427b35649ffa2]
Failed updating volume 5846f529-ff3a-4d5f-bd01-0860dbb9bcf6 metadata
using the provided image 3b56e3c1-02c3-4713-9248-d1581f1f7221
metadata: GlanceMetadataExists: Glance metadata cannot be updated, key
signature_verified exists for volume id
5846f529-ff3a-4d5f-bd01-0860dbb9bcf6
2019-04-06 00:21:29.792 137 ERROR cinder.volume.flows.manager.create_volume Traceback (most recent call last):
2019-04-06 00:21:29.792 137 ERROR cinder.volume.flows.manager.create_volume File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py", line 434, in _handle_bootable_volume_glance_meta
2019-04-06 00:21:29.792 137 ERROR cinder.volume.flows.manager.create_volume image_id, image_meta)
2019-04-06 00:21:29.792 137 ERROR cinder.volume.flows.manager.create_volume File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py", line 593, in _capture_volume_image_metadata
2019-04-06 00:21:29.792 137 ERROR cinder.volume.flows.manager.create_volume volume_metadata)
2019-04-06 00:21:29.792 137 ERROR cinder.volume.flows.manager.create_volume File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/db/api.py", line 971, in volume_glance_metadata_bulk_create
2019-04-06 00:21:29.792 137 ERROR cinder.volume.flows.manager.create_volume metadata)
2019-04-06 00:21:29.792 137 ERROR cinder.volume.flows.manager.create_volume File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/db/sqlalchemy/api.py", line 183, in wrapper
2019-04-06 00:21:29.792 137 ERROR cinder.volume.flows.manager.create_volume return f(*args, **kwargs)
2019-04-06 00:21:29.792 137 ERROR cinder.volume.flows.manager.create_volume File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/db/sqlalchemy/api.py", line 198, in wrapper
2019-04-06 00:21:29.792 137 ERROR cinder.volume.flows.manager.create_volume return f(context, volume_id, *args, **kwargs)
2019-04-06 00:21:29.792 137 ERROR cinder.volume.flows.manager.create_volume File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/db/sqlalchemy/api.py", line 5042, in volume_glance_metadata_bulk_create
2019-04-06 00:21:29.792 137 ERROR cinder.volume.flows.manager.create_volume volume_id=volume_id)
2019-04-06 00:21:29.792 137 ERROR cinder.volume.flows.manager.create_volume GlanceMetadataExists: Glance metadata cannot be updated, key signature_verified exists for volume id 5846f529-ff3a-4d5f-bd01-0860dbb9bcf6
2019-04-06 00:21:29.792 137 ERROR cinder.volume.flows.manager.create_volume
2019-04-06 00:21:29.796 137 WARNING cinder.volume.manager [req-2b67aef2-a543-48b4-b5f8-b19ecef9fe0d f6f81f8e87994ca785b746af4f39f6ee 6baee5fa402b4850a313b7d55d7b1aba - edcea72f06da4d5a879427b35649ffa2 edcea72f06da4d5a879427b35649ffa2] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (e01844d0-4736-4762-89ff-93338d70f38b) transitioned into state 'FAILURE' from state 'RUNNING'
============
SRU TEMPLATE
============
===============
SRU DESCRIPTION
===============
[Impact]
This issue has already been addressed by another LP (LP: #1945500) but
it hadn't included all possible string properties affected to be
filtered. We stumbled across another one that hits the same issue so
we need to add the property to the list so filtered properties. The
impact is very annoying to customers because their users unknownly hit
this issue by creating a volume from an image, then creating an image
from the volume, and then attempting to create a new volume from the
derived image.
[Test case]
1) Deploy OpenStack with Cinder and have a test image in glance
openstack image create --file ~/images/cirros-0.4.0-x86_64-disk.img
cirros --disk-format qcow2
2) Create a volume from the test image
openstack volume create --image cirros --size 1 v1
3) Create a new image from the volume
openstack image create --volume v1 v1
4) Attempt to create a new volume from the derived image
openstack volume create --image v1 --size 1 v2
5) Verify in logs the error message in logs or volume show:
GlanceMetadataExists: Glance metadata cannot be updated, key
signature_verified exists
6) install updated package
7) repeat steps 2-4
8) Verify volume created successfully this time
[Regression Potential]
Given the simplicity of this particular fix, and that it completements
an existing fix (by just adding an extra filter property) that has
been in production for quite a while, we can estimate the regression
potential to be very close to zero. The "signature_verified" property
workflow that is disrupted here by being filtered is actually intended
behavior, so we can be comfortable of not breaking something else that
expects the property to be there.
[Other Info]
None.
To manage notifications about this bug go to:
https://bugs.launchpad.net/cinder/+bug/1823445/+subscriptions
More information about the Ubuntu-openstack-bugs
mailing list