[Bug 1945500] Re: [SRU] It's not possible to upload a volume that was build from an image back to glance, if multistore (glance) is enabled.
Rodrigo Barbieri
1945500 at bugs.launchpad.net
Fri Apr 14 15:08:22 UTC 2023
** Tags removed: verification-needed verification-needed-jammy verification-needed-kinetic verification-yoga-needed verification-zed-needed
** Tags added: verification-failed verification-failed-jammy verification-failed-kinetic verification-yoga-failed verification-zed-failed
--
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to cinder in Ubuntu.
https://bugs.launchpad.net/bugs/1945500
Title:
[SRU] It's not possible to upload a volume that was build from an
image back to glance, if multistore (glance) is enabled.
Status in Cinder:
Triaged
Status in Ubuntu Cloud Archive:
Fix Released
Status in Ubuntu Cloud Archive yoga series:
In Progress
Status in Ubuntu Cloud Archive zed series:
In Progress
Status in cinder package in Ubuntu:
Fix Released
Status in cinder source package in Jammy:
Fix Committed
Status in cinder source package in Kinetic:
Fix Committed
Bug description:
* SRU TEMPLATE AT THE BOTTOM *
Brief description:
Cinder (including Wallaby release) is not able to upload a volume that was build from an image back to glance, if multistore (glance) is enabled.
Details:
After enabling glance multistore there will be two extra properties with every image. Those are `os_glance_failed_import`and `os_glance_importing_to_stores`.
If cinder creates a volume from an imageRef it will store all image_metadata with that volume, including `os_glance_failed_import`and `os_glance_importing_to_stores`.
Using the cinder action `volume_client.volumes.upload_to_image` cinder will try to upload that volume to glance including those two properties. But they are "reserved attributes" (in glance). The upload will fail with:
INFO cinder.volume.api [req-321adb05-189b-49f9-aa54-62432ee25907 917f4ddabf954b92b84babc613f0292d a69fdf3fa6654adcb8de23c803df6712 - f9f86f7344e942a5a66cec801aa60dbe f9f86f7344e942a5a66cec801aa60dbe] Volume info retrieved successfully.
INFO cinder.volume.api [req-321adb05-189b-49f9-aa54-62432ee25907 917f4ddabf954b92b84babc613f0292d a69fdf3fa6654adcb8de23c803df6712 - f9f86f7344e942a5a66cec801aa60dbe f9f86f7344e942a5a66cec801aa60dbe] Get volume image-metadata completed successfully.
DEBUG glanceclient.common.http [req-321adb05-189b-49f9-aa54-62432ee25907 917f4ddabf954b92b84babc613f0292d a69fdf3fa6654adcb8de23c803df6712 - f9f86f7344e942a5a66cec801aa60dbe f9f86f7344e942a5a66cec801aa60dbe] Request returned failure status 403. _handle_response /var/lib/kolla/venv/lib/python2.7/site-packages/glanceclient/common/http.py:125
ERROR cinder.volume.api [req-321adb05-189b-49f9-aa54-62432ee25907 917f4ddabf954b92b84babc613f0292d a69fdf3fa6654adcb8de23c803df6712 - f9f86f7344e942a5a66cec801aa60dbe f9f86f7344e942a5a66cec801aa60dbe] Error while doing something: HTTPForbidden: 403 Forbidden: Access was denied to this resource.: Attribute 'os_glance_importing_to_stores' is reserved. (HTTP 403)
ERROR cinder.volume.api Traceback (most recent call last):
ERROR cinder.volume.api File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/volume/api.py", line 1322, in copy_volume_to_image
ERROR cinder.volume.api context, self.image_service._translate_to_glance(metadata))
ERROR cinder.volume.api File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/image/glance.py", line 378, in create
ERROR cinder.volume.api **sent_service_image_meta)
ERROR cinder.volume.api File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/image/glance.py", line 225, in call
ERROR cinder.volume.api return getattr(controller, method)(*args, **kwargs)
ERROR cinder.volume.api File "/var/lib/kolla/venv/lib/python2.7/site-packages/glanceclient/common/utils.py", line 598, in inner
ERROR cinder.volume.api return RequestIdProxy(wrapped(*args, **kwargs))
ERROR cinder.volume.api File "/var/lib/kolla/venv/lib/python2.7/site-packages/glanceclient/v2/images.py", line 361, in create
ERROR cinder.volume.api resp, body = self.http_client.post(url, headers=headers, data=image)
ERROR cinder.volume.api File "/var/lib/kolla/venv/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 334, in post
ERROR cinder.volume.api return self.request(url, 'POST', **kwargs)
ERROR cinder.volume.api File "/var/lib/kolla/venv/lib/python2.7/site-packages/glanceclient/common/http.py", line 377, in request
ERROR cinder.volume.api return self._handle_response(resp)
ERROR cinder.volume.api File "/var/lib/kolla/venv/lib/python2.7/site-packages/glanceclient/common/http.py", line 126, in _handle_response
ERROR cinder.volume.api raise exc.from_response(resp, resp.content)
ERROR cinder.volume.api HTTPForbidden: 403 Forbidden: Access was denied to this resource.: Attribute 'os_glance_importing_to_stores' is reserved. (HTTP 403)
ERROR cinder.volume.api
INFO cinder.api.openstack.wsgi [req-321adb05-189b-49f9-aa54-62432ee25907 917f4ddabf954b92b84babc613f0292d a69fdf3fa6654adcb8de23c803df6712 - f9f86f7344e942a5a66cec801aa60dbe f9f86f7344e942a5a66cec801aa60dbe] HTTP exception thrown: 403 Forbidden: Access was denied to this resource.: Attribute 'os_glance_importing_to_stores' is reserved. (HTTP 403)
INFO cinder.api.openstack.wsgi [req-321adb05-189b-49f9-aa54-62432ee25907 917f4ddabf954b92b84babc613f0292d a69fdf3fa6654adcb8de23c803df6712 - f9f86f7344e942a5a66cec801aa60dbe f9f86f7344e942a5a66cec801aa60dbe] http://cinder.service.stage.ewcs.ch/v2/a69fdf3fa6654adcb8de23c803df6712/volumes/16a09728-a1d9-4032-88a4-16a32536f481/action returned with HTTP 400
This issue is known by Nova and was fixed in Nova with:
https://github.com/openstack/nova/commit/dda179d3f901e4f23091f3095f1af58bc26e222e
It looks like the issue is still unknown in cinder?
Howto reproduce:
1. Install devstack stable/wallaby
2. Change glance to enable multistore:
#####################
*** /etc/glance/glance-api.conf.org 2021-09-29 16:39:41.813610795 +0200
--- /etc/glance/glance-api.conf 2021-09-29 16:40:23.397360914 +0200
*************** image_cache_dir = /opt/stack/data/glance
*** 11,16 ****
--- 11,20 ----
use_syslog = False
debug = True
+ enabled_backends = az1:file, az2:file
+ show_multiple_locations = True
+ show_image_direct_url = True
+
[database]
connection = mysql+pymysql://xxxxxxxxx:xyzxyz@127.0.0.1/glance?charset=utf8
*************** auth_type = password
*** 35,42 ****
--- 39,58 ----
[oslo_messaging_notifications]
driver = messagingv2
+ [os_glance_staging_store]
+ filesystem_store_datadir = /opt/stack/data/glance//os_glance_staging_store
+
+ [os_glance_tasks_store]
+ filesystem_store_datadir = /opt/stack/data/glance/os_glance_tasks_store
+
[glance_store]
+ default_backend = az1
+
+ [az1]
filesystem_store_datadir = /opt/stack/data/glance/images/
+ [az2]
+ filesystem_store_datadir = /opt/stack/data/glance/images2/
+
[cors]
allowed_origin = http://172.16.0.6
#######################
mkdir /opt/stack/data/glance/images2
mkdir /opt/stack/data/glance/os_glance_tasks_store
mkdir /opt/stack/data/glance/os_glance_staging_store
systemctl restart devstack at g-api.service
3. Copy image to second store:
glance image-import --stores az2 --import-method copy-image $(openstack image show cirros-0.5.2-x86_64-disk -c id -f value)
4. Create a volume from an image:
openstack volume create --size 1 --image cirros-0.5.2-x86_64-disk testvol
5. Try to upload that volume to glance:
openstack image create --volume testvol --disk-format raw image-from-vol-from-image --debug
This will result in a 403:
HTTP 403 Forbidden: Access was denied to this resource.: Attribute 'os_glance_failed_import' is reserved. (HTTP 400)
Fix: Best would be to not store os_glance* properties with a volume
but we should also remove those properties when a volume is uploaded
to glance.
============
SRU TEMPLATE
============
[Impact]
The issue impacts workflows when downloading an image from glance and
then uploading it back to glance, because of the extra metadata. The
easy workaround is to manually delete the metadata for every image
downloaded from glance prior to uploading the volumes. The fix changed
code only on the upload-volume-to-image API to not submit the metadata
back. Additionally, the behavior can be controlled through a config
option.
[TestCase]
1. Setting up env
1a. Deploy an environment with more than 1 store for glance, such as ceph + swift
1b. Upload cirros image to glance
1c. Add the cirros image to swift store
glance image-import <image-id> --stores swift --import-method copy-
image
1d. List images including store to confirm
glance image-list --include-store
1e. Create a volume using the image
openstack volume create --size 1 --image <image-id/name> testvol
1f. Confirm the "os_glance..." image metadata is in the volume
openstack volume show testvol
2. Reproducing the issue
openstack image create --volume testvol --disk-format raw image-from-
vol-from-image
Result should be:
HTTP 403 Forbidden: Access was denied to this resource.: Attribute
'os_glance_importing_to_stores' is reserved. (HTTP 400)
3. Cleanup not needed
4. Install package that contains the fixed code
5. Repeat command in (2), result should now succeed.
[Regression Potential]
Fix has been tested in the Upstream Cinder CI (not specific scenario)
and through unit tests. Behavior is configurable through config
option. In case the code for the upload-volume-to-image API breaks,
then there is no other way to upload a volume to an image.
To manage notifications about this bug go to:
https://bugs.launchpad.net/cinder/+bug/1945500/+subscriptions
More information about the Ubuntu-openstack-bugs
mailing list