[Bug 1945500] Re: [SRU] It's not possible to upload a volume that was build from an image back to glance, if multistore (glance) is enabled.
Rodrigo Barbieri
1945500 at bugs.launchpad.net
Wed May 3 17:09:03 UTC 2023
I have another update that will flip things over with regards to the SRU
(again).
I performed validation for jammy-yoga following the update SRU template,
therefore applying the config option change after upgrading the package
with the original failed fix, following my previous understanding from
reading the code that the original failed fix enabled this config option
workaround. This worked fine.
However, I proceeded to validate jammy-zed, but I decided to apply the
config option change before the upgrade package and retry the image-
create command, just to test. To my surprise, it worked. I performed
lots of testing following this and concluded that the config option
would have worked from the start and not need the original fix in the
first place.
Therefore, I am once again aborting this SRU because the original failed
fix as it is it is not adding anything useful, and the workaround works
without it.
Talking to Brian Rosmaita a few minutes ago and sharing the new findings
we agreed that the new fix is still very desirable so to "do things
correctly" and avoid having to do this config option workaround approach
that is not ideal.
For SRU purposes, the new fix will eventually be included in a point
release SRU and no longer need a specific SRU for this. So, again,
aborting this SRU.
--
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to Ubuntu Cloud Archive.
https://bugs.launchpad.net/bugs/1945500
Title:
[SRU] It's not possible to upload a volume that was build from an
image back to glance, if multistore (glance) is enabled.
Status in Cinder:
In Progress
Status in Ubuntu Cloud Archive:
Fix Released
Status in Ubuntu Cloud Archive yoga series:
Fix Released
Status in Ubuntu Cloud Archive zed series:
Fix Released
Status in cinder package in Ubuntu:
Fix Released
Status in cinder source package in Jammy:
Fix Committed
Status in cinder source package in Kinetic:
Fix Committed
Bug description:
* SRU TEMPLATE AT THE BOTTOM *
Brief description:
Cinder (including Wallaby release) is not able to upload a volume that was build from an image back to glance, if multistore (glance) is enabled.
Details:
After enabling glance multistore there will be two extra properties with every image. Those are `os_glance_failed_import`and `os_glance_importing_to_stores`.
If cinder creates a volume from an imageRef it will store all image_metadata with that volume, including `os_glance_failed_import`and `os_glance_importing_to_stores`.
Using the cinder action `volume_client.volumes.upload_to_image` cinder will try to upload that volume to glance including those two properties. But they are "reserved attributes" (in glance). The upload will fail with:
INFO cinder.volume.api [req-321adb05-189b-49f9-aa54-62432ee25907 917f4ddabf954b92b84babc613f0292d a69fdf3fa6654adcb8de23c803df6712 - f9f86f7344e942a5a66cec801aa60dbe f9f86f7344e942a5a66cec801aa60dbe] Volume info retrieved successfully.
INFO cinder.volume.api [req-321adb05-189b-49f9-aa54-62432ee25907 917f4ddabf954b92b84babc613f0292d a69fdf3fa6654adcb8de23c803df6712 - f9f86f7344e942a5a66cec801aa60dbe f9f86f7344e942a5a66cec801aa60dbe] Get volume image-metadata completed successfully.
DEBUG glanceclient.common.http [req-321adb05-189b-49f9-aa54-62432ee25907 917f4ddabf954b92b84babc613f0292d a69fdf3fa6654adcb8de23c803df6712 - f9f86f7344e942a5a66cec801aa60dbe f9f86f7344e942a5a66cec801aa60dbe] Request returned failure status 403. _handle_response /var/lib/kolla/venv/lib/python2.7/site-packages/glanceclient/common/http.py:125
ERROR cinder.volume.api [req-321adb05-189b-49f9-aa54-62432ee25907 917f4ddabf954b92b84babc613f0292d a69fdf3fa6654adcb8de23c803df6712 - f9f86f7344e942a5a66cec801aa60dbe f9f86f7344e942a5a66cec801aa60dbe] Error while doing something: HTTPForbidden: 403 Forbidden: Access was denied to this resource.: Attribute 'os_glance_importing_to_stores' is reserved. (HTTP 403)
ERROR cinder.volume.api Traceback (most recent call last):
ERROR cinder.volume.api File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/volume/api.py", line 1322, in copy_volume_to_image
ERROR cinder.volume.api context, self.image_service._translate_to_glance(metadata))
ERROR cinder.volume.api File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/image/glance.py", line 378, in create
ERROR cinder.volume.api **sent_service_image_meta)
ERROR cinder.volume.api File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/image/glance.py", line 225, in call
ERROR cinder.volume.api return getattr(controller, method)(*args, **kwargs)
ERROR cinder.volume.api File "/var/lib/kolla/venv/lib/python2.7/site-packages/glanceclient/common/utils.py", line 598, in inner
ERROR cinder.volume.api return RequestIdProxy(wrapped(*args, **kwargs))
ERROR cinder.volume.api File "/var/lib/kolla/venv/lib/python2.7/site-packages/glanceclient/v2/images.py", line 361, in create
ERROR cinder.volume.api resp, body = self.http_client.post(url, headers=headers, data=image)
ERROR cinder.volume.api File "/var/lib/kolla/venv/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 334, in post
ERROR cinder.volume.api return self.request(url, 'POST', **kwargs)
ERROR cinder.volume.api File "/var/lib/kolla/venv/lib/python2.7/site-packages/glanceclient/common/http.py", line 377, in request
ERROR cinder.volume.api return self._handle_response(resp)
ERROR cinder.volume.api File "/var/lib/kolla/venv/lib/python2.7/site-packages/glanceclient/common/http.py", line 126, in _handle_response
ERROR cinder.volume.api raise exc.from_response(resp, resp.content)
ERROR cinder.volume.api HTTPForbidden: 403 Forbidden: Access was denied to this resource.: Attribute 'os_glance_importing_to_stores' is reserved. (HTTP 403)
ERROR cinder.volume.api
INFO cinder.api.openstack.wsgi [req-321adb05-189b-49f9-aa54-62432ee25907 917f4ddabf954b92b84babc613f0292d a69fdf3fa6654adcb8de23c803df6712 - f9f86f7344e942a5a66cec801aa60dbe f9f86f7344e942a5a66cec801aa60dbe] HTTP exception thrown: 403 Forbidden: Access was denied to this resource.: Attribute 'os_glance_importing_to_stores' is reserved. (HTTP 403)
INFO cinder.api.openstack.wsgi [req-321adb05-189b-49f9-aa54-62432ee25907 917f4ddabf954b92b84babc613f0292d a69fdf3fa6654adcb8de23c803df6712 - f9f86f7344e942a5a66cec801aa60dbe f9f86f7344e942a5a66cec801aa60dbe] http://cinder.service.stage.ewcs.ch/v2/a69fdf3fa6654adcb8de23c803df6712/volumes/16a09728-a1d9-4032-88a4-16a32536f481/action returned with HTTP 400
This issue is known by Nova and was fixed in Nova with:
https://github.com/openstack/nova/commit/dda179d3f901e4f23091f3095f1af58bc26e222e
It looks like the issue is still unknown in cinder?
Howto reproduce:
1. Install devstack stable/wallaby
2. Change glance to enable multistore:
#####################
*** /etc/glance/glance-api.conf.org 2021-09-29 16:39:41.813610795 +0200
--- /etc/glance/glance-api.conf 2021-09-29 16:40:23.397360914 +0200
*************** image_cache_dir = /opt/stack/data/glance
*** 11,16 ****
--- 11,20 ----
use_syslog = False
debug = True
+ enabled_backends = az1:file, az2:file
+ show_multiple_locations = True
+ show_image_direct_url = True
+
[database]
connection = mysql+pymysql://xxxxxxxxx:xyzxyz@127.0.0.1/glance?charset=utf8
*************** auth_type = password
*** 35,42 ****
--- 39,58 ----
[oslo_messaging_notifications]
driver = messagingv2
+ [os_glance_staging_store]
+ filesystem_store_datadir = /opt/stack/data/glance//os_glance_staging_store
+
+ [os_glance_tasks_store]
+ filesystem_store_datadir = /opt/stack/data/glance/os_glance_tasks_store
+
[glance_store]
+ default_backend = az1
+
+ [az1]
filesystem_store_datadir = /opt/stack/data/glance/images/
+ [az2]
+ filesystem_store_datadir = /opt/stack/data/glance/images2/
+
[cors]
allowed_origin = http://172.16.0.6
#######################
mkdir /opt/stack/data/glance/images2
mkdir /opt/stack/data/glance/os_glance_tasks_store
mkdir /opt/stack/data/glance/os_glance_staging_store
systemctl restart devstack at g-api.service
3. Copy image to second store:
glance image-import --stores az2 --import-method copy-image $(openstack image show cirros-0.5.2-x86_64-disk -c id -f value)
4. Create a volume from an image:
openstack volume create --size 1 --image cirros-0.5.2-x86_64-disk testvol
5. Try to upload that volume to glance:
openstack image create --volume testvol --disk-format raw image-from-vol-from-image --debug
This will result in a 403:
HTTP 403 Forbidden: Access was denied to this resource.: Attribute 'os_glance_failed_import' is reserved. (HTTP 400)
Fix: Best would be to not store os_glance* properties with a volume
but we should also remove those properties when a volume is uploaded
to glance.
============
SRU TEMPLATE
============
[Impact]
The issue impacts workflows when downloading an image from glance and
then uploading it back to glance, because of the extra metadata. The
easy workaround is to manually delete the metadata for every image
downloaded from glance prior to uploading the volumes. The fix changed
code only on the upload-volume-to-image API to not submit the metadata
back. Additionally, the behavior can be controlled through a config
option.
[TestCase]
1. Setting up env
1a. Deploy an environment with more than 1 store for glance, such as ceph + swift
1b. Upload cirros image to glance
1c. Add the cirros image to swift store
glance image-import <image-id> --stores swift --import-method copy-
image
1d. List images including store to confirm
glance image-list --include-store
1e. Create a volume using the image
openstack volume create --size 1 --image <image-id/name> testvol
1f. Confirm the "os_glance..." image metadata is in the volume
openstack volume show testvol
2. Reproducing the issue
openstack image create --volume testvol --disk-format raw image-from-
vol-from-image
Result should be:
HTTP 403 Forbidden: Access was denied to this resource.: Attribute
'os_glance_importing_to_stores' is reserved. (HTTP 400)
3. Cleanup not needed
4. Install package that contains the fixed code
5. Adjust cinder.conf as a workaround to issue in comment #21
glance_core_properties = checksum, container_format, disk_format,
image_name, image_id, min_disk, min_ram, name, size,
os_glance_failed_import,os_glance_importing_to_stores
6. Restart cinder services if needed
7. Repeat command in (2), result should now succeed.
[Regression Potential]
Fix has been tested in the Upstream Cinder CI (not specific scenario)
and through unit tests. Behavior is configurable through config
option. In case the code for the upload-volume-to-image API breaks,
then there is no other way to upload a volume to an image.
To manage notifications about this bug go to:
https://bugs.launchpad.net/cinder/+bug/1945500/+subscriptions
More information about the Ubuntu-openstack-bugs
mailing list