[Bug 1752511] Re: snapshot size 0 and image size 0
Launchpad Bug Tracker
1752511 at bugs.launchpad.net
Tue Sep 25 04:17:51 UTC 2018
[Expired for ceph (Ubuntu) because there has been no activity for 60
days.]
** Changed in: ceph (Ubuntu)
Status: Incomplete => Expired
--
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to ceph in Ubuntu.
https://bugs.launchpad.net/bugs/1752511
Title:
snapshot size 0 and image size 0
Status in ceph package in Ubuntu:
Expired
Bug description:
Hi,everyone:
ceph - jewel 10.2.6
We use ceph as openstack storage backend, and use ceph snapshot.But
recently we met a problem about snapshot usage.
[root at flexhcs_osd_2 /]# rbd du volumes/72786dd9-ee5d-4dab-b2c6-63bb58ee2d54
NAME PROVISIONED USED
72786dd9-ee5d-4dab-b2c6-63bb58ee2d54 at snapshot-ef4c937f-c587-4039-9c66-16819c5351fa 2000G 14380M
72786dd9-ee5d-4dab-b2c6-63bb58ee2d54 at snapshot-b0c7d4af-6fe7-48f2-8f94-49e445ba5bd9 2000G 176M
72786dd9-ee5d-4dab-b2c6-63bb58ee2d54 at snapshot-804f9d90-f642-4e20-b261-0ee8b2b3392b 2000G 284M
72786dd9-ee5d-4dab-b2c6-63bb58ee2d54 at snapshot-1dc897cb-be4b-4fa9-8ed5-8ec2561b5b7f 2000G 53248k
72786dd9-ee5d-4dab-b2c6-63bb58ee2d54 at snapshot-58191be1-ef94-4860-b27f-5ba5513ba7b0 2000G 12288k
72786dd9-ee5d-4dab-b2c6-63bb58ee2d54 at snapshot-41030624-5268-4249-9ca1-41a84bdacd3e 2000G 12288k
72786dd9-ee5d-4dab-b2c6-63bb58ee2d54 at snapshot-0458fe2c-6e61-4874-81e1-afb28f80ebbb 2000G 12288k
72786dd9-ee5d-4dab-b2c6-63bb58ee2d54 at snapshot-7e4693a4-646b-4ee0-926b-a1a97d417376 2000G 0
72786dd9-ee5d-4dab-b2c6-63bb58ee2d54 at snapshot-7cecace0-2f93-48ae-902a-1eb9003c61d6 2000G 0
72786dd9-ee5d-4dab-b2c6-63bb58ee2d54 at snapshot-cccec3f9-944c-4da6-97fc-07fcc6ab1eb4 2000G 0
72786dd9-ee5d-4dab-b2c6-63bb58ee2d54 at snapshot-d87e3738-9d14-41ee-8b3e-4d1ad7485bbe 2000G 0
72786dd9-ee5d-4dab-b2c6-63bb58ee2d54 at snapshot-5f9d5cc4-e8de-4722-a7bc-08a8b9288d65 2000G 0
72786dd9-ee5d-4dab-b2c6-63bb58ee2d54 at snapshot-faae70ee-9712-4908-92f9-289ef742f181 2000G 0
72786dd9-ee5d-4dab-b2c6-63bb58ee2d54 at snapshot-39e69a9a-3d49-4369-87c5-80ffa7a2515d 2000G 9668M
72786dd9-ee5d-4dab-b2c6-63bb58ee2d54 at snapshot-06b27d8c-896a-47fb-87ba-f05a89002c73 2000G 0
72786dd9-ee5d-4dab-b2c6-63bb58ee2d54 at snapshot-48d2c155-8f2a-467d-b531-2f1b6c986e0f 2000G 0
72786dd9-ee5d-4dab-b2c6-63bb58ee2d54 at snapshot-zero-20180227 2000G 0
72786dd9-ee5d-4dab-b2c6-63bb58ee2d54 at snapshot-287fdd44-06d8-4f10-af7f-2ae3d38a0aaa 2000G 244M
72786dd9-ee5d-4dab-b2c6-63bb58ee2d54 at snapshot-a3194cc3-ccdb-4740-b627-f22741379172 2000G 0
72786dd9-ee5d-4dab-b2c6-63bb58ee2d54 at snapshot-e239e7c0-2c07-435a-957b-ba0e352bc1cb 2000G 0
72786dd9-ee5d-4dab-b2c6-63bb58ee2d54 at snapshot-0b9f3a67-7f77-4eb0-9b6c-3c53e3b8f0cc 2000G 0
72786dd9-ee5d-4dab-b2c6-63bb58ee2d54 at snapshot-9ee3354b-6482-4670-9075-502881116658 2000G 0
72786dd9-ee5d-4dab-b2c6-63bb58ee2d54 at snapshot-c568a0d4-33ca-44cf-ab86-5e1db076d8fc 2000G 0
72786dd9-ee5d-4dab-b2c6-63bb58ee2d54 at snapshot-6e7fe680-fd95-4021-807a-b6dd4aa93638 2000G 0
72786dd9-ee5d-4dab-b2c6-63bb58ee2d54 at snapshot-6fe77060-3587-41ea-9972-dc88f05fec62 2000G 0
72786dd9-ee5d-4dab-b2c6-63bb58ee2d54 at snapshot-282363cd-3652-4a23-bde8-5bebf5df16b7 2000G 0
72786dd9-ee5d-4dab-b2c6-63bb58ee2d54 at snapshot-1ac22c13-5945-4d05-bef7-14ca93795739 2000G 0
72786dd9-ee5d-4dab-b2c6-63bb58ee2d54 at snapshot-9da8bc53-deec-4687-b1b3-02ccc7b48946 2000G 0
72786dd9-ee5d-4dab-b2c6-63bb58ee2d54 at snapshot-0dffe4ad-02a4-4344-876c-af24532f356c 2000G 0
72786dd9-ee5d-4dab-b2c6-63bb58ee2d54 at snapshot-6a733d71-7d24-4b23-8142-7d3f6c00d9ee 2000G 0
72786dd9-ee5d-4dab-b2c6-63bb58ee2d54 at snapshot-165a93b7-f662-42e9-8946-f44307cfca54 2000G 0
72786dd9-ee5d-4dab-b2c6-63bb58ee2d54 at snapshot-a28bd0d2-ae8a-4787-879f-969b623f70e4 2000G 0
72786dd9-ee5d-4dab-b2c6-63bb58ee2d54 at snapshot-e8b95dc3-69c5-4ea9-a7fb-5fb7c5608b55 2000G 0
72786dd9-ee5d-4dab-b2c6-63bb58ee2d54 at snapshot-33a0e603-6c91-48bf-896e-79b87a45ecc1 2000G 0
72786dd9-ee5d-4dab-b2c6-63bb58ee2d54 at snapshot-574e89de-f7d1-4d64-905d-2c239603bbf5 2000G 0
72786dd9-ee5d-4dab-b2c6-63bb58ee2d54 at snapshot-ae91ac52-2383-42a5-b1fb-5483d46660b9 2000G 0
72786dd9-ee5d-4dab-b2c6-63bb58ee2d54 at snapshot-426f6994-cba1-4ce9-b939-bb427efa5c3c 2000G 0
72786dd9-ee5d-4dab-b2c6-63bb58ee2d54 2000G 0
<TOTAL> 2000G 24840M
Here you can see the volumes/72786dd9-ee5d-4dab-b2c6-63bb58ee2d54 has
no changes,but in fact ,we do already make changes in virtual machine
,and write data to the virtual disk.
I use the following steps to detect this problem:
[root at flexhcs_osd_2 /]# rbd info volumes/72786dd9-ee5d-4dab-b2c6-63bb58ee2d54
rbd image '72786dd9-ee5d-4dab-b2c6-63bb58ee2d54':
size 2000 GB in 512000 objects
order 22 (4096 kB objects)
block_name_prefix: rbd_data.e3bee4366f29eb
format: 2
features: layering, exclusive-lock, object-map, fast-diff, deep-flatten
flags:
parent: volumes/72786dd9-ee5d-4dab-b2c6-63bb58ee2d54_1514318054178 at snapshot-7070e07e-4aba-42ec-b4e9-3b32ce3ac8d0
overlap: 2000 GB
1)
[root at flexhcs_osd_2 /]# rados -p volumes ls | grep rbd_data.e3bee4366f29eb > ./1521.txt
[root at flexhcs_osd_2 /]# rados -p volumes ls | grep rbd_data.e3bee4366f29eb > ./1523.txt
do the command before and after my changes on the disk;
2)find the different:
[root at flexhcs_osd_2 /]# diff ./1521.txt ./1523.txt
1500a1501
> rbd_data.e3bee4366f29eb.0000000000025554
2303a2305
> rbd_data.e3bee4366f29eb.0000000000046afe
2581a2584
> rbd_data.e3bee4366f29eb.0000000000046afd
3)find which osd and pg the real data stored in fs:
[root at flexhcs_osd_2 /]# ceph osd map volumes rbd_data.e3bee4366f29eb.0000000000046afd
osdmap e25385 pool 'volumes' (2) object 'rbd_data.e3bee4366f29eb.0000000000046afd' -> pg 2.1f22456e (2.16e) -> up ([2,5], p2) acting ([2,5], p2)
4)cd osd 2 and the pg 2.16,find the "e3bee4366f29eb.0000000000046afd" data:
[root at flexhcs_osd_2 2.16e_head]# pwd
/var/lib/ceph/osd/ceph-2/current/2.16e_head
[root at flexhcs_osd_2 2.16e_head]# find ./ -name *e3bee4366f29eb.0000000000046afd*
./DIR_E/DIR_6/DIR_5/DIR_4/rbd\udata.e3bee4366f29eb.0000000000046afd__head_1F22456E__2
./DIR_E/DIR_6/DIR_5/DIR_4/rbd\udata.e3bee4366f29eb.0000000000046afd__487e_1F22456E__2
As you can see, the
"udata.e3bee4366f29eb.0000000000046afd__487e_1F22456E__2" is the snap
"COW" data.
But In Fact, the 487e snapshot id has already been deleted!
0x487e = 18558
[root at flexhcs_osd_2 current]# rbd snap ls volumes/72786dd9-ee5d-4dab-b2c6-63bb58ee2d54
SNAPID NAME SIZE
11098 snapshot-ef4c937f-c587-4039-9c66-16819c5351fa 2000 GB
14259 snapshot-b0c7d4af-6fe7-48f2-8f94-49e445ba5bd9 2000 GB
18492 snapshot-804f9d90-f642-4e20-b261-0ee8b2b3392b 2000 GB
18503 snapshot-1dc897cb-be4b-4fa9-8ed5-8ec2561b5b7f 2000 GB
18512 snapshot-58191be1-ef94-4860-b27f-5ba5513ba7b0 2000 GB
18548 snapshot-41030624-5268-4249-9ca1-41a84bdacd3e 2000 GB
18664 snapshot-0458fe2c-6e61-4874-81e1-afb28f80ebbb 2000 GB
18673 snapshot-7e4693a4-646b-4ee0-926b-a1a97d417376 2000 GB
18682 snapshot-7cecace0-2f93-48ae-902a-1eb9003c61d6 2000 GB
18718 snapshot-cccec3f9-944c-4da6-97fc-07fcc6ab1eb4 2000 GB
18729 snapshot-d87e3738-9d14-41ee-8b3e-4d1ad7485bbe 2000 GB
18776 snapshot-5f9d5cc4-e8de-4722-a7bc-08a8b9288d65 2000 GB
18788 snapshot-faae70ee-9712-4908-92f9-289ef742f181 2000 GB
18794 snapshot-39e69a9a-3d49-4369-87c5-80ffa7a2515d 2000 GB
18817 snapshot-06b27d8c-896a-47fb-87ba-f05a89002c73 2000 GB
18826 snapshot-48d2c155-8f2a-467d-b531-2f1b6c986e0f 2000 GB
18832 snapshot-zero-20180227 2000 GB
18836 snapshot-287fdd44-06d8-4f10-af7f-2ae3d38a0aaa 2000 GB
18874 snapshot-a3194cc3-ccdb-4740-b627-f22741379172 2000 GB
18884 snapshot-e239e7c0-2c07-435a-957b-ba0e352bc1cb 2000 GB
18896 snapshot-0b9f3a67-7f77-4eb0-9b6c-3c53e3b8f0cc 2000 GB
18932 snapshot-9ee3354b-6482-4670-9075-502881116658 2000 GB
18944 snapshot-c568a0d4-33ca-44cf-ab86-5e1db076d8fc 2000 GB
18972 snapshot-6e7fe680-fd95-4021-807a-b6dd4aa93638 2000 GB
18983 snapshot-6fe77060-3587-41ea-9972-dc88f05fec62 2000 GB
18992 snapshot-282363cd-3652-4a23-bde8-5bebf5df16b7 2000 GB
19028 snapshot-1ac22c13-5945-4d05-bef7-14ca93795739 2000 GB
19040 snapshot-9da8bc53-deec-4687-b1b3-02ccc7b48946 2000 GB
19050 snapshot-0dffe4ad-02a4-4344-876c-af24532f356c 2000 GB
19090 snapshot-6a733d71-7d24-4b23-8142-7d3f6c00d9ee 2000 GB
19100 snapshot-165a93b7-f662-42e9-8946-f44307cfca54 2000 GB
19128 snapshot-a28bd0d2-ae8a-4787-879f-969b623f70e4 2000 GB
19139 snapshot-e8b95dc3-69c5-4ea9-a7fb-5fb7c5608b55 2000 GB
19145 snapshot-33a0e603-6c91-48bf-896e-79b87a45ecc1 2000 GB
19146 snapshot-574e89de-f7d1-4d64-905d-2c239603bbf5 2000 GB
19150 snapshot-ae91ac52-2383-42a5-b1fb-5483d46660b9 2000 GB
19187 snapshot-426f6994-cba1-4ce9-b939-bb427efa5c3c 2000 GB
As you can see ,the 18558 snapshot id has removed.
So, We think ,maybe something wrong with the volume/72786dd9-ee5d-
4dab-b2c6-63bb58ee2d54 , it should use the latest snapshot seq id,not
the 0x487e snapshot seq id.
Maybe somebody can give some suggestion, thanks a lot ahead !!
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/1752511/+subscriptions
More information about the Ubuntu-openstack-bugs
mailing list