[Bug 1607694] Re: Exclusive-Lock Issue
Daniel Kraft
daniel.kraft at d9t.de
Thu Aug 4 06:52:38 UTC 2016
We had this again last night. It hit the 14.04 machine again. This bug
seems to prefer this special VM...
librbd/ExclusiveLock.cc: In function 'std::__cxx11::string librbd::ExclusiveLock<ImageCtxT>::encode_lock_cookie() const [with ImageCtxT = librbd::ImageCtx; std::__cxx11::string = std::__cxx11::basic_string<char>]' thread 7efe88abfc00 time 2016-08-03 22:40:44.069765
librbd/ExclusiveLock.cc: 217: FAILED assert(m_watch_handle != 0)
ceph version 10.2.2 (45107e21c568dd033c2f0a3107dec8f0b0e58374)
1: (()+0x28d5d0) [0x7efe703335d0]
2: (()+0x8c785) [0x7efe70132785]
3: (()+0x8deab) [0x7efe70133eab]
4: (()+0x8d6c7) [0x7efe701336c7]
5: (()+0x74167) [0x7efe7011a167]
6: (()+0x7456d) [0x7efe7011a56d]
7: (rbd_aio_write()+0x4c) [0x7efe701067fc]
8: (()+0x2359) [0x7efe7999c359]
9: (()+0x241b) [0x7efe7999c41b]
10: (()+0x495ada) [0x564d75434ada]
11: (()+0x49809d) [0x564d7543709d]
12: (()+0x498a87) [0x564d75437a87]
13: (()+0x4633b6) [0x564d754023b6]
14: (()+0x49809d) [0x564d7543709d]
15: (()+0x498a87) [0x564d75437a87]
16: (()+0x499730) [0x564d75438730]
17: (()+0x4dd5ca) [0x564d7547c5ca]
18: (()+0x4a590) [0x7efe83724590]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
Might this correlate with creating snapshots or reading from them?
Because 4 seconds prior to this error, we took a snapshot of this block
device and created a rbd diff --whole-diff between the old snapshot and
the new snapshot...
--
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to ceph in Ubuntu.
https://bugs.launchpad.net/bugs/1607694
Title:
Exclusive-Lock Issue
Status in ceph package in Ubuntu:
Confirmed
Bug description:
After few days running a VM over RBD, under heavy loads. We get freeze
the VMs and get this log:
2016-07-26 16:20:47.069+0000: starting up libvirt version: 1.3.1, package: 1ubuntu10~cloud0 (Openstack Ubuntu Testing Bot <openstack-testing-bot at ubuntu.com> Fri, 15 Apr 2016 08:05:47 +0000), qemu version: 2.0.0 (Debian 2.0.0+dfsg-2ubuntu1.24), hostname: tckvmhv01.todoencloud
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin QEMU_AUDIO_DRV=none /usr/bin/kvm-spice -name opennebula -S -machine pc-i440fx-trusty,accel=kvm,usb=off -m 8192 -realtime mlock=off -smp 4,sockets=4,cores=1,threads=1 -uuid 7733f4c6-c72d-af36-2e84-cbc44c5c8da0 -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-opennebula/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -boot order=c,menu=on,strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive 'file=rbd:sata/opennebula.img:id=sata:key=AQB69U1Xq+bxHhAAcw7ZWZlOjE0Bf88QMsTqMQ==:auth_supported=cephx\;none:mon_host=10.255.128.238\:6789,format=raw,if=none,id=drive-virtio-disk0' -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0 -drive file=/usr/src/systemrescuecd.iso,format=raw,if=none,id=drive-ide0-1-0,readonly=on -device ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -netdev tap,fd=28,id=hostnet0,vhost=on,vhostfd=31 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:1c:07:7f,bus=pci.0,addr=0x3 -netdev tap,fd=35,id=hostnet1,vhost=on,vhostfd=36 -device virtio-net-pci,netdev=hostnet1,id=net1,mac=52:54:00:42:b6:74,bus=pci.0,addr=0x6 -netdev tap,fd=37,id=hostnet2,vhost=on,vhostfd=38 -device virtio-net-pci,netdev=hostnet2,id=net2,mac=52:54:00:2b:bb:89,bus=pci.0,addr=0x7 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -device usb-tablet,id=input0 -vnc 127.0.0.1:2 -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 -msg timestamp=on
char device redirected to /dev/pts/1 (label charserial0)
librbd/ExclusiveLock.cc: In function 'std::string librbd::ExclusiveLock<ImageCtxT>::encode_lock_cookie() const [with ImageCtxT = librbd::ImageCtx; std::string = std::basic_string<char>]' thread 7fded06a2a00 time 2016-07-28 23:44:09.845595
librbd/ExclusiveLock.cc: 197: FAILED assert(m_watch_handle != 0)
ceph version 10.2.0 (3a9fba20ec743699b69bd0181dd6c54dc01c64b9)
1: (()+0x24c4bb) [0x7fdebcbde4bb]
2: (()+0x7cf54) [0x7fdebca0ef54]
3: (()+0x7dadb) [0x7fdebca0fadb]
4: (()+0x7e827) [0x7fdebca10827]
5: (()+0x65d46) [0x7fdebc9f7d46]
6: (()+0x66218) [0x7fdebc9f8218]
7: (rbd_aio_write()+0x29) [0x7fdebc9ec629]
8: (()+0xde816) [0x7fded0108816]
9: (()+0xde910) [0x7fded0108910]
10: (()+0xb69b0) [0x7fded00e09b0]
11: (()+0xbbc44) [0x7fded00e5c44]
12: (()+0xbbc44) [0x7fded00e5c44]
13: (()+0xbc8fc) [0x7fded00e68fc]
14: (()+0xfc36a) [0x7fded012636a]
15: (()+0x49800) [0x7fdeb84ac800]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
2016-07-28 21:44:10.296+0000: shutting down
root at tckvmhv01:/var/log/libvirt/qemu# dpkg -l | grep ceph
ii ceph-common 10.2.0-0ubuntu0.16.04.1~cloud0 amd64 common utilities to mount and interact with a ceph storage cluster
ii libcephfs1 10.2.0-0ubuntu0.16.04.1~cloud0 amd64 Ceph distributed file system client library
ii python-ceph 10.2.0-0ubuntu0.16.04.1~cloud0 all Meta-package for python libraries for the Ceph libraries
ii python-cephfs 10.2.0-0ubuntu0.16.04.1~cloud0 amd64 Python libraries for the Ceph libcephfs library
root at tckvmhv01:/var/log/libvirt/qemu# dpkg -l | grep libvirt
ii libvirt-bin 1.3.1-1ubuntu10.1~cloud0 amd64 programs for the libvirt library
ii libvirt0:amd64 1.3.1-1ubuntu10.1~cloud0 amd64 library for interfacing with different virtualization systems
ii python-libvirt 1.3.1-1ubuntu1~cloud0 amd64 libvirt Python bindings
root at tckvmhv01:/var/log/libvirt/qemu# uname -a
Linux tckvmhv01 3.13.0-86-generic #131-Ubuntu SMP Thu May 12 23:33:13 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/1607694/+subscriptions
More information about the Ubuntu-openstack-bugs
mailing list