[Bug 1607694] Re: Exclusive-Lock Issue

Wed Aug 17 07:52:18 UTC 2016

Thanks for the possible workaround. But fast-diff is a very important
feature for us and unfortunately disabling it is not an options for us.
The virtual machine gets killed almost every night at the moment. Is
there an estimated time window for the fix in sight?

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to ceph in Ubuntu.
https://bugs.launchpad.net/bugs/1607694

Title:
  Exclusive-Lock Issue

Status in ceph package in Ubuntu:
  Confirmed

Bug description:
  After few days running a VM over RBD, under heavy loads. We get freeze
  the VMs and get this log:

  2016-07-26 16:20:47.069+0000: starting up libvirt version: 1.3.1, package: 1ubuntu10~cloud0 (Openstack Ubuntu Testing Bot <openstack-testing-bot at ubuntu.com> Fri, 15 Apr 2016 08:05:47 +0000), qemu version: 2.0.0 (Debian 2.0.0+dfsg-2ubuntu1.24), hostname: tckvmhv01.todoencloud
  LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin QEMU_AUDIO_DRV=none /usr/bin/kvm-spice -name opennebula -S -machine pc-i440fx-trusty,accel=kvm,usb=off -m 8192 -realtime mlock=off -smp 4,sockets=4,cores=1,threads=1 -uuid 7733f4c6-c72d-af36-2e84-cbc44c5c8da0 -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-opennebula/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -boot order=c,menu=on,strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive 'file=rbd:sata/opennebula.img:id=sata:key=AQB69U1Xq+bxHhAAcw7ZWZlOjE0Bf88QMsTqMQ==:auth_supported=cephx\;none:mon_host=10.255.128.238\:6789,format=raw,if=none,id=drive-virtio-disk0' -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0 -drive file=/usr/src/systemrescuecd.iso,format=raw,if=none,id=drive-ide0-1-0,readonly=on -device ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -netdev tap,fd=28,id=hostnet0,vhost=on,vhostfd=31 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:1c:07:7f,bus=pci.0,addr=0x3 -netdev tap,fd=35,id=hostnet1,vhost=on,vhostfd=36 -device virtio-net-pci,netdev=hostnet1,id=net1,mac=52:54:00:42:b6:74,bus=pci.0,addr=0x6 -netdev tap,fd=37,id=hostnet2,vhost=on,vhostfd=38 -device virtio-net-pci,netdev=hostnet2,id=net2,mac=52:54:00:2b:bb:89,bus=pci.0,addr=0x7 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -device usb-tablet,id=input0 -vnc 127.0.0.1:2 -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 -msg timestamp=on
  char device redirected to /dev/pts/1 (label charserial0)
  librbd/ExclusiveLock.cc: In function 'std::string librbd::ExclusiveLock<ImageCtxT>::encode_lock_cookie() const [with ImageCtxT = librbd::ImageCtx; std::string = std::basic_string<char>]' thread 7fded06a2a00 time 2016-07-28 23:44:09.845595
  librbd/ExclusiveLock.cc: 197: FAILED assert(m_watch_handle != 0)
   ceph version 10.2.0 (3a9fba20ec743699b69bd0181dd6c54dc01c64b9)
   1: (()+0x24c4bb) [0x7fdebcbde4bb]
   2: (()+0x7cf54) [0x7fdebca0ef54]
   3: (()+0x7dadb) [0x7fdebca0fadb]
   4: (()+0x7e827) [0x7fdebca10827]
   5: (()+0x65d46) [0x7fdebc9f7d46]
   6: (()+0x66218) [0x7fdebc9f8218]
   7: (rbd_aio_write()+0x29) [0x7fdebc9ec629]
   8: (()+0xde816) [0x7fded0108816]
   9: (()+0xde910) [0x7fded0108910]
   10: (()+0xb69b0) [0x7fded00e09b0]
   11: (()+0xbbc44) [0x7fded00e5c44]
   12: (()+0xbbc44) [0x7fded00e5c44]
   13: (()+0xbc8fc) [0x7fded00e68fc]
   14: (()+0xfc36a) [0x7fded012636a]
   15: (()+0x49800) [0x7fdeb84ac800]
   NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
  2016-07-28 21:44:10.296+0000: shutting down

  root at tckvmhv01:/var/log/libvirt/qemu# dpkg -l | grep ceph
  ii  ceph-common                          10.2.0-0ubuntu0.16.04.1~cloud0        amd64        common utilities to mount and interact with a ceph storage cluster
  ii  libcephfs1                           10.2.0-0ubuntu0.16.04.1~cloud0        amd64        Ceph distributed file system client library
  ii  python-ceph                          10.2.0-0ubuntu0.16.04.1~cloud0        all          Meta-package for python libraries for the Ceph libraries
  ii  python-cephfs                        10.2.0-0ubuntu0.16.04.1~cloud0        amd64        Python libraries for the Ceph libcephfs library

  root at tckvmhv01:/var/log/libvirt/qemu# dpkg -l | grep libvirt
  ii  libvirt-bin                          1.3.1-1ubuntu10.1~cloud0              amd64        programs for the libvirt library
  ii  libvirt0:amd64                       1.3.1-1ubuntu10.1~cloud0              amd64        library for interfacing with different virtualization systems
  ii  python-libvirt                       1.3.1-1ubuntu1~cloud0                 amd64        libvirt Python bindings

  root at tckvmhv01:/var/log/libvirt/qemu# uname -a
  Linux tckvmhv01 3.13.0-86-generic #131-Ubuntu SMP Thu May 12 23:33:13 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/1607694/+subscriptions