[Bug 1607694] Re: Exclusive-Lock Issue

Wed Aug 10 20:51:30 UTC 2016

I have added

[client]
debug rbd = 20
log path = /tmp/ceph-$pid.log

to /etc/ceph/ceph.conf
but this seems to have no effect for logging. At least, I can't find anything relevant below /tmp.

We're using virsh/libvirt for kvm virtualization on a hybrid
configuration (osd hosts == VM hosts). Any hints?

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to ceph in Ubuntu.
https://bugs.launchpad.net/bugs/1607694

Title:
  Exclusive-Lock Issue

Status in ceph package in Ubuntu:
  Confirmed

Bug description:
  After few days running a VM over RBD, under heavy loads. We get freeze
  the VMs and get this log:

  2016-07-26 16:20:47.069+0000: starting up libvirt version: 1.3.1, package: 1ubuntu10~cloud0 (Openstack Ubuntu Testing Bot <openstack-testing-bot at ubuntu.com> Fri, 15 Apr 2016 08:05:47 +0000), qemu version: 2.0.0 (Debian 2.0.0+dfsg-2ubuntu1.24), hostname: tckvmhv01.todoencloud
  LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin QEMU_AUDIO_DRV=none /usr/bin/kvm-spice -name opennebula -S -machine pc-i440fx-trusty,accel=kvm,usb=off -m 8192 -realtime mlock=off -smp 4,sockets=4,cores=1,threads=1 -uuid 7733f4c6-c72d-af36-2e84-cbc44c5c8da0 -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-opennebula/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -boot order=c,menu=on,strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive 'file=rbd:sata/opennebula.img:id=sata:key=AQB69U1Xq+bxHhAAcw7ZWZlOjE0Bf88QMsTqMQ==:auth_supported=cephx\;none:mon_host=10.255.128.238\:6789,format=raw,if=none,id=drive-virtio-disk0' -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0 -drive file=/usr/src/systemrescuecd.iso,format=raw,if=none,id=drive-ide0-1-0,readonly=on -device ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -netdev tap,fd=28,id=hostnet0,vhost=on,vhostfd=31 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:1c:07:7f,bus=pci.0,addr=0x3 -netdev tap,fd=35,id=hostnet1,vhost=on,vhostfd=36 -device virtio-net-pci,netdev=hostnet1,id=net1,mac=52:54:00:42:b6:74,bus=pci.0,addr=0x6 -netdev tap,fd=37,id=hostnet2,vhost=on,vhostfd=38 -device virtio-net-pci,netdev=hostnet2,id=net2,mac=52:54:00:2b:bb:89,bus=pci.0,addr=0x7 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -device usb-tablet,id=input0 -vnc 127.0.0.1:2 -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 -msg timestamp=on
  char device redirected to /dev/pts/1 (label charserial0)
  librbd/ExclusiveLock.cc: In function 'std::string librbd::ExclusiveLock<ImageCtxT>::encode_lock_cookie() const [with ImageCtxT = librbd::ImageCtx; std::string = std::basic_string<char>]' thread 7fded06a2a00 time 2016-07-28 23:44:09.845595
  librbd/ExclusiveLock.cc: 197: FAILED assert(m_watch_handle != 0)
   ceph version 10.2.0 (3a9fba20ec743699b69bd0181dd6c54dc01c64b9)
   1: (()+0x24c4bb) [0x7fdebcbde4bb]
   2: (()+0x7cf54) [0x7fdebca0ef54]
   3: (()+0x7dadb) [0x7fdebca0fadb]
   4: (()+0x7e827) [0x7fdebca10827]
   5: (()+0x65d46) [0x7fdebc9f7d46]
   6: (()+0x66218) [0x7fdebc9f8218]
   7: (rbd_aio_write()+0x29) [0x7fdebc9ec629]
   8: (()+0xde816) [0x7fded0108816]
   9: (()+0xde910) [0x7fded0108910]
   10: (()+0xb69b0) [0x7fded00e09b0]
   11: (()+0xbbc44) [0x7fded00e5c44]
   12: (()+0xbbc44) [0x7fded00e5c44]
   13: (()+0xbc8fc) [0x7fded00e68fc]
   14: (()+0xfc36a) [0x7fded012636a]
   15: (()+0x49800) [0x7fdeb84ac800]
   NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
  2016-07-28 21:44:10.296+0000: shutting down

  root at tckvmhv01:/var/log/libvirt/qemu# dpkg -l | grep ceph
  ii  ceph-common                          10.2.0-0ubuntu0.16.04.1~cloud0        amd64        common utilities to mount and interact with a ceph storage cluster
  ii  libcephfs1                           10.2.0-0ubuntu0.16.04.1~cloud0        amd64        Ceph distributed file system client library
  ii  python-ceph                          10.2.0-0ubuntu0.16.04.1~cloud0        all          Meta-package for python libraries for the Ceph libraries
  ii  python-cephfs                        10.2.0-0ubuntu0.16.04.1~cloud0        amd64        Python libraries for the Ceph libcephfs library

  root at tckvmhv01:/var/log/libvirt/qemu# dpkg -l | grep libvirt
  ii  libvirt-bin                          1.3.1-1ubuntu10.1~cloud0              amd64        programs for the libvirt library
  ii  libvirt0:amd64                       1.3.1-1ubuntu10.1~cloud0              amd64        library for interfacing with different virtualization systems
  ii  python-libvirt                       1.3.1-1ubuntu1~cloud0                 amd64        libvirt Python bindings

  root at tckvmhv01:/var/log/libvirt/qemu# uname -a
  Linux tckvmhv01 3.13.0-86-generic #131-Ubuntu SMP Thu May 12 23:33:13 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/1607694/+subscriptions