[Bug 1731819] Re: rbd snap_unprotect deadlock

[James Page]

Work inflight upstream to fix and then backport to L and J; Ubuntu will
pick those up with the next set of point releases.

** Changed in: ceph (Ubuntu)
       Status: New => Triaged

** Changed in: ceph (Ubuntu)
   Importance: Undecided => Medium

** Also affects: ceph (Ubuntu Artful)
   Importance: Undecided
       Status: New

** Also affects: ceph (Ubuntu Xenial)
   Importance: Undecided
       Status: New

** Also affects: ceph (Ubuntu Bionic)
   Importance: Medium
       Status: Triaged

** Also affects: ceph (Ubuntu Zesty)
   Importance: Undecided
       Status: New

** Changed in: ceph (Ubuntu Artful)
       Status: New => Triaged

** Changed in: ceph (Ubuntu Zesty)
       Status: New => Triaged

** Changed in: ceph (Ubuntu Xenial)
       Status: New => Triaged

** Changed in: ceph (Ubuntu Xenial)
   Importance: Undecided => Medium

** Changed in: ceph (Ubuntu Zesty)
   Importance: Undecided => Medium

** Changed in: ceph (Ubuntu Artful)
   Importance: Undecided => Medium

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to ceph in Ubuntu.
https://bugs.launchpad.net/bugs/1731819

Title:
  rbd snap_unprotect deadlock

Status in ceph package in Ubuntu:
  Triaged
Status in ceph source package in Xenial:
  Triaged
Status in ceph source package in Zesty:
  Triaged
Status in ceph source package in Artful:
  Triaged
Status in ceph source package in Bionic:
  Triaged

Bug description:
  Hello everyone:

  I'm using openstack mitaka and ceph-jewel-10.2.10 to do snapshot
  actions. And sometimes it occurs below deadlock condition.

  ps -ef | grep cinder-volume
  gdb -q python-dbg -p xx

  I found 2 frames are racing the lock:

  Thread 14 (Thread 0x7f510784c700 (LWP 759193)):
  #0  0x00007f513272603e in pthread_rwlock_wrlock () from /lib64/libpthread.so.0
  #1  0x00007f5112a4a83c in RWLock::get_write (this=0x5db1258, lockdep=<optimized out>) at ./common/RWLock.h:123
  #2  0x00007f5112ad77c5 in WLocker (lock=..., this=<synthetic pointer>) at ./common/RWLock.h:183
  #3  librbd::image::RefreshRequest<librbd::ImageCtx>::apply (this=this at entry=0x7f507c02bf10) at librbd/image/RefreshRequest.cc:855
  #4  0x00007f5112ad87f8 in librbd::image::RefreshRequest<librbd::ImageCtx>::handle_v2_apply (this=0x7f507c02bf10, result=result at entry=0x7f510784bb2c) at librbd/image/RefreshRequest.cc:655
  #5  0x00007f5112ad89ab in librbd::util::detail::C_StateCallbackAdapter<librbd::image::RefreshRequest<librbd::ImageCtx>, &librbd::image::RefreshRequest<librbd::ImageCtx>::handle_v2_apply, true>::complete (this=0x7f507c31e7b0, r=0) at ./librbd/Utils.h:66
  #6  0x00007f5112a3eb54 in ContextWQ::process (this=0x6e96b20, ctx=0x7f507c31e7b0) at ./common/WorkQueue.h:611
  #7  0x00007f5112c37a7e in ThreadPool::worker (this=0x7c222b0, wt=0x60fe290) at common/WorkQueue.cc:128
  #8  0x00007f5112c38950 in ThreadPool::WorkThread::entry (this=<optimized out>) at common/WorkQueue.h:448
  #9  0x00007f5132722dc5 in start_thread () from /lib64/libpthread.so.0
  #10 0x00007f5131d4873d in clone () from /lib64/libc.so.6

  
  Thread 1 (Thread 0x7f5132f10740 (LWP 2617826)):
  #0  0x00007f51327266d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  #1  0x00007f5112a14b60 in Wait (mutex=..., this=0x7ffd0e2ba8f0) at ./common/Cond.h:56
  #2  C_SaferCond::wait (this=this at entry=0x7ffd0e2ba890) at ./common/Cond.h:202
  #3  0x00007f5112ab5b8e in librbd::Operations<librbd::ImageCtx>::snap_unprotect (this=0x562bdb0, snap_name=snap_name at entry=0x58fa894 "snapshot-90259d85-3edc-40e7-b306-cfff1b855cd6")
      at librbd/Operations.cc:1079
  #4  0x00007f51129fb0d4 in rbd_snap_unprotect (image=0x5db10c0, snap_name=snap_name at entry=0x58fa894 "snapshot-90259d85-3edc-40e7-b306-cfff1b855cd6") at librbd/librbd.cc:2385
  #5  0x00007f511c32f427 in __pyx_pf_3rbd_5Image_50unprotect_snap (__pyx_v_self=0x7017260, __pyx_v_self=0x7017260, __pyx_v_name=0x58fa870) at rbd.c:12928
  #6  __pyx_pw_3rbd_5Image_51unprotect_snap (__pyx_v_self=0x7017260, __pyx_v_name=<optimized out>) at rbd.c:12843
  #7  0x00007f5132a1ba62 in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
  ......

  
  The full backtrace is in the attachment.

  Thancks a lot if anyone can give advise, ^ _ ^

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/1731819/+subscriptions