[Bug 1616213] Re: Core dump on multipathd shutdown - trusty 14.04.4

[Dragan S.]

PPA with the fix is available at:

https://launchpad.net/~dragan-s/+archive/ubuntu/lp1616213

** Description changed:

+ [Impact]
+ 
+  * During "service multipath-tools stop" multipath daemon
+    is trying to cleanup and shut down several concurrent
+    threads. At times depending on a race condition between
+    two threads, one thread might free resources that are still
+    used by another thread.
+ 
+    This is causing the multipathd to dump crash core on
+    stop events.
+ 
+  * Fix should be backported to trusty to avoid more support
+    issues being filed.
+ 
+  * This change delays freeing resources that another thread is
+    still using.
+ 
+ [Test Case]
+ 
+  * install multipath-tools, create a basic multipath.conf with
+    devices under management. Run: "service multipath-tools start"
+    run I/O on devices and keep the system CPU busy, then run
+    "service multipath-tools stop".
+ 
+ [Regression Potential]
+ 
+  * There should be no regression potential with this change,
+    this problem happens on the exit path and we are only delaying
+    a free call.
+ 
+ [Original Description]
+ 
  On ubuntu trusty 14.04.4 in multipath-tools version 0.4.9-3ubuntu7.14
  there is bug in multipathd on shutdown.
  
  The code will access pathvec pointer which is a valid address:
  
  Reading symbols from /sbin/multipathd...Reading symbols from /usr/lib/debug//sbin/multipathd...done.
  done.
  [New LWP 41631]
  [New LWP 41584]
  [New LWP 41633]
  [New LWP 41632]
  [New LWP 41582]
  [New LWP 41583]
  [Thread debugging using libthread_db enabled]
  Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
  Core was generated by `/sbin/multipathd'.
  Program terminated with signal SIGSEGV, Segmentation fault.
- #0  0x00000000004075db in checkerloop (ap=0x1b81040) at main.c:1150
+ #0 0x00000000004075db in checkerloop (ap=0x1b81040) at main.c:1150
  
- 1150				vector_foreach_slot (vecs->pathvec, pp, i) {
+ 1150	vector_foreach_slot (vecs->pathvec, pp, i) {
  (gdb) list
- 1145			pthread_cleanup_push(cleanup_lock, &vecs->lock);
- 1146			lock(vecs->lock);
- 1147			condlog(4, "tick");
- 1148	
- 1149			if (vecs->pathvec) {
- 1150				vector_foreach_slot (vecs->pathvec, pp, i) {
- 1151					check_path(vecs, pp);
- 1152				}
- 1153			}
- 1154			if (vecs->mpvec) {
+ 1145	pthread_cleanup_push(cleanup_lock, &vecs->lock);
+ 1146	lock(vecs->lock);
+ 1147	condlog(4, "tick");
+ 1148
+ 1149	if (vecs->pathvec) {
+ 1150	vector_foreach_slot (vecs->pathvec, pp, i) {
+ 1151	check_path(vecs, pp);
+ 1152	}
+ 1153	}
+ 1154	if (vecs->mpvec) {
  
  Pathvec is a valid pointer:
  (gdb) p vecs->pathvec
  $1 = (vector) 0x1b81280
  
  But the contents of the structure are just garbage:
  
  (gdb) p *vecs->pathvec
  $2 = {allocated = 1651076143, slot = 0x756e696c2d34365f}
  (gdb)

-- 
You received this bug notification because you are a member of Ubuntu
Sponsors Team, which is subscribed to the bug report.
https://bugs.launchpad.net/bugs/1616213

Title:
  Core dump on multipathd shutdown - trusty 14.04.4

Status in multipath-tools package in Ubuntu:
  In Progress
Status in multipath-tools source package in Trusty:
  In Progress

Bug description:
  [Impact]

   * During "service multipath-tools stop" multipath daemon
     is trying to cleanup and shut down several concurrent
     threads. At times depending on a race condition between
     two threads, one thread might free resources that are still
     used by another thread.

     This is causing the multipathd to dump crash core on
     stop events.

   * Fix should be backported to trusty to avoid more support
     issues being filed.

   * This change delays freeing resources that another thread is
     still using.

  [Test Case]

   * install multipath-tools, create a basic multipath.conf with
     devices under management. Run: "service multipath-tools start"
     run I/O on devices and keep the system CPU busy, then run
     "service multipath-tools stop".

  [Regression Potential]

   * There should be no regression potential with this change,
     this problem happens on the exit path and we are only delaying
     a free call.

  [Original Description]

  On ubuntu trusty 14.04.4 in multipath-tools version 0.4.9-3ubuntu7.14
  there is bug in multipathd on shutdown.

  The code will access pathvec pointer which is a valid address:

  Reading symbols from /sbin/multipathd...Reading symbols from /usr/lib/debug//sbin/multipathd...done.
  done.
  [New LWP 41631]
  [New LWP 41584]
  [New LWP 41633]
  [New LWP 41632]
  [New LWP 41582]
  [New LWP 41583]
  [Thread debugging using libthread_db enabled]
  Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
  Core was generated by `/sbin/multipathd'.
  Program terminated with signal SIGSEGV, Segmentation fault.
  #0 0x00000000004075db in checkerloop (ap=0x1b81040) at main.c:1150

  1150	vector_foreach_slot (vecs->pathvec, pp, i) {
  (gdb) list
  1145	pthread_cleanup_push(cleanup_lock, &vecs->lock);
  1146	lock(vecs->lock);
  1147	condlog(4, "tick");
  1148
  1149	if (vecs->pathvec) {
  1150	vector_foreach_slot (vecs->pathvec, pp, i) {
  1151	check_path(vecs, pp);
  1152	}
  1153	}
  1154	if (vecs->mpvec) {

  Pathvec is a valid pointer:
  (gdb) p vecs->pathvec
  $1 = (vector) 0x1b81280

  But the contents of the structure are just garbage:

  (gdb) p *vecs->pathvec
  $2 = {allocated = 1651076143, slot = 0x756e696c2d34365f}
  (gdb)

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1616213/+subscriptions