[Bug 1687004] Re: multipath crash generating core dump

[Eric Desrochers]

** Also affects: multipath-tools (Ubuntu Trusty)
   Importance: Undecided
       Status: New

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to multipath-tools in Ubuntu.
https://bugs.launchpad.net/bugs/1687004

Title:
  multipath crash generating core dump

Status in multipath-tools package in Ubuntu:
  In Progress
Status in multipath-tools source package in Trusty:
  New

Bug description:
  [Impact]

   * mutexes are being allocated from memory with no checks and, if allocation fails, it causes these kind of segfaults later in the code execution.
   expect multipath daemon to crash and not run any checkers on path groups.
   * not checking path groups, in an event of failure, the mpath won't change path prios.
   * openstack relies on flushing device maps frequently when using iscsi.

  [Test Case]

   * i'm fixing this based on a dump analysis and not on reproduction.
   * if you disallow memory overcommit - facilitating memory exhaustion - you would be able to reproduce that by stressing multipathd with paths being flushed, but that is theory only.

  [Regression Potential]

   * the patch is changing the locking mechanism for log thread, based on upstream commit.
   * major change is to use the mutexes from stack instead of allocating from heap.
   * multipath log thread could not work as designed.
   * tested by reported and reported to be good.

  [Other Info]

  It was brought to my attention that:

  multipath-tools: 0.4.9-3ubuntu7.15

  Faced a crash and generated a dump.

  ## multipath (trusty) crashed and its dump shows:

  (gdb) bt full
  #0  __GI___pthread_mutex_lock (mutex=0x0) at ../nptl/pthread_mutex_lock.c:66
          __PRETTY_FUNCTION__ = "__pthread_mutex_lock"
          type = 0
  #1  0x00007f48700b606e in flush_logqueue () at log_pthread.c:39
          empty = 0
  #2  0x00007f48700b611b in log_thread (et=0x0) at log_pthread.c:57
  No locals.
  #3  0x00007f4870964184 in start_thread (arg=0x7f4870d8b700) at pthread_create.c:312
          __res = <optimized out>
          pd = 0x7f4870d8b700
          now = <optimized out>
          unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139949107623680, -3200163692152804016, 0, 0, 139949107624384, 139949107623680, 3244383534590274896,
                  3244383107817352528}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
          not_first_call = <optimized out>
          pagesize_m1 = <optimized out>
          sp = <optimized out>
          freesize = <optimized out>
          __PRETTY_FUNCTION__ = "start_thread"
  #4  0x00007f486fdb537d in __ecvt_r (value=9.532824124368238e-130, ndigit=0, decpt=0x0, sign=0x0, buf=0x7f4870d8b9c0 "\220R\267pH\177", len=139949107623680)
      at efgcvt_r.c:218
          d = 0
          f = 3.2378592100206092e-319
          exponent = 1893250816
  #5  0x0000000000000000 in ?? ()
  No symbol table info available.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1687004/+subscriptions