[Bug 1695789] Re: multipath random crashes on use-after-free

[Rafael David Tinoco]

Example of misbehavior:

Jul 03 20:45:53 | lun02: devmap event #2
Jul 03 20:45:53 | lun02: table does not exist
Jul 03 20:45:53 | lun02: event checker exit
Jul 03 20:45:53 | uevent 'remove' from '/devices/virtual/block/dm-0'
Jul 03 20:45:53 | uevent 'remove' from '/devices/virtual/block/dm-0'
Jul 03 20:45:53 | uevent 'change' from '/devices/platform/host5/session4/target5:0:0/5:0:0:1/block/sdd'
Jul 03 20:45:54 | uevent 'remove' from '/devices/virtual/block/dm-1'
Jul 03 20:45:54 | dm-0: remove map (uevent)
Jul 03 20:45:54 | lun01: minor number mismatch (map 0, event 0)
Jul 03 20:45:54 | dm-0: remove map (uevent)
Jul 03 20:45:54 | dm-0: No DM_NAME in uevent, ignoring
Jul 03 20:45:54 | uevent trigger error
Jul 03 20:45:54 | dm-1: remove map (uevent)
Jul 03 20:45:54 | dm-1: devmap not registered, can't remove
Jul 03 20:45:54 | uevent 'remove' from '/devices/virtual/block/dm-1'
Jul 03 20:45:54 | dm-1: remove map (uevent)
Jul 03 20:45:54 | dm-1: No DM_NAME in uevent, ignoring

DM_NAME broken.

** Attachment added: "verification_old_multipathd.txt"
   https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1695789/+attachment/4908833/+files/verification_old_multipathd.txt

-- 
You received this bug notification because you are a member of Ubuntu
Sponsors Team, which is subscribed to the bug report.
https://bugs.launchpad.net/bugs/1695789

Title:
  multipath random crashes on use-after-free

Status in multipath-tools package in Ubuntu:
  Fix Released
Status in multipath-tools source package in Trusty:
  Fix Committed

Bug description:
  [Impact]

   * multipath crashes when device-mapper is modified. DM_NAME was being freed twice.
   * expect multipath daemon to crash and not run any checkers on path groups.
   * not checking path groups, in an event of failure, the mpath won't change path prios.
   * openstack relies on flushing device maps frequently when using iscsi.

  [Test Case]

   * having a multipathed environment (4 paths, 2 and 2, to a lun):
     - while true; do multipath -F ; multipath -r ; multipath -ll; done
   * run multipath with valgrind and see:

  ==31831== Invalid read of size 1
  ==31831== at 0x4C2E902: strncmp (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
  ==31831== by 0x56FC26E: find_mp_by_alias (structs.c:296)
  ==31831== by 0x404B2F: ev_add_map (main.c:264)
  ==31831== by 0x404A8B: uev_add_map (main.c:244)
  ...
  ==31831== Address 0x728d8d1 is 1 bytes inside a block of size 6 free'd
  ==31831== at 0x4C2BDEC: free (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
  ==31831== by 0x404A9A: uev_add_map (main.c:245)
  ==31831== by 0x40623C: uev_trigger (main.c:756)

  [Regression Potential]

   * using strdup for this char *, if there was no double free - like i
  discovered, would cause a slight memory leak of the size of DM_NAME
  every time a device mapper disappears and is re-created. it wouldn't
  be an important regression.

  * based on upstream commit and tested by the reported. fixes initial
  issue.

  * What releases are affected ?

   The following releases already got the fix 
   - Xenial/Yakkety/Zesty/Artful

   Note that Debian also has the fix.
   Meaning that ONLY Trusty is affected by this bug.

  * This SRU contained fixes for 2 LP bugs:
  https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1695789
  ​https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1687004

  
  [Other Info]

  It has brought to my attention that multipath in trusty has been
  crashing randomly. Some dumps were given to me and I was able to
  generate some others. I have also generated valgrind output to help me
  with these random crashes.

  Crashes:

  #0  malloc_consolidate (av=av at entry=0x7f5b58000020) at malloc.c:4149
  #1  0x00007f5b62df3cf8 in _int_malloc (av=0x7f5b58000020, bytes=16384) at malloc.c:3423
  #2  0x00007f5b62df66d0 in __GI___libc_malloc (bytes=16384) at malloc.c:2891
  #3  0x00007f5b638134d7 in dm_task_run () from /lib/x86_64-linux-gnu/libdevmapper.so.1.02.1
  #4  0x00007f5b6314be5c in dm_map_present (str=0x7f5b58000990 "lun02") at devmapper.c:304
  #5  0x0000000000404ac7 in ev_add_map (dev=, alias=, vecs=) at main.c:257
  #6  0x0000000000000000 in ?? ()

  And:

  #0  0x00007f13a5933c37 in __GI_raise (sig=sig at entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
  #1  0x00007f13a5937028 in __GI_abort () at abort.c:89
  #2  0x00007f13a59702a4 in __libc_message (do_abort=do_abort at entry=1, fmt=fmt at entry=0x7f13a5a81ef0 "") at ../sysdeps/posix/libc_fatal.c:175
  #3  0x00007f13a597c56e in malloc_printerr (ptr=<optimized out>, str=0x7f13a5a82020 "double free or corruption (out)", action=1) at malloc.c:4996
  #4  _int_free (av=<optimized out>, p=<optimized out>, have_lock=0) at malloc.c:3840
  #5  0x00007f13a5cdbe86 in free_multipath (mpp=0x7f138c033d60, free_paths=0) at structs.c:174
  #6  0x00007f13a5cfe117 in _remove_map (mpp=0x7f138c033d60, vecs=0x8adaa0, stop_waiter=1, purge_vec=1) at structs_vec.c:143
  #7  0x00007f13a5cfe175 in remove_map_and_stop_waiter (mpp=0x7f138c033d60, vecs=0x8adaa0, purge_vec=1) at structs_vec.c:156
  #8  0x0000000000406b4d in mpvec_garbage_collector (vecs=<error reading variable: can't compute CFA for this frame>) at main.c:950
  ...
  #14 0x00000000004076b7 in checkerloop (ap=<error reading variable: can't compute CFA for this frame>) at main.c:1163

  Please follow my analysis in the subsequent comments.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1695789/+subscriptions