[Bug 1695789] Re: multipath random crashes on use-after-free

Mon Jul 3 20:50:56 UTC 2017

Valgrind showing wrong memory behavior due to a double free() (explained
in this bug)

==10019== Thread 3:
==10019== Invalid read of size 1
==10019==    at 0x4C2E0E2: strlen (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==10019==    by 0x56FC243: find_mp_by_alias (structs.c:295)
==10019==    by 0x571F066: update_multipath (structs_vec.c:495)
==10019==    by 0x5720986: waiteventloop (waiter.c:130)
==10019==    by 0x5720AE3: waitevent (waiter.c:162)
==10019==    by 0x4E3F183: start_thread (pthread_create.c:312)
==10019==    by 0x5A2EFFC: clone (clone.S:111)
==10019==  Address 0x731ada0 is 0 bytes inside a block of size 6 free'd
==10019==    at 0x4C2BDEC: free (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==10019==    by 0x404A9A: uev_add_map (main.c:245)
==10019==    by 0x40623C: uev_trigger (main.c:756)
==10019==    by 0x5713958: service_uevq (uevent.c:118)
==10019==    by 0x5713B67: uevent_dispatch (uevent.c:167)
==10019==    by 0x406485: uevqloop (main.c:815)
==10019==    by 0x4E3F183: start_thread (pthread_create.c:312)
==10019==    by 0x5A2EFFC: clone (clone.S:111)

==10019== Invalid read of size 2
==10019==    at 0x4C2FDC0: __GI_memcpy (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==10019==    by 0x505DB81: ??? (in /lib/x86_64-linux-gnu/libdevmapper.so.1.02.1)
==10019==    by 0x56F50E9: dm_get_status (devmapper.c:402)
==10019==    by 0x571E6FF: update_multipath_status (structs_vec.c:262)
==10019==    by 0x571E7D9: update_multipath_strings (structs_vec.c:283)
==10019==    by 0x571EA57: setup_multipath (structs_vec.c:338)
==10019==    by 0x571F0DD: update_multipath (structs_vec.c:505)
==10019==    by 0x5720986: waiteventloop (waiter.c:130)
==10019==    by 0x5720AE3: waitevent (waiter.c:162)
==10019==    by 0x4E3F183: start_thread (pthread_create.c:312)
==10019==    by 0x5A2EFFC: clone (clone.S:111)
==10019==  Address 0x7e200d0 is 0 bytes inside a block of size 6 free'd
==10019==    at 0x4C2BDEC: free (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==10019==    by 0x404A9A: uev_add_map (main.c:245)
==10019==    by 0x40623C: uev_trigger (main.c:756)
==10019==    by 0x5713958: service_uevq (uevent.c:118)
==10019==    by 0x5713B67: uevent_dispatch (uevent.c:167)
==10019==    by 0x406485: uevqloop (main.c:815)
==10019==    by 0x4E3F183: start_thread (pthread_create.c:312)
==10019==    by 0x5A2EFFC: clone (clone.S:111)

** Attachment added: "verification_old_valgrind.txt"
   https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1695789/+attachment/4908834/+files/verification_old_valgrind.txt

-- 
You received this bug notification because you are a member of Ubuntu
Sponsors Team, which is subscribed to the bug report.
https://bugs.launchpad.net/bugs/1695789

Title:
  multipath random crashes on use-after-free

Status in multipath-tools package in Ubuntu:
  Fix Released
Status in multipath-tools source package in Trusty:
  Fix Committed

Bug description:
  [Impact]

   * multipath crashes when device-mapper is modified. DM_NAME was being freed twice.
   * expect multipath daemon to crash and not run any checkers on path groups.
   * not checking path groups, in an event of failure, the mpath won't change path prios.
   * openstack relies on flushing device maps frequently when using iscsi.

  [Test Case]

   * having a multipathed environment (4 paths, 2 and 2, to a lun):
     - while true; do multipath -F ; multipath -r ; multipath -ll; done
   * run multipath with valgrind and see:

  ==31831== Invalid read of size 1
  ==31831== at 0x4C2E902: strncmp (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
  ==31831== by 0x56FC26E: find_mp_by_alias (structs.c:296)
  ==31831== by 0x404B2F: ev_add_map (main.c:264)
  ==31831== by 0x404A8B: uev_add_map (main.c:244)
  ...
  ==31831== Address 0x728d8d1 is 1 bytes inside a block of size 6 free'd
  ==31831== at 0x4C2BDEC: free (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
  ==31831== by 0x404A9A: uev_add_map (main.c:245)
  ==31831== by 0x40623C: uev_trigger (main.c:756)

  [Regression Potential]

   * using strdup for this char *, if there was no double free - like i
  discovered, would cause a slight memory leak of the size of DM_NAME
  every time a device mapper disappears and is re-created. it wouldn't
  be an important regression.

  * based on upstream commit and tested by the reported. fixes initial
  issue.

  * What releases are affected ?

   The following releases already got the fix 
   - Xenial/Yakkety/Zesty/Artful

   Note that Debian also has the fix.
   Meaning that ONLY Trusty is affected by this bug.

  * This SRU contained fixes for 2 LP bugs:
  https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1695789
  https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1687004

  [Other Info]

  It has brought to my attention that multipath in trusty has been
  crashing randomly. Some dumps were given to me and I was able to
  generate some others. I have also generated valgrind output to help me
  with these random crashes.

  Crashes:

  #0  malloc_consolidate (av=av at entry=0x7f5b58000020) at malloc.c:4149
  #1  0x00007f5b62df3cf8 in _int_malloc (av=0x7f5b58000020, bytes=16384) at malloc.c:3423
  #2  0x00007f5b62df66d0 in __GI___libc_malloc (bytes=16384) at malloc.c:2891
  #3  0x00007f5b638134d7 in dm_task_run () from /lib/x86_64-linux-gnu/libdevmapper.so.1.02.1
  #4  0x00007f5b6314be5c in dm_map_present (str=0x7f5b58000990 "lun02") at devmapper.c:304
  #5  0x0000000000404ac7 in ev_add_map (dev=, alias=, vecs=) at main.c:257
  #6  0x0000000000000000 in ?? ()

  And:

  #0  0x00007f13a5933c37 in __GI_raise (sig=sig at entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
  #1  0x00007f13a5937028 in __GI_abort () at abort.c:89
  #2  0x00007f13a59702a4 in __libc_message (do_abort=do_abort at entry=1, fmt=fmt at entry=0x7f13a5a81ef0 "") at ../sysdeps/posix/libc_fatal.c:175
  #3  0x00007f13a597c56e in malloc_printerr (ptr=<optimized out>, str=0x7f13a5a82020 "double free or corruption (out)", action=1) at malloc.c:4996
  #4  _int_free (av=<optimized out>, p=<optimized out>, have_lock=0) at malloc.c:3840
  #5  0x00007f13a5cdbe86 in free_multipath (mpp=0x7f138c033d60, free_paths=0) at structs.c:174
  #6  0x00007f13a5cfe117 in _remove_map (mpp=0x7f138c033d60, vecs=0x8adaa0, stop_waiter=1, purge_vec=1) at structs_vec.c:143
  #7  0x00007f13a5cfe175 in remove_map_and_stop_waiter (mpp=0x7f138c033d60, vecs=0x8adaa0, purge_vec=1) at structs_vec.c:156
  #8  0x0000000000406b4d in mpvec_garbage_collector (vecs=<error reading variable: can't compute CFA for this frame>) at main.c:950
  ...
  #14 0x00000000004076b7 in checkerloop (ap=<error reading variable: can't compute CFA for this frame>) at main.c:1163

  Please follow my analysis in the subsequent comments.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1695789/+subscriptions