[Bug 1020210] Re: Race condition using ATOMIC_FASTBINS in _int_free causes crash or heap corruption

[Dariusz Gadomski]

** Description changed:

  [Impact]
  
-  * This bug is likely to cause a crash with a SEGV in multithreading
+  * This bug is likely to cause a crash with a SEGV in multithreading
  applications doing many memory deallocations with ATOMIC_FASTBINS
  feature enabled.
  
  [Test Case]
  
-  * Since this is a race condition issue there is no simple path of reproducing it, however one could try to follow the instruction in the upstream bug (https://www.sourceware.org/bugzilla/show_bug.cgi?id=15073):
+  * Since this is a race condition issue there is no simple path of reproducing it, however one could try to follow the instructions in the upstream bug (https://www.sourceware.org/bugzilla/show_bug.cgi?id=15073):
  https://www.sourceware.org/bugzilla/attachment.cgi?id=7331
  
  [Regression Potential]
  
-  * This issue has been merged upstream with no further issues reported.
+  * This issue has been merged upstream with no further issues reported.
  
  [Other Info]
  
  * Original bug description:
  
  We have an application which makes heavy allocation and de-allocation
  demands from multiple threads.  We run this application continuously on
  many servers, and once every several CPU months or years, we were
  getting a crash in _int_free that did not look like vanilla heap
  corruption.  I believe I have narrowed it down to a race condition in
  _int_free due to the ATOMIC_FASTBINS feature.  Basically, in the
  lockless FASTBIN _int_free path, a chunk is pulled into a local variable
  with the intent to add it to the fastbins list.  However, the heap
  consolidation/trim code can race with this, and can coalesce the entire
  block and/or give it back to the OS before _int_free has a chance to try
  and store it into the fastbins list.
  
  The problem is very challenging to reproduce in situ, but using gdb I
  have a recipe which demonstrates the crash 100% of the time on my 12.04
  x64 system running eglibc 2.15.  It relies on malloc_trim, although in
  our in situ data, the consolidation is triggered as a result of a normal
  free.  malloc_trim is just easier to control.
  
  While I am not a glibc developer, I could not see any easy ways to fix
  the situation shy of disabling ATOMIC_FASTBINS.
  
  I am attaching the reproduction source.  Other pertinent information
  follows:
  
  > jpieper at calculon:~/downloads$ lsb_release -rd
  > Description:	Ubuntu 12.04 LTS
  > Release:	12.04
  
  > jpieper at calculon:~/downloads$ apt-cache policy libc6
  > libc6:
  >   Installed: 2.15-0ubuntu10
  >   Candidate: 2.15-0ubuntu10
  >   Version table:
  >  *** 2.15-0ubuntu10 0
  >        500 http://us.archive.ubuntu.com/ubuntu/ precise/main amd64 Packages
  >        100 /var/lib/dpkg/status
  
  What I expect: I expect the attached application, when run using the gdb script in the comments, to complete with no failures.
  What happened: A SIGSEGV after the final continue.

-- 
You received this bug notification because you are a member of Ubuntu
Sponsors Team, which is subscribed to the bug report.
https://bugs.launchpad.net/bugs/1020210

Title:
  Race condition using ATOMIC_FASTBINS in _int_free causes crash or heap
  corruption

Status in Embedded GLIBC:
  Fix Released
Status in “eglibc” package in Ubuntu:
  Fix Released
Status in “eglibc” source package in Precise:
  In Progress

Bug description:
  [Impact]

   * This bug is likely to cause a crash with a SEGV in multithreading
  applications doing many memory deallocations with ATOMIC_FASTBINS
  feature enabled.

  [Test Case]

   * Since this is a race condition issue there is no simple path of reproducing it, however one could try to follow the instructions in the upstream bug (https://www.sourceware.org/bugzilla/show_bug.cgi?id=15073):
  https://www.sourceware.org/bugzilla/attachment.cgi?id=7331

  [Regression Potential]

   * This issue has been merged upstream with no further issues
  reported.

  [Other Info]

  * Original bug description:

  We have an application which makes heavy allocation and de-allocation
  demands from multiple threads.  We run this application continuously
  on many servers, and once every several CPU months or years, we were
  getting a crash in _int_free that did not look like vanilla heap
  corruption.  I believe I have narrowed it down to a race condition in
  _int_free due to the ATOMIC_FASTBINS feature.  Basically, in the
  lockless FASTBIN _int_free path, a chunk is pulled into a local
  variable with the intent to add it to the fastbins list.  However, the
  heap consolidation/trim code can race with this, and can coalesce the
  entire block and/or give it back to the OS before _int_free has a
  chance to try and store it into the fastbins list.

  The problem is very challenging to reproduce in situ, but using gdb I
  have a recipe which demonstrates the crash 100% of the time on my
  12.04 x64 system running eglibc 2.15.  It relies on malloc_trim,
  although in our in situ data, the consolidation is triggered as a
  result of a normal free.  malloc_trim is just easier to control.

  While I am not a glibc developer, I could not see any easy ways to fix
  the situation shy of disabling ATOMIC_FASTBINS.

  I am attaching the reproduction source.  Other pertinent information
  follows:

  > jpieper at calculon:~/downloads$ lsb_release -rd
  > Description:	Ubuntu 12.04 LTS
  > Release:	12.04

  > jpieper at calculon:~/downloads$ apt-cache policy libc6
  > libc6:
  >   Installed: 2.15-0ubuntu10
  >   Candidate: 2.15-0ubuntu10
  >   Version table:
  >  *** 2.15-0ubuntu10 0
  >        500 http://us.archive.ubuntu.com/ubuntu/ precise/main amd64 Packages
  >        100 /var/lib/dpkg/status

  What I expect: I expect the attached application, when run using the gdb script in the comments, to complete with no failures.
  What happened: A SIGSEGV after the final continue.

To manage notifications about this bug go to:
https://bugs.launchpad.net/eglibc/+bug/1020210/+subscriptions