[Bug 1020210] [NEW] Race condition using ATOMIC_FASTBINS in _int_free causes crash or heap corruption

[Josh Pieper]

Public bug reported:

We have an application which makes heavy allocation and de-allocation
demands from multiple threads.  We run this application continuously on
many servers, and once every several CPU months or years, we were
getting a crash in _int_free that did not look like vanilla heap
corruption.  I believe I have narrowed it down to a race condition in
_int_free due to the ATOMIC_FASTBINS feature.  Basically, in the
lockless FASTBIN _int_free path, a chunk is pulled into a local variable
with the intent to add it to the fastbins list.  However, the heap
consolidation/trim code can race with this, and can coalesce the entire
block and/or give it back to the OS before _int_free has a chance to try
and store it into the fastbins list.

The problem is very challenging to reproduce in situ, but using gdb I
have a recipe which demonstrates the crash 100% of the time on my 12.04
x64 system running eglibc 2.15.  It relies on malloc_trim, although in
our in situ data, the consolidation is triggered as a result of a normal
free.  malloc_trim is just easier to control.

While I am not a glibc developer, I could not see any easy ways to fix
the situation shy of disabling ATOMIC_FASTBINS.

I am attaching the reproduction source.  Other pertinent information
follows:

> jpieper at calculon:~/downloads$ lsb_release -rd
> Description:	Ubuntu 12.04 LTS
> Release:	12.04

> jpieper at calculon:~/downloads$ apt-cache policy libc6
> libc6:
>   Installed: 2.15-0ubuntu10
>   Candidate: 2.15-0ubuntu10
>   Version table:
>  *** 2.15-0ubuntu10 0
>        500 http://us.archive.ubuntu.com/ubuntu/ precise/main amd64 Packages
>        100 /var/lib/dpkg/status

What I expect: I expect the attached application, when run using the gdb script in the comments, to complete with no failures.
What happened: A SIGSEGV after the final continue.

** Affects: eglibc (Ubuntu)
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to eglibc in Ubuntu.
https://bugs.launchpad.net/bugs/1020210

Title:
  Race condition using ATOMIC_FASTBINS in _int_free causes crash or heap
  corruption

Status in “eglibc” package in Ubuntu:
  New

Bug description:
  We have an application which makes heavy allocation and de-allocation
  demands from multiple threads.  We run this application continuously
  on many servers, and once every several CPU months or years, we were
  getting a crash in _int_free that did not look like vanilla heap
  corruption.  I believe I have narrowed it down to a race condition in
  _int_free due to the ATOMIC_FASTBINS feature.  Basically, in the
  lockless FASTBIN _int_free path, a chunk is pulled into a local
  variable with the intent to add it to the fastbins list.  However, the
  heap consolidation/trim code can race with this, and can coalesce the
  entire block and/or give it back to the OS before _int_free has a
  chance to try and store it into the fastbins list.

  The problem is very challenging to reproduce in situ, but using gdb I
  have a recipe which demonstrates the crash 100% of the time on my
  12.04 x64 system running eglibc 2.15.  It relies on malloc_trim,
  although in our in situ data, the consolidation is triggered as a
  result of a normal free.  malloc_trim is just easier to control.

  While I am not a glibc developer, I could not see any easy ways to fix
  the situation shy of disabling ATOMIC_FASTBINS.

  I am attaching the reproduction source.  Other pertinent information
  follows:

  > jpieper at calculon:~/downloads$ lsb_release -rd
  > Description:	Ubuntu 12.04 LTS
  > Release:	12.04

  > jpieper at calculon:~/downloads$ apt-cache policy libc6
  > libc6:
  >   Installed: 2.15-0ubuntu10
  >   Candidate: 2.15-0ubuntu10
  >   Version table:
  >  *** 2.15-0ubuntu10 0
  >        500 http://us.archive.ubuntu.com/ubuntu/ precise/main amd64 Packages
  >        100 /var/lib/dpkg/status

  What I expect: I expect the attached application, when run using the gdb script in the comments, to complete with no failures.
  What happened: A SIGSEGV after the final continue.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/eglibc/+bug/1020210/+subscriptions