[Bug 1462853] Re: Free list can store enormous amounts of memory

[Sworddragon]

** Description changed:

  I'm using Ubuntu 15.10 dev (x86_64) with libc6 2.21-0ubuntu4 and I have
  noticed that the free list can store a huge amount of memory and on
  analyzing this and reading manpages I have found some strange things.
  First here is a testcase to reproduce this issue (compiled with "gcc
  -Wall -pedantic -o test test.c"):
  
  #include <stdlib.h>
  #include <string.h>
  #include <unistd.h>
  #define BLOCKSIZE 4096
  #define NUMBER_BLOCKS 524288
  
  int main()
  {
- 	char *block[NUMBER_BLOCKS];
- 	int unsigned i;
+  char *block[NUMBER_BLOCKS];
+  int unsigned i;
  
- 	for(i = 0; i < NUMBER_BLOCKS; ++i)
- 		memset(block[i] = malloc(BLOCKSIZE), 0, BLOCKSIZE);
- 	for(i = 0; i < NUMBER_BLOCKS - 1; ++i)
- 		free(block[i]);
- 	pause();
- 	return 0;
+  for(i = 0; i < NUMBER_BLOCKS; ++i)
+   memset(block[i] = malloc(BLOCKSIZE), 0, BLOCKSIZE);
+  for(i = 0; i < NUMBER_BLOCKS - 1; ++i)
+   free(block[i]);
+  pause();
+  return 0;
  }
  
- 
- On executing this the application uses actively only 4 KiB on the heap while RES is ~2 GiB because of glibc's caching methods. But there are some strange things:
+ On executing this the application uses actively only 4 KiB on the heap
+ while RES is ~2 GiB because of glibc's caching methods. But there are
+ some strange things:
  
  - Freeing up the memory by decrementing in the testcase results that RES is ~5 MiB. On reading the manpage of mallopt() I'm assuming this happens because in the original testcase the last allocated block is at the end of the heap which prevents trimming it. I'm thinking that this part describes it: "(By contrast, the heap can be
-               trimmed only if memory is freed at the top end.)". If I'm right maybe this could be stated in a better context as directly before it is talked about mmap().
+               trimmed only if memory is freed at the top end.)". If I'm right maybe this could be stated in a better context as directly before it is talked about mmap().
  - Using malloc_trim() before pause() in the testcase causes that RES is ~5 MiB too. The manpage of malloc_trim() says that releasing is done at the top of the heap which should not cause this result if my previous assumption should be correct.
  
- 
- In the end I think it is not healthy that the caching behavior from glibc can reserve such huge amounts of memory. The potential performance penalty if this would cause any swapping could be insane. For example I'm seeing this caching behavior on GIMP if I'm making huge scaling operations which causes ~9 GiB to be in the free list. Who knows how many other applications are caching moderate amounts of memory without being as suspicious as in this case with GIMP.
+ In the end I think it is not healthy that the caching behavior from
+ glibc can reserve such huge amounts of memory without automatically
+ releasing it. The potential performance penalty if this would cause any
+ swapping could be insane. For example I'm seeing this caching behavior
+ on GIMP if I'm making huge scaling operations which causes ~9 GiB to be
+ in the free list. Who knows how many other applications are caching
+ moderate amounts of memory without being as suspicious as in this case
+ with GIMP.

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to glibc in Ubuntu.
https://bugs.launchpad.net/bugs/1462853

Title:
  Free list can store enormous amounts of memory

Status in glibc package in Ubuntu:
  New

Bug description:
  I'm using Ubuntu 15.10 dev (x86_64) with libc6 2.21-0ubuntu4 and I
  have noticed that the free list can store a huge amount of memory and
  on analyzing this and reading manpages I have found some strange
  things. First here is a testcase to reproduce this issue (compiled
  with "gcc -Wall -pedantic -o test test.c"):

  #include <stdlib.h>
  #include <string.h>
  #include <unistd.h>
  #define BLOCKSIZE 4096
  #define NUMBER_BLOCKS 524288

  int main()
  {
   char *block[NUMBER_BLOCKS];
   int unsigned i;

   for(i = 0; i < NUMBER_BLOCKS; ++i)
    memset(block[i] = malloc(BLOCKSIZE), 0, BLOCKSIZE);
   for(i = 0; i < NUMBER_BLOCKS - 1; ++i)
    free(block[i]);
   pause();
   return 0;
  }

  On executing this the application uses actively only 4 KiB on the heap
  while RES is ~2 GiB because of glibc's caching methods. But there are
  some strange things:

  - Freeing up the memory by decrementing in the testcase results that RES is ~5 MiB. On reading the manpage of mallopt() I'm assuming this happens because in the original testcase the last allocated block is at the end of the heap which prevents trimming it. I'm thinking that this part describes it: "(By contrast, the heap can be
                trimmed only if memory is freed at the top end.)". If I'm right maybe this could be stated in a better context as directly before it is talked about mmap().
  - Using malloc_trim() before pause() in the testcase causes that RES is ~5 MiB too. The manpage of malloc_trim() says that releasing is done at the top of the heap which should not cause this result if my previous assumption should be correct.

  In the end I think it is not healthy that the caching behavior from
  glibc can reserve such huge amounts of memory without automatically
  releasing it. The potential performance penalty if this would cause
  any swapping could be insane. For example I'm seeing this caching
  behavior on GIMP if I'm making huge scaling operations which causes ~9
  GiB to be in the free list. Who knows how many other applications are
  caching moderate amounts of memory without being as suspicious as in
  this case with GIMP.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/glibc/+bug/1462853/+subscriptions