[Bug 1215014] Re: ceph-osd from Folsom leaks memory constantly

[James Troup]

James Page <james.page at ubuntu.com> writes:

> Any chance you could enable the profiler following:
>
> http://ceph.com/docs/master/rados/troubleshooting/memory-profiling/
>
> I have a hunch that this might actually be related to the version of
> google-perftools with have in precise - but this at least will give us
> some more information about what's chewing memory.

So, given:

57274 root      20   0 1427m 713m 9932 S    1  4.5 250:20.11 /usr/bin/ceph-osd --cluster=ceph -i 34 -f
58731 root      20   0 1001m 470m  10m S   80  2.9 229:51.78 /usr/bin/ceph-osd --cluster=ceph -i 36 -f
60881 root      20   0  998m 461m  11m S    0  2.9 181:30.65 /usr/bin/ceph-osd --cluster=ceph -i 37 -f
64272 root      20   0  910m 351m  11m S    1  2.2 197:16.14 /usr/bin/ceph-osd --cluster=ceph -i 35 -f
64584 root      20   0  855m 321m  10m S    1  2.0 101:52.52 /usr/bin/ceph-osd --cluster=ceph -i 2 -f
60613 root      20   0  824m 307m 9744 S    0  1.9 164:26.79 /usr/bin/ceph-osd --cluster=ceph -i 3 -f
58493 root      20   0  785m 296m 8852 S    0  1.9 147:28.98 /usr/bin/ceph-osd --cluster=ceph -i 4 -f
64948 root      20   0  773m 273m 9.9m S    0  1.7 150:23.20 /usr/bin/ceph-osd --cluster=ceph -i 33 -f

I choose OSD 34.

2013-08-29 13:09:57.574853 7fcdbcdeb700  0 log [INF] : osd.34 started profiler 
2013-08-29 13:09:57.576890 7fcdbcdeb700  0 osd.34 6002 do_command r=0 
2013-08-29 13:10:06.076818 7fcdbcdeb700  0 log [INF] : osd.34tcmalloc heap stats:------------------------------------------------
2013-08-29 13:10:06.138029 7fcdbcdeb700  0 log [INF] : MALLOC:      666882640 (  636.0 MB) Bytes in use by application
2013-08-29 13:10:06.138057 7fcdbcdeb700  0 log [INF] : MALLOC: +     62881792 (   60.0 MB) Bytes in page heap freelist
2013-08-29 13:10:06.138080 7fcdbcdeb700  0 log [INF] : MALLOC: +      3149808 (    3.0 MB) Bytes in central cache freelist
2013-08-29 13:10:06.138097 7fcdbcdeb700  0 log [INF] : MALLOC: +        83456 (    0.1 MB) Bytes in transfer cache freelist
2013-08-29 13:10:06.208103 7fcdbcdeb700  0 log [INF] : MALLOC: +      5633984 (    5.4 MB) Bytes in thread cache freelists
2013-08-29 13:10:06.209350 7fcdbcdeb700  0 log [INF] : MALLOC: +      6160384 (    5.9 MB) Bytes in malloc metadata
2013-08-29 13:10:06.209386 7fcdbcdeb700  0 log [INF] : MALLOC:   ------------
2013-08-29 13:10:06.209404 7fcdbcdeb700  0 log [INF] : MALLOC: =    744792064 (  710.3 MB) Actual memory used (physical + swap)
2013-08-29 13:10:06.210670 7fcdbcdeb700  0 log [INF] : MALLOC: +    143736832 (  137.1 MB) Bytes released to OS (aka unmapped)
2013-08-29 13:10:06.210685 7fcdbcdeb700  0 log [INF] : MALLOC:   ------------
2013-08-29 13:10:06.210903 7fcdbcdeb700  0 log [INF] : MALLOC: =    888528896 (  847.4 MB) Virtual address space used
2013-08-29 13:10:06.210919 7fcdbcdeb700  0 log [INF] : MALLOC:
2013-08-29 13:10:06.211010 7fcdbcdeb700  0 log [INF] : MALLOC:          75893              Spans in use
2013-08-29 13:10:06.211338 7fcdbcdeb700  0 log [INF] : MALLOC:            193              Thread heaps in use
2013-08-29 13:10:06.211350 7fcdbcdeb700  0 log [INF] : MALLOC:           4096              Tcmalloc page size
2013-08-29 13:10:06.211361 7fcdbcdeb700  0 log [INF] : ------------------------------------------------
2013-08-29 13:10:06.211373 7fcdbcdeb700  0 log [INF] : Call ReleaseFreeMemory() to release freelist memory to the OS (via madvise()).
2013-08-29 13:10:06.211396 7fcdbcdeb700  0 log [INF] : Bytes released to the OS take 

I've uploaded the heap dump to

  https://chinstrap.canonical.com/~james/nx/osd.34.profile.0001.heap.gz

(This is only available to Canonical employees as it's not clear to me
 if there's any potential leak of data in the heap dump.)

Please let me know if you need any more information.

-- 
James


** Changed in: ceph (Ubuntu)
       Status: Incomplete => New

-- 
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to ceph in Ubuntu.
https://bugs.launchpad.net/bugs/1215014

Title:
  ceph-osd from Folsom leaks memory constantly

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/1215014/+subscriptions