[Bug 1198180] Re: possible leak in upstart 1.5

[Dawid Stawiarski]

To clarify: we have about 200-300 containers running simultaneously, and
the average lifetime of a single container is 2-10 mins; the network
interfaces are also destroyed when the container goes down. So, in a
specific point in time, we have about 200-300 network interfaces, and
they're created and destroyed dynamicly (with lxc-start and lxc-stop).

strace'ing only mmap and unmap for init (-p 1) you can see, that memory
is mapped for EVERY event, but only parts of it are unmapped:

[pid  3058] 08:43:07.444810 mmap(NULL, 677, PROT_READ, MAP_SHARED, 11, 0) = 0x7ffe40a7e000
[pid  3058] 08:43:07.444942 munmap(0x7ffe40a7e000, 677) = 0
[pid  3058] 08:43:07.446329 mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f88f1b38000
[pid  3058] 08:43:07.446604 mmap(NULL, 26711, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f88f1b31000
[pid  3058] 08:43:07.446806 mmap(NULL, 3925208, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f88f1559000
[pid  3058] 08:43:07.446856 mmap(0x7f88f190d000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1b4000) = 0x7f88f190d000
[pid  3058] 08:43:07.446965 mmap(0x7f88f1913000, 17624, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f88f1913000
[pid  3058] 08:43:07.447137 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f88f1b30000
[pid  3058] 08:43:07.447200 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f88f1b2f000
[pid  3058] 08:43:07.447250 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f88f1b2e000
[pid  3058] 08:43:07.447490 munmap(0x7f88f1b31000, 26711) = 0

We do not have any custom upstart jobs, so only /etc/init/network-
interface*.conf are run. We have also machines, where init uses more
resources (after creating/destroying about 200K containers) :

    1 root      20   0  978m 951m 1360 S   77  0.7   3410:06 /sbin/init

it uses more, and more with every event served - and no, the memory is
NOT freed after shutting down every container. Also, after some time,
init uses almost one core (77% above), and it gets sluggish -i  takes
more time to serve new events. The only way for us right now, is to
reboot the machine.

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to upstart in Ubuntu.
https://bugs.launchpad.net/bugs/1198180

Title:
  possible leak in upstart 1.5

Status in “upstart” package in Ubuntu:
  New

Bug description:
  Description:    Ubuntu 12.04.2 LTS
  Release:        12.04
  Package:    1.5-0ubuntu7.2
  Kernel:      Linux XXX 3.8.0-23-generic #34~precise1-Ubuntu SMP Wed May 29 21:12:31 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux

  We're starting lxc containers on demand, so there's lots and lots of
  network interfaces going up and down. Udev emits signal, that goes to
  upstart-udev-bridge, and next to the upstart itself. After about 40K
  containers created/destroyed, upstart takes about 250MB or RAM, and it
  grows with every event received/send. Not only RAM is beeing used, but
  also init spends more time for every new event on the socket.

  What happens:
  init uses more RAM and more CPU during lifetime, when lots of network interfaces are beeing created/destroyed

  What is expected:
  init work should not change depending on number of events served

  technical detail:
  # pmap -x 1
  00007f446f865000       0  237036  237036 rw---    [ anon ]
  00007fffc694d000       0      24      24 rw---    [ stack ]
  00007fffc69fe000       0       4       0 r-x--    [ anon ]
  ffffffffff600000       0       0       0 r-x--    [ anon ]

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/upstart/+bug/1198180/+subscriptions