[Bug 565101] Re: walrus reports java.lang.OutOfMemoryError: Java heap space

[Dustin Kirkland]

** Description changed:

  stress-testing, 400 instances to be started in total. After a while the
  walrus starts reporting out-of-memory, and all new instances fail.
  
  Ubuntu-server 2010-04-16, UEC 2010-04-16.
  
  ProblemType: Bug
  DistroRelease: Ubuntu 10.04
  Package: eucalyptus-walrus 1.6.2-0ubuntu30
  ProcVersionSignature: Ubuntu 2.6.32-21.31-server 2.6.32.11+drm33.2
  Uname: Linux 2.6.32-21-server x86_64
  Architecture: amd64
  Date: Fri Apr 16 19:18:49 2010
  ProcEnviron:
-  LANG=en_US.UTF-8
-  SHELL=/bin/bash
+  LANG=en_US.UTF-8
+  SHELL=/bin/bash
  SourcePackage: eucalyptus
+ 
+ ========
+ IMPACT:
+  * This bug affects UEC installs, with the CLC and Walrus on separate machines.  After running a few hundred (or thousands) of instances, the Walrus will eventually run out of memory.
+ 
+ ADDRESSED:
+  * The bug is addressed in two ways.  First, a configuration parameter has been added to /etc/eucalyptus/eucalyptus.conf, JVM_MEM, defaulting to 512M.  Admins can tailor this value to the needs of their cloud (more memory mean more instances in a shorter period of time, before the garbage collector has to run).  Secondly, we have cherry picked several commits from the upstream stable source branch that address various memory leaks in the Walrus Java code.  Both of these are necessary to fully address the problem.
+ 
+ PATCH:
+  * The patch isn't very minimal.  Sorry.  But it's what's required to solve the problem.  And it's already in the upstream stable branch.
+ 
+ REPRODUCE:
+  * Reproducing this bug is not easy.  We most frequently see this happen when the CLC and Walrus are not co-located on the same machine.  And then we run several thousand instances.  Eventually, Walrus falls over with OOM.
+ 
+ REGRESSION POTENTIAL:
+  * There's certainly regression potential in the Walrus itself, as that's the code that's changed.  However, we have put this code through a very heavy barrage of scalability testing (which also exercises many of the base features).  We believe that we have proactively tested this code thoroughly.
+ ========

** Summary changed:

- walrus reports java.lang.OutOfMemoryError: Java heap space
+ [scalability] walrus reports java.lang.OutOfMemoryError: Java heap space

-- 
[SRU] [scalability] walrus reports java.lang.OutOfMemoryError: Java heap space
https://bugs.launchpad.net/bugs/565101
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to eucalyptus in ubuntu.