Unable to start new processes

[Hakan Koseoglu]

Chris,

On 24 August 2010 12:09, Karl Larsen <klarsen1 at gmail.com> wrote:
> On 08/23/2010 09:31 PM, Chris MacDonald wrote:
First let's take care of this.
>         It appears you have a problem with ssh. Please give details on
> how you have set up ssh. You should have zero problems using Tomcat on
> the remote machine.
Wrong, wrong, so wrong it's stupid.

>>  From my machine the problem manifests itself as an inability to
>> request much in the way of data from the remote machine, for instance,
>> when I SSH in (ssh -v) it opens a connection, attempts to negotiate a
>> session (I get a response from the remote machine), but then promptly
>> closes the connection remotely before I get prompted for a password.
>> Likewise for the running instance of Tomcat, I'll connect to the http
>> port, it will accept my connection, but before I get anything back it
>> closes the connection on me. I can ping the remote machine, it shows
>> ports as open, I just can't seem to get any data.
It looks like you cannot spawn any new processes. This can happen
because of a couple of main reasons. First being the ulimits being
reached. Typical Ubuntu installation does not have any limits on the
amount of memory & processes a user can consume. You can check the
limits by executing "ulimit -a". With the information given this
sounds like a memory leak where the server is starved and any new
processes are being killed. One other possibility is breaching the max
amount of open files. You can use various tools to check these. My
favourite is nmon, you can also use sar for checking cpu usage stats.

The best action is figuring out what's running on your server and how
do they behave as time goes. Nmon's capacity planning will give you
the necessary overview although you might like to collect more data.

One other thing to check is if your applications are consuming too
many ports! You might like to have a look at
net.ipv4.ip_local_port_range configuration you have. Regardless, This
is usually quite a high range, if this is happening, you have an other
problem like your processes not closing their ports after in use.

Reducing the amount of memory allocated to Tomcat might be a starting
point since that's the process most likely ballooning and leaking.
Also look for OOM killer in the message files.
-- 
Hakan (m1fcj) - http://www.hititgunesi.org