Server stops responding

[Siggy Brentrup]

On Sun, Jul 26, 2009 at 09:38 -0400, Hal Burgiss wrote:
> I have an issue with an 8.04 server, that about once a month, stops
> responding. It doesn't "crash", really, it just stops responding.
> 
> Testing open ports:
> 
> $ nmap example.com
> 
> Starting nmap 3.70 ( http://www.insecure.org/nmap/ ) at 2009-07-26 08:33 EDT
> Interesting ports on example.com:
> (The 1655 ports scanned but not shown below are in state: closed)
> PORT     STATE SERVICE
> 22/tcp   open  ssh
> 25/tcp   open  smtp
> 80/tcp   open  http
> 443/tcp  open  https
> 3306/tcp open  mysql
> 
> Looks good. Problem is none of those will fully establish connection. An
> attempt to connect via ssh:

[snip]

> 123.149.115.33 - - [26/Jul/2009:06:41:32 -0400] "GET
> /academic-programs/academic-calendar/ HTTP/1.1" 404 -
> 
> 123.149.115.33 - - [26/Jul/2009:06:41:32 -0400] "GET
> /academic-programs/academic-calendar/ HTTP/1.1" 404 - "-" "-"
> 
> This is a pretty active site. The correct time was 6:41.
> 
> Typically there is not anything interesting in syslog, but this time there was
> a bunch oom-killer actions against apache processes at 7:45. The time is wrong
> and after the wierdness started so I don't know whether to trust this. Or
> whether its an effect or a cause of another problem.
> 
> This server is headless in a datacenter, so I am limited with what I can do
> remotely (especially if I can't connect).
> 
> Any ideas how to hunt this down?

My first bet is failing memory, impossible to hunt down remotely.

Did you truncate the access.log lines? They are incomplete as
shown.

If it's such a busy site, why don't you have a backup server to
jump in in case of failure?  This incident might provide you
with the arguments to convince ${authority} you need more HW.

my 2¢
 Siggy
-- 
Please don't Cc: me when replying, I might not see either copy.
               bsb-at-psycho-dot-informationsanarchistik-dot-de
               or:                bsb-at-psycho-dot-i21k-dot-de
O< ascii ribbon campaign - stop html mail - www.asciiribbon.org
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 197 bytes
Desc: Digital signature
URL: <https://lists.ubuntu.com/archives/ubuntu-users/attachments/20090726/cb62df88/attachment.sig>