[ec2] swap?

[Eric Hammond]

Darren:

Did you mention which AMI (id) you are running?  This is important when reporting problems on EC2.

Amazon has the ability to perform a "hard reboot" of an instance, if a normal reboot does not work.  You can post the request with instance id to to the EC2 forum, though it might take a while for them to get around to it.  

  http://ec2forum.notlong.com

Not suitable for emergency response, but might be useful to know if this helps the situation for debugging purposes.

--
Eric Hammond


-----Original Message-----
From: Darren Govoni <darren at ontrenet.com>
Date: Sunday, Jul 26, 2009 2:07 pm
Subject: Re: [ec2] swap?
To: Robert Coup <robert.coup at koordinates.com>
CC: ec2 at lists.ubuntu.com

Hi Rob,
>   According to the launchpad bug, it seems related to a bug in the
>kernel at the moment, but as yet I am only assuming my issue is related to the documented bug because of the description.
>
>My new data shows this.
>
>My system runs fine for about 15 minutes and I see the memory dwindle down (I'm running 32 supervisord python app threads that make system
>calls). For the most part, memory is fairly static. But inches down.
>
>At one point it jumps from about 250MB free to 34MB free and that is the last update I see after it freezes. My 'top' indicates that kswapd with a nice of -5 has jumped to the top of the process list. Not sure if
>thats because of the memory situation or the cause of it.
>
>After that, the instance cannot be used, rebooted or contacted in
>anyway.
>
>I will post this on the launchpad thread too, which seems to indicate a bug in the process scheduler.
>
>But if kswapd is kicking in and there is no swap, I wonder what that would mean.
>
>Darren
>
>On Mon, 2009-07-27 at 08:23 +1200, Robert Coup wrote:
> Hi Darren,
> 
> 
> On Mon, Jul 27, 2009 at 5:44 AM, Darren Govoni <darren at ontrenet.com>
> wrote:
>         
>         The problem I am running into may be related to the other
>         posted issue
>         here when instances freeze.
>         
>         https://bugs.launchpad.net/ubuntu/+source/linux/+bug/276476
>         
>         Basically, all of my instances will freeze up after some time
>         running
>         where they are reading/writing AWS services heavily as well as
>         local
>         disk. I was concerned if its related to no virtual memory
>         swap.
> 
> 
> If you have no swap things will get killed, be it apache, a database,
> or something else. The system won't go down if it runs out of memory.
> If the system hangs it implies (to me) that it's a filesystem or
> kernel problem. What filesystem are you running? Any RAID/LVM? Are you
> using EBS or is this the local instance storage?
> 
> 
>         
>         once it freezes I cannot even reboot it. Its gone forever.
> 
> 
> So rebooting via the AWS console/tools doesn't help? Note that it can
> easily take 5 minutes or longer to reboot an instance via the AWS
> restart command. After your instance has rebooted, there may be
> something informative (about the problem) in the EC2 console output.
> 
> 
> Rob :)
> 
>