[ec2-beta] hung processes

[Eric Hammond]

Anna:

The behavior you describe sounds a lot like a hung file system.  I'm
guessing that a reboot would clear it up, but it would be good to track
down the cause.

--
Eric Hammond
ehammond at thinksome.com



Anna Hentzel wrote:
> On Wed, Feb 25, 2009 at 2:03 PM, Eric Hammond <ehammond at thinksome.com> wrote:
>> Anna:
>>
>> Are you snapshotting the EBS volumes?
>>
> 
> I'm not snapshotting yet.
> 
>> Does your snapshot process freeze the file system (e.g., xfs_freeze)?
>>
>> If so, it's possible that the snapshot process failed and left the file
>> system frozen which would cause some commands to hang, especially if
>> your current working directory might be on that file system.
>>
>> If this fits with your experience, try a command like
>>  xfs_freeze -u /mountpoint
>> or the equivalent for your file system.
> 
> I tried this anyway, and it didn't seem to affect anything.
> 
>> Also note that you generally don't want to include the EBS volume file
>> systems when rebundling to a new AMI.  Use the --exclude option to
>> exclude the mount point.
> 
> Ah, thanks for this. For some reason I hadn't realized the bundles
> were including the EBS volumes.
> 
> -Anna
> 
>> --
>> Eric Hammond
>> ehammond at thinksome.com
>>
>>
>>
>> Anna Hentzel wrote:
>>> I'm running django, lighttpd, and postgis/postgres.
>>>
>>> The postgres database is on two EBS volumes, mounted raid 0.
>>>
>>> I tried to cluster a large table last night, which I think might have
>>> been when this problem started.
>>>
>>> -Anna
>>>
>>> On Wed, Feb 25, 2009 at 1:46 PM, Eric Hammond <ehammond at thinksome.com> wrote:
>>>> Anna:
>>>>
>>>> Can you provide more background on what you are doing on the instance?
>>>>
>>>> What kinds of software are you running?
>>>>
>>>> Are you using EBS (Elastic Block Store) or any other persistent storage
>>>> mechanism?
>>>>
>>>> --
>>>> Eric Hammond
>>>> ehammond at thinksome.com
>>>>
>>>>
>>>>
>>>> Anna Hentzel wrote:
>>>>> Hi,
>>>>>
>>>>> I don't know if this is related to the ubuntu beta or not, but I'm
>>>>> getting a lot of hung processes that can't be killed even with -9.
>>>>>
>>>>> These commands hang:
>>>>> ps -ef
>>>>> sudo lsof -i tcp:3033
>>>>> strace -p 7679
>>>>>
>>>>> Also, once I started getting the hung processes, all attempts to
>>>>> upload bundles are failing. Anyone else see anything like that?
>>>>>
>>>>> Thanks
>>>>> Anna
>>>>>
>