[ec2-beta] data corruption

[Ben Hendrickson]

On Tue, Apr 14, 2009 at 4:10 PM, Eric Hammond <ehammond at thinksome.com> wrote:
> Ben, it wasn't quite clear to me exactly how many instances you were
> running for how long on each of the different images and how many
> failures you experienced.  I'm trying to figure out if the failure rate
> is statistically significant or if it might simply be an

About 85% of the time I have 16-20 large instances running, and 15% of
the time I have 0 machines running.  We switched to the beta images
towards the end of January, and switched back to the Alestic images on
March 29th.  So we were running the beta images for 8 weeks.  I don't
actually have a record when I upgraded from Beta 1 to Beta 2, but I
think I switched to the Beta 1 in early March, and afterward I
continued to see corruptions at roughly the same pace.  I didn't keep
track of exactly how many corruptions I saw, but roughly 12.  Most of
the corruptions were on different machines (I started swapping
machines out as soon as they had their first corruption).  Thinking it
might be a data center issue, I swapped availability zones, but
continued to see corruptions.

Regarding time with the Alestic AMIs, I had been using the Alestic
images for several months prior to switching to the beta, and since
March 29th I've had 20 machines running continually with the Alestic
images.  And during the time with the Alestic images, there have been
zero corruptions.  Currently I'm using the 8.10 Alestic AMI
(ami-bcfe19d5), but previously was using the Alestic 8.04 AMI without
issue.

On Tue, Apr 14, 2009 at 12:54 PM, Mark Shuttleworth <mark at ubuntu.com> wrote:
> Is ReiserFS integral to the solution, or a personal preference? It jumped
> out at me as an area of risk, as it's not a filesystem we're particularly
> focused on. Ext3, and the newer ext4 and ultimately btrfs would be the
> "stable, next, future" default filesystems we'd recommend unless there was a
> specific technical reason to do otherwise. If Reiser isn't integral I'd be
> interested in your results with ext3, both performance and stability wise.

I selected Reiser prior to the guilty verdict and before it was clear
to me developers were backing away from it.  I wouldn't choose it now,
but at the time, I went with it because it had a slight performance
advantage when I benchmarked our code on it vs ext2.  I was thinking
to change over to ext4 when that is available, as we do have a lot of
large files which the extents might be good for.  Eric's comment he
uses XFS suddenly makes that more attractive.

For the record, I really appreciate your guy's work.  It is great
having up-to-date Ubuntu AMIs.

Ben