[ec2-beta] data corruption

[Mark Shuttleworth]

I'll ask Pete Graner, who leads the Ubuntu kernel team, for a definitive
statement (or a pointer to the person who can provide it :-)).

Mark

John Hampton wrote:
> Hi Mark,
>
> Interesting information about file system preferences. I was wondering
> what is the status of XFS? We use it because of the quicker time to
> format.
>
> Thanks
> John
>
> On Tue, Apr 14, 2009 at 12:54 PM, Mark Shuttleworth <mark at ubuntu.com
> <mailto:mark at ubuntu.com>> wrote:
>
>     Ben Hendrickson wrote:
>>     I use around 16 large instances for data processing tasks.  When using
>>     the beta AMIs, I had around a dozen instances of data corruption.
>>     Previously when using the alestic.com <http://alestic.com> images, and now that I've
>>     switched back to them, I haven't seen any corruption.  Is there any
>>     known issue this could be related to?
>>
>>     I use machines by raiding together the disks via RAID 0, and then
>>     installing ReiserFS on top of that.  The list of commands I use to do
>>     this is at the bottom of this email.  The workload of the machines
>>     changes somewhat, but generally it maxes out both of the cores, use
>>     around 15MB/s of disk throughput (split even read/writing), and has
>>     the disks around 60% full.  Our data is always compressed on disk
>>     (LZO), and we have checksums every 64KB of uncompressed data.  What I
>>     would see is that at a seemingly random point in a file the checksum
>>     wouldn't match, although the checksums for the rest of the file both
>>     before and after this point would be fine.  I didn't notice anything
>>     unusual in the system logs.
>>       
>     Thanks for the detailed information, that may help to narrow the
>     search for the problem substantially.
>
>     Is ReiserFS integral to the solution, or a personal preference? It
>     jumped out at me as an area of risk, as it's not a filesystem
>     we're particularly focused on. Ext3, and the newer ext4 and
>     ultimately btrfs would be the "stable, next, future" default
>     filesystems we'd recommend unless there was a specific technical
>     reason to do otherwise. If Reiser isn't integral I'd be interested
>     in your results with ext3, both performance and stability wise.
>
>     There are different kernels, as I understand it, between the
>     Alestic images. Chuck and Eric will be able to say in detail but
>     AIUI the beta AMI's use newer kernels, which bring some benefits
>     but are also quite possibly the source of new gotchas. You may
>     have triggered one of those.
>
>     Mark
>
>     --
>     Ec2-beta mailing list
>     Ec2-beta at lists.ubuntu.com <mailto:Ec2-beta at lists.ubuntu.com>
>     Modify settings or unsubscribe at:
>     https://lists.ubuntu.com/mailman/listinfo/ec2-beta
>
>
>
>
> -- 
> John O. Hampton, Jr., CTO
> CleanOffer, Inc.
> 101 California Street
> Suite 2450 #612
> San Francisco, CA  94111
>
> Homepage: http://www.cleanoffer.com
> Blog: http://blog.cleanoffer.com
>
> (415) 240-4532 (office)

-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://lists.ubuntu.com/mailman/private/ec2/attachments/20090414/1794458e/attachment-0002.htm