[ec2-beta] EC2 I/O Failure

[Chuck Short]

Hi John,


I was wondering how I can reproduce this also could you open up a bug in 
launchpad (http://bugs.launchpad.net/ubuntu-on-ec2) so we can track this?

Regards
chuck

John Hampton wrote:
> Just loaded one of our first ubuntu beta instances (ami-18d73071). Is 
> this due to a driver problem or a hardware failure at amazon? The 
> instance is dying due to I/O failures found this in the logs: 
>
> Mar 26 12:25:31 ubuntu kernel: [183609.913991] lost page write due to 
> I/O error on dm-0
> Mar 26 12:25:31 ubuntu kernel: [183609.947241] lost page write due to 
> I/O error on dm-0
> Mar 26 12:25:31 ubuntu kernel: [183609.997125] lost page write due to 
> I/O error on dm-0
> Mar 26 12:25:31 ubuntu kernel: [183610.047034] lost page write due to 
> I/O error on dm-0
> Mar 26 12:25:31 ubuntu kernel: [183610.096987] lost page write due to 
> I/O error on sda1
> Mar 26 12:25:31 ubuntu kernel: [183610.096997] lost page write due to 
> I/O error on sda1
> Mar 26 12:25:31 ubuntu kernel: [183610.097005] lost page write due to 
> I/O error on sda1
> Mar 26 12:25:31 ubuntu kernel: [183610.097013] lost page write due to 
> I/O error on sda1
> Mar 26 12:25:31 ubuntu kernel: [183610.097039] JBD: Detected IO errors 
> while flushing file data on sda1
> Mar 26 12:25:31 ubuntu kernel: [183610.146889] lost page write due to 
> I/O error on dm-0
> Mar 26 12:25:31 ubuntu kernel: [183610.180132] lost page write due to 
> I/O error on dm-0
> Mar 26 12:25:31 ubuntu kernel: [183610.280729] JBD: Detected IO errors 
> while flushing file data on dm-0
> Mar 26 12:25:31 ubuntu kernel: [183610.327166] ------------[ cut here 
> ]------------
> Mar 26 12:25:31 ubuntu kernel: [183610.327169] WARNING: at 
> /build/buildd/linux-xen-2.6.27/fs/buffer.c:1186 
> mark_buffer_dirty+0x84/0xb0()
> Mar 26 12:25:31 ubuntu kernel: [183610.327171] Modules linked in: 
> iptable_filter ip_tables x_tables fbcon tileblit font bitblit 
> softcursor uvesafb xfs binfmt_misc dm_multipath dm_mod scsi_dh 
> scsi_mod ipv6 af_packet loop ext3 jbd fuse
> Mar 26 12:25:31 ubuntu kernel: [183610.327188] Pid: 32062, comm: vi 
> Not tainted 2.6.27-22-xen #1
> Mar 26 12:25:31 ubuntu kernel: [183610.327190] 
> Mar 26 12:25:31 ubuntu kernel: [183610.327191] Call Trace:
> Mar 26 12:25:31 ubuntu kernel: [183610.327197]  [<ffffffff8023679f>] 
> warn_on_slowpath+0x5f/0x90
> Mar 26 12:25:31 ubuntu kernel: [183610.327201]  [<ffffffff80236b1f>] ? 
> __call_console_drivers+0x6f/0x90
> Mar 26 12:25:31 ubuntu kernel: [183610.327207]  [<ffffffff804e26e3>] ? 
> printk+0x67/0x6c
> Mar 26 12:25:31 ubuntu kernel: [183610.327210]  [<ffffffff802e26e4>] 
> mark_buffer_dirty+0x84/0xb0
> Mar 26 12:25:31 ubuntu kernel: [183610.327224]  [<ffffffffa002fae0>] 
> ext3_commit_super+0x60/0x90 [ext3]
> Mar 26 12:25:31 ubuntu kernel: [183610.327233]  [<ffffffffa0032127>] 
> ext3_handle_error+0x57/0xd0 [ext3]
> Mar 26 12:25:31 ubuntu kernel: [183610.327241]  [<ffffffffa00322c2>] 
> ext3_error+0xa2/0xb0 [ext3]
> Mar 26 12:25:31 ubuntu kernel: [183610.327245]  [<ffffffff804e395b>] ? 
> __wait_on_bit+0x7b/0x90
> Mar 26 12:25:31 ubuntu kernel: [183610.327248]  [<ffffffff802e1b00>] ? 
> sync_buffer+0x0/0x50
> Mar 26 12:25:31 ubuntu kernel: [183610.327251]  [<ffffffff804e39e4>] ? 
> out_of_line_wait_on_bit+0x74/0x90
> Mar 26 12:25:31 ubuntu kernel: [183610.327256]  [<ffffffff8024fbd0>] ? 
> wake_bit_function+0x0/0x40
> Mar 26 12:25:31 ubuntu kernel: [183610.327259]  [<ffffffff802e1ea6>] ? 
> __wait_on_buffer+0x26/0x30
> Mar 26 12:25:31 ubuntu kernel: [183610.327266]  [<ffffffffa00226a2>] 
> read_block_bitmap+0x162/0x170 [ext3]
> Mar 26 12:25:31 ubuntu kernel: [183610.327274]  [<ffffffffa00228e7>] 
> ext3_free_blocks_sb+0xd7/0x4c0 [ext3]
> Mar 26 12:25:31 ubuntu kernel: [183610.327277]  [<ffffffff802e3703>] ? 
> __find_get_block_slow+0x43/0x150
> Mar 26 12:25:31 ubuntu kernel: [183610.327284]  [<ffffffffa0023911>] 
> ext3_free_blocks+0x21/0x90 [ext3]
> Mar 26 12:25:31 ubuntu kernel: [183610.327292]  [<ffffffffa0029785>] 
> ext3_clear_blocks+0x195/0x1d0 [ext3]
> Mar 26 12:25:31 ubuntu kernel: [183610.327300]  [<ffffffffa00298d7>] 
> ext3_free_data+0x117/0x160 [ext3]
> Mar 26 12:25:31 ubuntu kernel: [183610.327308]  [<ffffffffa002a4a3>] 
> ext3_truncate+0x563/0x650 [ext3]
> Mar 26 12:25:31 ubuntu kernel: [183610.327317]  [<ffffffffa0011c76>] ? 
> journal_start+0xa6/0xf0 [jbd]
> Mar 26 12:25:31 ubuntu kernel: [183610.327325]  [<ffffffffa002a690>] 
> ext3_delete_inode+0x100/0x110 [ext3]
> Mar 26 12:25:31 ubuntu kernel: [183610.327333]  [<ffffffffa002a590>] ? 
> ext3_delete_inode+0x0/0x110 [ext3]
> Mar 26 12:25:31 ubuntu kernel: [183610.327338]  [<ffffffff802cf9af>] 
> generic_delete_inode+0xbf/0x1a0
> Mar 26 12:25:31 ubuntu kernel: [183610.327341]  [<ffffffff802cfab5>] 
> generic_drop_inode+0x25/0x30
> Mar 26 12:25:31 ubuntu kernel: [183610.327345]  [<ffffffff802ce48d>] 
> iput+0x5d/0x70
> Mar 26 12:25:31 ubuntu kernel: [183610.327347]  [<ffffffff802c304c>] 
> do_unlinkat+0x10c/0x1d0
> Mar 26 12:25:31 ubuntu kernel: [183610.327351]  [<ffffffff802b78fd>] ? 
> fput+0x1d/0x30
> Mar 26 12:25:31 ubuntu kernel: [183610.327354]  [<ffffffff802b3d6b>] ? 
> filp_close+0x5b/0x90
> Mar 26 12:25:31 ubuntu kernel: [183610.327356]  [<ffffffff802c3121>] 
> sys_unlink+0x11/0x20
> Mar 26 12:25:31 ubuntu kernel: [183610.327360]  [<ffffffff8020c088>] 
> system_call_fastpath+0x16/0x1b
> Mar 26 12:25:31 ubuntu kernel: [183610.327363]  [<ffffffff8020c020>] ? 
> system_call+0x0/0x52
> Mar 26 12:25:31 ubuntu kernel: [183610.327364] 
> Mar 26 12:25:31 ubuntu kernel: [183610.327366] ---[ end trace 
> 27412ec0c2aa53a8 ]---
> Mar 26 12:25:36 ubuntu kernel: [183613.270600] JBD: Detected IO errors 
> while flushing file data on sda1
>
>
>