[ec2-beta] EC2 I/O Failure

[John Hampton]

Just loaded one of our first ubuntu beta instances (ami-18d73071). Is this
due to a driver problem or a hardware failure at amazon? The instance is
dying due to I/O failures found this in the logs:
Mar 26 12:25:31 ubuntu kernel: [183609.913991] lost page write due to I/O
error on dm-0
Mar 26 12:25:31 ubuntu kernel: [183609.947241] lost page write due to I/O
error on dm-0
Mar 26 12:25:31 ubuntu kernel: [183609.997125] lost page write due to I/O
error on dm-0
Mar 26 12:25:31 ubuntu kernel: [183610.047034] lost page write due to I/O
error on dm-0
Mar 26 12:25:31 ubuntu kernel: [183610.096987] lost page write due to I/O
error on sda1
Mar 26 12:25:31 ubuntu kernel: [183610.096997] lost page write due to I/O
error on sda1
Mar 26 12:25:31 ubuntu kernel: [183610.097005] lost page write due to I/O
error on sda1
Mar 26 12:25:31 ubuntu kernel: [183610.097013] lost page write due to I/O
error on sda1
Mar 26 12:25:31 ubuntu kernel: [183610.097039] JBD: Detected IO errors while
flushing file data on sda1
Mar 26 12:25:31 ubuntu kernel: [183610.146889] lost page write due to I/O
error on dm-0
Mar 26 12:25:31 ubuntu kernel: [183610.180132] lost page write due to I/O
error on dm-0
Mar 26 12:25:31 ubuntu kernel: [183610.280729] JBD: Detected IO errors while
flushing file data on dm-0
Mar 26 12:25:31 ubuntu kernel: [183610.327166] ------------[ cut here
]------------
Mar 26 12:25:31 ubuntu kernel: [183610.327169] WARNING: at
/build/buildd/linux-xen-2.6.27/fs/buffer.c:1186
mark_buffer_dirty+0x84/0xb0()
Mar 26 12:25:31 ubuntu kernel: [183610.327171] Modules linked in:
iptable_filter ip_tables x_tables fbcon tileblit font bitblit softcursor
uvesafb xfs binfmt_misc dm_multipath dm_mod scsi_dh scsi_mod ipv6 af_packet
loop ext3 jbd fuse
Mar 26 12:25:31 ubuntu kernel: [183610.327188] Pid: 32062, comm: vi Not
tainted 2.6.27-22-xen #1
Mar 26 12:25:31 ubuntu kernel: [183610.327190]
Mar 26 12:25:31 ubuntu kernel: [183610.327191] Call Trace:
Mar 26 12:25:31 ubuntu kernel: [183610.327197]  [<ffffffff8023679f>]
warn_on_slowpath+0x5f/0x90
Mar 26 12:25:31 ubuntu kernel: [183610.327201]  [<ffffffff80236b1f>] ?
__call_console_drivers+0x6f/0x90
Mar 26 12:25:31 ubuntu kernel: [183610.327207]  [<ffffffff804e26e3>] ?
printk+0x67/0x6c
Mar 26 12:25:31 ubuntu kernel: [183610.327210]  [<ffffffff802e26e4>]
mark_buffer_dirty+0x84/0xb0
Mar 26 12:25:31 ubuntu kernel: [183610.327224]  [<ffffffffa002fae0>]
ext3_commit_super+0x60/0x90 [ext3]
Mar 26 12:25:31 ubuntu kernel: [183610.327233]  [<ffffffffa0032127>]
ext3_handle_error+0x57/0xd0 [ext3]
Mar 26 12:25:31 ubuntu kernel: [183610.327241]  [<ffffffffa00322c2>]
ext3_error+0xa2/0xb0 [ext3]
Mar 26 12:25:31 ubuntu kernel: [183610.327245]  [<ffffffff804e395b>] ?
__wait_on_bit+0x7b/0x90
Mar 26 12:25:31 ubuntu kernel: [183610.327248]  [<ffffffff802e1b00>] ?
sync_buffer+0x0/0x50
Mar 26 12:25:31 ubuntu kernel: [183610.327251]  [<ffffffff804e39e4>] ?
out_of_line_wait_on_bit+0x74/0x90
Mar 26 12:25:31 ubuntu kernel: [183610.327256]  [<ffffffff8024fbd0>] ?
wake_bit_function+0x0/0x40
Mar 26 12:25:31 ubuntu kernel: [183610.327259]  [<ffffffff802e1ea6>] ?
__wait_on_buffer+0x26/0x30
Mar 26 12:25:31 ubuntu kernel: [183610.327266]  [<ffffffffa00226a2>]
read_block_bitmap+0x162/0x170 [ext3]
Mar 26 12:25:31 ubuntu kernel: [183610.327274]  [<ffffffffa00228e7>]
ext3_free_blocks_sb+0xd7/0x4c0 [ext3]
Mar 26 12:25:31 ubuntu kernel: [183610.327277]  [<ffffffff802e3703>] ?
__find_get_block_slow+0x43/0x150
Mar 26 12:25:31 ubuntu kernel: [183610.327284]  [<ffffffffa0023911>]
ext3_free_blocks+0x21/0x90 [ext3]
Mar 26 12:25:31 ubuntu kernel: [183610.327292]  [<ffffffffa0029785>]
ext3_clear_blocks+0x195/0x1d0 [ext3]
Mar 26 12:25:31 ubuntu kernel: [183610.327300]  [<ffffffffa00298d7>]
ext3_free_data+0x117/0x160 [ext3]
Mar 26 12:25:31 ubuntu kernel: [183610.327308]  [<ffffffffa002a4a3>]
ext3_truncate+0x563/0x650 [ext3]
Mar 26 12:25:31 ubuntu kernel: [183610.327317]  [<ffffffffa0011c76>] ?
journal_start+0xa6/0xf0 [jbd]
Mar 26 12:25:31 ubuntu kernel: [183610.327325]  [<ffffffffa002a690>]
ext3_delete_inode+0x100/0x110 [ext3]
Mar 26 12:25:31 ubuntu kernel: [183610.327333]  [<ffffffffa002a590>] ?
ext3_delete_inode+0x0/0x110 [ext3]
Mar 26 12:25:31 ubuntu kernel: [183610.327338]  [<ffffffff802cf9af>]
generic_delete_inode+0xbf/0x1a0
Mar 26 12:25:31 ubuntu kernel: [183610.327341]  [<ffffffff802cfab5>]
generic_drop_inode+0x25/0x30
Mar 26 12:25:31 ubuntu kernel: [183610.327345]  [<ffffffff802ce48d>]
iput+0x5d/0x70
Mar 26 12:25:31 ubuntu kernel: [183610.327347]  [<ffffffff802c304c>]
do_unlinkat+0x10c/0x1d0
Mar 26 12:25:31 ubuntu kernel: [183610.327351]  [<ffffffff802b78fd>] ?
fput+0x1d/0x30
Mar 26 12:25:31 ubuntu kernel: [183610.327354]  [<ffffffff802b3d6b>] ?
filp_close+0x5b/0x90
Mar 26 12:25:31 ubuntu kernel: [183610.327356]  [<ffffffff802c3121>]
sys_unlink+0x11/0x20
Mar 26 12:25:31 ubuntu kernel: [183610.327360]  [<ffffffff8020c088>]
system_call_fastpath+0x16/0x1b
Mar 26 12:25:31 ubuntu kernel: [183610.327363]  [<ffffffff8020c020>] ?
system_call+0x0/0x52
Mar 26 12:25:31 ubuntu kernel: [183610.327364]
Mar 26 12:25:31 ubuntu kernel: [183610.327366] ---[ end trace
27412ec0c2aa53a8 ]---
Mar 26 12:25:36 ubuntu kernel: [183613.270600] JBD: Detected IO errors while
flushing file data on sda1
-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://lists.ubuntu.com/mailman/private/ec2/attachments/20090326/d76f1527/attachment-0002.htm