No subject


Fri Dec 18 02:33:18 GMT 2009


I have a similar server on another location ( though a newer hwarid card , but again LSI ) that doesn't have those problems.

I have managed to reproduce the bug today, by issuing a kvm-img convert command from qcow2 to raw format ( 15G image ).  All guests were shut off at that time.

Since i have first encountered this problem i have reinstalled the server with a clean karmic install.  Same thing happens. Here are some logs :

Mar 24 10:10:05 blackwing kernel: [  322.304064] sd 0:2:0:0: [sda] megasas: RESET -17491 cmd=2a retries=0
Mar 24 10:10:05 blackwing kernel: [  322.304069] megasas: [ 0]waiting for 96 commands to complete
Mar 24 10:10:10 blackwing kernel: [  327.344068] megasas: [ 5]waiting for 96 commands to complete
Mar 24 10:10:15 blackwing kernel: [  332.384185] megasas: [10]waiting for 96 commands to complete
Mar 24 10:10:20 blackwing kernel: [  337.424291] megasas: [15]waiting for 96 commands to complete
Mar 24 10:10:25 blackwing kernel: [  342.464406] megasas: [20]waiting for 96 commands to complete
Mar 24 10:10:30 blackwing kernel: [  347.504518] megasas: [25]waiting for 96 commands to complete
Mar 24 10:10:35 blackwing kernel: [  352.544640] megasas: [30]waiting for 96 commands to complete
Mar 24 10:10:40 blackwing kernel: [  357.584755] megasas: [35]waiting for 96 commands to complete
Mar 24 10:10:45 blackwing kernel: [  362.365467] INFO: task kdmflush:563 blocked for more than 120 seconds.
Mar 24 10:10:45 blackwing kernel: [  362.365539] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 24 10:10:45 blackwing kernel: [  362.365627] kdmflush      D 0000000000000000     0   563      2 0x00000000
Mar 24 10:10:45 blackwing kernel: [  362.365632]  ffff8801b5483d20 0000000000000046 ffff8801b61a65c0 0000000000015880
Mar 24 10:10:45 blackwing kernel: [  362.365637]  ffff8801b61ec7c0 0000000000015880 0000000000015880 0000000000015880
Mar 24 10:10:45 blackwing kernel: [  362.365641]  0000000000015880 ffff8801b61ec7c0 0000000000015880 0000000000015880
Mar 24 10:10:45 blackwing kernel: [  362.365645] Call Trace:
Mar 24 10:10:45 blackwing kernel: [  362.365657]  [<ffffffff81528488>] io_schedule+0x28/0x40
Mar 24 10:10:45 blackwing kernel: [  362.365664]  [<ffffffff813e87a7>] dm_wait_for_completion+0x87/0x150
Mar 24 10:10:45 blackwing kernel: [  362.365671]  [<ffffffff81053890>] ? default_wake_function+0x0/0x10
Mar 24 10:10:45 blackwing kernel: [  362.365675]  [<ffffffff813e91de>] dm_flush+0x5e/0x70
Mar 24 10:10:45 blackwing kernel: [  362.365678]  [<ffffffff813e9219>] process_barrier+0x29/0xf0
Mar 24 10:10:45 blackwing kernel: [  362.365682]  [<ffffffff813e931b>] dm_wq_work+0x3b/0x100
Mar 24 10:10:45 blackwing kernel: [  362.365685]  [<ffffffff813e92e0>] ? dm_wq_work+0x0/0x100
Mar 24 10:10:45 blackwing kernel: [  362.365692]  [<ffffffff810731f5>] run_workqueue+0x95/0x170
Mar 24 10:10:45 blackwing kernel: [  362.365696]  [<ffffffff81073374>] worker_thread+0xa4/0x120
Mar 24 10:10:45 blackwing kernel: [  362.365701]  [<ffffffff81078500>] ? autoremove_wake_function+0x0/0x40
Mar 24 10:10:45 blackwing kernel: [  362.365705]  [<ffffffff810732d0>] ? worker_thread+0x0/0x120
Mar 24 10:10:45 blackwing kernel: [  362.365708]  [<ffffffff81078116>] kthread+0xa6/0xb0
Mar 24 10:10:45 blackwing kernel: [  362.365714]  [<ffffffff8101312a>] child_rip+0xa/0x20
Mar 24 10:10:45 blackwing kernel: [  362.365717]  [<ffffffff81078070>] ? kthread+0x0/0xb0
Mar 24 10:10:45 blackwing kernel: [  362.365720]  [<ffffffff81013120>] ? child_rip+0x0/0x20
Mar 24 10:10:45 blackwing kernel: [  362.365732] INFO: task kjournald2:1122 blocked for more than 120 seconds.
Mar 24 10:10:45 blackwing kernel: [  362.365800] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 24 10:10:45 blackwing kernel: [  362.365886] kjournald2    D 0000000000000000     0  1122      2 0x00000000
Mar 24 10:10:45 blackwing kernel: [  362.365890]  ffff8801b7801bd0 0000000000000046 ffff8801b605af70 0000000000015880
Mar 24 10:10:45 blackwing kernel: [  362.365895]  ffff8801b81d3110 0000000000015880 0000000000015880 0000000000015880
Mar 24 10:10:45 blackwing kernel: [  362.365898]  0000000000015880 ffff8801b81d3110 0000000000015880 0000000000015880
Mar 24 10:10:45 blackwing kernel: [  362.365902] Call Trace:
Mar 24 10:10:45 blackwing kernel: [  362.365910]  [<ffffffff81145b30>] ? sync_buffer+0x0/0x50
Mar 24 10:10:45 blackwing kernel: [  362.365914]  [<ffffffff81528488>] io_schedule+0x28/0x40
Mar 24 10:10:45 blackwing kernel: [  362.365918]  [<ffffffff81145b6b>] sync_buffer+0x3b/0x50
Mar 24 10:10:45 blackwing kernel: [  362.365921]  [<ffffffff815289a7>] __wait_on_bit+0x57/0x80
Mar 24 10:10:45 blackwing kernel: [  362.365925]  [<ffffffff81145b30>] ? sync_buffer+0x0/0x50
Mar 24 10:10:45 blackwing kernel: [  362.365929]  [<ffffffff81528a43>] out_of_line_wait_on_bit+0x73/0x90
Mar 24 10:10:45 blackwing kernel: [  362.365933]  [<ffffffff81078540>] ? wake_bit_function+0x0/0x40
Mar 24 10:10:45 blackwing kernel: [  362.365937]  [<ffffffff81145b26>] __wait_on_buffer+0x26/0x30
Mar 24 10:10:45 blackwing kernel: [  362.365943]  [<ffffffff811eedd7>] journal_wait_on_commit_record+0x47/0xf0
Mar 24 10:10:45 blackwing kernel: [  362.365946]  [<ffffffff811f009a>] jbd2_journal_commit_transaction+0xfba/0x1100
Mar 24 10:10:45 blackwing kernel: [  362.365952]  [<ffffffff811f5233>] kjournald2+0x103/0x270
Mar 24 10:10:45 blackwing kernel: [  362.365955]  [<ffffffff81078500>] ? autoremove_wake_function+0x0/0x40
Mar 24 10:10:45 blackwing kernel: [  362.365959]  [<ffffffff811f5130>] ? kjournald2+0x0/0x270
Mar 24 10:10:45 blackwing kernel: [  362.365962]  [<ffffffff81078116>] kthread+0xa6/0xb0
Mar 24 10:10:45 blackwing kernel: [  362.365966]  [<ffffffff8101312a>] child_rip+0xa/0x20
Mar 24 10:10:45 blackwing kernel: [  362.365969]  [<ffffffff81078070>] ? kthread+0x0/0xb0
Mar 24 10:10:45 blackwing kernel: [  362.365972]  [<ffffffff81013120>] ? child_rip+0x0/0x20
Mar 24 10:10:45 blackwing kernel: [  362.365978] INFO: task kjournald2:1348 blocked for more than 120 seconds.
Mar 24 10:10:45 blackwing kernel: [  362.366046] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 24 10:10:45 blackwing kernel: [  362.366133] kjournald2    D 0000000000000000     0  1348      2 0x00000000
Mar 24 10:10:45 blackwing kernel: [  362.366137]  ffff8801b614dad0 0000000000000046 0000000000000000 0000000000015880
Mar 24 10:10:45 blackwing kernel: [  362.366141]  ffff8801b2123110 0000000000015880 0000000000015880 0000000000015880
Mar 24 10:10:45 blackwing kernel: [  362.366145]  0000000000015880 ffff8801b2123110 0000000000015880 0000000000015880
Mar 24 10:10:45 blackwing kernel: [  362.366149] Call Trace:
Mar 24 10:10:45 blackwing kernel: [  362.366154]  [<ffffffff810da010>] ? sync_page+0x0/0x50
Mar 24 10:10:45 blackwing kernel: [  362.366158]  [<ffffffff81528488>] io_schedule+0x28/0x40
Mar 24 10:10:45 blackwing kernel: [  362.366161]  [<ffffffff810da04d>] sync_page+0x3d/0x50
Mar 24 10:10:45 blackwing kernel: [  362.366164]  [<ffffffff815289a7>] __wait_on_bit+0x57/0x80
Mar 24 10:10:45 blackwing kernel: [  362.366168]  [<ffffffff810da1be>] wait_on_page_bit+0x6e/0x80
Mar 24 10:10:45 blackwing kernel: [  362.366171]  [<ffffffff81078540>] ? wake_bit_function+0x0/0x40
Mar 24 10:10:45 blackwing kernel: [  362.366176]  [<ffffffff810e4480>] ? pagevec_lookup_tag+0x20/0x30
Mar 24 10:10:45 blackwing kernel: [  362.366180]  [<ffffffff810da665>] wait_on_page_writeback_range+0xf5/0x190
Mar 24 10:10:45 blackwing kernel: [  362.366184]  [<ffffffff810da727>] filemap_fdatawait+0x27/0x30
Mar 24 10:10:45 blackwing kernel: [  362.366188]  [<ffffffff811eec96>] journal_finish_inode_data_buffers+0x56/0x150
Mar 24 10:10:45 blackwing kernel: [  362.366192]  [<ffffffff811ef7b6>] jbd2_journal_commit_transaction+0x6d6/0x1100
Mar 24 10:10:45 blackwing kernel: [  362.366197]  [<ffffffff811f5233>] kjournald2+0x103/0x270
Mar 24 10:10:45 blackwing kernel: [  362.366200]  [<ffffffff81078500>] ? autoremove_wake_function+0x0/0x40
Mar 24 10:10:45 blackwing kernel: [  362.366204]  [<ffffffff811f5130>] ? kjournald2+0x0/0x270
Mar 24 10:10:45 blackwing kernel: [  362.366207]  [<ffffffff81078116>] kthread+0xa6/0xb0
Mar 24 10:10:45 blackwing kernel: [  362.366211]  [<ffffffff8101312a>] child_rip+0xa/0x20
Mar 24 10:10:45 blackwing kernel: [  362.366214]  [<ffffffff81078070>] ? kthread+0x0/0xb0
Mar 24 10:10:45 blackwing kernel: [  362.366217]  [<ffffffff81013120>] ? child_rip+0x0/0x20
Mar 24 10:10:45 blackwing kernel: [  362.624882] megasas: [40]waiting for 96 commands to complete
Mar 24 10:10:50 blackwing kernel: [  367.664995] megasas: [45]waiting for 96 commands to complete
Mar 24 10:10:55 blackwing kernel: [  372.705116] megasas: [50]waiting for 96 commands to complete
Mar 24 10:11:00 blackwing kernel: [  377.745234] megasas: [55]waiting for 96 commands to complete
Mar 24 10:11:05 blackwing kernel: [  382.785363] megasas: [60]waiting for 96 commands to complete
Mar 24 10:11:10 blackwing kernel: [  387.825484] megasas: [65]waiting for 96 commands to complete
Mar 24 10:11:15 blackwing kernel: [  392.865598] megasas: [70]waiting for 96 commands to complete
Mar 24 10:11:20 blackwing kernel: [  397.905716] megasas: [75]waiting for 96 commands to complete
Mar 24 10:11:26 blackwing kernel: [  402.945836] megasas: [80]waiting for 96 commands to complete
Mar 24 10:11:31 blackwing kernel: [  407.985954] megasas: [85]waiting for 96 commands to complete
Mar 24 10:11:36 blackwing kernel: [  413.026074] megasas: [90]waiting for 96 commands to complete
Mar 24 10:11:41 blackwing kernel: [  418.066189] megasas: [95]waiting for 96 commands to complete
Mar 24 10:11:46 blackwing kernel: [  423.106308] megasas: [100]waiting for 96 commands to complete
Mar 24 10:11:51 blackwing kernel: [  428.146420] megasas: [105]waiting for 96 commands to complete
Mar 24 10:11:56 blackwing kernel: [  433.186537] megasas: [110]waiting for 96 commands to complete



When the above appears in the logs, all disk i/o stops and i cannot launch anything ( htop, dstat, iotop, aptitude do not start at all ).
I can however do cat /proc/loadavg and i see an increasing load on the system ( last time it went up to 10 before the logs below started )




[ 1009.717634] end_request: I/O error, dev sda, sector 49095082
[ 1009.717711] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.717712] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.717715] end_request: I/O error, dev sda, sector 49095090
[ 1009.717977] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.717979] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.717982] end_request: I/O error, dev sda, sector 49095730
[ 1009.718179] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.718181] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.718184] end_request: I/O error, dev sda, sector 49096370
[ 1009.718330] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.718332] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.718335] end_request: I/O error, dev sda, sector 49115578
[ 1009.718620] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.718623] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.718626] end_request: I/O error, dev sda, sector 49116218
[ 1009.718704] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.718706] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.718709] end_request: I/O error, dev sda, sector 49116226
[ 1009.718976] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.718978] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.718981] end_request: I/O error, dev sda, sector 49116866
[ 1009.719252] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.719254] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.719257] end_request: I/O error, dev sda, sector 49117506
[ 1009.719527] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.719529] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.719532] end_request: I/O error, dev sda, sector 49118146
[ 1009.719607] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.719609] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.719612] end_request: I/O error, dev sda, sector 49118154
[ 1009.719883] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.719885] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.719888] end_request: I/O error, dev sda, sector 49118794
[ 1009.720156] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.720157] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.720160] end_request: I/O error, dev sda, sector 49119434
[ 1009.720425] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.720427] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.720430] end_request: I/O error, dev sda, sector 49120074
[ 1009.720506] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.720508] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.720511] end_request: I/O error, dev sda, sector 49120082
[ 1009.720792] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.720794] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.720797] end_request: I/O error, dev sda, sector 49120722
[ 1009.721073] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.721075] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.721078] end_request: I/O error, dev sda, sector 49121362
[ 1009.721285] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.721287] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.721290] end_request: I/O error, dev sda, sector 49122002
[ 1009.721363] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.721365] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.721368] end_request: I/O error, dev sda, sector 49122010
[ 1009.721623] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.721625] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.721628] end_request: I/O error, dev sda, sector 49122650
[ 1009.721901] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.721903] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.721906] end_request: I/O error, dev sda, sector 49123290
[ 1009.722173] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.722175] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.722178] end_request: I/O error, dev sda, sector 49123930
[ 1009.722253] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.722255] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.722258] end_request: I/O error, dev sda, sector 49123938
[ 1009.722539] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.722541] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.722544] end_request: I/O error, dev sda, sector 49124578
[ 1009.722816] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.722818] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.722821] end_request: I/O error, dev sda, sector 49125218
[ 1009.722897] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.722899] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.722902] end_request: I/O error, dev sda, sector 49125226
[ 1009.723171] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.723172] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.723176] end_request: I/O error, dev sda, sector 49125866
[ 1009.723450] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.723452] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.723455] end_request: I/O error, dev sda, sector 49126506
[ 1009.723719] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.723721] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.723724] end_request: I/O error, dev sda, sector 49127146
[ 1009.723798] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.723800] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.723803] end_request: I/O error, dev sda, sector 49127154
[ 1009.724083] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.724085] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.724088] end_request: I/O error, dev sda, sector 49127794
[ 1009.724352] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.724354] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.724357] end_request: I/O error, dev sda, sector 49128434
[ 1009.724596] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.724598] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.724601] end_request: I/O error, dev sda, sector 49129074
[ 1009.724677] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.724679] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.724682] end_request: I/O error, dev sda, sector 49129082
[ 1009.724958] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.724960] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.724963] end_request: I/O error, dev sda, sector 49129722
[ 1009.725235] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.725237] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.725240] end_request: I/O error, dev sda, sector 49130362
[ 1009.725511] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.725513] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.725516] end_request: I/O error, dev sda, sector 49131002
[ 1009.725591] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.725593] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.725596] end_request: I/O error, dev sda, sector 49131010
[ 1009.725868] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.725870] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.725873] end_request: I/O error, dev sda, sector 49131650
[ 1009.726143] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.726145] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.726148] end_request: I/O error, dev sda, sector 49132290
[ 1009.726420] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.726422] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.726425] end_request: I/O error, dev sda, sector 49132930
[ 1009.726499] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.726501] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.726504] end_request: I/O error, dev sda, sector 49132938
[ 1009.726781] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.726783] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.726786] end_request: I/O error, dev sda, sector 49133578
[ 1009.727061] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.727063] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.727066] end_request: I/O error, dev sda, sector 49134218
[ 1009.727328] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.727330] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.727333] end_request: I/O error, dev sda, sector 49134858
[ 1009.727407] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.727409] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.727412] end_request: I/O error, dev sda, sector 49134866
[ 1009.727681] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.727683] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.727686] end_request: I/O error, dev sda, sector 49135506
[ 1009.727958] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.727960] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.727963] end_request: I/O error, dev sda, sector 49136146
[ 1009.728233] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.728235] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.728238] end_request: I/O error, dev sda, sector 49136786
[ 1009.728512] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.728513] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.728516] end_request: I/O error, dev sda, sector 49137426
[ 1009.728791] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.728793] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.728796] end_request: I/O error, dev sda, sector 49138066
[ 1009.729068] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.729070] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.729073] end_request: I/O error, dev sda, sector 49138706
[ 1009.729339] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.729341] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.729344] end_request: I/O error, dev sda, sector 49139346
[ 1009.729612] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.729614] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.729617] end_request: I/O error, dev sda, sector 49139986
[ 1009.729882] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.729884] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.729887] end_request: I/O error, dev sda, sector 49140626
[ 1009.730157] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.730159] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.730162] end_request: I/O error, dev sda, sector 49141266
[ 1009.730430] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.730432] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.730435] end_request: I/O error, dev sda, sector 49141906
[ 1009.730711] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.730713] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.730716] end_request: I/O error, dev sda, sector 49142546
[ 1009.730986] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.730988] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.730991] end_request: I/O error, dev sda, sector 49143186
[ 1009.731256] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.731258] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.731261] end_request: I/O error, dev sda, sector 49143826
[ 1009.731535] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.731537] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.731540] end_request: I/O error, dev sda, sector 49144466
[ 1009.731817] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.731819] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.731822] end_request: I/O error, dev sda, sector 49145106
[ 1009.732094] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.732096] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.732099] end_request: I/O error, dev sda, sector 49145746
[ 1009.732370] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.732372] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.732375] end_request: I/O error, dev sda, sector 49146386
[ 1009.732644] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.732646] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.732649] end_request: I/O error, dev sda, sector 49147026
[ 1009.732918] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.732920] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.732923] end_request: I/O error, dev sda, sector 49147666
[ 1009.733200] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.733202] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.733205] end_request: I/O error, dev sda, sector 49148306
[ 1009.733482] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.733483] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.733486] end_request: I/O error, dev sda, sector 49148946
[ 1009.733760] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.733762] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.733765] end_request: I/O error, dev sda, sector 49149586
[ 1009.734029] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.734031] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.734034] end_request: I/O error, dev sda, sector 49150226
[ 1009.734290] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.734292] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.734295] end_request: I/O error, dev sda, sector 49150866
[ 1009.734543] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.734545] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.734548] end_request: I/O error, dev sda, sector 49151506
[ 1009.734816] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.734818] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.734821] end_request: I/O error, dev sda, sector 49152146
[ 1009.735090] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.735092] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.735095] end_request: I/O error, dev sda, sector 49152786
[ 1009.735362] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.735364] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.735367] end_request: I/O error, dev sda, sector 49153426
[ 1009.735636] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.735638] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.735641] end_request: I/O error, dev sda, sector 49154066
[ 1009.735913] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.735915] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.735918] end_request: I/O error, dev sda, sector 49154706
[ 1009.736189] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.736191] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.736194] end_request: I/O error, dev sda, sector 49155346
[ 1009.736474] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.736476] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.736479] end_request: I/O error, dev sda, sector 49155986
[ 1009.736761] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.736763] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.736766] end_request: I/O error, dev sda, sector 49156626
[ 1009.737045] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.737047] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.737050] end_request: I/O error, dev sda, sector 49157266
[ 1009.737322] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.737324] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.737327] end_request: I/O error, dev sda, sector 49157906
[ 1009.737550] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.737552] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.737555] end_request: I/O error, dev sda, sector 49158546
[ 1009.737851] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.737853] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.737856] end_request: I/O error, dev sda, sector 49159186
[ 1009.738129] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.738131] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.738135] end_request: I/O error, dev sda, sector 49159826
[ 1009.738409] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.738411] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.738414] end_request: I/O error, dev sda, sector 49160466
[ 1009.738693] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.738695] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.738698] end_request: I/O error, dev sda, sector 49161106
[ 1009.738963] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.738965] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.738968] end_request: I/O error, dev sda, sector 49161746
[ 1009.739233] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.739235] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.739238] end_request: I/O error, dev sda, sector 49162386
[ 1009.739507] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.739509] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.739512] end_request: I/O error, dev sda, sector 49163026
[ 1009.739779] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.739781] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.739784] end_request: I/O error, dev sda, sector 49163666
[ 1009.740049] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.740051] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.740054] end_request: I/O error, dev sda, sector 49164306
[ 1009.740333] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.740335] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.740338] end_request: I/O error, dev sda, sector 49164946
[ 1009.740606] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.740608] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.740611] end_request: I/O error, dev sda, sector 49165586
[ 1009.740886] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.740888] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.740891] end_request: I/O error, dev sda, sector 49166226
[ 1009.741157] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.741159] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.741162] end_request: I/O error, dev sda, sector 49166866
[ 1009.741437] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.741439] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.741442] end_request: I/O error, dev sda, sector 49167506
[ 1009.741714] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.741716] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.741719] end_request: I/O error, dev sda, sector 49168146
[ 1009.741989] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.741991] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.741994] end_request: I/O error, dev sda, sector 49168786
[ 1009.742264] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.742266] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.742269] end_request: I/O error, dev sda, sector 49169426
[ 1009.742545] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.742547] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.742550] end_request: I/O error, dev sda, sector 49170066
[ 1009.742814] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.742816] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.742819] end_request: I/O error, dev sda, sector 49170706
[ 1009.743090] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.743092] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.743095] end_request: I/O error, dev sda, sector 49171346
[ 1009.743373] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.743375] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.743378] end_request: I/O error, dev sda, sector 49171986
[ 1009.743644] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.743646] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.743649] end_request: I/O error, dev sda, sector 49172626
[ 1009.743929] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.743931] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.743934] end_request: I/O error, dev sda, sector 49173266
[ 1009.744203] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.744205] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.744208] end_request: I/O error, dev sda, sector 49173906
[ 1009.744461] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.744463] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.744466] end_request: I/O error, dev sda, sector 49174546
[ 1009.744723] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.744725] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.744728] end_request: I/O error, dev sda, sector 49175186
[ 1009.745001] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.745003] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.745006] end_request: I/O error, dev sda, sector 49175826
[ 1009.745274] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.745276] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.745279] end_request: I/O error, dev sda, sector 49176466
[ 1009.745547] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.745549] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.745552] end_request: I/O error, dev sda, sector 49177106
[ 1009.745709] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.745711] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.745714] end_request: I/O error, dev sda, sector 49177386
[ 1009.745991] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.745993] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.745996] end_request: I/O error, dev sda, sector 49178026
[ 1009.746279] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.746281] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.746284] end_request: I/O error, dev sda, sector 49178666
[ 1009.746475] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.746477] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.746480] end_request: I/O error, dev sda, sector 49179050
[ 1009.746551] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.746553] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.746556] end_request: I/O error, dev sda, sector 49179066
[ 1009.746628] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.746630] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.746633] end_request: I/O error, dev sda, sector 49179082
[ 1009.746708] JBD2: Detected IO errors while flushing file data on dm-7:8
[ 1009.746723] Aborting journal on device dm-7:8.
[ 1009.746799] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.746872] JBD2: I/O error detected when updating journal superblock for dm-7:8.
[ 1009.746972] mpage_da_map_blocks block allocation failed for inode 2596879 at logical offset 1902607 with max blocks 1024 with error -30
[ 1009.747075] This should not happen.!! Data will be lost
[ 1009.747718] EXT4-fs error (device dm-7) in ext4_da_writepages: Journal has aborted
[ 1009.747819] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.747897] EXT4-fs error (device dm-7): ext4_journal_start_sb: Detected aborted journal
[ 1009.748064] EXT4-fs (dm-7): Remounting filesystem read-only
[ 1009.748208] ext4_da_writepages: jbd2_start: 1024 pages, ino 2596879; err -30
[ 1009.748282] Pid: 130, comm: pdflush Not tainted 2.6.31-20-server #58-Ubuntu
[ 1009.748289] Call Trace:
[ 1009.748301]  [<ffffffff811b5f57>] ext4_da_writepages+0x587/0x5c0
[ 1009.748311]  [<ffffffff8106ab37>] ? lock_timer_base+0x37/0x70
[ 1009.748322]  [<ffffffff810e3158>] do_writepages+0x28/0x50
[ 1009.748331]  [<ffffffff8113eb5c>] writeback_single_inode+0x1bc/0x450
[ 1009.748340]  [<ffffffff8113f478>] generic_sync_sb_inodes+0x418/0x530
[ 1009.748350]  [<ffffffff8113f69b>] writeback_inodes+0x5b/0x100
[ 1009.748358]  [<ffffffff810e1eec>] wb_kupdate+0xbc/0x140
[ 1009.748367]  [<ffffffff810e395e>] __pdflush+0x13e/0x260
[ 1009.748376]  [<ffffffff810e3a80>] ? pdflush+0x0/0x50
[ 1009.748384]  [<ffffffff810e3ac8>] pdflush+0x48/0x50
[ 1009.748392]  [<ffffffff810e1e30>] ? wb_kupdate+0x0/0x140
[ 1009.748400]  [<ffffffff810e3a80>] ? pdflush+0x0/0x50
[ 1009.748409]  [<ffffffff81078116>] kthread+0xa6/0xb0
[ 1009.748419]  [<ffffffff8101312a>] child_rip+0xa/0x20
[ 1009.748427]  [<ffffffff81078070>] ? kthread+0x0/0xb0
[ 1009.748435]  [<ffffffff81013120>] ? child_rip+0x0/0x20
[ 1009.748480] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.748552] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.748628] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.748697] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.748782] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.748852] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.748923] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.748993] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.749073] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.749144] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.749214] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.749285] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.749371] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.749444] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.749516] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.749586] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.749674] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.749744] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.751025] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.751095] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.751174] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.751245] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.751315] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.751384] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.751463] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.751532] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.751602] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.751672] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.751753] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.751824] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.751894] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.751964] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.752043] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.752114] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.752184] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.752253] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.752334] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.752405] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.752475] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.752545] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.752638] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.752710] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.752780] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.752851] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.752930] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.753000] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.753070] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.753140] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.753219] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.753289] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.753360] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.753430] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.753511] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.753581] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.753651] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.753720] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.753799] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.753869] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.753939] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.754009] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.754087] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.754157] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.754228] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.754297] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.754376] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.754446] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.754515] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.754584] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.754754] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.754825] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.754894] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.754963] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.755043] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.755114] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.755184] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.755254] sd 0:2:0:0: rejecting I/O to offline device
[ 1010.752509] EXT4-fs error (device dm-3): ext4_journal_start_sb: Detected aborted journal
[ 1010.752671] EXT4-fs (dm-3): Remounting filesystem read-only
[ 1010.752808] ext4_da_writepages: jbd2_start: 1024 pages, ino 4221; err -30
[ 1010.752877] Pid: 130, comm: pdflush Not tainted 2.6.31-20-server #58-Ubuntu
[ 1010.752879] Call Trace:
[ 1010.752884]  [<ffffffff811b5f57>] ext4_da_writepages+0x587/0x5c0
[ 1010.752890]  [<ffffffff810e3158>] do_writepages+0x28/0x50
[ 1010.752894]  [<ffffffff8113eb5c>] writeback_single_inode+0x1bc/0x450
[ 1010.752898]  [<ffffffff8113f478>] generic_sync_sb_inodes+0x418/0x530
[ 1010.752904]  [<ffffffff81036419>] ? default_spin_lock_flags+0x9/0x10
[ 1010.752908]  [<ffffffff8113f69b>] writeback_inodes+0x5b/0x100
[ 1010.752912]  [<ffffffff810e1eec>] wb_kupdate+0xbc/0x140
[ 1010.752916]  [<ffffffff810e395e>] __pdflush+0x13e/0x260
[ 1010.752920]  [<ffffffff810e3a80>] ? pdflush+0x0/0x50
[ 1010.752923]  [<ffffffff810e3ac8>] pdflush+0x48/0x50
[ 1010.752927]  [<ffffffff810e1e30>] ? wb_kupdate+0x0/0x140
[ 1010.752930]  [<ffffffff810e3a80>] ? pdflush+0x0/0x50
[ 1010.752934]  [<ffffffff81078116>] kthread+0xa6/0xb0
[ 1010.752938]  [<ffffffff8101312a>] child_rip+0xa/0x20
[ 1010.752941]  [<ffffffff81078070>] ? kthread+0x0/0xb0
[ 1010.752944]  [<ffffffff81013120>] ? child_rip+0x0/0x20
[ 1010.753020] sd 0:2:0:0: rejecting I/O to offline device
[ 1010.753089] sd 0:2:0:0: rejecting I/O to offline device
[ 1010.753155] sd 0:2:0:0: rejecting I/O to offline device
[ 1010.753223] sd 0:2:0:0: rejecting I/O to offline device
[ 1010.753301] sd 0:2:0:0: rejecting I/O to offline device
[ 1010.753369] sd 0:2:0:0: rejecting I/O to offline device
[ 1010.753435] sd 0:2:0:0: rejecting I/O to offline device
[ 1010.753500] sd 0:2:0:0: rejecting I/O to offline device
[ 1010.753593] sd 0:2:0:0: rejecting I/O to offline device
[ 1010.753660] sd 0:2:0:0: rejecting I/O to offline device
[ 1010.753726] sd 0:2:0:0: rejecting I/O to offline device
[ 1010.753798] sd 0:2:0:0: rejecting I/O to offline device
[ 1010.753874] sd 0:2:0:0: rejecting I/O to offline device
[ 1010.753941] sd 0:2:0:0: rejecting I/O to offline device
[ 1010.754006] sd 0:2:0:0: rejecting I/O to offline device
[ 1010.754072] sd 0:2:0:0: rejecting I/O to offline device
[ 1010.754148] sd 0:2:0:0: rejecting I/O to offline device
[ 1010.754214] sd 0:2:0:0: rejecting I/O to offline device
[ 1010.754279] sd 0:2:0:0: rejecting I/O to offline device
[ 1010.754345] sd 0:2:0:0: rejecting I/O to offline device
[ 1010.762505] sd 0:2:0:0: rejecting I/O to offline device
[ 1014.844041] sd 0:2:0:0: rejecting I/O to offline device
[ 1014.844116] Aborting journal on device dm-1:8.
[ 1014.844183] sd 0:2:0:0: rejecting I/O to offline device
[ 1014.844248] __ratelimit: 11943 callbacks suppressed
[ 1014.844251] Buffer I/O error on device dm-1, logical block 196608
[ 1014.844318] lost page write due to I/O error on dm-1
[ 1014.844321] JBD2: I/O error detected when updating journal superblock for dm-1:8.
[ 1026.690197] sd 0:2:0:0: rejecting I/O to offline device
[ 1026.690280] sd 0:2:0:0: rejecting I/O to offline device
[ 1026.690388] sd 0:2:0:0: rejecting I/O to offline device
[ 1041.839247] sd 0:2:0:0: rejecting I/O to offline device
[ 1041.839353] sd 0:2:0:0: rejecting I/O to offline device




Any ideas ? The LSI controller doesn't log any errors in his event log. The disk come up clean, and i have even ran a consistency check on both raid arrays.

Is there any known problem with the specific raid controller. I could try replacing it with the model that i have on the other server and where i don't see that problem.



More information about the ubuntu-server mailing list