Karmic disk/raid troubles
Nikolai K. Bochev
n.bochev at grandstarco.com
Wed Mar 24 20:32:12 UTC 2010
I am running Karmic on a Supermicro server with 7 KVM guests on it.
Hardware specs :
H/W path Device Class Description
=======================================================
system X8DT3
/0 bus X8DT3
/0/0 memory 64KiB BIOS
/0/4 processor Intel(R) Xeon(R) CPU E5520 @ 2.27GHz
/0/4/5 memory 128KiB L1 cache
/0/4/6 memory 1MiB L2 cache
/0/4/7 memory 8MiB L3 cache
/0/8 processor Intel(R) Xeon(R) CPU E5520 @ 2.27GHz
/0/8/9 memory 128KiB L1 cache
/0/8/a memory 1MiB L2 cache
/0/8/b memory 8MiB L3 cache
/0/39 memory System Memory
/0/39/0 memory 2GiB DIMM 1333 MHz (0.8 ns)
/0/39/1 memory DIMM 1333 MHz (0.8 ns) [empty]
/0/39/2 memory 2GiB DIMM 1333 MHz (0.8 ns)
/0/39/3 memory DIMM 1333 MHz (0.8 ns) [empty]
/0/39/4 memory 2GiB DIMM 1333 MHz (0.8 ns)
/0/39/5 memory DIMM 1333 MHz (0.8 ns) [empty]
/0/47 memory System Memory
/0/47/0 memory 2GiB DIMM 1333 MHz (0.8 ns)
/0/47/1 memory DIMM 1333 MHz (0.8 ns) [empty]
/0/47/2 memory 2GiB DIMM 1333 MHz (0.8 ns)
/0/47/3 memory DIMM 1333 MHz (0.8 ns) [empty]
/0/47/4 memory 2GiB DIMM 1333 MHz (0.8 ns)
/0/47/5 memory DIMM 1333 MHz (0.8 ns) [empty]
/0/55 memory Flash Memory
/0/55/0 memory 4MiB FLASH Non-volatile 33 MHz (30.3 ns)
/0/1 memory
/0/2 memory
/0/100 bridge 5520 I/O Hub to ESI Port
/0/100/1 bridge 5520/5500/X58 I/O Hub PCI Express Root Port 1
/0/100/1/0 eth0 network 82576 Gigabit Network Connection
/0/100/1/0.1 eth1 network 82576 Gigabit Network Connection
/0/100/3 bridge 5520/5500/X58 I/O Hub PCI Express Root Port 3
/0/100/3/0 scsi0 storage MegaRAID SAS 1078
/0/100/3/0/2.0.0 /dev/sda disk 997GB MegaRAID SAS RMB
/0/100/3/0/2.0.0/1 /dev/sda1 volume 1906MiB EXT4 volume
/0/100/3/0/2.0.0/2 /dev/sda2 volume 927GiB Extended partition
/0/100/3/0/2.0.0/2/5 /dev/sda5 volume 927GiB Linux LVM Physical Volume partition
/0/100/3/0/2.1.0 /dev/sdb disk 498GB MegaRAID SAS RMB
/0/100/3/0/2.1.0/1 /dev/sdb1 volume 464GiB Linux LVM Physical Volume partition
/0/100/5 bridge 5520/X58 I/O Hub PCI Express Root Port 5
/0/100/7 bridge 5520/5500/X58 I/O Hub PCI Express Root Port 7
/0/100/8 bridge 5520/5500/X58 I/O Hub PCI Express Root Port 8
/0/100/9 bridge 5520/5500/X58 I/O Hub PCI Express Root Port 9
/0/100/14 generic 5520/5500/X58 I/O Hub System Management Registers
/0/100/14.1 generic 5520/5500/X58 I/O Hub GPIO and Scratch Pad Registers
/0/100/14.2 generic 5520/5500/X58 I/O Hub Control Status and RAS Registers
/0/100/14.3 generic 5520/5500/X58 I/O Hub Throttle Registers
/0/100/16 generic 5520/5500/X58 Chipset QuickData Technology Device
/0/100/16.1 generic 5520/5500/X58 Chipset QuickData Technology Device
/0/100/16.2 generic 5520/5500/X58 Chipset QuickData Technology Device
/0/100/16.3 generic 5520/5500/X58 Chipset QuickData Technology Device
/0/100/16.4 generic 5520/5500/X58 Chipset QuickData Technology Device
/0/100/16.5 generic 5520/5500/X58 Chipset QuickData Technology Device
/0/100/16.6 generic 5520/5500/X58 Chipset QuickData Technology Device
/0/100/16.7 generic 5520/5500/X58 Chipset QuickData Technology Device
/0/100/1a bus 82801JI (ICH10 Family) USB UHCI Controller #4
/0/100/1a.1 bus 82801JI (ICH10 Family) USB UHCI Controller #5
/0/100/1a.2 bus 82801JI (ICH10 Family) USB UHCI Controller #6
/0/100/1a.7 bus 82801JI (ICH10 Family) USB2 EHCI Controller #2
/0/100/1d bus 82801JI (ICH10 Family) USB UHCI Controller #1
/0/100/1d.1 bus 82801JI (ICH10 Family) USB UHCI Controller #2
/0/100/1d.2 bus 82801JI (ICH10 Family) USB UHCI Controller #3
/0/100/1d.7 bus 82801JI (ICH10 Family) USB2 EHCI Controller #1
/0/100/1e bridge 82801 PCI Bridge
/0/100/1e/3 display MGA G200eW WPCM450
/0/100/1f bridge 82801JIR (ICH10R) LPC Interface Controller
/0/100/1f.3 bus 82801JI (ICH10 Family) SMBus Controller
/0/101 bridge Intel Corporation
/0/102 bridge Intel Corporation
/0/103 bridge Intel Corporation
/0/104 bridge Intel Corporation
/0/105 bridge 5520/5500/X58 Physical Layer Port 0
/0/106 bridge 5520/5500 Physical Layer Port 1
/0/107 bridge Intel Corporation
/0/108 bridge Intel Corporation
/0/109 bridge Intel Corporation
/0/10a bridge Intel Corporation
/0/10b bridge Intel Corporation
/1 vnet0 network Ethernet interface
/2 vnet1 network Ethernet interface
/3 vnet2 network Ethernet interface
/4 vnet3 network Ethernet interface
/5 vnet4 network Ethernet interface
/6 vnet5 network Ethernet interface
/7 vnet6 network Ethernet interface
/8 vnet7 network Ethernet interface
/9 vnet8 network Ethernet interface
/a vnet9 network Ethernet interface
/b vnet10 network Ethernet interface
/c vnet11 network Ethernet interface
mount
/dev/mapper/srv--kvm-root on / type ext4 (rw,errors=remount-ro)
proc on /proc type proc (rw)
none on /sys type sysfs (rw,noexec,nosuid,nodev)
none on /sys/fs/fuse/connections type fusectl (rw)
none on /sys/kernel/debug type debugfs (rw)
none on /sys/kernel/security type securityfs (rw)
udev on /dev type tmpfs (rw,mode=0755)
none on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620)
none on /dev/shm type tmpfs (rw,nosuid,nodev)
none on /var/run type tmpfs (rw,nosuid,mode=0755)
none on /var/lock type tmpfs (rw,noexec,nosuid,nodev)
none on /lib/init/rw type tmpfs (rw,nosuid,mode=0755)
/dev/sda1 on /boot type ext4 (rw)
/dev/mapper/srv--kvm--backup-backup on /srv/backup type ext4 (rw)
/dev/mapper/srv--kvm-home on /home type ext4 (rw)
/dev/mapper/srv--kvm-usr on /usr type ext4 (rw)
/dev/mapper/srv--kvm-var on /var type ext4 (rw)
/dev/mapper/srv--kvm-tmp on /tmp type ext4 (rw)
/dev/mapper/srv--kvm-storage on /srv/storage type ext4 (rw)
lvs
LV VG Attr LSize Origin Snap% Move Log Copy% Convert
home srv-kvm -wi-ao 1.86G
root srv-kvm -wi-ao 1.86G
storage srv-kvm -wi-ao 196.56G
swap srv-kvm -wi-ao 1.86G
tmp srv-kvm -wi-ao 1.86G
usr srv-kvm -wi-ao 1.86G
var srv-kvm -wi-ao 51.86G
backup srv-kvm-backup -wi-ao 146.56G
df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/srv--kvm-root
1.9G 287M 1.5G 17% /
udev 5.9G 228K 5.9G 1% /dev
none 5.9G 0 5.9G 0% /dev/shm
none 5.9G 100K 5.9G 1% /var/run
none 5.9G 0 5.9G 0% /var/lock
none 5.9G 0 5.9G 0% /lib/init/rw
/dev/sda1 1.9G 64M 1.7G 4% /boot
/dev/mapper/srv--kvm--backup-backup
145G 80G 65G 56% /srv/backup
/dev/mapper/srv--kvm-home
1.9G 35M 1.8G 2% /home
/dev/mapper/srv--kvm-usr
1.9G 441M 1.4G 25% /usr
/dev/mapper/srv--kvm-var
52G 547M 48G 2% /var
/dev/mapper/srv--kvm-tmp
1.9G 35M 1.8G 2% /tmp
/dev/mapper/srv--kvm-storage
194G 83G 111G 43% /srv/storage
The disks are split on 2 raids :
4 disks in RAID 6 - boot and kvm storage
2 disks in RAID 1 - backup
>From a month or so i started getting hangs on the server. Disk I/O failures, the server becomes unresponsive, i cannot even connect to the monitor through the IPMI KVM over LAN .
I have a similar server on another location ( though a newer hwarid card , but again LSI ) that doesn't have those problems.
I have managed to reproduce the bug today, by issuing a kvm-img convert command from qcow2 to raw format ( 15G image ). All guests were shut off at that time.
Since i have first encountered this problem i have reinstalled the server with a clean karmic install. Same thing happens. Here are some logs :
Mar 24 10:10:05 blackwing kernel: [ 322.304064] sd 0:2:0:0: [sda] megasas: RESET -17491 cmd=2a retries=0
Mar 24 10:10:05 blackwing kernel: [ 322.304069] megasas: [ 0]waiting for 96 commands to complete
Mar 24 10:10:10 blackwing kernel: [ 327.344068] megasas: [ 5]waiting for 96 commands to complete
Mar 24 10:10:15 blackwing kernel: [ 332.384185] megasas: [10]waiting for 96 commands to complete
Mar 24 10:10:20 blackwing kernel: [ 337.424291] megasas: [15]waiting for 96 commands to complete
Mar 24 10:10:25 blackwing kernel: [ 342.464406] megasas: [20]waiting for 96 commands to complete
Mar 24 10:10:30 blackwing kernel: [ 347.504518] megasas: [25]waiting for 96 commands to complete
Mar 24 10:10:35 blackwing kernel: [ 352.544640] megasas: [30]waiting for 96 commands to complete
Mar 24 10:10:40 blackwing kernel: [ 357.584755] megasas: [35]waiting for 96 commands to complete
Mar 24 10:10:45 blackwing kernel: [ 362.365467] INFO: task kdmflush:563 blocked for more than 120 seconds.
Mar 24 10:10:45 blackwing kernel: [ 362.365539] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 24 10:10:45 blackwing kernel: [ 362.365627] kdmflush D 0000000000000000 0 563 2 0x00000000
Mar 24 10:10:45 blackwing kernel: [ 362.365632] ffff8801b5483d20 0000000000000046 ffff8801b61a65c0 0000000000015880
Mar 24 10:10:45 blackwing kernel: [ 362.365637] ffff8801b61ec7c0 0000000000015880 0000000000015880 0000000000015880
Mar 24 10:10:45 blackwing kernel: [ 362.365641] 0000000000015880 ffff8801b61ec7c0 0000000000015880 0000000000015880
Mar 24 10:10:45 blackwing kernel: [ 362.365645] Call Trace:
Mar 24 10:10:45 blackwing kernel: [ 362.365657] [<ffffffff81528488>] io_schedule+0x28/0x40
Mar 24 10:10:45 blackwing kernel: [ 362.365664] [<ffffffff813e87a7>] dm_wait_for_completion+0x87/0x150
Mar 24 10:10:45 blackwing kernel: [ 362.365671] [<ffffffff81053890>] ? default_wake_function+0x0/0x10
Mar 24 10:10:45 blackwing kernel: [ 362.365675] [<ffffffff813e91de>] dm_flush+0x5e/0x70
Mar 24 10:10:45 blackwing kernel: [ 362.365678] [<ffffffff813e9219>] process_barrier+0x29/0xf0
Mar 24 10:10:45 blackwing kernel: [ 362.365682] [<ffffffff813e931b>] dm_wq_work+0x3b/0x100
Mar 24 10:10:45 blackwing kernel: [ 362.365685] [<ffffffff813e92e0>] ? dm_wq_work+0x0/0x100
Mar 24 10:10:45 blackwing kernel: [ 362.365692] [<ffffffff810731f5>] run_workqueue+0x95/0x170
Mar 24 10:10:45 blackwing kernel: [ 362.365696] [<ffffffff81073374>] worker_thread+0xa4/0x120
Mar 24 10:10:45 blackwing kernel: [ 362.365701] [<ffffffff81078500>] ? autoremove_wake_function+0x0/0x40
Mar 24 10:10:45 blackwing kernel: [ 362.365705] [<ffffffff810732d0>] ? worker_thread+0x0/0x120
Mar 24 10:10:45 blackwing kernel: [ 362.365708] [<ffffffff81078116>] kthread+0xa6/0xb0
Mar 24 10:10:45 blackwing kernel: [ 362.365714] [<ffffffff8101312a>] child_rip+0xa/0x20
Mar 24 10:10:45 blackwing kernel: [ 362.365717] [<ffffffff81078070>] ? kthread+0x0/0xb0
Mar 24 10:10:45 blackwing kernel: [ 362.365720] [<ffffffff81013120>] ? child_rip+0x0/0x20
Mar 24 10:10:45 blackwing kernel: [ 362.365732] INFO: task kjournald2:1122 blocked for more than 120 seconds.
Mar 24 10:10:45 blackwing kernel: [ 362.365800] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 24 10:10:45 blackwing kernel: [ 362.365886] kjournald2 D 0000000000000000 0 1122 2 0x00000000
Mar 24 10:10:45 blackwing kernel: [ 362.365890] ffff8801b7801bd0 0000000000000046 ffff8801b605af70 0000000000015880
Mar 24 10:10:45 blackwing kernel: [ 362.365895] ffff8801b81d3110 0000000000015880 0000000000015880 0000000000015880
Mar 24 10:10:45 blackwing kernel: [ 362.365898] 0000000000015880 ffff8801b81d3110 0000000000015880 0000000000015880
Mar 24 10:10:45 blackwing kernel: [ 362.365902] Call Trace:
Mar 24 10:10:45 blackwing kernel: [ 362.365910] [<ffffffff81145b30>] ? sync_buffer+0x0/0x50
Mar 24 10:10:45 blackwing kernel: [ 362.365914] [<ffffffff81528488>] io_schedule+0x28/0x40
Mar 24 10:10:45 blackwing kernel: [ 362.365918] [<ffffffff81145b6b>] sync_buffer+0x3b/0x50
Mar 24 10:10:45 blackwing kernel: [ 362.365921] [<ffffffff815289a7>] __wait_on_bit+0x57/0x80
Mar 24 10:10:45 blackwing kernel: [ 362.365925] [<ffffffff81145b30>] ? sync_buffer+0x0/0x50
Mar 24 10:10:45 blackwing kernel: [ 362.365929] [<ffffffff81528a43>] out_of_line_wait_on_bit+0x73/0x90
Mar 24 10:10:45 blackwing kernel: [ 362.365933] [<ffffffff81078540>] ? wake_bit_function+0x0/0x40
Mar 24 10:10:45 blackwing kernel: [ 362.365937] [<ffffffff81145b26>] __wait_on_buffer+0x26/0x30
Mar 24 10:10:45 blackwing kernel: [ 362.365943] [<ffffffff811eedd7>] journal_wait_on_commit_record+0x47/0xf0
Mar 24 10:10:45 blackwing kernel: [ 362.365946] [<ffffffff811f009a>] jbd2_journal_commit_transaction+0xfba/0x1100
Mar 24 10:10:45 blackwing kernel: [ 362.365952] [<ffffffff811f5233>] kjournald2+0x103/0x270
Mar 24 10:10:45 blackwing kernel: [ 362.365955] [<ffffffff81078500>] ? autoremove_wake_function+0x0/0x40
Mar 24 10:10:45 blackwing kernel: [ 362.365959] [<ffffffff811f5130>] ? kjournald2+0x0/0x270
Mar 24 10:10:45 blackwing kernel: [ 362.365962] [<ffffffff81078116>] kthread+0xa6/0xb0
Mar 24 10:10:45 blackwing kernel: [ 362.365966] [<ffffffff8101312a>] child_rip+0xa/0x20
Mar 24 10:10:45 blackwing kernel: [ 362.365969] [<ffffffff81078070>] ? kthread+0x0/0xb0
Mar 24 10:10:45 blackwing kernel: [ 362.365972] [<ffffffff81013120>] ? child_rip+0x0/0x20
Mar 24 10:10:45 blackwing kernel: [ 362.365978] INFO: task kjournald2:1348 blocked for more than 120 seconds.
Mar 24 10:10:45 blackwing kernel: [ 362.366046] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 24 10:10:45 blackwing kernel: [ 362.366133] kjournald2 D 0000000000000000 0 1348 2 0x00000000
Mar 24 10:10:45 blackwing kernel: [ 362.366137] ffff8801b614dad0 0000000000000046 0000000000000000 0000000000015880
Mar 24 10:10:45 blackwing kernel: [ 362.366141] ffff8801b2123110 0000000000015880 0000000000015880 0000000000015880
Mar 24 10:10:45 blackwing kernel: [ 362.366145] 0000000000015880 ffff8801b2123110 0000000000015880 0000000000015880
Mar 24 10:10:45 blackwing kernel: [ 362.366149] Call Trace:
Mar 24 10:10:45 blackwing kernel: [ 362.366154] [<ffffffff810da010>] ? sync_page+0x0/0x50
Mar 24 10:10:45 blackwing kernel: [ 362.366158] [<ffffffff81528488>] io_schedule+0x28/0x40
Mar 24 10:10:45 blackwing kernel: [ 362.366161] [<ffffffff810da04d>] sync_page+0x3d/0x50
Mar 24 10:10:45 blackwing kernel: [ 362.366164] [<ffffffff815289a7>] __wait_on_bit+0x57/0x80
Mar 24 10:10:45 blackwing kernel: [ 362.366168] [<ffffffff810da1be>] wait_on_page_bit+0x6e/0x80
Mar 24 10:10:45 blackwing kernel: [ 362.366171] [<ffffffff81078540>] ? wake_bit_function+0x0/0x40
Mar 24 10:10:45 blackwing kernel: [ 362.366176] [<ffffffff810e4480>] ? pagevec_lookup_tag+0x20/0x30
Mar 24 10:10:45 blackwing kernel: [ 362.366180] [<ffffffff810da665>] wait_on_page_writeback_range+0xf5/0x190
Mar 24 10:10:45 blackwing kernel: [ 362.366184] [<ffffffff810da727>] filemap_fdatawait+0x27/0x30
Mar 24 10:10:45 blackwing kernel: [ 362.366188] [<ffffffff811eec96>] journal_finish_inode_data_buffers+0x56/0x150
Mar 24 10:10:45 blackwing kernel: [ 362.366192] [<ffffffff811ef7b6>] jbd2_journal_commit_transaction+0x6d6/0x1100
Mar 24 10:10:45 blackwing kernel: [ 362.366197] [<ffffffff811f5233>] kjournald2+0x103/0x270
Mar 24 10:10:45 blackwing kernel: [ 362.366200] [<ffffffff81078500>] ? autoremove_wake_function+0x0/0x40
Mar 24 10:10:45 blackwing kernel: [ 362.366204] [<ffffffff811f5130>] ? kjournald2+0x0/0x270
Mar 24 10:10:45 blackwing kernel: [ 362.366207] [<ffffffff81078116>] kthread+0xa6/0xb0
Mar 24 10:10:45 blackwing kernel: [ 362.366211] [<ffffffff8101312a>] child_rip+0xa/0x20
Mar 24 10:10:45 blackwing kernel: [ 362.366214] [<ffffffff81078070>] ? kthread+0x0/0xb0
Mar 24 10:10:45 blackwing kernel: [ 362.366217] [<ffffffff81013120>] ? child_rip+0x0/0x20
Mar 24 10:10:45 blackwing kernel: [ 362.624882] megasas: [40]waiting for 96 commands to complete
Mar 24 10:10:50 blackwing kernel: [ 367.664995] megasas: [45]waiting for 96 commands to complete
Mar 24 10:10:55 blackwing kernel: [ 372.705116] megasas: [50]waiting for 96 commands to complete
Mar 24 10:11:00 blackwing kernel: [ 377.745234] megasas: [55]waiting for 96 commands to complete
Mar 24 10:11:05 blackwing kernel: [ 382.785363] megasas: [60]waiting for 96 commands to complete
Mar 24 10:11:10 blackwing kernel: [ 387.825484] megasas: [65]waiting for 96 commands to complete
Mar 24 10:11:15 blackwing kernel: [ 392.865598] megasas: [70]waiting for 96 commands to complete
Mar 24 10:11:20 blackwing kernel: [ 397.905716] megasas: [75]waiting for 96 commands to complete
Mar 24 10:11:26 blackwing kernel: [ 402.945836] megasas: [80]waiting for 96 commands to complete
Mar 24 10:11:31 blackwing kernel: [ 407.985954] megasas: [85]waiting for 96 commands to complete
Mar 24 10:11:36 blackwing kernel: [ 413.026074] megasas: [90]waiting for 96 commands to complete
Mar 24 10:11:41 blackwing kernel: [ 418.066189] megasas: [95]waiting for 96 commands to complete
Mar 24 10:11:46 blackwing kernel: [ 423.106308] megasas: [100]waiting for 96 commands to complete
Mar 24 10:11:51 blackwing kernel: [ 428.146420] megasas: [105]waiting for 96 commands to complete
Mar 24 10:11:56 blackwing kernel: [ 433.186537] megasas: [110]waiting for 96 commands to complete
When the above appears in the logs, all disk i/o stops and i cannot launch anything ( htop, dstat, iotop, aptitude do not start at all ).
I can however do cat /proc/loadavg and i see an increasing load on the system ( last time it went up to 10 before the logs below started )
[ 1009.717634] end_request: I/O error, dev sda, sector 49095082
[ 1009.717711] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.717712] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.717715] end_request: I/O error, dev sda, sector 49095090
[ 1009.717977] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.717979] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.717982] end_request: I/O error, dev sda, sector 49095730
[ 1009.718179] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.718181] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.718184] end_request: I/O error, dev sda, sector 49096370
[ 1009.718330] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.718332] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.718335] end_request: I/O error, dev sda, sector 49115578
[ 1009.718620] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.718623] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.718626] end_request: I/O error, dev sda, sector 49116218
[ 1009.718704] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.718706] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.718709] end_request: I/O error, dev sda, sector 49116226
[ 1009.718976] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.718978] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.718981] end_request: I/O error, dev sda, sector 49116866
[ 1009.719252] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.719254] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.719257] end_request: I/O error, dev sda, sector 49117506
[ 1009.719527] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.719529] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.719532] end_request: I/O error, dev sda, sector 49118146
[ 1009.719607] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.719609] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.719612] end_request: I/O error, dev sda, sector 49118154
[ 1009.719883] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.719885] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.719888] end_request: I/O error, dev sda, sector 49118794
[ 1009.720156] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.720157] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.720160] end_request: I/O error, dev sda, sector 49119434
[ 1009.720425] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.720427] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.720430] end_request: I/O error, dev sda, sector 49120074
[ 1009.720506] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.720508] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.720511] end_request: I/O error, dev sda, sector 49120082
[ 1009.720792] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.720794] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.720797] end_request: I/O error, dev sda, sector 49120722
[ 1009.721073] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.721075] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.721078] end_request: I/O error, dev sda, sector 49121362
[ 1009.721285] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.721287] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.721290] end_request: I/O error, dev sda, sector 49122002
[ 1009.721363] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.721365] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.721368] end_request: I/O error, dev sda, sector 49122010
[ 1009.721623] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.721625] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.721628] end_request: I/O error, dev sda, sector 49122650
[ 1009.721901] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.721903] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.721906] end_request: I/O error, dev sda, sector 49123290
[ 1009.722173] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.722175] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.722178] end_request: I/O error, dev sda, sector 49123930
[ 1009.722253] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.722255] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.722258] end_request: I/O error, dev sda, sector 49123938
[ 1009.722539] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.722541] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.722544] end_request: I/O error, dev sda, sector 49124578
[ 1009.722816] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.722818] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.722821] end_request: I/O error, dev sda, sector 49125218
[ 1009.722897] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.722899] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.722902] end_request: I/O error, dev sda, sector 49125226
[ 1009.723171] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.723172] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.723176] end_request: I/O error, dev sda, sector 49125866
[ 1009.723450] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.723452] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.723455] end_request: I/O error, dev sda, sector 49126506
[ 1009.723719] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.723721] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.723724] end_request: I/O error, dev sda, sector 49127146
[ 1009.723798] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.723800] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.723803] end_request: I/O error, dev sda, sector 49127154
[ 1009.724083] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.724085] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.724088] end_request: I/O error, dev sda, sector 49127794
[ 1009.724352] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.724354] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.724357] end_request: I/O error, dev sda, sector 49128434
[ 1009.724596] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.724598] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.724601] end_request: I/O error, dev sda, sector 49129074
[ 1009.724677] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.724679] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.724682] end_request: I/O error, dev sda, sector 49129082
[ 1009.724958] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.724960] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.724963] end_request: I/O error, dev sda, sector 49129722
[ 1009.725235] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.725237] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.725240] end_request: I/O error, dev sda, sector 49130362
[ 1009.725511] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.725513] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.725516] end_request: I/O error, dev sda, sector 49131002
[ 1009.725591] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.725593] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.725596] end_request: I/O error, dev sda, sector 49131010
[ 1009.725868] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.725870] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.725873] end_request: I/O error, dev sda, sector 49131650
[ 1009.726143] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.726145] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.726148] end_request: I/O error, dev sda, sector 49132290
[ 1009.726420] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.726422] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.726425] end_request: I/O error, dev sda, sector 49132930
[ 1009.726499] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.726501] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.726504] end_request: I/O error, dev sda, sector 49132938
[ 1009.726781] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.726783] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.726786] end_request: I/O error, dev sda, sector 49133578
[ 1009.727061] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.727063] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.727066] end_request: I/O error, dev sda, sector 49134218
[ 1009.727328] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.727330] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.727333] end_request: I/O error, dev sda, sector 49134858
[ 1009.727407] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.727409] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.727412] end_request: I/O error, dev sda, sector 49134866
[ 1009.727681] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.727683] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.727686] end_request: I/O error, dev sda, sector 49135506
[ 1009.727958] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.727960] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.727963] end_request: I/O error, dev sda, sector 49136146
[ 1009.728233] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.728235] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.728238] end_request: I/O error, dev sda, sector 49136786
[ 1009.728512] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.728513] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.728516] end_request: I/O error, dev sda, sector 49137426
[ 1009.728791] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.728793] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.728796] end_request: I/O error, dev sda, sector 49138066
[ 1009.729068] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.729070] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.729073] end_request: I/O error, dev sda, sector 49138706
[ 1009.729339] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.729341] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.729344] end_request: I/O error, dev sda, sector 49139346
[ 1009.729612] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.729614] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.729617] end_request: I/O error, dev sda, sector 49139986
[ 1009.729882] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.729884] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.729887] end_request: I/O error, dev sda, sector 49140626
[ 1009.730157] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.730159] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.730162] end_request: I/O error, dev sda, sector 49141266
[ 1009.730430] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.730432] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.730435] end_request: I/O error, dev sda, sector 49141906
[ 1009.730711] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.730713] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.730716] end_request: I/O error, dev sda, sector 49142546
[ 1009.730986] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.730988] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.730991] end_request: I/O error, dev sda, sector 49143186
[ 1009.731256] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.731258] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.731261] end_request: I/O error, dev sda, sector 49143826
[ 1009.731535] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.731537] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.731540] end_request: I/O error, dev sda, sector 49144466
[ 1009.731817] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.731819] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.731822] end_request: I/O error, dev sda, sector 49145106
[ 1009.732094] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.732096] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.732099] end_request: I/O error, dev sda, sector 49145746
[ 1009.732370] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.732372] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.732375] end_request: I/O error, dev sda, sector 49146386
[ 1009.732644] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.732646] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.732649] end_request: I/O error, dev sda, sector 49147026
[ 1009.732918] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.732920] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.732923] end_request: I/O error, dev sda, sector 49147666
[ 1009.733200] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.733202] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.733205] end_request: I/O error, dev sda, sector 49148306
[ 1009.733482] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.733483] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.733486] end_request: I/O error, dev sda, sector 49148946
[ 1009.733760] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.733762] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.733765] end_request: I/O error, dev sda, sector 49149586
[ 1009.734029] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.734031] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.734034] end_request: I/O error, dev sda, sector 49150226
[ 1009.734290] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.734292] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.734295] end_request: I/O error, dev sda, sector 49150866
[ 1009.734543] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.734545] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.734548] end_request: I/O error, dev sda, sector 49151506
[ 1009.734816] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.734818] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.734821] end_request: I/O error, dev sda, sector 49152146
[ 1009.735090] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.735092] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.735095] end_request: I/O error, dev sda, sector 49152786
[ 1009.735362] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.735364] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.735367] end_request: I/O error, dev sda, sector 49153426
[ 1009.735636] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.735638] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.735641] end_request: I/O error, dev sda, sector 49154066
[ 1009.735913] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.735915] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.735918] end_request: I/O error, dev sda, sector 49154706
[ 1009.736189] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.736191] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.736194] end_request: I/O error, dev sda, sector 49155346
[ 1009.736474] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.736476] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.736479] end_request: I/O error, dev sda, sector 49155986
[ 1009.736761] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.736763] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.736766] end_request: I/O error, dev sda, sector 49156626
[ 1009.737045] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.737047] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.737050] end_request: I/O error, dev sda, sector 49157266
[ 1009.737322] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.737324] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.737327] end_request: I/O error, dev sda, sector 49157906
[ 1009.737550] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.737552] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.737555] end_request: I/O error, dev sda, sector 49158546
[ 1009.737851] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.737853] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.737856] end_request: I/O error, dev sda, sector 49159186
[ 1009.738129] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.738131] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.738135] end_request: I/O error, dev sda, sector 49159826
[ 1009.738409] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.738411] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.738414] end_request: I/O error, dev sda, sector 49160466
[ 1009.738693] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.738695] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.738698] end_request: I/O error, dev sda, sector 49161106
[ 1009.738963] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.738965] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.738968] end_request: I/O error, dev sda, sector 49161746
[ 1009.739233] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.739235] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.739238] end_request: I/O error, dev sda, sector 49162386
[ 1009.739507] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.739509] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.739512] end_request: I/O error, dev sda, sector 49163026
[ 1009.739779] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.739781] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.739784] end_request: I/O error, dev sda, sector 49163666
[ 1009.740049] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.740051] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.740054] end_request: I/O error, dev sda, sector 49164306
[ 1009.740333] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.740335] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.740338] end_request: I/O error, dev sda, sector 49164946
[ 1009.740606] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.740608] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.740611] end_request: I/O error, dev sda, sector 49165586
[ 1009.740886] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.740888] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.740891] end_request: I/O error, dev sda, sector 49166226
[ 1009.741157] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.741159] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.741162] end_request: I/O error, dev sda, sector 49166866
[ 1009.741437] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.741439] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.741442] end_request: I/O error, dev sda, sector 49167506
[ 1009.741714] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.741716] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.741719] end_request: I/O error, dev sda, sector 49168146
[ 1009.741989] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.741991] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.741994] end_request: I/O error, dev sda, sector 49168786
[ 1009.742264] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.742266] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.742269] end_request: I/O error, dev sda, sector 49169426
[ 1009.742545] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.742547] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.742550] end_request: I/O error, dev sda, sector 49170066
[ 1009.742814] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.742816] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.742819] end_request: I/O error, dev sda, sector 49170706
[ 1009.743090] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.743092] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.743095] end_request: I/O error, dev sda, sector 49171346
[ 1009.743373] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.743375] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.743378] end_request: I/O error, dev sda, sector 49171986
[ 1009.743644] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.743646] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.743649] end_request: I/O error, dev sda, sector 49172626
[ 1009.743929] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.743931] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.743934] end_request: I/O error, dev sda, sector 49173266
[ 1009.744203] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.744205] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.744208] end_request: I/O error, dev sda, sector 49173906
[ 1009.744461] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.744463] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.744466] end_request: I/O error, dev sda, sector 49174546
[ 1009.744723] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.744725] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.744728] end_request: I/O error, dev sda, sector 49175186
[ 1009.745001] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.745003] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.745006] end_request: I/O error, dev sda, sector 49175826
[ 1009.745274] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.745276] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.745279] end_request: I/O error, dev sda, sector 49176466
[ 1009.745547] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.745549] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.745552] end_request: I/O error, dev sda, sector 49177106
[ 1009.745709] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.745711] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.745714] end_request: I/O error, dev sda, sector 49177386
[ 1009.745991] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.745993] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.745996] end_request: I/O error, dev sda, sector 49178026
[ 1009.746279] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.746281] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.746284] end_request: I/O error, dev sda, sector 49178666
[ 1009.746475] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.746477] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.746480] end_request: I/O error, dev sda, sector 49179050
[ 1009.746551] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.746553] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.746556] end_request: I/O error, dev sda, sector 49179066
[ 1009.746628] sd 0:2:0:0: [sda] Unhandled error code
[ 1009.746630] sd 0:2:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[ 1009.746633] end_request: I/O error, dev sda, sector 49179082
[ 1009.746708] JBD2: Detected IO errors while flushing file data on dm-7:8
[ 1009.746723] Aborting journal on device dm-7:8.
[ 1009.746799] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.746872] JBD2: I/O error detected when updating journal superblock for dm-7:8.
[ 1009.746972] mpage_da_map_blocks block allocation failed for inode 2596879 at logical offset 1902607 with max blocks 1024 with error -30
[ 1009.747075] This should not happen.!! Data will be lost
[ 1009.747718] EXT4-fs error (device dm-7) in ext4_da_writepages: Journal has aborted
[ 1009.747819] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.747897] EXT4-fs error (device dm-7): ext4_journal_start_sb: Detected aborted journal
[ 1009.748064] EXT4-fs (dm-7): Remounting filesystem read-only
[ 1009.748208] ext4_da_writepages: jbd2_start: 1024 pages, ino 2596879; err -30
[ 1009.748282] Pid: 130, comm: pdflush Not tainted 2.6.31-20-server #58-Ubuntu
[ 1009.748289] Call Trace:
[ 1009.748301] [<ffffffff811b5f57>] ext4_da_writepages+0x587/0x5c0
[ 1009.748311] [<ffffffff8106ab37>] ? lock_timer_base+0x37/0x70
[ 1009.748322] [<ffffffff810e3158>] do_writepages+0x28/0x50
[ 1009.748331] [<ffffffff8113eb5c>] writeback_single_inode+0x1bc/0x450
[ 1009.748340] [<ffffffff8113f478>] generic_sync_sb_inodes+0x418/0x530
[ 1009.748350] [<ffffffff8113f69b>] writeback_inodes+0x5b/0x100
[ 1009.748358] [<ffffffff810e1eec>] wb_kupdate+0xbc/0x140
[ 1009.748367] [<ffffffff810e395e>] __pdflush+0x13e/0x260
[ 1009.748376] [<ffffffff810e3a80>] ? pdflush+0x0/0x50
[ 1009.748384] [<ffffffff810e3ac8>] pdflush+0x48/0x50
[ 1009.748392] [<ffffffff810e1e30>] ? wb_kupdate+0x0/0x140
[ 1009.748400] [<ffffffff810e3a80>] ? pdflush+0x0/0x50
[ 1009.748409] [<ffffffff81078116>] kthread+0xa6/0xb0
[ 1009.748419] [<ffffffff8101312a>] child_rip+0xa/0x20
[ 1009.748427] [<ffffffff81078070>] ? kthread+0x0/0xb0
[ 1009.748435] [<ffffffff81013120>] ? child_rip+0x0/0x20
[ 1009.748480] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.748552] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.748628] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.748697] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.748782] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.748852] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.748923] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.748993] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.749073] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.749144] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.749214] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.749285] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.749371] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.749444] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.749516] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.749586] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.749674] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.749744] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.751025] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.751095] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.751174] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.751245] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.751315] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.751384] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.751463] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.751532] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.751602] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.751672] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.751753] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.751824] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.751894] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.751964] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.752043] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.752114] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.752184] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.752253] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.752334] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.752405] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.752475] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.752545] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.752638] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.752710] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.752780] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.752851] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.752930] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.753000] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.753070] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.753140] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.753219] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.753289] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.753360] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.753430] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.753511] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.753581] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.753651] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.753720] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.753799] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.753869] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.753939] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.754009] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.754087] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.754157] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.754228] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.754297] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.754376] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.754446] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.754515] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.754584] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.754754] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.754825] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.754894] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.754963] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.755043] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.755114] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.755184] sd 0:2:0:0: rejecting I/O to offline device
[ 1009.755254] sd 0:2:0:0: rejecting I/O to offline device
[ 1010.752509] EXT4-fs error (device dm-3): ext4_journal_start_sb: Detected aborted journal
[ 1010.752671] EXT4-fs (dm-3): Remounting filesystem read-only
[ 1010.752808] ext4_da_writepages: jbd2_start: 1024 pages, ino 4221; err -30
[ 1010.752877] Pid: 130, comm: pdflush Not tainted 2.6.31-20-server #58-Ubuntu
[ 1010.752879] Call Trace:
[ 1010.752884] [<ffffffff811b5f57>] ext4_da_writepages+0x587/0x5c0
[ 1010.752890] [<ffffffff810e3158>] do_writepages+0x28/0x50
[ 1010.752894] [<ffffffff8113eb5c>] writeback_single_inode+0x1bc/0x450
[ 1010.752898] [<ffffffff8113f478>] generic_sync_sb_inodes+0x418/0x530
[ 1010.752904] [<ffffffff81036419>] ? default_spin_lock_flags+0x9/0x10
[ 1010.752908] [<ffffffff8113f69b>] writeback_inodes+0x5b/0x100
[ 1010.752912] [<ffffffff810e1eec>] wb_kupdate+0xbc/0x140
[ 1010.752916] [<ffffffff810e395e>] __pdflush+0x13e/0x260
[ 1010.752920] [<ffffffff810e3a80>] ? pdflush+0x0/0x50
[ 1010.752923] [<ffffffff810e3ac8>] pdflush+0x48/0x50
[ 1010.752927] [<ffffffff810e1e30>] ? wb_kupdate+0x0/0x140
[ 1010.752930] [<ffffffff810e3a80>] ? pdflush+0x0/0x50
[ 1010.752934] [<ffffffff81078116>] kthread+0xa6/0xb0
[ 1010.752938] [<ffffffff8101312a>] child_rip+0xa/0x20
[ 1010.752941] [<ffffffff81078070>] ? kthread+0x0/0xb0
[ 1010.752944] [<ffffffff81013120>] ? child_rip+0x0/0x20
[ 1010.753020] sd 0:2:0:0: rejecting I/O to offline device
[ 1010.753089] sd 0:2:0:0: rejecting I/O to offline device
[ 1010.753155] sd 0:2:0:0: rejecting I/O to offline device
[ 1010.753223] sd 0:2:0:0: rejecting I/O to offline device
[ 1010.753301] sd 0:2:0:0: rejecting I/O to offline device
[ 1010.753369] sd 0:2:0:0: rejecting I/O to offline device
[ 1010.753435] sd 0:2:0:0: rejecting I/O to offline device
[ 1010.753500] sd 0:2:0:0: rejecting I/O to offline device
[ 1010.753593] sd 0:2:0:0: rejecting I/O to offline device
[ 1010.753660] sd 0:2:0:0: rejecting I/O to offline device
[ 1010.753726] sd 0:2:0:0: rejecting I/O to offline device
[ 1010.753798] sd 0:2:0:0: rejecting I/O to offline device
[ 1010.753874] sd 0:2:0:0: rejecting I/O to offline device
[ 1010.753941] sd 0:2:0:0: rejecting I/O to offline device
[ 1010.754006] sd 0:2:0:0: rejecting I/O to offline device
[ 1010.754072] sd 0:2:0:0: rejecting I/O to offline device
[ 1010.754148] sd 0:2:0:0: rejecting I/O to offline device
[ 1010.754214] sd 0:2:0:0: rejecting I/O to offline device
[ 1010.754279] sd 0:2:0:0: rejecting I/O to offline device
[ 1010.754345] sd 0:2:0:0: rejecting I/O to offline device
[ 1010.762505] sd 0:2:0:0: rejecting I/O to offline device
[ 1014.844041] sd 0:2:0:0: rejecting I/O to offline device
[ 1014.844116] Aborting journal on device dm-1:8.
[ 1014.844183] sd 0:2:0:0: rejecting I/O to offline device
[ 1014.844248] __ratelimit: 11943 callbacks suppressed
[ 1014.844251] Buffer I/O error on device dm-1, logical block 196608
[ 1014.844318] lost page write due to I/O error on dm-1
[ 1014.844321] JBD2: I/O error detected when updating journal superblock for dm-1:8.
[ 1026.690197] sd 0:2:0:0: rejecting I/O to offline device
[ 1026.690280] sd 0:2:0:0: rejecting I/O to offline device
[ 1026.690388] sd 0:2:0:0: rejecting I/O to offline device
[ 1041.839247] sd 0:2:0:0: rejecting I/O to offline device
[ 1041.839353] sd 0:2:0:0: rejecting I/O to offline device
Any ideas ? The LSI controller doesn't log any errors in his event log. The disk come up clean, and i have even ran a consistency check on both raid arrays.
Is there any known problem with the specific raid controller. I could try replacing it with the model that i have on the other server and where i don't see that problem.
More information about the ubuntu-server
mailing list