[Bug 1001019] Re: Mdadm crash on raid5 reshape

Sun Mar 17 16:32:52 UTC 2013

I've seen this bug too - I was also doing a reshape of a RAID5 array
(but without changing chunk size - just adding a new drive).

I'm also seeing this intermittently only.  One thing that might be
relevant is that I'm reshaping several arrays in succession through a
script (doing them in series, but with no waiting between reshapes).
I've attached my dmesg output - similar error and stack to the above.

Let me know if there are any other diags which would help here, I can
hopefully repro this without too much trouble.

** Attachment added: "Sample dmesg output showing a repro of the bug"
   https://bugs.launchpad.net/ubuntu/+source/mdadm/+bug/1001019/+attachment/3579754/+files/dmesg.txt

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to mdadm in Ubuntu.
https://bugs.launchpad.net/bugs/1001019

Title:
  Mdadm crash on raid5 reshape

Status in “mdadm” package in Ubuntu:
  Confirmed

Bug description:
  Ubuntu 12.04 LTS
  mdadm 3.2.3-2ubuntu1 0

  I try to grow mdadm raid5 array with changing device number and chunk size on same time:
  mdadm --grow /dev/md2 --raid-devices=4 --chunk=512 --backup-file=/root/md2_backup.img

  After this /proc/mdstat don't respond and I can't gen any info about
  raid status

  May 17 23:27:17 protoss kernel: [40835.240145] RAID conf printout:
  May 17 23:27:17 protoss kernel: [40835.240153]  --- level:5 rd:4 wd:4
  May 17 23:27:17 protoss kernel: [40835.240160]  disk 0, o:1, dev:sdc1
  May 17 23:27:17 protoss kernel: [40835.240166]  disk 1, o:1, dev:sdd1
  May 17 23:27:17 protoss kernel: [40835.240171]  disk 2, o:1, dev:sdf1
  May 17 23:27:17 protoss kernel: [40835.240176]  disk 3, o:1, dev:sdg1
  May 17 23:27:17 protoss kernel: [40835.240332] md: reshape of RAID array md2
  May 17 23:27:17 protoss kernel: [40835.240342] md: minimum _guaranteed_  speed: 1000 KB/sec/disk.
  May 17 23:27:17 protoss kernel: [40835.240348] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for reshape.
  May 17 23:27:17 protoss kernel: [40835.240365] md: using 128k window, over a total of 1953513472k.
  May 17 23:27:17 protoss kernel: [40835.507761] md: md_do_sync() got signal ... exiting
  May 17 23:30:42 protoss kernel: [41040.652048] INFO: task md2_raid5:259 blocked for more than 120 seconds.
  May 17 23:30:42 protoss kernel: [41040.652055] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  May 17 23:30:42 protoss kernel: [41040.652062] md2_raid5       D ffffffff81806240     0   259      2 0x00000000
  May 17 23:30:42 protoss kernel: [41040.652074]  ffff8801336a1c40 0000000000000046 ffffea0001d7f180 ffffffff7fffffff
  May 17 23:30:42 protoss kernel: [41040.652086]  ffff8801336a1fd8 ffff8801336a1fd8 ffff8801336a1fd8 0000000000013780
  May 17 23:30:42 protoss kernel: [41040.652097]  ffff880138b796f0 ffff880133ba2de0 ffff8801336a1c30 ffff880133ba2de0
  May 17 23:30:42 protoss kernel: [41040.652108] Call Trace:
  May 17 23:30:42 protoss kernel: [41040.652123]  [<ffffffff8165a55f>] schedule+0x3f/0x60
  May 17 23:30:42 protoss kernel: [41040.652157]  [<ffffffffa00a3d2e>] resize_stripes+0x51e/0x590 [raid456]
  May 17 23:30:42 protoss kernel: [41040.652167]  [<ffffffff81056c9c>] ? update_shares+0xcc/0x100
  May 17 23:30:42 protoss kernel: [41040.652176]  [<ffffffff8105f990>] ? try_to_wake_up+0x200/0x200
  May 17 23:30:42 protoss kernel: [41040.652192]  [<ffffffffa00a3e1f>] check_reshape+0x7f/0xd0 [raid456]
  May 17 23:38:42 protoss kernel: [41520.652310]  [<ffffffff8108a3a0>] ? flush_kthread_worker+0xa0/0xa0
  May 17 23:38:42 protoss kernel: [41520.652319]  [<ffffffff81666bf0>] ? gs_change+0x13/0x13
  May 17 23:38:42 protoss kernel: [41520.652357] INFO: task lvm:25938 blocked for more than 120 seconds.
  May 17 23:38:42 protoss kernel: [41520.652362] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  May 17 23:38:42 protoss kernel: [41520.652367] lvm             D ffffffff81806240     0 25938  25935 0x00000000
  May 17 23:38:42 protoss kernel: [41520.652377]  ffff88006c6b1ae8 0000000000000086 0000000000000000 ffff88006c6b1aa8
  May 17 23:38:42 protoss kernel: [41520.652388]  ffff88006c6b1fd8 ffff88006c6b1fd8 ffff88006c6b1fd8 0000000000013780
  May 17 23:38:42 protoss kernel: [41520.652398]  ffff880138b616f0 ffff88010c02dbc0 ffff88006c6b1ab8 ffff88013fc94040
  May 17 23:38:42 protoss kernel: [41520.652409] Call Trace:
  May 17 23:38:42 protoss kernel: [41520.652417]  [<ffffffff8165a55f>] schedule+0x3f/0x60
  May 17 23:38:42 protoss kernel: [41520.652424]  [<ffffffff8165a60f>] io_schedule+0x8f/0xd0
  May 17 23:38:42 protoss kernel: [41520.652434]  [<ffffffff811b0e74>] dio_await_completion+0x54/0xd0
  May 17 23:38:42 protoss kernel: [41520.652442]  [<ffffffff811b3484>] __blockdev_direct_IO+0x954/0xd90
  May 17 23:38:42 protoss kernel: [41520.652451]  [<ffffffff811af6f0>] ? blkdev_get_block+0x80/0x80
  May 17 23:38:42 protoss kernel: [41520.652461]  [<ffffffff811af1b7>] blkdev_direct_IO+0x57/0x60
  May 17 23:38:42 protoss kernel: [41520.652468]  [<ffffffff811af6f0>] ? blkdev_get_block+0x80/0x80
  May 17 23:38:42 protoss kernel: [41520.652478]  [<ffffffff811196bb>] generic_file_aio_read+0x24b/0x280
  May 17 23:38:42 protoss kernel: [41520.652488]  [<ffffffff811874ac>] ? path_openat+0xfc/0x3f0
  May 17 23:38:42 protoss kernel: [41520.652496]  [<ffffffff81177452>] do_sync_read+0xd2/0x110
  May 17 23:38:42 protoss kernel: [41520.652506]  [<ffffffff8129cd03>] ? security_file_permission+0x93/0xb0
  May 17 23:38:42 protoss kernel: [41520.652514]  [<ffffffff811778d1>] ? rw_verify_area+0x61/0xf0
  May 17 23:38:42 protoss kernel: [41520.652521]  [<ffffffff81177db0>] vfs_read+0xb0/0x180
  May 17 23:38:42 protoss kernel: [41520.652529]  [<ffffffff81177eca>] sys_read+0x4a/0x90
  May 17 23:38:42 protoss kernel: [41520.652537]  [<ffffffff81664a82>] system_call_fastpath+0x16/0x1b

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/mdadm/+bug/1001019/+subscriptions