[Bug 1850540] Re: multi-zone raid0 corruption

Fri Dec 6 22:10:12 UTC 2019

= disco verification =
ubuntu at ip-172-30-0-118:~$ sudo mdadm --create /dev/md0 --run --metadata=default --level=0 --raid-devices=2 /dev/xvdb /dev/xvdc
mdadm: Fail create md0 when using /sys/module/md_mod/parameters/new_array
mdadm: array /dev/md0 started.
ubuntu at ip-172-30-0-118:~$ cat /proc/mdstat
Personalities : [raid0] 
md0 : active raid0 xvdc[1] xvdb[0]
      3141632 blocks super 1.2 512k chunks

unused devices: <none>

## kernel & mdadm upgrade
ubuntu at ip-172-30-0-118:~$ dmesg | grep raid
[    2.419732] md: If you don't use raid, use raid=noautodetect
[    4.990204] md/raid0:md0: cannot assemble multi-zone RAID0 with default_layout setting
[    4.997283] md/raid0: please set raid0.default_layout to 1 or 2
ubuntu at ip-172-30-0-118:~$ sudo mdadm --stop /dev/md0
mdadm: stopped /dev/md0
ubuntu at ip-172-30-0-118:~$ sudo mdadm --assemble -U layout-original /dev/md0 /dev/xvdb /dev/xvdc
mdadm: /dev/md0 has been started with 2 drives.
ubuntu at ip-172-30-0-118:~$ cat /proc/mdstat
Personalities : [raid0] 
md0 : active raid0 xvdb[0] xvdc[1]
      3141632 blocks super 1.2 512k chunks

unused devices: <none>

## reboot

ubuntu at ip-172-30-0-118:~$ dmesg | grep raid
[    2.545479] md: If you don't use raid, use raid=noautodetect
ubuntu at ip-172-30-0-118:~$ cat /proc/mdstat
Personalities : [raid0] 
md0 : active raid0 xvdb[0] xvdc[1]
      3141632 blocks super 1.2 512k chunks

unused devices: <none>
ubuntu at ip-172-30-0-118:~$ sudo mdadm --stop /dev/md0
mdadm: stopped /dev/md0
ubuntu at ip-172-30-0-118:~$ sudo mdadm --create /dev/md0 --run --metadata=default --level=0 --raid-devices=2 /dev/xvdc /dev/xvdb
mdadm: /dev/xvdc appears to be part of a raid array:
       level=raid0 devices=2 ctime=Fri Dec  6 21:57:50 2019
mdadm: /dev/xvdb appears to be part of a raid array:
       level=raid0 devices=2 ctime=Fri Dec  6 21:57:50 2019
mdadm: array /dev/md0 started.
ubuntu at ip-172-30-0-118:~$ sudo mdadm --detail /dev/md0 | grep Layout
            Layout : original

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to mdadm in Ubuntu.
https://bugs.launchpad.net/bugs/1850540

Title:
  multi-zone raid0 corruption

Status in Release Notes for Ubuntu:
  New
Status in linux package in Ubuntu:
  Confirmed
Status in mdadm package in Ubuntu:
  Fix Released
Status in linux source package in Precise:
  New
Status in mdadm source package in Precise:
  New
Status in linux source package in Trusty:
  Confirmed
Status in mdadm source package in Trusty:
  Confirmed
Status in linux source package in Xenial:
  Confirmed
Status in mdadm source package in Xenial:
  Confirmed
Status in linux source package in Bionic:
  Confirmed
Status in mdadm source package in Bionic:
  Fix Committed
Status in linux source package in Disco:
  Confirmed
Status in mdadm source package in Disco:
  Fix Committed
Status in linux source package in Eoan:
  Confirmed
Status in mdadm source package in Eoan:
  Fix Committed
Status in linux source package in Focal:
  Confirmed
Status in mdadm source package in Focal:
  Fix Released
Status in mdadm package in Debian:
  Fix Released

Bug description:
  Bug 1849682 tracks the temporarily revert of the fix for this issue,
  while this bug tracks the re-application of that fix once we have a
  full solution.

  [Impact]
  (cut & paste from https://marc.info/?l=linux-raid&m=157360088014027&w=2)
  An unintentional RAID0 layout change was introduced in the v3.14 kernel. This effectively means there are 2 different layouts Linux will use to write data to RAID0 arrays in the wild - the “pre-3.14” way and the “3.14 and later” way. Mixing these layouts by writing to an array while booted on these different kernel versions can lead to corruption.

  Note that this only impacts RAID0 arrays that include devices of
  different sizes. If your devices are all the same size, both layouts
  are equivalent, and your array is not at risk of corruption due to
  this issue.

  Unfortunately, the kernel cannot detect which layout was used for
  writes to pre-existing arrays, and therefore requires input from the
  administrator. This input can be provided via the kernel command line
  with the raid0.default_layout=<N> parameter, or by setting the
  default_layout module parameter when loading the raid0 module. With a
  new enough version of mdadm (>= 4.2, or equivalent distro backports),
  you can set the layout version when assembling a stopped array. For
  example:

  mdadm --stop /dev/md0
  mdadm --assemble -U layout-alternate /dev/md0 /dev/sda1 /dev/sda2
  See the mdadm manpage for more details. Once set in this manner, the layout will be recorded in the array and will not need to be explicitly specified in the future.

  (The mdadm part of this SRU is for the above support ^)

  [Test Case]
  = mdadm =
  Confirm that a multi-zone raid0 created w/ older mdadm is able to be started on a fixed kernel by setting a layout.
  1) Ex: w/ old kernel/mdadm:
    mdadm --create /dev/md0 --run --metadata=default \
          --level=0 --raid-devices=2 /dev/vdb1 /dev/vdc1
  2) Reboot onto fixed kernel & update mdadm
  3) sudo mdadm --stop /dev/md0 &&
     sudo mdadm --assemble -U layout-alternate \
       /dev/md0 /dev/vdb1 /dev/vdc1
  4) Confirm that the array autostarts on reboot
  5) Confirm that w/ new kernel & new mdadm, a user can create and start an array in a backwards-compatible fashion (i.e. w/o an explicit layout).
  6) Verify that 'mdadm --detail /dev/md0' displays the layout

  = linux =
  Similar to above, but using kernel command line options.

  [Regression Risk]
  The kernel side of things will break starting pre-existing arrays. That's intentional.

  Although I've done due-diligence to check for backwards compatibility
  issues, the mdadm side may still present some.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu-release-notes/+bug/1850540/+subscriptions