[Bug 1817097] Re: pvmove causes file system corruption without notice upon move from 512 -> 4096 logical block size devices
Dimitri John Ledkov
launchpad at surgut.co.uk
Mon Mar 11 15:11:53 UTC 2019
This is a well-known upstream issue/bug.
This is not s390x, Ubuntu 18.10, any other Ubuntu release specific.
There is no dataloss -> one can execute pvmove operation in reverse (or i guess onto any 512 sector size PV) to mount the filesystems again.
Thus this is not critical at all.
Also, I am failing to understand what is the expectation for Canonical
to do, w.r.t. this bug report?
If you want support, as a workaround one can force using 4k sizes, with
vgcreate and ext4, then moving volumes to/from 512/4k physical volumes
appears to work seamlessly:
$ sudo vgcreate --physicalextentsize 4k newtestvg /dev/...
$ sudo mkfs.ext4 -b 4096 /dev/mapper/...
For a more general solution, do create stand-alone new VGs/LVs/FSs, and
migrate data over using higther level tools - e.g. dump/restore, rsync,
etc.
But note, that launchpad should not be used for support requests. Please
use your UA account (salesforce) for support request for your production
systems.
This is discussed upstream, where they are trying to introduce a soft
check to prevent from moving data across.
https://bugzilla.redhat.com/show_bug.cgi?id=1669751 But it's not a real
solution, just a weak safety check. As one can still force create ext4fs
of either 512 or 4k, and move the volume to the "wrong" size. As ideally
it would be user friendly if moving to/from mixed sector sizes would
just work(tm) but that's unlikely to happen upstream, thus is wont-fix
downstream too.
Was there anything in particular that you were expecting for us to
change?
We could change the cloud-images (if they don't already), installers
(i.e. d-i / subiquity) or the utils (i.e. vgcreate, mkfs.ext4) to
default to 4k minimum sector sizes. But at the moment, these utils try
to guess the sector sizes based on heuristics at creation time, and
obviously get is "wrong" if the underlying device is swapped away from
under their feet post creation time. Thus this is expected.
References:
The upstream bug report is https://bugzilla.redhat.com/show_bug.cgi?id=1669751
The upstream overridable weak safety-net check is https://sourceware.org/git/?p=lvm2.git;a=commitdiff;h=dd6ff9e3a75801fc5c6166aa0983fa8df098e91a
And that will make it into ubuntu eventually, when released in a stable lvm2 release and integration into ubuntu.
Please remove severity critical
Please remove target ubuntu 18.10
Please provide explanation as to why this issue was filed
** Bug watch added: Red Hat Bugzilla #1669751
https://bugzilla.redhat.com/show_bug.cgi?id=1669751
** Changed in: linux (Ubuntu)
Status: New => Invalid
** Changed in: ubuntu-z-systems
Status: New => Incomplete
** Changed in: lvm2 (Ubuntu)
Status: New => Incomplete
** Also affects: lvm2 via
https://bugzilla.redhat.com/show_bug.cgi?id=1669751
Importance: Unknown
Status: Unknown
--
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to lvm2 in Ubuntu.
https://bugs.launchpad.net/bugs/1817097
Title:
pvmove causes file system corruption without notice upon move from 512
-> 4096 logical block size devices
Status in lvm2:
Unknown
Status in Ubuntu on IBM z Systems:
Incomplete
Status in linux package in Ubuntu:
Invalid
Status in lvm2 package in Ubuntu:
Incomplete
Bug description:
Problem Description---
Summary
=======
Environment: IBM Z13 LPAR and z/VM Guest
IBM Type: 2964 Model: 701 NC9
OS: Ubuntu 18.10 (GNU/Linux 4.18.0-13-generic s390x)
Package: lvm2 version 2.02.176-4.1ubuntu3
LVM: pvmove operation corrupts file system when using 4096 (4k) logical block size
and default block size being 512 bytes in the underlying devices
The problem is immediately reproducible.
We see a real usability issue with data destruction as consequence - which is not acceptable.
We expect 'pvmove' to fail with error in such situations to prevent fs destruction,
which might possibly be overridden by a force flag.
Details
=======
After a 'pvmove' operation is run to move a physical volume onto an ecrypted
device with 4096 bytes logical block size we experience a file system corruption.
There is no need for the file system to be mounted, but the problem surfaces
differently if so.
Either, the 'pvs' command after the pvmove shows
/dev/LOOP_VG/LV: read failed after 0 of 1024 at 0: Invalid argument
/dev/LOOP_VG/LV: read failed after 0 of 1024 at 314507264: Invalid argument
/dev/LOOP_VG/LV: read failed after 0 of 1024 at 314564608: Invalid argument
/dev/LOOP_VG/LV: read failed after 0 of 1024 at 4096: Invalid argument
or
a subsequent mount shows (after umount if the fs had previously been mounted as in our
setup)
mount: /mnt: wrong fs type, bad option, bad superblock on /dev/mapper/LOOP_VG-LV, missing codepage or helper program, or other error.
A minimal setup of LVM using one volume group with one logical volume defined,
based on one physical volume is sufficient to raise the problem. One more physical
volume of the same size is needed to run the pvmove operation to.
LV
|
VG: LOOP_VG [ ]
|
PV: /dev/loop0 --> /dev/mapper/enc-loop
( backed by /dev/mapper/enc-loop )
The physical volumes are backed by loopback devices (losetup) to base the
problem report on, but we have seen the error on real SCSI multipath volumes
also, with and without cryptsetup mapper devices in use.
Further discussion
==================
https://www.saout.de/pipermail/dm-crypt/2019-February/006078.html
The problem does not occur on block devices with native size of 4k.
E.g. DASDs, or file systems with mkfs -b 4096 option.
Terminal output
===============
See attached file pvmove-error.txt
Debug data
==========
pvmove was run with -dddddd (maximum debug level)
See attached journal file.
Contact Information = Christian.Rund at de.ibm.com
---uname output---
Linux system 4.18.0-13-generic #14-Ubuntu SMP Wed Dec 5 09:00:35 UTC 2018 s390x s390x s390x GNU/Linux
Machine Type = IBM Type: 2964 Model: 701 NC9
---Debugger---
A debugger is not configured
---Steps to Reproduce---
1.) Create two image files of 500MB in size
and set up two loopback devices with 'losetup -fP FILE'
2.) Create one physical volume and one volume group 'LOOP_VG',
and one logical volume 'VG'
Run:
pvcreate /dev/loop0
vgcreate LOOP_VG /dev/loop0
lvcreate -L 300MB LOOP_VG -n LV /dev/loop0
3.) Create a file system on the logical volume device:
mkfs.ext4 /dev/mapper/LOOP_VG-LV
4.) mount the file system created in the previous step to some empty available directory:
mount /dev/mapper/LOOP_VG-LV /mnt
5.) Set up a second physical volume, this time encrypted with LUKS2,
and open the volume to make it available:
cryptsetup luksFormat --type luks2 --sector-size 4096 /dev/loop1
cryptsetup luksOpen /dev/loop1 enc-loop
6.) Create the second physical volume, and add it to the LOOP_VG
pvcreate /dev/mapper/enc-loop
vgextend LOOP_VG /dev/mapper/enc-loop
7.) Ensure the new physical volume is part of the volume group:
pvs
8.) Move the /dev/loop0 volume onto the encrypted volume with maximum debug option:
pvmove -dddddd /dev/loop0 /dev/mapper/enc-loop
9.) The previous step succeeds, but corrupts the file system on the logical volume
We expect an error here.
There might be a command line flag to override used because corruption does not cause a data loss.
Userspace tool common name: pvmove
The userspace tool has the following bit modes: 64bit
Userspace rpm: lvm2 in versoin 2.02.176-4.1ubuntu3
Userspace tool obtained from project website: na
*Additional Instructions for Christian.Rund at de.ibm.com:
-Attach ltrace and strace of userspace application.
To manage notifications about this bug go to:
https://bugs.launchpad.net/lvm2/+bug/1817097/+subscriptions
More information about the foundations-bugs
mailing list