[Bug 1925211] Re: Hot-unplug of disks leaves broken block devices around in Hirsute on s390x

[Frank Heimes]

With that we can close the entire bug, since hirsute / 5.11 is now Fix
Released (#34), groovy / 5.8 is not affected (#17) and the patches are
upstream with 5.13 (#33), hence will end up sooner or later in impish.

** Changed in: ubuntu-z-systems
       Status: Fix Committed => Fix Released

** Changed in: linux (Ubuntu)
       Status: Fix Committed => Fix Released

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1925211

Title:
  Hot-unplug of disks leaves broken block devices around in Hirsute on
  s390x

Status in Ubuntu on IBM z Systems:
  Fix Released
Status in linux package in Ubuntu:
  Fix Released
Status in systemd package in Ubuntu:
  Invalid
Status in udev package in Ubuntu:
  Invalid
Status in linux source package in Hirsute:
  Fix Released
Status in systemd source package in Hirsute:
  Invalid
Status in udev source package in Hirsute:
  Invalid

Bug description:
  SRU Justification

  [Impact]

  Hot removal of disks under kvm on s390 does not result in the kernel
  removing the block device, which can lead to hung tasks and other
  issues.

  [Test Plan]

  See steps to reproduce the bug in the original description below. To
  test, execute these steps and confirm that the block device gets
  removed as expected.

  [Where problems could occur]

  The fix is a revert of the changes which introduced this regression.
  The original commit was a removal of supposedly unused code, but it
  seems a mistake was made in the logic around unregistering of disks.
  Reverting the changes could have potential to introduce bugs related
  to other virt devices, especially if it interacts badly with
  subsequent driver changes. However, the patch reverted cleanly, and
  reverting restores the code to the state which has been working well
  in previous kernels and seems like the lowest risk option until a
  proper fix is available upstream.

  ---

  Repro:
  #1 Get a guest
  $ uvt-kvm create --disk 5  --password=ubuntu h release=hirsute arch=s390x label=daily
  $ uvt-kvm wait h release=hirsute arch=s390x label=daily

  #2 Attach and Detach disk
  $ sudo qemu-img create -f qcow2 /var/lib/libvirt/images/test.qcow2 10M
  $ virsh attach-disk h /var/lib/libvirt/images/test.qcow2 vdc
  $ virsh detach-disk h vdc

  From libvirts POV it is gone at this point
  $ virsh domblklist h
   Target   Source
  ------------------------------------------------------------------
   vda      /var/lib/uvtool/libvirt/images/hirsute-2nd-zfs.qcow
   vdb      /var/lib/uvtool/libvirt/images/hirsute-2nd-zfs-ds.qcow

  But the guest thinks still it is present
  $ uvt-kvm ssh --insecure hirsute-2nd-zfs lsblk
    ...
    vdc    252:32   0   20M  0 disk

  This even remains a while after (not a race).

  Any access to it in the guest will hang (as you'd expect of a non-existing blockdev)
  4     0    1758    1739  20   0  12140  4800 -      S+   pts/0      0:00  |           \_ sudo mkfs.ext4 /dev/vdc
  4     0    1759    1758  20   0   6924  1044 -      D+   pts/0      0:00  |               \_ mkfs.ext4 /dev/vdc

  The result above was originally found with hirsute-guest at hirsute-host
  on s390x

  I do NOT see the same with  groovy-guest at hirsute-host on s390x
  I DO see the same with hirsute-guest at groovy-host on s390x
    => Guest version dependent not Host/Hipervisor dependent
  I DO see the same with ZFS disks AND LVM disks being added&removed
    => not type dependent
  I do NOT see the same on x86.
    => Arch dependent ??

  ... the evidence slowly points towards an issue in the guest, damn we are so
  close to release - but non-fully detaching disks are critical in my POV :-/

  Filing this as-is for awareness, but certainly this will need more debugging.
  Unsure where this is going to eventually I'll now file it for kernel/udev/systemd.
  If there are any known issues/components that are related let me know please!
  ---
  ProblemType: Bug
  AlsaDevices: Error: command ['ls', '-l', '/dev/snd/'] failed with exit code 2: ls: cannot access '/dev/snd/': No such file or directory
  AplayDevices: Error: [Errno 2] No such file or directory: 'aplay'
  ApportVersion: 2.20.11-0ubuntu65
  Architecture: s390x
  ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord'
  CRDA: N/A
  CasperMD5CheckResult: unknown
  DistroRelease: Ubuntu 21.04
  IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig'
  Lspci:

  Lspci-vt: -[0000:00]-
  Lsusb: Error: command ['lsusb'] failed with exit code 1:
  Lsusb-t: Error: command ['lsusb', '-t'] failed with exit code 1: /sys/bus/usb/devices: No such file or directory
  Lsusb-v: Error: command ['lsusb', '-v'] failed with exit code 1:
  Package: udev
  PackageArchitecture: s390x
  PciMultimedia:

  ProcFB:

  ProcKernelCmdLine: root=LABEL=cloudimg-rootfs
  ProcVersionSignature: User Name 5.11.0-14.15-generic 5.11.12
  RelatedPackageVersions:
   linux-restricted-modules-5.11.0-14-generic N/A
   linux-backports-modules-5.11.0-14-generic  N/A
   linux-firmware                             N/A
  RfKill: Error: [Errno 2] No such file or directory: 'rfkill'
  Tags:  hirsute uec-images
  Uname: Linux 5.11.0-14-generic s390x
  UpgradeStatus: No upgrade log present (probably fresh install)
  UserGroups: adm audio cdrom dialout dip floppy lxd netdev plugdev sudo video
  _MarkForUpload: True
  acpidump:

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu-z-systems/+bug/1925211/+subscriptions