[Bug 1925211] Re: Hot-unplug of disks leaves broken block devices around in Hirsute

Tue Apr 20 13:31:09 UTC 2021

Hirsute - udevadm

# attach
KERNEL[319.020043] add      /devices/css0/0.0.0005 (css)
KERNEL[319.020088] add      /devices/css0/0.0.0005/0.0.0005 (ccw)
KERNEL[319.020103] bind     /devices/css0/0.0.0005/0.0.0005 (ccw)
UDEV  [319.022297] add      /devices/css0/0.0.0005 (css)
UDEV  [319.022802] add      /devices/css0/0.0.0005/0.0.0005 (ccw)
UDEV  [319.023039] bind     /devices/css0/0.0.0005/0.0.0005 (ccw)
KERNEL[319.025073] add      /devices/css0/0.0.0005/0.0.0005/virtio5 (virtio)
UDEV  [319.025527] add      /devices/css0/0.0.0005/0.0.0005/virtio5 (virtio)
KERNEL[319.027524] add      /devices/virtual/bdi/252:32 (bdi)
UDEV  [319.028389] add      /devices/virtual/bdi/252:32 (bdi)
KERNEL[319.048685] add      /devices/css0/0.0.0005/0.0.0005/virtio5/block/vdc (block)
KERNEL[319.048743] bind     /devices/css0/0.0.0005/0.0.0005/virtio5 (virtio)
UDEV  [319.072936] add      /devices/css0/0.0.0005/0.0.0005/virtio5/block/vdc (block)
UDEV  [319.075862] bind     /devices/css0/0.0.0005/0.0.0005/virtio5 (virtio)

# detach
<no entry>

Groovy - udevadm

# attach
KERNEL[719.986637] add      /devices/css0/0.0.0005 (css)
KERNEL[719.986669] add      /devices/css0/0.0.0005/0.0.0005 (ccw)
KERNEL[719.986685] bind     /devices/css0/0.0.0005/0.0.0005 (ccw)
UDEV  [719.988667] add      /devices/css0/0.0.0005 (css)
KERNEL[719.992750] add      /devices/css0/0.0.0005/0.0.0005/virtio5 (virtio)
KERNEL[719.992757] add      /devices/virtual/bdi/252:32 (bdi)
UDEV  [719.993298] add      /devices/css0/0.0.0005/0.0.0005 (ccw)
UDEV  [719.993520] bind     /devices/css0/0.0.0005/0.0.0005 (ccw)
UDEV  [719.994097] add      /devices/virtual/bdi/252:32 (bdi)
UDEV  [719.995568] add      /devices/css0/0.0.0005/0.0.0005/virtio5 (virtio)
KERNEL[720.009523] add      /devices/css0/0.0.0005/0.0.0005/virtio5/block/vdc (block)
KERNEL[720.009544] bind     /devices/css0/0.0.0005/0.0.0005/virtio5 (virtio)
UDEV  [720.058301] add      /devices/css0/0.0.0005/0.0.0005/virtio5/block/vdc (block)
UDEV  [720.059128] bind     /devices/css0/0.0.0005/0.0.0005/virtio5 (virtio)

# detach
KERNEL[780.673663] remove   /devices/virtual/bdi/252:32 (bdi)
KERNEL[780.673928] remove   /devices/css0/0.0.0005/0.0.0005/virtio5/block/vdc (block)
UDEV  [780.676469] remove   /devices/css0/0.0.0005/0.0.0005/virtio5/block/vdc (block)
UDEV  [780.678185] remove   /devices/virtual/bdi/252:32 (bdi)
KERNEL[780.708055] unbind   /devices/css0/0.0.0005/0.0.0005/virtio5 (virtio)
KERNEL[780.708078] remove   /devices/css0/0.0.0005/0.0.0005/virtio5 (virtio)
KERNEL[780.708088] unbind   /devices/css0/0.0.0005/0.0.0005 (ccw)
KERNEL[780.708101] remove   /devices/css0/0.0.0005/0.0.0005 (ccw)
KERNEL[780.708109] unbind   /devices/css0/0.0.0005 (css)
KERNEL[780.708117] remove   /devices/css0/0.0.0005 (css)
UDEV  [780.708779] unbind   /devices/css0/0.0.0005/0.0.0005/virtio5 (virtio)
UDEV  [780.709099] remove   /devices/css0/0.0.0005/0.0.0005/virtio5 (virtio)
UDEV  [780.709397] unbind   /devices/css0/0.0.0005/0.0.0005 (ccw)
UDEV  [780.709670] remove   /devices/css0/0.0.0005/0.0.0005 (ccw)
UDEV  [780.709971] unbind   /devices/css0/0.0.0005 (css)
UDEV  [780.710194] remove   /devices/css0/0.0.0005 (css)

The events on attach are exactly the same, but in slightly different order.
On detach no events are fired at all - so I guess userspace isn't guilty of
not doing anything if there is no event?
As far as i am right now I'd point to the guest kernel, let me try a few other
kernels ...

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1925211

Title:
  Hot-unplug of disks leaves broken block devices around in Hirsute

Status in linux package in Ubuntu:
  Confirmed
Status in systemd package in Ubuntu:
  New
Status in udev package in Ubuntu:
  New

Bug description:
  Repro:
  #1 Get a guest
  $ uvt-kvm create --disk 5  --password=ubuntu h release=hirsute arch=s390x label=daily
  $ uvt-kvm wait h release=hirsute arch=s390x label=daily

  #2 Attach and Detach disk
  $ sudo qemu-img create -f qcow2 /var/lib/libvirt/images/test.qcow2 10M
  $ virsh attach-disk h /var/lib/libvirt/images/test.qcow2 vdc
  $ virsh detach-disk h vdc

  From libvirts POV it is gone at this point
  $ virsh domblklist h
   Target   Source
  ------------------------------------------------------------------
   vda      /var/lib/uvtool/libvirt/images/hirsute-2nd-zfs.qcow
   vdb      /var/lib/uvtool/libvirt/images/hirsute-2nd-zfs-ds.qcow

  But the guest thinks still it is present
  $ uvt-kvm ssh --insecure hirsute-2nd-zfs lsblk
    ...
    vdc    252:32   0   20M  0 disk

  This even remains a while after (not a race).

  Any access to it in the guest will hang (as you'd expect of a non-existing blockdev)
  4     0    1758    1739  20   0  12140  4800 -      S+   pts/0      0:00  |           \_ sudo mkfs.ext4 /dev/vdc
  4     0    1759    1758  20   0   6924  1044 -      D+   pts/0      0:00  |               \_ mkfs.ext4 /dev/vdc

  The result above was originally found with hirsute-guest at hirsute-host
  on s390x

  I do NOT see the same with  groovy-guest at hirsute-host on s390x
  I DO see the same with hirsute-guest at groovy-host on s390x
    => Guest version dependent not Host/Hipervisor dependent
  I DO see the same with ZFS disks AND LVM disks being added&removed
    => not type dependent
  I do NOT see the same on x86.
    => Arch dependent ??

  ... the evidence slowly points towards an issue in the guest, damn we are so
  close to release - but non-fully detaching disks are critical in my POV :-/

  Filing this as-is for awareness, but certainly this will need more debugging.
  Unsure where this is going to eventually I'll now file it for kernel/udev/systemd.
  If there are any known issues/components that are related let me know please!
  --- 
  ProblemType: Bug
  AlsaDevices: Error: command ['ls', '-l', '/dev/snd/'] failed with exit code 2: ls: cannot access '/dev/snd/': No such file or directory
  AplayDevices: Error: [Errno 2] No such file or directory: 'aplay'
  ApportVersion: 2.20.11-0ubuntu65
  Architecture: s390x
  ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord'
  CRDA: N/A
  CasperMD5CheckResult: unknown
  DistroRelease: Ubuntu 21.04
  IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig'
  Lspci:

  Lspci-vt: -[0000:00]-
  Lsusb: Error: command ['lsusb'] failed with exit code 1:
  Lsusb-t: Error: command ['lsusb', '-t'] failed with exit code 1: /sys/bus/usb/devices: No such file or directory
  Lsusb-v: Error: command ['lsusb', '-v'] failed with exit code 1:
  Package: udev
  PackageArchitecture: s390x
  PciMultimedia:

  ProcFB:

  ProcKernelCmdLine: root=LABEL=cloudimg-rootfs
  ProcVersionSignature: User Name 5.11.0-14.15-generic 5.11.12
  RelatedPackageVersions:
   linux-restricted-modules-5.11.0-14-generic N/A
   linux-backports-modules-5.11.0-14-generic  N/A
   linux-firmware                             N/A
  RfKill: Error: [Errno 2] No such file or directory: 'rfkill'
  Tags:  hirsute uec-images
  Uname: Linux 5.11.0-14-generic s390x
  UpgradeStatus: No upgrade log present (probably fresh install)
  UserGroups: adm audio cdrom dialout dip floppy lxd netdev plugdev sudo video
  _MarkForUpload: True
  acpidump:

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1925211/+subscriptions