[Bug 1925211] Re: Hot-unplug of disks leaves broken block devices around in Hirsute

Tue Apr 20 13:25:32 UTC 2021

Hirsute - dmesg
# attach
[  264.065866] crw_info : CRW reports slct=0, oflw=0, chn=1, rsc=3, anc=0, erc=4, rsid=5
[  264.065906] crw_info : CRW reports slct=0, oflw=0, chn=0, rsc=3, anc=0, erc=4, rsid=0
[  264.099347] virtio_blk virtio5: [vdc] 385 512-byte logical blocks (197 kB/193 KiB)
# detach
[  289.702243] crw_info : CRW reports slct=0, oflw=0, chn=1, rsc=3, anc=0, erc=4, rsid=5
[  289.702267] crw_info : CRW reports slct=0, oflw=0, chn=0, rsc=3, anc=0, erc=4, rsid=0

Groovy - dmesg
# attach
[  719.712747] crw_info : CRW reports slct=0, oflw=0, chn=1, rsc=3, anc=0, erc=4, rsid=5
[  719.712758] crw_info : CRW reports slct=0, oflw=0, chn=0, rsc=3, anc=0, erc=4, rsid=0
[  719.745538] virtio_blk virtio5: [vdc] 385 512-byte logical blocks (197 kB/193 KiB)
[  719.745542] vdc: detected capacity change from 0 to 197120
# detach
[  780.425222] crw_info : CRW reports slct=0, oflw=0, chn=1, rsc=3, anc=0, erc=4, rsid=5
[  780.425233] crw_info : CRW reports slct=0, oflw=0, chn=0, rsc=3, anc=0, erc=4, rsid=0

There is a difference in hirsute missing the capacity change. The newer kernel
might immediately do that instead of detecting as zero and then bumping it up.

The sizes reported in the Hirsute guest are correct (e.g. 20M for a 20M zfs case).
So this might be a red herring unless the missing message makes you very suspicious

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1925211

Title:
  Hot-unplug of disks leaves broken block devices around in Hirsute

Status in linux package in Ubuntu:
  Confirmed
Status in systemd package in Ubuntu:
  New
Status in udev package in Ubuntu:
  New

Bug description:
  Repro:
  #1 Get a guest
  $ uvt-kvm create --disk 5  --password=ubuntu h release=hirsute arch=s390x label=daily
  $ uvt-kvm wait h release=hirsute arch=s390x label=daily

  #2 Attach and Detach disk
  $ sudo qemu-img create -f qcow2 /var/lib/libvirt/images/test.qcow2 10M
  $ virsh attach-disk h /var/lib/libvirt/images/test.qcow2 vdc
  $ virsh detach-disk h vdc

  From libvirts POV it is gone at this point
  $ virsh domblklist h
   Target   Source
  ------------------------------------------------------------------
   vda      /var/lib/uvtool/libvirt/images/hirsute-2nd-zfs.qcow
   vdb      /var/lib/uvtool/libvirt/images/hirsute-2nd-zfs-ds.qcow

  But the guest thinks still it is present
  $ uvt-kvm ssh --insecure hirsute-2nd-zfs lsblk
    ...
    vdc    252:32   0   20M  0 disk

  This even remains a while after (not a race).

  Any access to it in the guest will hang (as you'd expect of a non-existing blockdev)
  4     0    1758    1739  20   0  12140  4800 -      S+   pts/0      0:00  |           \_ sudo mkfs.ext4 /dev/vdc
  4     0    1759    1758  20   0   6924  1044 -      D+   pts/0      0:00  |               \_ mkfs.ext4 /dev/vdc

  The result above was originally found with hirsute-guest at hirsute-host
  on s390x

  I do NOT see the same with  groovy-guest at hirsute-host on s390x
  I DO see the same with hirsute-guest at groovy-host on s390x
    => Guest version dependent not Host/Hipervisor dependent
  I DO see the same with ZFS disks AND LVM disks being added&removed
    => not type dependent
  I do NOT see the same on x86.
    => Arch dependent ??

  ... the evidence slowly points towards an issue in the guest, damn we are so
  close to release - but non-fully detaching disks are critical in my POV :-/

  Filing this as-is for awareness, but certainly this will need more debugging.
  Unsure where this is going to eventually I'll now file it for kernel/udev/systemd.
  If there are any known issues/components that are related let me know please!
  --- 
  ProblemType: Bug
  AlsaDevices: Error: command ['ls', '-l', '/dev/snd/'] failed with exit code 2: ls: cannot access '/dev/snd/': No such file or directory
  AplayDevices: Error: [Errno 2] No such file or directory: 'aplay'
  ApportVersion: 2.20.11-0ubuntu65
  Architecture: s390x
  ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord'
  CRDA: N/A
  CasperMD5CheckResult: unknown
  DistroRelease: Ubuntu 21.04
  IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig'
  Lspci:

  Lspci-vt: -[0000:00]-
  Lsusb: Error: command ['lsusb'] failed with exit code 1:
  Lsusb-t: Error: command ['lsusb', '-t'] failed with exit code 1: /sys/bus/usb/devices: No such file or directory
  Lsusb-v: Error: command ['lsusb', '-v'] failed with exit code 1:
  Package: udev
  PackageArchitecture: s390x
  PciMultimedia:

  ProcFB:

  ProcKernelCmdLine: root=LABEL=cloudimg-rootfs
  ProcVersionSignature: User Name 5.11.0-14.15-generic 5.11.12
  RelatedPackageVersions:
   linux-restricted-modules-5.11.0-14-generic N/A
   linux-backports-modules-5.11.0-14-generic  N/A
   linux-firmware                             N/A
  RfKill: Error: [Errno 2] No such file or directory: 'rfkill'
  Tags:  hirsute uec-images
  Uname: Linux 5.11.0-14-generic s390x
  UpgradeStatus: No upgrade log present (probably fresh install)
  UserGroups: adm audio cdrom dialout dip floppy lxd netdev plugdev sudo video
  _MarkForUpload: True
  acpidump:

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1925211/+subscriptions