[SRU][F][G][PATCH 0/3] zPCI attach/detach issues with PF/VF linking support (LP: 1892849)

[frank.heimes at canonical.com]

Buglink: https://bugs.launchpad.net/bugs/1892849

SRU Justification:

[Impact]

* There is a zPCI attach/detach issue in combination with PF/VF linking support.

* On IBM Z with zPCI, VFs can be enabled/disabled individually with /sys/bus/pci/slots/<vf_fid>/power.

* If this was done with a VF that is linked to a parent PF, the PF symlink would become stale while the VF is disabled and
when turned back to the VF, it would not be properly linked back to the PF.

* The PF link is removed together with the whole VF directory.

* Hence for example qemu cannot be used since it relies on such links.

* Additionally there is a missing pci_dev_put() when searching for the parent PF. This potentially results in a reference count of the parent PFs that is becoming too high.

[Fix]

* 3cddb79afc60bcdb5fd9dd7a1c64a8d03bdd460f 3cddb79afc60 "s390/pci: fix zpci_bus_link_virtfn()"

* 2f0230b2f2d5fd287a85583eefb5aed35b6fe510 2f0230b2f2d5 "390/pci: re-introduce zpci_remove_device()"

* b97bf44f99155e57088e16974afb1f2d7b5287aa b97bf44f9915 "s390/pci: fix PF/VF linking on hot plug"

[Test Case]

* Assign a zPCI device, that is capable of handling PFs/VFs (like a RoCE adapter / Connect-X5) to a z15 or LinuxONE III LPAR (usually using the HMC).

* Enable/disable a VF with /sys/bus/pci/slots/<vf_fid>/power

* Try to pass it through to qemu.

* The test needs to be done at IBM due to the special hardware requirements.

[Regression Potential]

* A larger subset of the zPCI files in arch/s390/pci is touched (pci_bus.{h,c}, pci_bus.c, pci.c, s390_pci_hpc.c and pci_event.c), hence there is a certain risk for regressions.

* zPCI is the s390x-specific PCI implementation and (aot) less wide spread compared to the traditional CCW hardware on s390x - no code is touched outside of arch/s390/.

* The modifications are mainly in the IOV and hotplug area of zPCI.

* So SR-IOV, like RoCE adapters, may be harmed by bugs or issues with hot-plugging PCI hardware on s390x.

* But on the other hand side that is the area where these patches fix existing problems.

* In worst case PCI events can be impacted as well, that may harm control and communication

* or changes in base pci_bus/pci code may even break zPCI entirely.

* Right now regular RoCE adapters (like Connect-X5) are currently not handled as real SR-IOV VFs in zPCI, but are treated as normal PCI devices.

* Hence these zPCI SR-IOV setup changes currently apply to PFs with SR-IOV capability only, and those are currently not yet available to customers outside IBM.

* The modifications were tested at IBM in house and a patched Ubuntu kernel was created and shared for further testing and got successfully verified (LP 1892849, comment #3).

[Other]

* This SRU depends on the SRU submitted for LP 1891437, which got already ACKed. So LP 1891437 is a prerequisite and needs to be applied before this one!

* The patches of the depending SRU and the ones here were successfully tested based on a patched Ubuntu test kernel (LP 1892849, comment #3).

* Since the above three patches got upstream accepted with 5.9-rc2, this SRU request is for Focal and Groovy.

Niklas Schnelle (3):
  s390/pci: fix zpci_bus_link_virtfn()
  s390/pci: re-introduce zpci_remove_device()
  s390/pci: fix PF/VF linking on hot plug

 arch/s390/pci/pci.c                | 22 +++++++++----
 arch/s390/pci/pci_bus.c            | 52 +++++++++++++++++-------------
 arch/s390/pci/pci_bus.h            | 13 ++++++++
 arch/s390/pci/pci_event.c          |  4 +--
 drivers/pci/hotplug/s390_pci_hpc.c | 12 +++----
 5 files changed, 65 insertions(+), 38 deletions(-)

-- 
2.25.1