[SRU][F][PATCH 0/12] s390x/pci: enumerate pci functions per physical adapter (LP: 1874056)

Mon May 18 18:24:26 UTC 2020

Buglink: https://bugs.launchpad.net/bugs/1874056

SRU Justification:

[Impact]

* On s390x the enumeration of PCI functions does not reflect which functions belongs to which physical adapter.

* Layout of a PCI function address on Linux:
  0000:00:00.0
  <root complex>:<bus>:<device>.<function>

* On s390x, each function is presented as individual root complex today, e.g.:
  PCHID 0100 VF1 0000:00:00.0
  PCHID 0100 VF23 0001:00:00.0
  PCHID 0200 VF1 0002:00:00.0
  OCHID 0100 VF17 0003:00:00.0

* On other platforms, the addresses correctly reflect the actual HW configuration.

* Some device drivers (like mlx5, for Mellanox adapters) group functions of one physical adapter by checking which PCI functions have identical values for <root complex>:<bus>:<device>.

* We need to use the same enumeration scheme to achieve this functionality on s390x.

* In this case, the two physical functions of a Mellanox adapter need to get function number 0 and 1,
  and all virtual functions need to use the same <root complex>:<bus> numbers with function/device numbers counting up.

* Required result (example with 4 VFs per PF):
  PCHID 0100 PF 0 0000:00:00.0
  PCHID 0100 PF 1 0000:00:00.1
  PCHID 0100 PF 0 VF 0 0000:00:00.2
  PCHID 0100 PF 0 VF 1 0000:00:00.3
  PCHID 0100 PF 0 VF 2 0000:00:00.4
  PCHID 0100 PF 0 VF 3 0000:00:00.5
  PCHID 0100 PF 1 VF 0 0000:00:00.6
  PCHID 0100 PF 1 VF 1 0000:00:00.7
  PCHID 0100 PF 1 VF 2 0000:00:00.8
  PCHID 0100 PF 1 VF 3 0000:00:00.9
  PCHID 0200 PF 0 0001:00:00.0

[Fix]

* Backport 1: https://launchpadlibrarian.net/479699471/0001-s390-pci-Improve-handling-of-unset-UID.patch

* Backport 2: https://launchpadlibrarian.net/479699482/0002-s390-pci-embedding-hotplug_slot-in-zdev.patch

* Backport 3: https://launchpadlibrarian.net/479699492/0003-s390-pci-Expose-new-port-attribute-for-PCIe-function.patch

* Backport 4: https://launchpadlibrarian.net/479699497/0004-s390-pci-adaptation-of-iommu-to-multifunction.patch

* Backport 5: https://launchpadlibrarian.net/479700706/0005-s390-pci-define-kernel-parameters-for-PCI-multifunct.patch

* Backport 6: https://launchpadlibrarian.net/479700712/0006-s390-pci-define-RID-and-RID-available.patch

* Backport 7: https://launchpadlibrarian.net/479700739/0007-s390-pci-create-zPCI-bus.patch

* Backport 8: https://launchpadlibrarian.net/479700769/0008-s390-pci-adapt-events-for-zbus.patch

* Backport 9: https://launchpadlibrarian.net/479700786/0009-s390-pci-Handling-multifunctions.patch

* Backport 10: https://launchpadlibrarian.net/479700794/0010-s390-pci-Do-not-disable-PF-when-VFs-exist.patch

* Backport 11: https://launchpadlibrarian.net/479700798/0011-s390-pci-Documentation-for-zPCI.patch

* Backport 12: https://launchpadlibrarian.net/479700799/0012-s390-pci-removes-wrong-PCI-multifunction-assignment.patch

[Test Case]

* Prepare an IBM z13 or LinuxONE III (or newer) system with two or more RoCE Express PCI 2(.1) adapters.

* Assign the adapters (and it's virtual functions) to an LPAR.

* Verify whether the physical and virtual functions are grouped in arbitrary order or in consecutive order - physical first (for example with lspci -t ...)

[Regression Potential] 

* The regression potential can be considered as moderate, since:

* It is purely s390x specific code (arch/s390/* drivers/iommu/s390-iommu.c and drivers/pci/hotplug/s390_pci_hpc.c - and some doc adjustments, too).

* It largely affects zPCI, the s390x specific PCI code layer.

* PCI cards available for s390x are optional cards (RoCE and zEDC) and not very wide-spread.

* The situation described above affects the RoCE adapters only (Mellanox based).

* The patches are also upstream accepted and available via linux-next, but to apply them to focal kernel 5.4 the above backports are needed.

* However, the code is modified by several patches (12), hence there is a chance to break zPCI with them.

* For upfront testing a PPA got created with a focal (master-next) kernel that incl. all the above patches.

Alexander Schmidt (1):
  s390/pci: Expose new port attribute for PCIe functions

Niklas Schnelle (1):
  s390/pci: Improve handling of unset UID

Pierre Morel (10):
  s390/pci: embedding hotplug_slot in zdev
  s390/pci: adaptation of iommu to multifunction
  s390/pci: define kernel parameters for PCI multifunction
  s390/pci: define RID and RID available
  s390/pci: create zPCI bus
  s390/pci: adapt events for zbus
  s390/pci: Handling multifunctions
  s390/pci: Do not disable PF when VFs exist
  s390/pci: Documentation for zPCI
  s390/pci: removes wrong PCI multifunction assignment

 .../admin-guide/kernel-parameters.txt         |   2 +
 Documentation/s390/index.rst                  |   1 +
 Documentation/s390/pci.rst                    | 126 +++++++++
 MAINTAINERS                                   |   1 +
 arch/s390/include/asm/pci.h                   |  45 +++-
 arch/s390/include/asm/pci_clp.h               |  12 +-
 arch/s390/pci/Makefile                        |   3 +-
 arch/s390/pci/pci.c                           | 198 ++++++--------
 arch/s390/pci/pci_bus.c                       | 255 ++++++++++++++++++
 arch/s390/pci/pci_bus.h                       |  31 +++
 arch/s390/pci/pci_clp.c                       |   6 +-
 arch/s390/pci/pci_event.c                     |  39 ++-
 arch/s390/pci/pci_sysfs.c                     |   4 +-
 drivers/iommu/s390-iommu.c                    |   8 +-
 drivers/pci/hotplug/s390_pci_hpc.c            | 105 +++-----
 15 files changed, 618 insertions(+), 218 deletions(-)
 create mode 100644 Documentation/s390/pci.rst
 create mode 100644 arch/s390/pci/pci_bus.c
 create mode 100644 arch/s390/pci/pci_bus.h

-- 
2.25.1