ACK: [SRU][F][PATCH 0/12] s390x/pci: enumerate pci functions per physical adapter (LP: 1874056)
Kleber Souza
kleber.souza at canonical.com
Fri May 29 06:31:36 UTC 2020
On 2020-05-18 20:24, frank.heimes at canonical.com wrote:
> Buglink: https://bugs.launchpad.net/bugs/1874056
>
> SRU Justification:
>
> [Impact]
>
> * On s390x the enumeration of PCI functions does not reflect which functions belongs to which physical adapter.
>
> * Layout of a PCI function address on Linux:
> 0000:00:00.0
> <root complex>:<bus>:<device>.<function>
>
> * On s390x, each function is presented as individual root complex today, e.g.:
> PCHID 0100 VF1 0000:00:00.0
> PCHID 0100 VF23 0001:00:00.0
> PCHID 0200 VF1 0002:00:00.0
> OCHID 0100 VF17 0003:00:00.0
>
> * On other platforms, the addresses correctly reflect the actual HW configuration.
>
> * Some device drivers (like mlx5, for Mellanox adapters) group functions of one physical adapter by checking which PCI functions have identical values for <root complex>:<bus>:<device>.
>
> * We need to use the same enumeration scheme to achieve this functionality on s390x.
>
> * In this case, the two physical functions of a Mellanox adapter need to get function number 0 and 1,
> and all virtual functions need to use the same <root complex>:<bus> numbers with function/device numbers counting up.
>
> * Required result (example with 4 VFs per PF):
> PCHID 0100 PF 0 0000:00:00.0
> PCHID 0100 PF 1 0000:00:00.1
> PCHID 0100 PF 0 VF 0 0000:00:00.2
> PCHID 0100 PF 0 VF 1 0000:00:00.3
> PCHID 0100 PF 0 VF 2 0000:00:00.4
> PCHID 0100 PF 0 VF 3 0000:00:00.5
> PCHID 0100 PF 1 VF 0 0000:00:00.6
> PCHID 0100 PF 1 VF 1 0000:00:00.7
> PCHID 0100 PF 1 VF 2 0000:00:00.8
> PCHID 0100 PF 1 VF 3 0000:00:00.9
> PCHID 0200 PF 0 0001:00:00.0
>
> [Fix]
>
> * Backport 1: https://launchpadlibrarian.net/479699471/0001-s390-pci-Improve-handling-of-unset-UID.patch
>
> * Backport 2: https://launchpadlibrarian.net/479699482/0002-s390-pci-embedding-hotplug_slot-in-zdev.patch
>
> * Backport 3: https://launchpadlibrarian.net/479699492/0003-s390-pci-Expose-new-port-attribute-for-PCIe-function.patch
>
> * Backport 4: https://launchpadlibrarian.net/479699497/0004-s390-pci-adaptation-of-iommu-to-multifunction.patch
>
> * Backport 5: https://launchpadlibrarian.net/479700706/0005-s390-pci-define-kernel-parameters-for-PCI-multifunct.patch
>
> * Backport 6: https://launchpadlibrarian.net/479700712/0006-s390-pci-define-RID-and-RID-available.patch
>
> * Backport 7: https://launchpadlibrarian.net/479700739/0007-s390-pci-create-zPCI-bus.patch
>
> * Backport 8: https://launchpadlibrarian.net/479700769/0008-s390-pci-adapt-events-for-zbus.patch
>
> * Backport 9: https://launchpadlibrarian.net/479700786/0009-s390-pci-Handling-multifunctions.patch
>
> * Backport 10: https://launchpadlibrarian.net/479700794/0010-s390-pci-Do-not-disable-PF-when-VFs-exist.patch
>
> * Backport 11: https://launchpadlibrarian.net/479700798/0011-s390-pci-Documentation-for-zPCI.patch
>
> * Backport 12: https://launchpadlibrarian.net/479700799/0012-s390-pci-removes-wrong-PCI-multifunction-assignment.patch
>
> [Test Case]
>
> * Prepare an IBM z13 or LinuxONE III (or newer) system with two or more RoCE Express PCI 2(.1) adapters.
>
> * Assign the adapters (and it's virtual functions) to an LPAR.
>
> * Verify whether the physical and virtual functions are grouped in arbitrary order or in consecutive order - physical first (for example with lspci -t ...)
>
> [Regression Potential]
>
> * The regression potential can be considered as moderate, since:
>
> * It is purely s390x specific code (arch/s390/* drivers/iommu/s390-iommu.c and drivers/pci/hotplug/s390_pci_hpc.c - and some doc adjustments, too).
>
> * It largely affects zPCI, the s390x specific PCI code layer.
>
> * PCI cards available for s390x are optional cards (RoCE and zEDC) and not very wide-spread.
>
> * The situation described above affects the RoCE adapters only (Mellanox based).
>
> * The patches are also upstream accepted and available via linux-next, but to apply them to focal kernel 5.4 the above backports are needed.
>
> * However, the code is modified by several patches (12), hence there is a chance to break zPCI with them.
>
> * For upfront testing a PPA got created with a focal (master-next) kernel that incl. all the above patches.
Limited to a single architecture, positive test results.
Acked-by: Kleber Sacilotto de Souza <kleber.souza at canonical.com>
>
> Alexander Schmidt (1):
> s390/pci: Expose new port attribute for PCIe functions
>
> Niklas Schnelle (1):
> s390/pci: Improve handling of unset UID
>
> Pierre Morel (10):
> s390/pci: embedding hotplug_slot in zdev
> s390/pci: adaptation of iommu to multifunction
> s390/pci: define kernel parameters for PCI multifunction
> s390/pci: define RID and RID available
> s390/pci: create zPCI bus
> s390/pci: adapt events for zbus
> s390/pci: Handling multifunctions
> s390/pci: Do not disable PF when VFs exist
> s390/pci: Documentation for zPCI
> s390/pci: removes wrong PCI multifunction assignment
>
> .../admin-guide/kernel-parameters.txt | 2 +
> Documentation/s390/index.rst | 1 +
> Documentation/s390/pci.rst | 126 +++++++++
> MAINTAINERS | 1 +
> arch/s390/include/asm/pci.h | 45 +++-
> arch/s390/include/asm/pci_clp.h | 12 +-
> arch/s390/pci/Makefile | 3 +-
> arch/s390/pci/pci.c | 198 ++++++--------
> arch/s390/pci/pci_bus.c | 255 ++++++++++++++++++
> arch/s390/pci/pci_bus.h | 31 +++
> arch/s390/pci/pci_clp.c | 6 +-
> arch/s390/pci/pci_event.c | 39 ++-
> arch/s390/pci/pci_sysfs.c | 4 +-
> drivers/iommu/s390-iommu.c | 8 +-
> drivers/pci/hotplug/s390_pci_hpc.c | 105 +++-----
> 15 files changed, 618 insertions(+), 218 deletions(-)
> create mode 100644 Documentation/s390/pci.rst
> create mode 100644 arch/s390/pci/pci_bus.c
> create mode 100644 arch/s390/pci/pci_bus.h
>
More information about the kernel-team
mailing list