APPLIED: [J][PULL] Enhanced Interpretation for PCI Functions on s390x

Roxana Nicolescu roxana.nicolescu at canonical.com
Fri Jul 7 13:03:27 UTC 2023


On 14/06/2023 17:50, frank.heimes at canonical.com wrote:
> BugLink: https://bugs.launchpad.net/bugs/1853306
>
> [ Impact ]
>
>   * Currently the PCI passthrough implementation for s390x is based on
>     intercepting PCI I/O instructions, which leads to a reduced I/O performance
>     compared to the execution of PCI instructions directly in LPAR.
>
>   * Hence users may face I/O bottlenecks when using PCI devices in passthrough
>     mode based on the current implementation.
>
>   * For avoiding this and to improve performance, the interpretive execution
>     of the PCI store and PCI load instructions get enabled.
>
>   * A further improvement is achieved by enabling the Adapter-Event-Notification
>     Interpretation (AENI).
>
>   * Since LTS releases are the main focus for stable and long running KVM
>     workloads, it is highly desired to get this backported to the jammy kernel
>     (and because the next LTS is still some time away).
>
> [ Test Plan ]
>
>   * Have an Ubuntu Server 22.04 installation on LPAR,
>     that is able to access (ideally multiple) PCI devices,
>     like RoCE Express (network) or NVMe disks.
>
>   * Setup KVM and pass through (ideally multiple) of these PCI devices
>     (that are otherwise unused on the KVM host).
>
>   * Generate IO load on these passed through PCI devices,
>     for example with stress-ng, using class network and/or device
>     and/or io stressors.
>
>   * This PR also introduces a new kernel config option 'VFIO_PCI_ZDEV_KVM'
>     that allows to enable support for the s390x-specific extensions and
>     enhancements to KVM passthrough, such as interpretive execution of zPCI
>     instructions and is with this PR and got enabled.
>
>   * The qemu autopkgtest (also needed due to LP#1853307) will be a got fit
>     to identify any potential regressions, also in the kvm kernel area.
>
>   * zPCI passthrough related test will be done by IBM.
>
> [ Where problems could occur ]
>
>   * The modifications do not change the way users or APIs have to make
>     use of PCI passthrough, only the internal implementation got modified.
>
>   * The vast majority of the code changes/or additional code is s390x-specific,
>     under arch/s390 and drivers/s390.
>
>   * However there is also common code touched:
>
>   * 'kvm: use kvfree() in kvm_arch_free_vm()' touches
>     arch/arm64/include/asm/kvm_host.h, arch/arm64/kvm/arm.c,
>     arch/x86/include/asm/kvm_host.h, arch/x86/kvm/x86.c,
>     include/linux/kvm_host.h switches in kvm_arch_free_vm() from kfree() to
>     kvfree() allowing to use the common variant, which is upstream since v5.16
>     and with that well established.
>
>   * And 'vfio-pci/zdev: add open/close device hooks' touches
>     drivers/vfio/pci/vfio_pci_core.c and drivers/vfio/pci/vfio_pci_zdev.c
>     include/linux/vfio_pci_core.h add now code to introduce device hooks.
>     It's upstream since kernel 6.0.
>
>   * 'KVM: s390: pci: provide routines for en-/disabling interrupt forwarding'
>     expands a single #if statement in include/linux/sched/user.h.
>
>   * 'KVM: s390: add KVM_S390_ZPCI_OP to manage guest zPCI devices'
>     adds s390x specific KVM_S390_ZPCI_OP and it's definition to
>     include/uapi/linux/kvm.h.
>
>   * And 'vfio-pci/zdev: different maxstbl for interpreted devices' and
>     'vfio-pci/zdev: add function handle to clp base capability' expand
>     s390x-specific (aka z-specific aka zdev) device structs in
>     include/uapi/linux/vfio_zdev.h.
>
>   * This shows that the vast majority of modifications are s390x specific,
>     even in most of the common code files.
>
>   * The remaining modifications in the (generally) common code files are
>     related to the newly introduced kernel option 'CONFIG_VFIO_PCI_ZDEV_KVM'
>     and documentation.
>
>   * The s390x changes are more significant, and could not only harm
>     passthrough itself for zPCI devices, but also KVM virtualization in general.
>
>   * In addition to these kernel changes, qemu modifications are needed
>     as well (that are addressed at LP#1853307), this modified kernel
>     must be tested in combination with the updated qemu package.
>     - The qemu autopkgtest will be a got fit to identify any regressions,
>     also in the kernel.
>     - In addition some passthrough related test will be done by IBM
>
> ---
>
> The following changes since commit 6e98058cb755053a370055f90721014a9d48c413:
>
>    UBUNTU: Upstream stable to v5.15.108 (2023-06-08 13:08:35 -0700)
>
> are available in the Git repository at:
>
>    https://git.launchpad.net/~fheimes/+git/lp1853306/ 13bfa01561d9014b1a8be5b0265eb7fd896689e4
>
> for you to fetch changes up to 13bfa01561d9014b1a8be5b0265eb7fd896689e4:
>
>    UBUNTU: [Config] enable VFIO zPCI pass-through for s390x (2023-06-13 12:02:23 +0200)
>
> ----------------------------------------------------------------
> Alexander Egorenkov (1):
>        s390/sclp: add detection of IPL-complete-control facility
>
> Bagas Sanjaya (1):
>        Documentation: kvm: extend KVM_S390_ZPCI_OP subheading underline
>
> Frank Heimes (1):
>        UBUNTU: [Config] enable VFIO zPCI pass-through for s390x
>
> Juergen Gross (1):
>        kvm: use kvfree() in kvm_arch_free_vm()
>
> Matthew Rosato (24):
>        s390/sclp: detect the zPCI load/store interpretation facility
>        s390/sclp: detect the AISII facility
>        s390/sclp: detect the AENI facility
>        s390/sclp: detect the AISI facility
>        s390/airq: pass more TPI info to airq handlers
>        s390/airq: allow for airq structure that uses an input vector
>        s390/pci: externalize the SIC operation controls and routine
>        s390/pci: stash associated GISA designation
>        s390/pci: stash dtsm and maxstbl
>        vfio/pci: introduce CONFIG_VFIO_PCI_ZDEV_KVM
>        KVM: s390: pci: add basic kvm_zdev structure
>        KVM: s390: pci: do initial setup for AEN interpretation
>        KVM: s390: pci: enable host forwarding of Adapter Event Notifications
>        KVM: s390: mechanism to enable guest zPCI Interpretation
>        KVM: s390: pci: provide routines for enabling/disabling interrupt forwarding
>        KVM: s390: pci: add routines to start/stop interpretive execution
>        vfio-pci/zdev: add open/close device hooks
>        vfio-pci/zdev: add function handle to clp base capability
>        vfio-pci/zdev: different maxstbl for interpreted devices
>        KVM: s390: add KVM_S390_ZPCI_OP to manage guest zPCI devices
>        MAINTAINERS: additional files related kvm s390 pci passthrough
>        KVM: s390: pci: fix plain integer as NULL pointer warnings
>        KVM: s390: pci: fix GAIT physical vs virtual pointers usage
>        KVM: s390: pci: register pci hooks without interpretation
>
> Niklas Schnelle (1):
>        s390/pci: use phys_to_virt() for AIBVs/DIBVs
>
> Pierre Morel (1):
>        KVM: s390: pci: Hook to access KVM lowlevel from VFIO
>
>   Documentation/virt/kvm/api.rst    |  47 +++
>   MAINTAINERS                       |   1 +
>   arch/arm64/include/asm/kvm_host.h |   1 -
>   arch/arm64/kvm/arm.c              |   8 -
>   arch/s390/include/asm/airq.h      |   7 +-
>   arch/s390/include/asm/kvm_host.h  |  18 +
>   arch/s390/include/asm/pci.h       |  12 +
>   arch/s390/include/asm/pci_clp.h   |   9 +-
>   arch/s390/include/asm/pci_insn.h  |  29 +-
>   arch/s390/include/asm/sclp.h      |   5 +
>   arch/s390/include/asm/tpi.h       |  13 +
>   arch/s390/kvm/Makefile            |   1 +
>   arch/s390/kvm/interrupt.c         |  96 +++++-
>   arch/s390/kvm/kvm-s390.c          |  83 ++++-
>   arch/s390/kvm/kvm-s390.h          |  10 +
>   arch/s390/kvm/pci.c               | 702 ++++++++++++++++++++++++++++++++++++++
>   arch/s390/kvm/pci.h               |  88 +++++
>   arch/s390/pci/Makefile            |   2 +-
>   arch/s390/pci/pci.c               |  16 +
>   arch/s390/pci/pci_clp.c           |   7 +
>   arch/s390/pci/pci_insn.c          |   4 +-
>   arch/s390/pci/pci_irq.c           |  54 +--
>   arch/s390/pci/pci_kvm_hook.c      |  11 +
>   arch/x86/include/asm/kvm_host.h   |   2 +
>   arch/x86/kvm/x86.c                |   2 +-
>   debian.master/config/annotations  |   3 +
>   drivers/s390/char/sclp_early.c    |   8 +-
>   drivers/s390/cio/airq.c           |  12 +-
>   drivers/s390/cio/qdio_thinint.c   |   6 +-
>   drivers/s390/crypto/ap_bus.c      |   9 +-
>   drivers/s390/virtio/virtio_ccw.c  |   6 +-
>   drivers/vfio/pci/Kconfig          |  12 +
>   drivers/vfio/pci/Makefile         |   2 +-
>   drivers/vfio/pci/vfio_pci_core.c  |  11 +
>   drivers/vfio/pci/vfio_pci_zdev.c  |  72 +++-
>   include/linux/kvm_host.h          |   9 +-
>   include/linux/sched/user.h        |   3 +-
>   include/linux/vfio_pci_core.h     |  12 +-
>   include/uapi/linux/kvm.h          |  31 ++
>   include/uapi/linux/vfio_zdev.h    |   7 +
>   40 files changed, 1360 insertions(+), 71 deletions(-)
>   create mode 100644 arch/s390/kvm/pci.c
>   create mode 100644 arch/s390/kvm/pci.h
>   create mode 100644 arch/s390/pci/pci_kvm_hook.c
>
Applied to jammy:master-next. I added your SOB in the last commit and 
the buglink. Thanks.

Roxana



More information about the kernel-team mailing list