ACK/Cmnt: [J][PULL] Enhanced Interpretation for PCI Functions on s390x
Tim Gardner
tim.gardner at canonical.com
Wed Jun 14 17:16:40 UTC 2023
On 6/14/23 08:50, frank.heimes at canonical.com wrote:
> BugLink: https://bugs.launchpad.net/bugs/1853306
>
> [ Impact ]
>
> * Currently the PCI passthrough implementation for s390x is based on
> intercepting PCI I/O instructions, which leads to a reduced I/O performance
> compared to the execution of PCI instructions directly in LPAR.
>
> * Hence users may face I/O bottlenecks when using PCI devices in passthrough
> mode based on the current implementation.
>
> * For avoiding this and to improve performance, the interpretive execution
> of the PCI store and PCI load instructions get enabled.
>
> * A further improvement is achieved by enabling the Adapter-Event-Notification
> Interpretation (AENI).
>
> * Since LTS releases are the main focus for stable and long running KVM
> workloads, it is highly desired to get this backported to the jammy kernel
> (and because the next LTS is still some time away).
>
> [ Test Plan ]
>
> * Have an Ubuntu Server 22.04 installation on LPAR,
> that is able to access (ideally multiple) PCI devices,
> like RoCE Express (network) or NVMe disks.
>
> * Setup KVM and pass through (ideally multiple) of these PCI devices
> (that are otherwise unused on the KVM host).
>
> * Generate IO load on these passed through PCI devices,
> for example with stress-ng, using class network and/or device
> and/or io stressors.
>
> * This PR also introduces a new kernel config option 'VFIO_PCI_ZDEV_KVM'
> that allows to enable support for the s390x-specific extensions and
> enhancements to KVM passthrough, such as interpretive execution of zPCI
> instructions and is with this PR and got enabled.
>
> * The qemu autopkgtest (also needed due to LP#1853307) will be a got fit
> to identify any potential regressions, also in the kvm kernel area.
>
> * zPCI passthrough related test will be done by IBM.
>
> [ Where problems could occur ]
>
> * The modifications do not change the way users or APIs have to make
> use of PCI passthrough, only the internal implementation got modified.
>
> * The vast majority of the code changes/or additional code is s390x-specific,
> under arch/s390 and drivers/s390.
>
> * However there is also common code touched:
>
> * 'kvm: use kvfree() in kvm_arch_free_vm()' touches
> arch/arm64/include/asm/kvm_host.h, arch/arm64/kvm/arm.c,
> arch/x86/include/asm/kvm_host.h, arch/x86/kvm/x86.c,
> include/linux/kvm_host.h switches in kvm_arch_free_vm() from kfree() to
> kvfree() allowing to use the common variant, which is upstream since v5.16
> and with that well established.
>
> * And 'vfio-pci/zdev: add open/close device hooks' touches
> drivers/vfio/pci/vfio_pci_core.c and drivers/vfio/pci/vfio_pci_zdev.c
> include/linux/vfio_pci_core.h add now code to introduce device hooks.
> It's upstream since kernel 6.0.
>
> * 'KVM: s390: pci: provide routines for en-/disabling interrupt forwarding'
> expands a single #if statement in include/linux/sched/user.h.
>
> * 'KVM: s390: add KVM_S390_ZPCI_OP to manage guest zPCI devices'
> adds s390x specific KVM_S390_ZPCI_OP and it's definition to
> include/uapi/linux/kvm.h.
>
> * And 'vfio-pci/zdev: different maxstbl for interpreted devices' and
> 'vfio-pci/zdev: add function handle to clp base capability' expand
> s390x-specific (aka z-specific aka zdev) device structs in
> include/uapi/linux/vfio_zdev.h.
>
> * This shows that the vast majority of modifications are s390x specific,
> even in most of the common code files.
>
> * The remaining modifications in the (generally) common code files are
> related to the newly introduced kernel option 'CONFIG_VFIO_PCI_ZDEV_KVM'
> and documentation.
>
> * The s390x changes are more significant, and could not only harm
> passthrough itself for zPCI devices, but also KVM virtualization in general.
>
> * In addition to these kernel changes, qemu modifications are needed
> as well (that are addressed at LP#1853307), this modified kernel
> must be tested in combination with the updated qemu package.
> - The qemu autopkgtest will be a got fit to identify any regressions,
> also in the kernel.
> - In addition some passthrough related test will be done by IBM
>
> ---
>
> The following changes since commit 6e98058cb755053a370055f90721014a9d48c413:
>
> UBUNTU: Upstream stable to v5.15.108 (2023-06-08 13:08:35 -0700)
>
> are available in the Git repository at:
>
> https://git.launchpad.net/~fheimes/+git/lp1853306/ 13bfa01561d9014b1a8be5b0265eb7fd896689e4
>
> for you to fetch changes up to 13bfa01561d9014b1a8be5b0265eb7fd896689e4:
>
> UBUNTU: [Config] enable VFIO zPCI pass-through for s390x (2023-06-13 12:02:23 +0200)
>
> ----------------------------------------------------------------
> Alexander Egorenkov (1):
> s390/sclp: add detection of IPL-complete-control facility
>
> Bagas Sanjaya (1):
> Documentation: kvm: extend KVM_S390_ZPCI_OP subheading underline
>
> Frank Heimes (1):
> UBUNTU: [Config] enable VFIO zPCI pass-through for s390x
>
> Juergen Gross (1):
> kvm: use kvfree() in kvm_arch_free_vm()
>
> Matthew Rosato (24):
> s390/sclp: detect the zPCI load/store interpretation facility
> s390/sclp: detect the AISII facility
> s390/sclp: detect the AENI facility
> s390/sclp: detect the AISI facility
> s390/airq: pass more TPI info to airq handlers
> s390/airq: allow for airq structure that uses an input vector
> s390/pci: externalize the SIC operation controls and routine
> s390/pci: stash associated GISA designation
> s390/pci: stash dtsm and maxstbl
> vfio/pci: introduce CONFIG_VFIO_PCI_ZDEV_KVM
> KVM: s390: pci: add basic kvm_zdev structure
> KVM: s390: pci: do initial setup for AEN interpretation
> KVM: s390: pci: enable host forwarding of Adapter Event Notifications
> KVM: s390: mechanism to enable guest zPCI Interpretation
> KVM: s390: pci: provide routines for enabling/disabling interrupt forwarding
> KVM: s390: pci: add routines to start/stop interpretive execution
> vfio-pci/zdev: add open/close device hooks
> vfio-pci/zdev: add function handle to clp base capability
> vfio-pci/zdev: different maxstbl for interpreted devices
> KVM: s390: add KVM_S390_ZPCI_OP to manage guest zPCI devices
> MAINTAINERS: additional files related kvm s390 pci passthrough
> KVM: s390: pci: fix plain integer as NULL pointer warnings
> KVM: s390: pci: fix GAIT physical vs virtual pointers usage
> KVM: s390: pci: register pci hooks without interpretation
>
> Niklas Schnelle (1):
> s390/pci: use phys_to_virt() for AIBVs/DIBVs
>
> Pierre Morel (1):
> KVM: s390: pci: Hook to access KVM lowlevel from VFIO
>
> Documentation/virt/kvm/api.rst | 47 +++
> MAINTAINERS | 1 +
> arch/arm64/include/asm/kvm_host.h | 1 -
> arch/arm64/kvm/arm.c | 8 -
> arch/s390/include/asm/airq.h | 7 +-
> arch/s390/include/asm/kvm_host.h | 18 +
> arch/s390/include/asm/pci.h | 12 +
> arch/s390/include/asm/pci_clp.h | 9 +-
> arch/s390/include/asm/pci_insn.h | 29 +-
> arch/s390/include/asm/sclp.h | 5 +
> arch/s390/include/asm/tpi.h | 13 +
> arch/s390/kvm/Makefile | 1 +
> arch/s390/kvm/interrupt.c | 96 +++++-
> arch/s390/kvm/kvm-s390.c | 83 ++++-
> arch/s390/kvm/kvm-s390.h | 10 +
> arch/s390/kvm/pci.c | 702 ++++++++++++++++++++++++++++++++++++++
> arch/s390/kvm/pci.h | 88 +++++
> arch/s390/pci/Makefile | 2 +-
> arch/s390/pci/pci.c | 16 +
> arch/s390/pci/pci_clp.c | 7 +
> arch/s390/pci/pci_insn.c | 4 +-
> arch/s390/pci/pci_irq.c | 54 +--
> arch/s390/pci/pci_kvm_hook.c | 11 +
> arch/x86/include/asm/kvm_host.h | 2 +
> arch/x86/kvm/x86.c | 2 +-
> debian.master/config/annotations | 3 +
> drivers/s390/char/sclp_early.c | 8 +-
> drivers/s390/cio/airq.c | 12 +-
> drivers/s390/cio/qdio_thinint.c | 6 +-
> drivers/s390/crypto/ap_bus.c | 9 +-
> drivers/s390/virtio/virtio_ccw.c | 6 +-
> drivers/vfio/pci/Kconfig | 12 +
> drivers/vfio/pci/Makefile | 2 +-
> drivers/vfio/pci/vfio_pci_core.c | 11 +
> drivers/vfio/pci/vfio_pci_zdev.c | 72 +++-
> include/linux/kvm_host.h | 9 +-
> include/linux/sched/user.h | 3 +-
> include/linux/vfio_pci_core.h | 12 +-
> include/uapi/linux/kvm.h | 31 ++
> include/uapi/linux/vfio_zdev.h | 7 +
> 40 files changed, 1360 insertions(+), 71 deletions(-)
> create mode 100644 arch/s390/kvm/pci.c
> create mode 100644 arch/s390/kvm/pci.h
> create mode 100644 arch/s390/pci/pci_kvm_hook.c
>
Acked-by: Tim Gardner <tim.gardner at canonical.com>
This is a lot of patches to review, though all but 4 code patches are
clean cherry picks. I am assuming this will be thoroughly tested by IBM.
The backport explanations for "vfio/pci: introduce
CONFIG_VFIO_PCI_ZDEV_KVM" and "KVM: s390: pci: add routines to
start/stop interpretive execution" are a bit thin.
--
-----------
Tim Gardner
Canonical, Inc
More information about the kernel-team
mailing list