[J][PULL] Enhanced Interpretation for PCI Functions on s390x
frank.heimes at canonical.com
frank.heimes at canonical.com
Wed Jun 14 15:50:21 UTC 2023
BugLink: https://bugs.launchpad.net/bugs/1853306
[ Impact ]
* Currently the PCI passthrough implementation for s390x is based on
intercepting PCI I/O instructions, which leads to a reduced I/O performance
compared to the execution of PCI instructions directly in LPAR.
* Hence users may face I/O bottlenecks when using PCI devices in passthrough
mode based on the current implementation.
* For avoiding this and to improve performance, the interpretive execution
of the PCI store and PCI load instructions get enabled.
* A further improvement is achieved by enabling the Adapter-Event-Notification
Interpretation (AENI).
* Since LTS releases are the main focus for stable and long running KVM
workloads, it is highly desired to get this backported to the jammy kernel
(and because the next LTS is still some time away).
[ Test Plan ]
* Have an Ubuntu Server 22.04 installation on LPAR,
that is able to access (ideally multiple) PCI devices,
like RoCE Express (network) or NVMe disks.
* Setup KVM and pass through (ideally multiple) of these PCI devices
(that are otherwise unused on the KVM host).
* Generate IO load on these passed through PCI devices,
for example with stress-ng, using class network and/or device
and/or io stressors.
* This PR also introduces a new kernel config option 'VFIO_PCI_ZDEV_KVM'
that allows to enable support for the s390x-specific extensions and
enhancements to KVM passthrough, such as interpretive execution of zPCI
instructions and is with this PR and got enabled.
* The qemu autopkgtest (also needed due to LP#1853307) will be a got fit
to identify any potential regressions, also in the kvm kernel area.
* zPCI passthrough related test will be done by IBM.
[ Where problems could occur ]
* The modifications do not change the way users or APIs have to make
use of PCI passthrough, only the internal implementation got modified.
* The vast majority of the code changes/or additional code is s390x-specific,
under arch/s390 and drivers/s390.
* However there is also common code touched:
* 'kvm: use kvfree() in kvm_arch_free_vm()' touches
arch/arm64/include/asm/kvm_host.h, arch/arm64/kvm/arm.c,
arch/x86/include/asm/kvm_host.h, arch/x86/kvm/x86.c,
include/linux/kvm_host.h switches in kvm_arch_free_vm() from kfree() to
kvfree() allowing to use the common variant, which is upstream since v5.16
and with that well established.
* And 'vfio-pci/zdev: add open/close device hooks' touches
drivers/vfio/pci/vfio_pci_core.c and drivers/vfio/pci/vfio_pci_zdev.c
include/linux/vfio_pci_core.h add now code to introduce device hooks.
It's upstream since kernel 6.0.
* 'KVM: s390: pci: provide routines for en-/disabling interrupt forwarding'
expands a single #if statement in include/linux/sched/user.h.
* 'KVM: s390: add KVM_S390_ZPCI_OP to manage guest zPCI devices'
adds s390x specific KVM_S390_ZPCI_OP and it's definition to
include/uapi/linux/kvm.h.
* And 'vfio-pci/zdev: different maxstbl for interpreted devices' and
'vfio-pci/zdev: add function handle to clp base capability' expand
s390x-specific (aka z-specific aka zdev) device structs in
include/uapi/linux/vfio_zdev.h.
* This shows that the vast majority of modifications are s390x specific,
even in most of the common code files.
* The remaining modifications in the (generally) common code files are
related to the newly introduced kernel option 'CONFIG_VFIO_PCI_ZDEV_KVM'
and documentation.
* The s390x changes are more significant, and could not only harm
passthrough itself for zPCI devices, but also KVM virtualization in general.
* In addition to these kernel changes, qemu modifications are needed
as well (that are addressed at LP#1853307), this modified kernel
must be tested in combination with the updated qemu package.
- The qemu autopkgtest will be a got fit to identify any regressions,
also in the kernel.
- In addition some passthrough related test will be done by IBM
---
The following changes since commit 6e98058cb755053a370055f90721014a9d48c413:
UBUNTU: Upstream stable to v5.15.108 (2023-06-08 13:08:35 -0700)
are available in the Git repository at:
https://git.launchpad.net/~fheimes/+git/lp1853306/ 13bfa01561d9014b1a8be5b0265eb7fd896689e4
for you to fetch changes up to 13bfa01561d9014b1a8be5b0265eb7fd896689e4:
UBUNTU: [Config] enable VFIO zPCI pass-through for s390x (2023-06-13 12:02:23 +0200)
----------------------------------------------------------------
Alexander Egorenkov (1):
s390/sclp: add detection of IPL-complete-control facility
Bagas Sanjaya (1):
Documentation: kvm: extend KVM_S390_ZPCI_OP subheading underline
Frank Heimes (1):
UBUNTU: [Config] enable VFIO zPCI pass-through for s390x
Juergen Gross (1):
kvm: use kvfree() in kvm_arch_free_vm()
Matthew Rosato (24):
s390/sclp: detect the zPCI load/store interpretation facility
s390/sclp: detect the AISII facility
s390/sclp: detect the AENI facility
s390/sclp: detect the AISI facility
s390/airq: pass more TPI info to airq handlers
s390/airq: allow for airq structure that uses an input vector
s390/pci: externalize the SIC operation controls and routine
s390/pci: stash associated GISA designation
s390/pci: stash dtsm and maxstbl
vfio/pci: introduce CONFIG_VFIO_PCI_ZDEV_KVM
KVM: s390: pci: add basic kvm_zdev structure
KVM: s390: pci: do initial setup for AEN interpretation
KVM: s390: pci: enable host forwarding of Adapter Event Notifications
KVM: s390: mechanism to enable guest zPCI Interpretation
KVM: s390: pci: provide routines for enabling/disabling interrupt forwarding
KVM: s390: pci: add routines to start/stop interpretive execution
vfio-pci/zdev: add open/close device hooks
vfio-pci/zdev: add function handle to clp base capability
vfio-pci/zdev: different maxstbl for interpreted devices
KVM: s390: add KVM_S390_ZPCI_OP to manage guest zPCI devices
MAINTAINERS: additional files related kvm s390 pci passthrough
KVM: s390: pci: fix plain integer as NULL pointer warnings
KVM: s390: pci: fix GAIT physical vs virtual pointers usage
KVM: s390: pci: register pci hooks without interpretation
Niklas Schnelle (1):
s390/pci: use phys_to_virt() for AIBVs/DIBVs
Pierre Morel (1):
KVM: s390: pci: Hook to access KVM lowlevel from VFIO
Documentation/virt/kvm/api.rst | 47 +++
MAINTAINERS | 1 +
arch/arm64/include/asm/kvm_host.h | 1 -
arch/arm64/kvm/arm.c | 8 -
arch/s390/include/asm/airq.h | 7 +-
arch/s390/include/asm/kvm_host.h | 18 +
arch/s390/include/asm/pci.h | 12 +
arch/s390/include/asm/pci_clp.h | 9 +-
arch/s390/include/asm/pci_insn.h | 29 +-
arch/s390/include/asm/sclp.h | 5 +
arch/s390/include/asm/tpi.h | 13 +
arch/s390/kvm/Makefile | 1 +
arch/s390/kvm/interrupt.c | 96 +++++-
arch/s390/kvm/kvm-s390.c | 83 ++++-
arch/s390/kvm/kvm-s390.h | 10 +
arch/s390/kvm/pci.c | 702 ++++++++++++++++++++++++++++++++++++++
arch/s390/kvm/pci.h | 88 +++++
arch/s390/pci/Makefile | 2 +-
arch/s390/pci/pci.c | 16 +
arch/s390/pci/pci_clp.c | 7 +
arch/s390/pci/pci_insn.c | 4 +-
arch/s390/pci/pci_irq.c | 54 +--
arch/s390/pci/pci_kvm_hook.c | 11 +
arch/x86/include/asm/kvm_host.h | 2 +
arch/x86/kvm/x86.c | 2 +-
debian.master/config/annotations | 3 +
drivers/s390/char/sclp_early.c | 8 +-
drivers/s390/cio/airq.c | 12 +-
drivers/s390/cio/qdio_thinint.c | 6 +-
drivers/s390/crypto/ap_bus.c | 9 +-
drivers/s390/virtio/virtio_ccw.c | 6 +-
drivers/vfio/pci/Kconfig | 12 +
drivers/vfio/pci/Makefile | 2 +-
drivers/vfio/pci/vfio_pci_core.c | 11 +
drivers/vfio/pci/vfio_pci_zdev.c | 72 +++-
include/linux/kvm_host.h | 9 +-
include/linux/sched/user.h | 3 +-
include/linux/vfio_pci_core.h | 12 +-
include/uapi/linux/kvm.h | 31 ++
include/uapi/linux/vfio_zdev.h | 7 +
40 files changed, 1360 insertions(+), 71 deletions(-)
create mode 100644 arch/s390/kvm/pci.c
create mode 100644 arch/s390/kvm/pci.h
create mode 100644 arch/s390/pci/pci_kvm_hook.c
More information about the kernel-team
mailing list