ACK: [SRU][J:linux-bluefield][PATCH v7 0/6] Add VFIO P2P support

[Agathe Porte]

2024-09-25 19:23 CEST, William Tu:
> BugLink: https://bugs.launchpad.net/bugs/2077887
> 
> The series adds support for VFIO P2P feature for NVMe target offload.
> We took the patch (not upstreamed) below with some fix to make it work.
> https://patchwork.kernel.org/project/linux-rdma/cover/0-v2-472615b3877e+28f7-vfio_dma_buf_jgg@nvidia.com/
> 
> * Kernel Config
> Need to enable
> CONFIG_PCI_P2PDMA (enable p2p dma transaction)
> CONFIG_DMABUF_MOVE_NOTIFY (no need to pin down the memory)
> 
> The feature doesn't work with IOMMU, so either we disable it
> for the system, "iommu.passthrough=1 iommu=off" in kernel boot
> parameter, or conditionally disable it for VFIO.
> CONFIG_VFIO_NOIOMMU (optional)
> - You should enable CONFIG_VFIO_NOIOMMU in the kernel config if your
> vfio module does not have vfio enable_unsafe_noiommu_mode in the parameter list.
> 
> * /etc/modprobe.d/vfio.conf
> $ cat /etc/modprobe.d/vfio.conf
> # The kernel hungs when 0000:06:00.0 (Non-Volatile memory controller: KIOXIA
> # Corporation Device 0001) is bound to the vfio-pci module, and the following
> # error is printed:
> #
> # vfio-pci 0000:06:00.0: Unable to change power state from D3hot to D0, device inaccessible
> #
> # The configuration below is a workarround for the issue.
> options vfio-pci disable_idle_d3=1
> 
> # Allow using VFIO without IOMMU. This is only required when SMMU is disabled.
> options vfio enable_unsafe_noiommu_mode=1
> 
> * Setup and Test
> 1. For ease of testing, we take a BF3 and boot the OS into mmc instead
> of NVMe drive. So leaving NVMe drive free to use.
> 
> 2. use a test program to verify its correctness.
> The program create ibv rdma program, open nvme drive, create dma vfio
> buf from nvme, and let the rdma program read (DMA P2P) the version
> number of the nvme drive.
> https://gitlab.com/Mellanox/spdk_team/vfio-dmabuf-test
> 
> * UAPI changes
> see "vfio/pci: Allow MMIO regions to be exported through
>     dma-buf" for detail
> 
> v7:
> - add back the missing vfio_pci_dma_buf_move code, suggested by Bartlomiej
> 
> v6: 
> - remove comments, fix the vfio_pci_set_power_state code
> 
> v5:
> - add back the missing power management code, suggested by Bartlomiej
> 
> v4:
> - fix Makefile
> 
> v1->v2:
> - introduce new ioctl uAPI for vfio dma-buf
> 
> v2->v3:
> - squash the ioctl uapi change into
>   "vfio/pci: Allow MMIO regions to be exported through dma-buf"
> 
> Jason Gunthorpe (4):
>   UBUNTU: SAUCE: dma-buf: Add dma_buf_try_get()
>   UBUNTU: SAUCE: vfio: Add vfio_device_get()
>   UBUNTU: SAUCE: vfio_pci: Do not open code pci_try_reset_function()
>   UBUNTU: SAUCE: vfio/pci: Allow MMIO regions to be exported through
>     dma-buf
> 
> Sergey Gorenko (1):
>   UBUNTU: SAUCE: vfio/pci: Fix p2p address
> 
> William Tu (1):
>   UBUNTU: [Config] bluefield: add config for VFIO P2P
> 
>  debian.bluefield/config/annotations |   2 +
>  drivers/vfio/pci/Makefile           |   1 +
>  drivers/vfio/pci/dma_buf.c          | 265 ++++++++++++++++++++++++++++
>  drivers/vfio/pci/vfio_pci_config.c  |  24 +--
>  drivers/vfio/pci/vfio_pci_core.c    |  76 +++++---
>  drivers/vfio/pci/vfio_pci_priv.h    |  28 +++
>  include/linux/dma-buf.h             |  13 ++
>  include/linux/vfio.h                |   5 +
>  include/linux/vfio_pci_core.h       |   1 +
>  include/uapi/linux/vfio.h           |  17 ++
>  10 files changed, 395 insertions(+), 37 deletions(-)
>  create mode 100644 drivers/vfio/pci/dma_buf.c
>  create mode 100644 drivers/vfio/pci/vfio_pci_priv.h

Acked-by: Agathe Porte <agathe.porte at canonical.com>