NACK/Cmt: [SRU][J:linux-bluefield][PATCH v2 0/7] Add VFIO P2P support

[Thibault Ferrante]

v3 sent to the mailing list

On 06-09-2024 02:40, William Tu wrote:
> BugLink: https://bugs.launchpad.net/bugs/2077887
> 
> The series adds support for VFIO P2P feature for NVMe target offload.
> We took the patch (not upstreamed) below with some fix to make it work.
> https://patchwork.kernel.org/project/linux-rdma/cover/0-v2-472615b3877e+28f7-vfio_dma_buf_jgg@nvidia.com/
> 
> * Kernel Config
> Need to enable
> CONFIG_PCI_P2PDMA (enable p2p dma transaction)
> CONFIG_DMABUF_MOVE_NOTIFY (no need to pin down the memory)
> 
> The feature doesn't work with IOMMU, so either we disable it
> for the system, "iommu.passthrough=1 iommu=off" in kernel boot
> parameter, or conditionally disable it for VFIO.
> CONFIG_VFIO_NOIOMMU (optional)
> - You should enable CONFIG_VFIO_NOIOMMU in the kernel config if your
> vfio module does not have vfio enable_unsafe_noiommu_mode in the parameter list.
> 
> * /etc/modprobe.d/vfio.conf
> $ cat /etc/modprobe.d/vfio.conf
> # The kernel hungs when 0000:06:00.0 (Non-Volatile memory controller: KIOXIA
> # Corporation Device 0001) is bound to the vfio-pci module, and the following
> # error is printed:
> #
> # vfio-pci 0000:06:00.0: Unable to change power state from D3hot to D0, device inaccessible
> #
> # The configuration below is a workarround for the issue.
> options vfio-pci disable_idle_d3=1
> 
> # Allow using VFIO without IOMMU. This is only required when SMMU is disabled.
> options vfio enable_unsafe_noiommu_mode=1
> 
> * Setup and Test
> 1. For ease of testing, we take a BF3 and boot the OS into mmc instead
> of NVMe drive. So leaving NVMe drive free to use.
> 
> 2. use a test program to verify its correctness.
> The program create ibv rdma program, open nvme drive, create dma vfio
> buf from nvme, and let the rdma program read (DMA P2P) the version
> number of the nvme drive.
> https://gitlab.com/Mellanox/spdk_team/vfio-dmabuf-test
> 
> * UAPI changes
> see "UBUNTU: SAUCE: vfio/pci: add new ioctl for vfio dma buf"
> 
> Jason Gunthorpe (4):
>    UBUNTU: SAUCE: dma-buf: Add dma_buf_try_get()
>    UBUNTU: SAUCE: vfio: Add vfio_device_get()
>    UBUNTU: SAUCE: vfio_pci: Do not open code pci_try_reset_function()
>    UBUNTU: SAUCE: vfio/pci: Allow MMIO regions to be exported through
>      dma-buf
> 
> Sergey Gorenko (1):
>    UBUNTU: SAUCE: vfio/pci: Fix p2p address
> 
> William Tu (2):
>    UBUNTU: [Config] bluefield: add config for VFIO P2P
>    UBUNTU: SAUCE: vfio/pci: add new ioctl for vfio dma buf
> 
>   debian.bluefield/config/annotations |   2 +
>   drivers/vfio/pci/Makefile           |   3 +-
>   drivers/vfio/pci/dma_buf.c          | 265 ++++++++++++++++++++++++++++
>   drivers/vfio/pci/vfio_pci_config.c  |  24 +--
>   drivers/vfio/pci/vfio_pci_core.c    |  64 ++++---
>   drivers/vfio/pci/vfio_pci_priv.h    |  28 +++
>   include/linux/dma-buf.h             |  13 ++
>   include/linux/vfio.h                |   5 +
>   include/linux/vfio_pci_core.h       |   1 +
>   include/uapi/linux/vfio.h           |  28 +++
>   10 files changed, 394 insertions(+), 39 deletions(-)
>   create mode 100644 drivers/vfio/pci/dma_buf.c
>   create mode 100644 drivers/vfio/pci/vfio_pci_priv.h
> 


-- 
--
Thibault