NACK/Cmnt: [SRU][J:linux-bluefield][PATCH v1 0/6] Add VFIO P2P support

[Bartlomiej Zolnierkiewicz]

v3 sent to the mailing list

On Mon, Aug 26, 2024 at 8:31 PM William Tu <witu at nvidia.com> wrote:
>
> BugLink: https://bugs.launchpad.net/bugs/2077887
>
> The series adds support for VFIO P2P feature for NVMe target offload.
> We took the patch (not upstreamed) below with some fix to make it work.
> https://patchwork.kernel.org/project/linux-rdma/cover/0-v2-472615b3877e+28f7-vfio_dma_buf_jgg@nvidia.com/
>
> * Kernel Config
> Need to enable
> CONFIG_PCI_P2PDMA (enable p2p dma transaction)
> CONFIG_DMABUF_MOVE_NOTIFY (no need to pin down the memory)
>
> The feature doesn't work with IOMMU, so either we disable it
> for the system, "iommu.passthrough=1 iommu=off" in kernel boot
> parameter, or conditionally disable it for VFIO.
> CONFIG_VFIO_NOIOMMU (optional)
> - You should enable CONFIG_VFIO_NOIOMMU in the kernel config if your
> vfio module does not have vfio enable_unsafe_noiommu_mode in the parameter list.
>
> * /etc/modprobe.d/vfio.conf
> $ cat /etc/modprobe.d/vfio.conf
> # The kernel hungs when 0000:06:00.0 (Non-Volatile memory controller: KIOXIA
> # Corporation Device 0001) is bound to the vfio-pci module, and the following
> # error is printed:
> #
> # vfio-pci 0000:06:00.0: Unable to change power state from D3hot to D0, device inaccessible
> #
> # The configuration below is a workarround for the issue.
> options vfio-pci disable_idle_d3=1
>
> # Allow using VFIO without IOMMU. This is only required when SMMU is disabled.
> options vfio enable_unsafe_noiommu_mode=1
>
> * Setup and Test
> 1. For ease of testing, we take a BF3 and boot the OS into mmc instead
> of NVMe drive. So leaving NVMe drive free to use.
>
> 2. use a test program to verify its correctness.
> The program create ibv rdma program, open nvme drive, create dma vfio
> buf from nvme, and let the rdma program read (DMA P2P) the version
> number of the nvme drive.
> https://gitlab.com/Mellanox/spdk_team/vfio-dmabuf-test
>
> * UAPI changes
> The backport patches are not upstreamed and we have to maintain it if
> conflict in the future.Specifically, the patch "vfio/pci: Allow MMIO
> regions to be exported through dma-buf" adds the following struct and define.
> /**
>  * Upon VFIO_DEVICE_FEATURE_GET create a dma_buf fd for the
>  * region selected.
>  *
>  * open_flags are the typical flags passed to open(2), eg O_RDWR, O_CLOEXEC,
>  * etc. offset/length specify a slice of the region to create the dmabuf from.
>  * If both are 0 then the whole region is used.
>  *
>  * Return: The fd number on success, -1 and errno is set on failure.
>  */
> struct vfio_device_feature_dma_buf {
>        __u32 region_index;
>        __u32 open_flags;
>        __u32 offset;
>        __u64 length;
> };
> #define VFIO_DEVICE_FEATURE_DMA_BUF 3
>
> Jason Gunthorpe (4):
>   UBUNTU: SAUCE: dma-buf: Add dma_buf_try_get()
>   UBUNTU: SAUCE: vfio: Add vfio_device_get()
>   UBUNTU: SAUCE: vfio_pci: Do not open code pci_try_reset_function()
>   UBUNTU: SAUCE: vfio/pci: Allow MMIO regions to be exported through
>     dma-buf
>
> Sergey Gorenko (1):
>   UBUNTU: SAUCE: vfio/pci: Fix p2p address
>
> William Tu (1):
>   UBUNTU: [Config] bluefield: add config for VFIO P2P
>
>  debian.bluefield/config/annotations |   2 +
>  drivers/vfio/pci/Makefile           |   3 +-
>  drivers/vfio/pci/dma_buf.c          | 267 ++++++++++++++++++++++++++++
>  drivers/vfio/pci/vfio_pci_config.c  |  24 +--
>  drivers/vfio/pci/vfio_pci_core.c    |  60 ++++---
>  drivers/vfio/pci/vfio_pci_priv.h    |  30 ++++
>  include/linux/dma-buf.h             |  13 ++
>  include/linux/vfio.h                |   5 +
>  include/linux/vfio_pci_core.h       |   1 +
>  include/uapi/linux/vfio.h           |  18 ++
>  10 files changed, 384 insertions(+), 39 deletions(-)
>  create mode 100644 drivers/vfio/pci/dma_buf.c
>  create mode 100644 drivers/vfio/pci/vfio_pci_priv.h
>