1.. SPDX-License-Identifier: GPL-2.0 2 3============== 4VFIO Selftests 5============== 6 7VFIO selftests are built on top of the kernel's selftests framework and are 8located in ``tools/testing/selftests/vfio``. 9 10VFIO selftests enable kernel developers to write and run tests that take the 11form of userspace programs that interact with VFIO and IOMMUFD uAPIs. VFIO 12selftests can be used to write functional tests for new features, regression 13tests for bugs, and performance tests for optimizations. 14 15These tests are designed to interact with real PCI devices, i.e. they do not 16rely on mocking out or faking any behavior in the kernel. This allows the tests 17to exercise not only VFIO but also IOMMUFD, the IOMMU driver, interrupt 18remapping, IRQ handling, etc. 19 20Running Tests 21============= 22 23Running VFIO selftests requires a device bound to the ``vfio-pci`` driver. 24There are scripts in ``tools/testing/selftests/vfio/scripts/`` to help with 25setting one up. 26 27Picking a Device 28---------------- 29 30Some VFIO selftests require a device with a supported driver implementation in 31the selftests library (see `Driver Framework`_), while other tests (such as 32basic PCI configuration space or reset tests) can use any arbitrary PCI device. 33 34This section will be expanded in the future to guide developers on how to find 35a device with a supported driver. In the meantime, the selftests library 36includes driver support for the following devices: 37 38- Intel IGB Gigabit Ethernet controllers 39- Intel IOAT (I/O Acceleration Technology) DMA engines 40- Intel DSA (Data Streaming Accelerator) 41- NVIDIA Falcon engines (found in many NVIDIA GPUs, e.g., GTX 1080 or RTX 2080) 42 43Note that using Intel DSA devices requires setting the module parameter 44``vfio_pci.disable_denylist=Y`` to allow ``vfio-pci`` to bind to DSA devices. 45 46Setting up Devices 47------------------ 48 49.. code-block:: sh 50 51 tools/testing/selftests/vfio/scripts/setup.sh 0000:01:00.0 52 53The script handles unbinding the device from its current driver and binding it 54to ``vfio-pci``. Metadata about this device is stored in 55``/tmp/vfio-selftests-devices`` so that the device can be cleaned up later 56(bound back to its original driver). 57 58Execution 59--------- 60 61VFIO selftests expect the PCI device BDF string as their final command-line 62argument. e.g. 63 64.. code-block:: sh 65 66 tools/testing/selftests/vfio/vfio_pci_device_test 0000:01:00.0 67 68Alternatively, the PCI device BDF can be passed via the ``VFIO_SELFTESTS_BDF`` 69environment variable, which can be useful when running all tests: 70 71.. code-block:: sh 72 73 export VFIO_SELFTESTS_BDF=0000:01:00.0 74 make -C tools/testing/selftests TARGETS=vfio install 75 tools/testing/selftests/kselftest_install/run_kselftest.sh -c vfio 76 77Cleanup 78------- 79 80.. code-block:: sh 81 82 tools/testing/selftests/vfio/scripts/cleanup.sh 83 84This script handles unbinding all previously setup devices from ``vfio-pci`` 85and binding them back to their original driver. Specific devices can be cleaned 86up by passing their BDF string to the cleanup script. 87 88Writing Tests 89============= 90 91VFIO selftests leverage the ``kselftest_harness.h`` macros to structure setup 92and teardown scenarios. A basic test involves: 93 941. Declaring a test FIXTURE containing the device and IOMMU state. 952. Generating test variants that span the different IOMMU modes utilizing 96 ``FIXTURE_VARIANT_ADD_ALL_IOMMU_MODES()``. 973. Initializing the device and binding the default driver ops. 98 99For a complete example of writing a VFIO selftest, refer to 100``tools/testing/selftests/vfio/vfio_pci_device_test.c``. 101 102VFIO Selftests Library (libvfio) 103================================ 104 105The selftests directory contains a helper library compiled from the ``lib/`` 106subdirectory. It provides structures and helper functions to handle standard 107operations like opening VFIO devices, creating VFIO containers, and interacting 108with VFIO and IOMMUFD. 109 110All VFIO selftests have libvfio linked in by default. Tests can access the 111library by including ``<libvfio.h>``. 112 113Core Objects 114------------ 115 116The VFIO selftests framework revolves around two distinct core objects: 117``struct iommu`` and ``struct vfio_pci_device``. It is important to understand 118the distinction and relationship between them: 119 120- ``struct iommu`` represents an IOMMU domain and the API used to program it 121 (e.g. legacy VFIO Type1 vs IOMMUFD). It is responsible for managing the I/O 122 address space, handling DMA mappings, and translating between host virtual 123 addresses (HVA) and IO virtual addresses (IOVA). 124 125- ``struct vfio_pci_device`` represents a single PCI device. It encapsulates 126 the VFIO device file descriptor, handles access to the PCI configuration 127 space, manages memory-mapped BARs, and drives device interrupts. 128 129A typical test initializes an IOMMU instance first, then passes it when 130initializing one or more PCI devices, linking the devices to the shared IOMMU 131domain. 132 133``struct iommu`` 134^^^^^^^^^^^^^^^^ 135 136The ``struct iommu`` abstracts away the complexity of managing VFIO containers, 137IOMMU groups, and IOMMUFD contexts. A test allocates its IOMMU handle using 138``iommu_init()`` passing one of the supported IOMMU modes: 139 140.. code-block:: c 141 142 struct iommu *iommu = iommu_init(MODE_IOMMUFD); 143 ... 144 iommu_cleanup(iommu); 145 146The selftest framework is designed to work uniformly across all standard kernel 147IOMMU APIs. The defined modes (in ``iommu.h``) include: 148 149- ``MODE_VFIO_TYPE1_IOMMU`` 150- ``MODE_VFIO_TYPE1V2_IOMMU`` 151- ``MODE_IOMMUFD_COMPAT_TYPE1`` 152- ``MODE_IOMMUFD_COMPAT_TYPE1V2`` 153- ``MODE_IOMMUFD`` 154 155VFIO selftests can replicate their test cases across all possible IOMMU modes 156using FIXTURE_VARIANT_ADD_ALL_IOMMU_MODES() to ensure kernel IOMMU APIs remain 157compatible. 158 159``struct vfio_pci_device`` 160^^^^^^^^^^^^^^^^^^^^^^^^^^ 161 162A PCI device is represented as a ``struct vfio_pci_device``. The device 163representation is initialized using ``vfio_pci_device_init()``, which takes a 164target BDF string, opens the device in VFIO, and binds it to an existing 165``struct iommu`` instance: 166 167.. code-block:: c 168 169 struct vfio_pci_device *device = vfio_pci_device_init(device_bdf, iommu); 170 ... 171 vfio_pci_device_cleanup(device); 172 173``struct iova_allocator`` 174^^^^^^^^^^^^^^^^^^^^^^^^^ 175 176To facilitate DMA mappings without hardcoding IOVAs (IO virtual addresses) that 177might conflict with reserved platform address ranges, the library provides a 178``struct iova_allocator``. 179 180The IOVA allocator queries the underlying IOMMU (via ``struct iommu``) for its 181valid target IOVA ranges. Tests can safely carve out unique contiguous chunks 182of IOVA space for device DMA by repeatedly calling: 183 184.. code-block:: c 185 186 struct iova_allocator *allocator = iova_allocator_init(iommu); 187 iova_t addr = iova_allocator_alloc(allocator, size); 188 189Driver Framework 190---------------- 191 192The primary goal of VFIO selftests is to verify the correctness of the kernel 193code handling VFIO devices, including VFIO core, IOMMUFD, IOMMU drivers, page 194pinning, DMA mapping/unmapping, interrupt remapping, and IRQ delivery. 195 196Because different physical devices have vastly different programming interfaces 197for triggering work, the selftests library implements a standard driver 198framework. This framework abstracts diverse hardware endpoints (such as Intel 199IGB, IOAT, or DSA) behind a uniform interface, allowing tests to provoke a 200broad range of devices to generate DMA operations and interrupts in a 201standardized manner. 202 203By exercising DMA and interrupts with real endpoints, tests can thoroughly 204stress and validate the underlying kernel paths, such as IOMMU page table 205management, dirty page tracking, Device TLB and IOTLB invalidations, and 206interrupt delivery under authentic hardware conditions. 207 208All endpoint driver implementations populate a common set of operations defined 209in ``struct vfio_pci_driver_ops``: 210 211- ``probe()``: Validate whether the driver supports the current PCI device 212 (e.g. matching vendor/device IDs). 213- ``init()``: Set up specific control registers or ring-buffer data structures. 214- ``remove()``: Clean up structures before destruction. 215- ``memcpy_start()``: Initiate DMA copies via the device. 216- ``memcpy_wait()``: Poll and wait until previously initiated copies complete 217 safely. 218- ``send_msi()``: Provoke the device to signal an MSI interrupt. 219 220Tests are required to set up a dedicated DMA region for the driver to use 221within ``struct vfio_pci_driver`` to serve as the driver's main memory 222allocation region (e.g. used for in-memory descriptor rings and hardware 223buffers). Tests must provision and map this DMA region before completing driver 224initialization. 225 226For a complete example of interacting with the driver framework from a test 227perspective, refer to ``tools/testing/selftests/vfio/vfio_pci_driver_test.c``. 228 229When contributing a new driver implementation to the selftests library, the 230primary acceptance criteria is that ``vfio_pci_driver_test`` passes using the 231new driver and target device. 232 233Integration with KVM Selftests 234------------------------------ 235 236libvfio is not strictly limited to VFIO selftests, it is also leveraged within 237KVM selftests (``tools/testing/selftests/kvm/``). KVM selftests build their own 238copy of libvfio and link against it, enabling libvfio to be used in KVM 239selftests. 240 241This integration enables testing the interaction between KVM and VFIO and 242IOMMUFD. For example, testing the end-to-end delivery of device interrupts into 243running vCPUs. 244 245Contribution Guidelines 246======================= 247 248- Prefer the shortlog prefix ``vfio: selftests: ...`` for commits that modify 249 files in ``tools/testing/selftests/vfio/``. 250- Follow the Coding Style, Comments, Changelogs, and Function References 251 guidelines from the 252 :doc:`KVM x86 maintainers handbook </process/maintainer-kvm-x86>`. 253