1.. SPDX-License-Identifier: GPL-2.0-only 2.. include:: <isonum.txt> 3 4===================== 5VFIO Mediated devices 6===================== 7 8:Copyright: |copy| 2016, NVIDIA CORPORATION. All rights reserved. 9:Author: Neo Jia <cjia@nvidia.com> 10:Author: Kirti Wankhede <kwankhede@nvidia.com> 11 12 13 14Virtual Function I/O (VFIO) Mediated devices[1] 15=============================================== 16 17The number of use cases for virtualizing DMA devices that do not have built-in 18SR_IOV capability is increasing. Previously, to virtualize such devices, 19developers had to create their own management interfaces and APIs, and then 20integrate them with user space software. To simplify integration with user space 21software, we have identified common requirements and a unified management 22interface for such devices. 23 24The VFIO driver framework provides unified APIs for direct device access. It is 25an IOMMU/device-agnostic framework for exposing direct device access to user 26space in a secure, IOMMU-protected environment. This framework is used for 27multiple devices, such as GPUs, network adapters, and compute accelerators. With 28direct device access, virtual machines or user space applications have direct 29access to the physical device. This framework is reused for mediated devices. 30 31The mediated core driver provides a common interface for mediated device 32management that can be used by drivers of different devices. This module 33provides a generic interface to perform these operations: 34 35* Create and destroy a mediated device 36* Add a mediated device to and remove it from a mediated bus driver 37* Add a mediated device to and remove it from an IOMMU group 38 39The mediated core driver also provides an interface to register a bus driver. 40For example, the mediated VFIO mdev driver is designed for mediated devices and 41supports VFIO APIs. The mediated bus driver adds a mediated device to and 42removes it from a VFIO group. 43 44The following high-level block diagram shows the main components and interfaces 45in the VFIO mediated driver framework. The diagram shows NVIDIA, Intel, and IBM 46devices as examples, as these devices are the first devices to use this module:: 47 48 +---------------+ 49 | | 50 | +-----------+ | mdev_register_driver() +--------------+ 51 | | | +<------------------------+ | 52 | | mdev | | | | 53 | | bus | +------------------------>+ vfio_mdev.ko |<-> VFIO user 54 | | driver | | probe()/remove() | | APIs 55 | | | | +--------------+ 56 | +-----------+ | 57 | | 58 | MDEV CORE | 59 | MODULE | 60 | mdev.ko | 61 | +-----------+ | mdev_register_parent() +--------------+ 62 | | | +<------------------------+ | 63 | | | | | ccw_device.ko|<-> physical 64 | | | +------------------------>+ | device 65 | | | | callbacks +--------------+ 66 | | Physical | | 67 | | device | | mdev_register_parent() +--------------+ 68 | | interface | |<------------------------+ | 69 | | | | | i915.ko |<-> physical 70 | | | +------------------------>+ | device 71 | | | | callbacks +--------------+ 72 | +-----------+ | 73 +---------------+ 74 75 76Registration Interfaces 77======================= 78 79The mediated core driver provides the following types of registration 80interfaces: 81 82* Registration interface for a mediated bus driver 83* Physical device driver interface 84 85Registration Interface for a Mediated Bus Driver 86------------------------------------------------ 87 88The registration interface for a mediated device driver provides the following 89structure to represent a mediated device's driver:: 90 91 /* 92 * struct mdev_driver [2] - Mediated device's driver 93 * @probe: called when new device created 94 * @remove: called when device removed 95 * @driver: device driver structure 96 */ 97 struct mdev_driver { 98 int (*probe) (struct mdev_device *dev); 99 void (*remove) (struct mdev_device *dev); 100 unsigned int (*get_available)(struct mdev_type *mtype); 101 ssize_t (*show_description)(struct mdev_type *mtype, char *buf); 102 struct device_driver driver; 103 }; 104 105A mediated bus driver for mdev should use this structure in the function calls 106to register and unregister itself with the core driver: 107 108* Register:: 109 110 int mdev_register_driver(struct mdev_driver *drv); 111 112* Unregister:: 113 114 void mdev_unregister_driver(struct mdev_driver *drv); 115 116The mediated bus driver's probe function should create a vfio_device on top of 117the mdev_device and connect it to an appropriate implementation of 118vfio_device_ops. 119 120When a driver wants to add the GUID creation sysfs to an existing device it has 121probe'd to then it should call:: 122 123 int mdev_register_parent(struct mdev_parent *parent, struct device *dev, 124 struct mdev_driver *mdev_driver); 125 126This will provide the 'mdev_supported_types/XX/create' files which can then be 127used to trigger the creation of a mdev_device. The created mdev_device will be 128attached to the specified driver. 129 130When the driver needs to remove itself it calls:: 131 132 void mdev_unregister_parent(struct mdev_parent *parent); 133 134Which will unbind and destroy all the created mdevs and remove the sysfs files. 135 136Mediated Device Management Interface Through sysfs 137================================================== 138 139The management interface through sysfs enables user space software, such as 140libvirt, to query and configure mediated devices in a hardware-agnostic fashion. 141This management interface provides flexibility to the underlying physical 142device's driver to support features such as: 143 144* Mediated device hot plug 145* Multiple mediated devices in a single virtual machine 146* Multiple mediated devices from different physical devices 147 148Links in the mdev_bus Class Directory 149------------------------------------- 150The /sys/class/mdev_bus/ directory contains links to devices that are registered 151with the mdev core driver. 152 153Directories and files under the sysfs for Each Physical Device 154-------------------------------------------------------------- 155 156:: 157 158 |- [parent physical device] 159 |--- Vendor-specific-attributes [optional] 160 |--- [mdev_supported_types] 161 | |--- [<type-id>] 162 | | |--- create 163 | | |--- name 164 | | |--- available_instances 165 | | |--- device_api 166 | | |--- description 167 | | |--- [devices] 168 | |--- [<type-id>] 169 | | |--- create 170 | | |--- name 171 | | |--- available_instances 172 | | |--- device_api 173 | | |--- description 174 | | |--- [devices] 175 | |--- [<type-id>] 176 | |--- create 177 | |--- name 178 | |--- available_instances 179 | |--- device_api 180 | |--- description 181 | |--- [devices] 182 183* [mdev_supported_types] 184 185 The list of currently supported mediated device types and their details. 186 187 [<type-id>], device_api, and available_instances are mandatory attributes 188 that should be provided by vendor driver. 189 190* [<type-id>] 191 192 The [<type-id>] name is created by adding the device driver string as a prefix 193 to the string provided by the vendor driver. This format of this name is as 194 follows:: 195 196 sprintf(buf, "%s-%s", dev_driver_string(parent->dev), group->name); 197 198* device_api 199 200 This attribute shows which device API is being created, for example, 201 "vfio-pci" for a PCI device. 202 203* available_instances 204 205 This attribute shows the number of devices of type <type-id> that can be 206 created. 207 208* [device] 209 210 This directory contains links to the devices of type <type-id> that have been 211 created. 212 213* name 214 215 This attribute shows a human readable name. 216 217* description 218 219 This attribute can show brief features/description of the type. This is an 220 optional attribute. 221 222Directories and Files Under the sysfs for Each mdev Device 223---------------------------------------------------------- 224 225:: 226 227 |- [parent phy device] 228 |--- [$MDEV_UUID] 229 |--- remove 230 |--- mdev_type {link to its type} 231 |--- vendor-specific-attributes [optional] 232 233* remove (write only) 234 235Writing '1' to the 'remove' file destroys the mdev device. The vendor driver can 236fail the remove() callback if that device is active and the vendor driver 237doesn't support hot unplug. 238 239Example:: 240 241 # echo 1 > /sys/bus/mdev/devices/$mdev_UUID/remove 242 243Mediated device Hot plug 244------------------------ 245 246Mediated devices can be created and assigned at runtime. The procedure to hot 247plug a mediated device is the same as the procedure to hot plug a PCI device. 248 249Translation APIs for Mediated Devices 250===================================== 251 252The following APIs are provided for translating user pfn to host pfn in a VFIO 253driver:: 254 255 int vfio_pin_pages(struct vfio_device *device, dma_addr_t iova, 256 int npage, int prot, struct page **pages); 257 258 void vfio_unpin_pages(struct vfio_device *device, dma_addr_t iova, 259 int npage); 260 261These functions call back into the back-end IOMMU module by using the pin_pages 262and unpin_pages callbacks of the struct vfio_iommu_driver_ops[4]. Currently 263these callbacks are supported in the TYPE1 IOMMU module. To enable them for 264other IOMMU backend modules, such as PPC64 sPAPR module, they need to provide 265these two callback functions. 266 267References 268========== 269 2701. See Documentation/driver-api/vfio.rst for more information on VFIO. 2712. struct mdev_driver in include/linux/mdev.h 2723. struct mdev_parent_ops in include/linux/mdev.h 2734. struct vfio_iommu_driver_ops in include/linux/vfio.h 274