xref: /linux/Documentation/driver-api/vfio-mediated-device.rst (revision e65e175b07bef5974045cc42238de99057669ca7) 
1.. SPDX-License-Identifier: GPL-2.0-only
2.. include:: <isonum.txt>
3
4=====================
5VFIO Mediated devices
6=====================
7
8:Copyright: |copy| 2016, NVIDIA CORPORATION. All rights reserved.
9:Author: Neo Jia <cjia@nvidia.com>
10:Author: Kirti Wankhede <kwankhede@nvidia.com>
11
12
13
14Virtual Function I/O (VFIO) Mediated devices[1]
15===============================================
16
17The number of use cases for virtualizing DMA devices that do not have built-in
18SR_IOV capability is increasing. Previously, to virtualize such devices,
19developers had to create their own management interfaces and APIs, and then
20integrate them with user space software. To simplify integration with user space
21software, we have identified common requirements and a unified management
22interface for such devices.
23
24The VFIO driver framework provides unified APIs for direct device access. It is
25an IOMMU/device-agnostic framework for exposing direct device access to user
26space in a secure, IOMMU-protected environment. This framework is used for
27multiple devices, such as GPUs, network adapters, and compute accelerators. With
28direct device access, virtual machines or user space applications have direct
29access to the physical device. This framework is reused for mediated devices.
30
31The mediated core driver provides a common interface for mediated device
32management that can be used by drivers of different devices. This module
33provides a generic interface to perform these operations:
34
35* Create and destroy a mediated device
36* Add a mediated device to and remove it from a mediated bus driver
37* Add a mediated device to and remove it from an IOMMU group
38
39The mediated core driver also provides an interface to register a bus driver.
40For example, the mediated VFIO mdev driver is designed for mediated devices and
41supports VFIO APIs. The mediated bus driver adds a mediated device to and
42removes it from a VFIO group.
43
44The following high-level block diagram shows the main components and interfaces
45in the VFIO mediated driver framework. The diagram shows NVIDIA, Intel, and IBM
46devices as examples, as these devices are the first devices to use this module::
47
48     +---------------+
49     |               |
50     | +-----------+ |  mdev_register_driver() +--------------+
51     | |           | +<------------------------+              |
52     | |  mdev     | |                         |              |
53     | |  bus      | +------------------------>+ vfio_mdev.ko |<-> VFIO user
54     | |  driver   | |     probe()/remove()    |              |    APIs
55     | |           | |                         +--------------+
56     | +-----------+ |
57     |               |
58     |  MDEV CORE    |
59     |   MODULE      |
60     |   mdev.ko     |
61     | +-----------+ |  mdev_register_parent() +--------------+
62     | |           | +<------------------------+              |
63     | |           | |                         |  nvidia.ko   |<-> physical
64     | |           | +------------------------>+              |    device
65     | |           | |        callbacks        +--------------+
66     | | Physical  | |
67     | |  device   | |  mdev_register_parent() +--------------+
68     | | interface | |<------------------------+              |
69     | |           | |                         |  i915.ko     |<-> physical
70     | |           | +------------------------>+              |    device
71     | |           | |        callbacks        +--------------+
72     | |           | |
73     | |           | |  mdev_register_parent() +--------------+
74     | |           | +<------------------------+              |
75     | |           | |                         | ccw_device.ko|<-> physical
76     | |           | +------------------------>+              |    device
77     | |           | |        callbacks        +--------------+
78     | +-----------+ |
79     +---------------+
80
81
82Registration Interfaces
83=======================
84
85The mediated core driver provides the following types of registration
86interfaces:
87
88* Registration interface for a mediated bus driver
89* Physical device driver interface
90
91Registration Interface for a Mediated Bus Driver
92------------------------------------------------
93
94The registration interface for a mediated device driver provides the following
95structure to represent a mediated device's driver::
96
97     /*
98      * struct mdev_driver [2] - Mediated device's driver
99      * @probe: called when new device created
100      * @remove: called when device removed
101      * @driver: device driver structure
102      */
103     struct mdev_driver {
104	     int  (*probe)  (struct mdev_device *dev);
105	     void (*remove) (struct mdev_device *dev);
106	     unsigned int (*get_available)(struct mdev_type *mtype);
107	     ssize_t (*show_description)(struct mdev_type *mtype, char *buf);
108	     struct device_driver    driver;
109     };
110
111A mediated bus driver for mdev should use this structure in the function calls
112to register and unregister itself with the core driver:
113
114* Register::
115
116    int mdev_register_driver(struct mdev_driver *drv);
117
118* Unregister::
119
120    void mdev_unregister_driver(struct mdev_driver *drv);
121
122The mediated bus driver's probe function should create a vfio_device on top of
123the mdev_device and connect it to an appropriate implementation of
124vfio_device_ops.
125
126When a driver wants to add the GUID creation sysfs to an existing device it has
127probe'd to then it should call::
128
129    int mdev_register_parent(struct mdev_parent *parent, struct device *dev,
130			struct mdev_driver *mdev_driver);
131
132This will provide the 'mdev_supported_types/XX/create' files which can then be
133used to trigger the creation of a mdev_device. The created mdev_device will be
134attached to the specified driver.
135
136When the driver needs to remove itself it calls::
137
138    void mdev_unregister_parent(struct mdev_parent *parent);
139
140Which will unbind and destroy all the created mdevs and remove the sysfs files.
141
142Mediated Device Management Interface Through sysfs
143==================================================
144
145The management interface through sysfs enables user space software, such as
146libvirt, to query and configure mediated devices in a hardware-agnostic fashion.
147This management interface provides flexibility to the underlying physical
148device's driver to support features such as:
149
150* Mediated device hot plug
151* Multiple mediated devices in a single virtual machine
152* Multiple mediated devices from different physical devices
153
154Links in the mdev_bus Class Directory
155-------------------------------------
156The /sys/class/mdev_bus/ directory contains links to devices that are registered
157with the mdev core driver.
158
159Directories and files under the sysfs for Each Physical Device
160--------------------------------------------------------------
161
162::
163
164  |- [parent physical device]
165  |--- Vendor-specific-attributes [optional]
166  |--- [mdev_supported_types]
167  |     |--- [<type-id>]
168  |     |   |--- create
169  |     |   |--- name
170  |     |   |--- available_instances
171  |     |   |--- device_api
172  |     |   |--- description
173  |     |   |--- [devices]
174  |     |--- [<type-id>]
175  |     |   |--- create
176  |     |   |--- name
177  |     |   |--- available_instances
178  |     |   |--- device_api
179  |     |   |--- description
180  |     |   |--- [devices]
181  |     |--- [<type-id>]
182  |          |--- create
183  |          |--- name
184  |          |--- available_instances
185  |          |--- device_api
186  |          |--- description
187  |          |--- [devices]
188
189* [mdev_supported_types]
190
191  The list of currently supported mediated device types and their details.
192
193  [<type-id>], device_api, and available_instances are mandatory attributes
194  that should be provided by vendor driver.
195
196* [<type-id>]
197
198  The [<type-id>] name is created by adding the device driver string as a prefix
199  to the string provided by the vendor driver. This format of this name is as
200  follows::
201
202	sprintf(buf, "%s-%s", dev_driver_string(parent->dev), group->name);
203
204* device_api
205
206  This attribute shows which device API is being created, for example,
207  "vfio-pci" for a PCI device.
208
209* available_instances
210
211  This attribute shows the number of devices of type <type-id> that can be
212  created.
213
214* [device]
215
216  This directory contains links to the devices of type <type-id> that have been
217  created.
218
219* name
220
221  This attribute shows a human readable name.
222
223* description
224
225  This attribute can show brief features/description of the type. This is an
226  optional attribute.
227
228Directories and Files Under the sysfs for Each mdev Device
229----------------------------------------------------------
230
231::
232
233  |- [parent phy device]
234  |--- [$MDEV_UUID]
235         |--- remove
236         |--- mdev_type {link to its type}
237         |--- vendor-specific-attributes [optional]
238
239* remove (write only)
240
241Writing '1' to the 'remove' file destroys the mdev device. The vendor driver can
242fail the remove() callback if that device is active and the vendor driver
243doesn't support hot unplug.
244
245Example::
246
247	# echo 1 > /sys/bus/mdev/devices/$mdev_UUID/remove
248
249Mediated device Hot plug
250------------------------
251
252Mediated devices can be created and assigned at runtime. The procedure to hot
253plug a mediated device is the same as the procedure to hot plug a PCI device.
254
255Translation APIs for Mediated Devices
256=====================================
257
258The following APIs are provided for translating user pfn to host pfn in a VFIO
259driver::
260
261	int vfio_pin_pages(struct vfio_device *device, dma_addr_t iova,
262				  int npage, int prot, struct page **pages);
263
264	void vfio_unpin_pages(struct vfio_device *device, dma_addr_t iova,
265				    int npage);
266
267These functions call back into the back-end IOMMU module by using the pin_pages
268and unpin_pages callbacks of the struct vfio_iommu_driver_ops[4]. Currently
269these callbacks are supported in the TYPE1 IOMMU module. To enable them for
270other IOMMU backend modules, such as PPC64 sPAPR module, they need to provide
271these two callback functions.
272
273Using the Sample Code
274=====================
275
276mtty.c in samples/vfio-mdev/ directory is a sample driver program to
277demonstrate how to use the mediated device framework.
278
279The sample driver creates an mdev device that simulates a serial port over a PCI
280card.
281
2821. Build and load the mtty.ko module.
283
284   This step creates a dummy device, /sys/devices/virtual/mtty/mtty/
285
286   Files in this device directory in sysfs are similar to the following::
287
288     # tree /sys/devices/virtual/mtty/mtty/
289        /sys/devices/virtual/mtty/mtty/
290        |-- mdev_supported_types
291        |   |-- mtty-1
292        |   |   |-- available_instances
293        |   |   |-- create
294        |   |   |-- device_api
295        |   |   |-- devices
296        |   |   `-- name
297        |   `-- mtty-2
298        |       |-- available_instances
299        |       |-- create
300        |       |-- device_api
301        |       |-- devices
302        |       `-- name
303        |-- mtty_dev
304        |   `-- sample_mtty_dev
305        |-- power
306        |   |-- autosuspend_delay_ms
307        |   |-- control
308        |   |-- runtime_active_time
309        |   |-- runtime_status
310        |   `-- runtime_suspended_time
311        |-- subsystem -> ../../../../class/mtty
312        `-- uevent
313
3142. Create a mediated device by using the dummy device that you created in the
315   previous step::
316
317     # echo "83b8f4f2-509f-382f-3c1e-e6bfe0fa1001" >	\
318              /sys/devices/virtual/mtty/mtty/mdev_supported_types/mtty-2/create
319
3203. Add parameters to qemu-kvm::
321
322     -device vfio-pci,\
323      sysfsdev=/sys/bus/mdev/devices/83b8f4f2-509f-382f-3c1e-e6bfe0fa1001
324
3254. Boot the VM.
326
327   In the Linux guest VM, with no hardware on the host, the device appears
328   as  follows::
329
330     # lspci -s 00:05.0 -xxvv
331     00:05.0 Serial controller: Device 4348:3253 (rev 10) (prog-if 02 [16550])
332             Subsystem: Device 4348:3253
333             Physical Slot: 5
334             Control: I/O+ Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr-
335     Stepping- SERR- FastB2B- DisINTx-
336             Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort-
337     <TAbort- <MAbort- >SERR- <PERR- INTx-
338             Interrupt: pin A routed to IRQ 10
339             Region 0: I/O ports at c150 [size=8]
340             Region 1: I/O ports at c158 [size=8]
341             Kernel driver in use: serial
342     00: 48 43 53 32 01 00 00 02 10 02 00 07 00 00 00 00
343     10: 51 c1 00 00 59 c1 00 00 00 00 00 00 00 00 00 00
344     20: 00 00 00 00 00 00 00 00 00 00 00 00 48 43 53 32
345     30: 00 00 00 00 00 00 00 00 00 00 00 00 0a 01 00 00
346
347     In the Linux guest VM, dmesg output for the device is as follows:
348
349     serial 0000:00:05.0: PCI INT A -> Link[LNKA] -> GSI 10 (level, high) -> IRQ 10
350     0000:00:05.0: ttyS1 at I/O 0xc150 (irq = 10) is a 16550A
351     0000:00:05.0: ttyS2 at I/O 0xc158 (irq = 10) is a 16550A
352
353
3545. In the Linux guest VM, check the serial ports::
355
356     # setserial -g /dev/ttyS*
357     /dev/ttyS0, UART: 16550A, Port: 0x03f8, IRQ: 4
358     /dev/ttyS1, UART: 16550A, Port: 0xc150, IRQ: 10
359     /dev/ttyS2, UART: 16550A, Port: 0xc158, IRQ: 10
360
3616. Using minicom or any terminal emulation program, open port /dev/ttyS1 or
362   /dev/ttyS2 with hardware flow control disabled.
363
3647. Type data on the minicom terminal or send data to the terminal emulation
365   program and read the data.
366
367   Data is loop backed from hosts mtty driver.
368
3698. Destroy the mediated device that you created::
370
371     # echo 1 > /sys/bus/mdev/devices/83b8f4f2-509f-382f-3c1e-e6bfe0fa1001/remove
372
373References
374==========
375
3761. See Documentation/driver-api/vfio.rst for more information on VFIO.
3772. struct mdev_driver in include/linux/mdev.h
3783. struct mdev_parent_ops in include/linux/mdev.h
3794. struct vfio_iommu_driver_ops in include/linux/vfio.h
380