1=============================== 2Adjunct Processor (AP) facility 3=============================== 4 5 6Introduction 7============ 8The Adjunct Processor (AP) facility is an IBM Z cryptographic facility comprised 9of three AP instructions and from 1 up to 256 PCIe cryptographic adapter cards. 10The AP devices provide cryptographic functions to all CPUs assigned to a 11linux system running in an IBM Z system LPAR. 12 13The AP adapter cards are exposed via the AP bus. The motivation for vfio-ap 14is to make AP cards available to KVM guests using the VFIO mediated device 15framework. This implementation relies considerably on the s390 virtualization 16facilities which do most of the hard work of providing direct access to AP 17devices. 18 19AP Architectural Overview 20========================= 21To facilitate the comprehension of the design, let's start with some 22definitions: 23 24* AP adapter 25 26 An AP adapter is an IBM Z adapter card that can perform cryptographic 27 functions. There can be from 0 to 256 adapters assigned to an LPAR. Adapters 28 assigned to the LPAR in which a linux host is running will be available to 29 the linux host. Each adapter is identified by a number from 0 to 255; however, 30 the maximum adapter number is determined by machine model and/or adapter type. 31 When installed, an AP adapter is accessed by AP instructions executed by any 32 CPU. 33 34 The AP adapter cards are assigned to a given LPAR via the system's Activation 35 Profile which can be edited via the HMC. When the linux host system is IPL'd 36 in the LPAR, the AP bus detects the AP adapter cards assigned to the LPAR and 37 creates a sysfs device for each assigned adapter. For example, if AP adapters 38 4 and 10 (0x0a) are assigned to the LPAR, the AP bus will create the following 39 sysfs device entries:: 40 41 /sys/devices/ap/card04 42 /sys/devices/ap/card0a 43 44 Symbolic links to these devices will also be created in the AP bus devices 45 sub-directory:: 46 47 /sys/bus/ap/devices/[card04] 48 /sys/bus/ap/devices/[card04] 49 50* AP domain 51 52 An adapter is partitioned into domains. An adapter can hold up to 256 domains 53 depending upon the adapter type and hardware configuration. A domain is 54 identified by a number from 0 to 255; however, the maximum domain number is 55 determined by machine model and/or adapter type.. A domain can be thought of 56 as a set of hardware registers and memory used for processing AP commands. A 57 domain can be configured with a secure private key used for clear key 58 encryption. A domain is classified in one of two ways depending upon how it 59 may be accessed: 60 61 * Usage domains are domains that are targeted by an AP instruction to 62 process an AP command. 63 64 * Control domains are domains that are changed by an AP command sent to a 65 usage domain; for example, to set the secure private key for the control 66 domain. 67 68 The AP usage and control domains are assigned to a given LPAR via the system's 69 Activation Profile which can be edited via the HMC. When a linux host system 70 is IPL'd in the LPAR, the AP bus module detects the AP usage and control 71 domains assigned to the LPAR. The domain number of each usage domain and 72 adapter number of each AP adapter are combined to create AP queue devices 73 (see AP Queue section below). The domain number of each control domain will be 74 represented in a bitmask and stored in a sysfs file 75 /sys/bus/ap/ap_control_domain_mask. The bits in the mask, from most to least 76 significant bit, correspond to domains 0-255. 77 78* AP Queue 79 80 An AP queue is the means by which an AP command is sent to a usage domain 81 inside a specific adapter. An AP queue is identified by a tuple 82 comprised of an AP adapter ID (APID) and an AP queue index (APQI). The 83 APQI corresponds to a given usage domain number within the adapter. This tuple 84 forms an AP Queue Number (APQN) uniquely identifying an AP queue. AP 85 instructions include a field containing the APQN to identify the AP queue to 86 which the AP command is to be sent for processing. 87 88 The AP bus will create a sysfs device for each APQN that can be derived from 89 the cross product of the AP adapter and usage domain numbers detected when the 90 AP bus module is loaded. For example, if adapters 4 and 10 (0x0a) and usage 91 domains 6 and 71 (0x47) are assigned to the LPAR, the AP bus will create the 92 following sysfs entries:: 93 94 /sys/devices/ap/card04/04.0006 95 /sys/devices/ap/card04/04.0047 96 /sys/devices/ap/card0a/0a.0006 97 /sys/devices/ap/card0a/0a.0047 98 99 The following symbolic links to these devices will be created in the AP bus 100 devices subdirectory:: 101 102 /sys/bus/ap/devices/[04.0006] 103 /sys/bus/ap/devices/[04.0047] 104 /sys/bus/ap/devices/[0a.0006] 105 /sys/bus/ap/devices/[0a.0047] 106 107* AP Instructions: 108 109 There are three AP instructions: 110 111 * NQAP: to enqueue an AP command-request message to a queue 112 * DQAP: to dequeue an AP command-reply message from a queue 113 * PQAP: to administer the queues 114 115 AP instructions identify the domain that is targeted to process the AP 116 command; this must be one of the usage domains. An AP command may modify a 117 domain that is not one of the usage domains, but the modified domain 118 must be one of the control domains. 119 120AP and SIE 121========== 122Let's now take a look at how AP instructions executed on a guest are interpreted 123by the hardware. 124 125A satellite control block called the Crypto Control Block (CRYCB) is attached to 126our main hardware virtualization control block. The CRYCB contains an AP Control 127Block (APCB) that has three fields to identify the adapters, usage domains and 128control domains assigned to the KVM guest: 129 130* The AP Mask (APM) field is a bit mask that identifies the AP adapters assigned 131 to the KVM guest. Each bit in the mask, from left to right, corresponds to 132 an APID from 0-255. If a bit is set, the corresponding adapter is valid for 133 use by the KVM guest. 134 135* The AP Queue Mask (AQM) field is a bit mask identifying the AP usage domains 136 assigned to the KVM guest. Each bit in the mask, from left to right, 137 corresponds to an AP queue index (APQI) from 0-255. If a bit is set, the 138 corresponding queue is valid for use by the KVM guest. 139 140* The AP Domain Mask field is a bit mask that identifies the AP control domains 141 assigned to the KVM guest. The ADM bit mask controls which domains can be 142 changed by an AP command-request message sent to a usage domain from the 143 guest. Each bit in the mask, from left to right, corresponds to a domain from 144 0-255. If a bit is set, the corresponding domain can be modified by an AP 145 command-request message sent to a usage domain. 146 147If you recall from the description of an AP Queue, AP instructions include 148an APQN to identify the AP queue to which an AP command-request message is to be 149sent (NQAP and PQAP instructions), or from which a command-reply message is to 150be received (DQAP instruction). The validity of an APQN is defined by the matrix 151calculated from the APM and AQM; it is the Cartesian product of all assigned 152adapter numbers (APM) with all assigned queue indexes (AQM). For example, if 153adapters 1 and 2 and usage domains 5 and 6 are assigned to a guest, the APQNs 154(1,5), (1,6), (2,5) and (2,6) will be valid for the guest. 155 156The APQNs can provide secure key functionality - i.e., a private key is stored 157on the adapter card for each of its domains - so each APQN must be assigned to 158at most one guest or to the linux host:: 159 160 Example 1: Valid configuration: 161 ------------------------------ 162 Guest1: adapters 1,2 domains 5,6 163 Guest2: adapter 1,2 domain 7 164 165 This is valid because both guests have a unique set of APQNs: 166 Guest1 has APQNs (1,5), (1,6), (2,5), (2,6); 167 Guest2 has APQNs (1,7), (2,7) 168 169 Example 2: Valid configuration: 170 ------------------------------ 171 Guest1: adapters 1,2 domains 5,6 172 Guest2: adapters 3,4 domains 5,6 173 174 This is also valid because both guests have a unique set of APQNs: 175 Guest1 has APQNs (1,5), (1,6), (2,5), (2,6); 176 Guest2 has APQNs (3,5), (3,6), (4,5), (4,6) 177 178 Example 3: Invalid configuration: 179 -------------------------------- 180 Guest1: adapters 1,2 domains 5,6 181 Guest2: adapter 1 domains 6,7 182 183 This is an invalid configuration because both guests have access to 184 APQN (1,6). 185 186The Design 187========== 188The design introduces three new objects: 189 1901. AP matrix device 1912. VFIO AP device driver (vfio_ap.ko) 1923. VFIO AP mediated pass-through device 193 194The VFIO AP device driver 195------------------------- 196The VFIO AP (vfio_ap) device driver serves the following purposes: 197 1981. Provides the interfaces to secure APQNs for exclusive use of KVM guests. 199 2002. Sets up the VFIO mediated device interfaces to manage a vfio_ap mediated 201 device and creates the sysfs interfaces for assigning adapters, usage 202 domains, and control domains comprising the matrix for a KVM guest. 203 2043. Configures the APM, AQM and ADM in the APCB contained in the CRYCB referenced 205 by a KVM guest's SIE state description to grant the guest access to a matrix 206 of AP devices 207 208Reserve APQNs for exclusive use of KVM guests 209--------------------------------------------- 210The following block diagram illustrates the mechanism by which APQNs are 211reserved:: 212 213 +------------------+ 214 7 remove | | 215 +--------------------> cex4queue driver | 216 | | | 217 | +------------------+ 218 | 219 | 220 | +------------------+ +----------------+ 221 | 5 register driver | | 3 create | | 222 | +----------------> Device core +----------> matrix device | 223 | | | | | | 224 | | +--------^---------+ +----------------+ 225 | | | 226 | | +-------------------+ 227 | | +-----------------------------------+ | 228 | | | 4 register AP driver | | 2 register device 229 | | | | | 230 +--------+---+-v---+ +--------+-------+-+ 231 | | | | 232 | ap_bus +--------------------- > vfio_ap driver | 233 | | 8 probe | | 234 +--------^---------+ +--^--^------------+ 235 6 edit | | | 236 apmask | +-----------------------------+ | 11 mdev create 237 aqmask | | 1 modprobe | 238 +--------+-----+---+ +----------------+-+ +----------------+ 239 | | | |10 create| mediated | 240 | admin | | VFIO device core |---------> matrix | 241 | + | | | device | 242 +------+-+---------+ +--------^---------+ +--------^-------+ 243 | | | | 244 | | 9 create vfio_ap-passthrough | | 245 | +------------------------------+ | 246 +-------------------------------------------------------------+ 247 12 assign adapter/domain/control domain 248 249The process for reserving an AP queue for use by a KVM guest is: 250 2511. The administrator loads the vfio_ap device driver 2522. The vfio-ap driver during its initialization will register a single 'matrix' 253 device with the device core. This will serve as the parent device for 254 all vfio_ap mediated devices used to configure an AP matrix for a guest. 2553. The /sys/devices/vfio_ap/matrix device is created by the device core 2564. The vfio_ap device driver will register with the AP bus for AP queue devices 257 of type 10 and higher (CEX4 and newer). The driver will provide the vfio_ap 258 driver's probe and remove callback interfaces. Devices older than CEX4 queues 259 are not supported to simplify the implementation by not needlessly 260 complicating the design by supporting older devices that will go out of 261 service in the relatively near future, and for which there are few older 262 systems around on which to test. 2635. The AP bus registers the vfio_ap device driver with the device core 2646. The administrator edits the AP adapter and queue masks to reserve AP queues 265 for use by the vfio_ap device driver. 2667. The AP bus removes the AP queues reserved for the vfio_ap driver from the 267 default zcrypt cex4queue driver. 2688. The AP bus probes the vfio_ap device driver to bind the queues reserved for 269 it. 2709. The administrator creates a passthrough type vfio_ap mediated device to be 271 used by a guest 27210. The administrator assigns the adapters, usage domains and control domains 273 to be exclusively used by a guest. 274 275Set up the VFIO mediated device interfaces 276------------------------------------------ 277The VFIO AP device driver utilizes the common interfaces of the VFIO mediated 278device core driver to: 279 280* Register an AP mediated bus driver to add a vfio_ap mediated device to and 281 remove it from a VFIO group. 282* Create and destroy a vfio_ap mediated device 283* Add a vfio_ap mediated device to and remove it from the AP mediated bus driver 284* Add a vfio_ap mediated device to and remove it from an IOMMU group 285 286The following high-level block diagram shows the main components and interfaces 287of the VFIO AP mediated device driver:: 288 289 +-------------+ 290 | | 291 | +---------+ | mdev_register_driver() +--------------+ 292 | | Mdev | +<-----------------------+ | 293 | | bus | | | vfio_mdev.ko | 294 | | driver | +----------------------->+ |<-> VFIO user 295 | +---------+ | probe()/remove() +--------------+ APIs 296 | | 297 | MDEV CORE | 298 | MODULE | 299 | mdev.ko | 300 | +---------+ | mdev_register_parent() +--------------+ 301 | |Physical | +<-----------------------+ | 302 | | device | | | vfio_ap.ko |<-> matrix 303 | |interface| +----------------------->+ | device 304 | +---------+ | callback +--------------+ 305 +-------------+ 306 307During initialization of the vfio_ap module, the matrix device is registered 308with an 'mdev_parent_ops' structure that provides the sysfs attribute 309structures, mdev functions and callback interfaces for managing the mediated 310matrix device. 311 312* sysfs attribute structures: 313 314 supported_type_groups 315 The VFIO mediated device framework supports creation of user-defined 316 mediated device types. These mediated device types are specified 317 via the 'supported_type_groups' structure when a device is registered 318 with the mediated device framework. The registration process creates the 319 sysfs structures for each mediated device type specified in the 320 'mdev_supported_types' sub-directory of the device being registered. Along 321 with the device type, the sysfs attributes of the mediated device type are 322 provided. 323 324 The VFIO AP device driver will register one mediated device type for 325 passthrough devices: 326 327 /sys/devices/vfio_ap/matrix/mdev_supported_types/vfio_ap-passthrough 328 329 Only the read-only attributes required by the VFIO mdev framework will 330 be provided:: 331 332 ... name 333 ... device_api 334 ... available_instances 335 ... device_api 336 337 Where: 338 339 * name: 340 specifies the name of the mediated device type 341 * device_api: 342 the mediated device type's API 343 * available_instances: 344 the number of vfio_ap mediated passthrough devices 345 that can be created 346 * device_api: 347 specifies the VFIO API 348 mdev_attr_groups 349 This attribute group identifies the user-defined sysfs attributes of the 350 mediated device. When a device is registered with the VFIO mediated device 351 framework, the sysfs attribute files identified in the 'mdev_attr_groups' 352 structure will be created in the vfio_ap mediated device's directory. The 353 sysfs attributes for a vfio_ap mediated device are: 354 355 assign_adapter / unassign_adapter: 356 Write-only attributes for assigning/unassigning an AP adapter to/from the 357 vfio_ap mediated device. To assign/unassign an adapter, the APID of the 358 adapter is echoed into the respective attribute file. 359 assign_domain / unassign_domain: 360 Write-only attributes for assigning/unassigning an AP usage domain to/from 361 the vfio_ap mediated device. To assign/unassign a domain, the domain 362 number of the usage domain is echoed into the respective attribute 363 file. 364 matrix: 365 A read-only file for displaying the APQNs derived from the Cartesian 366 product of the adapter and domain numbers assigned to the vfio_ap mediated 367 device. 368 guest_matrix: 369 A read-only file for displaying the APQNs derived from the Cartesian 370 product of the adapter and domain numbers assigned to the APM and AQM 371 fields respectively of the KVM guest's CRYCB. This may differ from the 372 the APQNs assigned to the vfio_ap mediated device if any APQN does not 373 reference a queue device bound to the vfio_ap device driver (i.e., the 374 queue is not in the host's AP configuration). 375 assign_control_domain / unassign_control_domain: 376 Write-only attributes for assigning/unassigning an AP control domain 377 to/from the vfio_ap mediated device. To assign/unassign a control domain, 378 the ID of the domain to be assigned/unassigned is echoed into the 379 respective attribute file. 380 control_domains: 381 A read-only file for displaying the control domain numbers assigned to the 382 vfio_ap mediated device. 383 ap_config: 384 A read/write file that, when written to, allows all three of the 385 vfio_ap mediated device's ap matrix masks to be replaced in one shot. 386 Three masks are given, one for adapters, one for domains, and one for 387 control domains. If the given state cannot be set then no changes are 388 made to the vfio-ap mediated device. 389 390 The format of the data written to ap_config is as follows: 391 {amask},{dmask},{cmask}\n 392 393 \n is a newline character. 394 395 amask, dmask, and cmask are masks identifying which adapters, domains, 396 and control domains should be assigned to the mediated device. 397 398 The format of a mask is as follows: 399 0xNN..NN 400 401 Where NN..NN is 64 hexadecimal characters representing a 256-bit value. 402 The leftmost (highest order) bit represents adapter/domain 0. 403 404 For an example set of masks that represent your mdev's current 405 configuration, simply cat ap_config. 406 407 Setting an adapter or domain number greater than the maximum allowed for 408 the system will result in an error. 409 410 This attribute is intended to be used by automation. End users would be 411 better served using the respective assign/unassign attributes for 412 adapters, domains, and control domains. 413 414* functions: 415 416 create: 417 allocates the ap_matrix_mdev structure used by the vfio_ap driver to: 418 419 * Store the reference to the KVM structure for the guest using the mdev 420 * Store the AP matrix configuration for the adapters, domains, and control 421 domains assigned via the corresponding sysfs attributes files 422 * Store the AP matrix configuration for the adapters, domains and control 423 domains available to a guest. A guest may not be provided access to APQNs 424 referencing queue devices that do not exist, or are not bound to the 425 vfio_ap device driver. 426 427 remove: 428 deallocates the vfio_ap mediated device's ap_matrix_mdev structure. 429 This will be allowed only if a running guest is not using the mdev. 430 431* callback interfaces 432 433 open_device: 434 the open_device callback is invoked by userspace to connect the 435 VFIO iommu group for the matrix mdev device to the MDEV bus. The 436 callback retrieves the KVM structure used to configure the KVM guest 437 and configures the guest's access to the AP matrix defined via the 438 vfio_ap mediated device's sysfs attribute files. 439 440 close_device: 441 this callback deconfigures the guest's AP matrix. 442 443 ioctl: 444 this callback handles the VFIO_DEVICE_GET_INFO and VFIO_DEVICE_RESET ioctls 445 defined by the vfio framework. 446 447Configure the guest's AP resources 448---------------------------------- 449Configuring the AP resources for a KVM guest will be performed at the 450time of ``open_device`` and ``close_device``. The guest's AP resources are 451configured via its APCB by: 452 453* Setting the bits in the APM corresponding to the APIDs assigned to the 454 vfio_ap mediated device via its 'assign_adapter' interface. 455* Setting the bits in the AQM corresponding to the domains assigned to the 456 vfio_ap mediated device via its 'assign_domain' interface. 457* Setting the bits in the ADM corresponding to the domain dIDs assigned to the 458 vfio_ap mediated device via its 'assign_control_domains' interface. 459 460The linux device model precludes passing a device through to a KVM guest that 461is not bound to the device driver facilitating its pass-through. Consequently, 462an APQN that does not reference a queue device bound to the vfio_ap device 463driver will not be assigned to a KVM guest's matrix. The AP architecture, 464however, does not provide a means to filter individual APQNs from the guest's 465matrix, so the adapters, domains and control domains assigned to vfio_ap 466mediated device via its sysfs 'assign_adapter', 'assign_domain' and 467'assign_control_domain' interfaces will be filtered before providing the AP 468configuration to a guest: 469 470* The APIDs of the adapters, the APQIs of the domains and the domain numbers of 471 the control domains assigned to the matrix mdev that are not also assigned to 472 the host's AP configuration will be filtered. 473 474* Each APQN derived from the Cartesian product of the APIDs and APQIs assigned 475 to the vfio_ap mdev is examined and if any one of them does not reference a 476 queue device bound to the vfio_ap device driver, the adapter will not be 477 plugged into the guest (i.e., the bit corresponding to its APID will not be 478 set in the APM of the guest's APCB). 479 480The CPU model features for AP 481----------------------------- 482The AP stack relies on the presence of the AP instructions as well as three 483facilities: The AP Facilities Test (APFT) facility; the AP Query 484Configuration Information (QCI) facility; and the AP Queue Interruption Control 485facility. These features/facilities are made available to a KVM guest via the 486following CPU model features: 487 4881. ap: Indicates whether the AP instructions are installed on the guest. This 489 feature will be enabled by KVM only if the AP instructions are installed 490 on the host. 491 4922. apft: Indicates the APFT facility is available on the guest. This facility 493 can be made available to the guest only if it is available on the host (i.e., 494 facility bit 15 is set). 495 4963. apqci: Indicates the AP QCI facility is available on the guest. This facility 497 can be made available to the guest only if it is available on the host (i.e., 498 facility bit 12 is set). 499 5004. apqi: Indicates AP Queue Interruption Control faclity is available on the 501 guest. This facility can be made available to the guest only if it is 502 available on the host (i.e., facility bit 65 is set). 503 504Note: If the user chooses to specify a CPU model different than the 'host' 505model to QEMU, the CPU model features and facilities need to be turned on 506explicitly; for example:: 507 508 /usr/bin/qemu-system-s390x ... -cpu z13,ap=on,apqci=on,apft=on,apqi=on 509 510A guest can be precluded from using AP features/facilities by turning them off 511explicitly; for example:: 512 513 /usr/bin/qemu-system-s390x ... -cpu host,ap=off,apqci=off,apft=off,apqi=off 514 515Note: If the APFT facility is turned off (apft=off) for the guest, the guest 516will not see any AP devices. The zcrypt device drivers on the guest that 517register for type 10 and newer AP devices - i.e., the cex4card and cex4queue 518device drivers - need the APFT facility to ascertain the facilities installed on 519a given AP device. If the APFT facility is not installed on the guest, then no 520adapter or domain devices will get created by the AP bus running on the 521guest because only type 10 and newer devices can be configured for guest use. 522 523Example 524======= 525Let's now provide an example to illustrate how KVM guests may be given 526access to AP facilities. For this example, we will show how to configure 527three guests such that executing the lszcrypt command on the guests would 528look like this: 529 530Guest1 531------ 532=========== ===== ============ 533CARD.DOMAIN TYPE MODE 534=========== ===== ============ 53505 CEX5C CCA-Coproc 53605.0004 CEX5C CCA-Coproc 53705.00ab CEX5C CCA-Coproc 53806 CEX5A Accelerator 53906.0004 CEX5A Accelerator 54006.00ab CEX5A Accelerator 541=========== ===== ============ 542 543Guest2 544------ 545=========== ===== ============ 546CARD.DOMAIN TYPE MODE 547=========== ===== ============ 54805 CEX5C CCA-Coproc 54905.0047 CEX5C CCA-Coproc 55005.00ff CEX5C CCA-Coproc 551=========== ===== ============ 552 553Guest3 554------ 555=========== ===== ============ 556CARD.DOMAIN TYPE MODE 557=========== ===== ============ 55806 CEX5A Accelerator 55906.0047 CEX5A Accelerator 56006.00ff CEX5A Accelerator 561=========== ===== ============ 562 563These are the steps: 564 5651. Install the vfio_ap module on the linux host. The dependency chain for the 566 vfio_ap module is: 567 * iommu 568 * s390 569 * zcrypt 570 * vfio 571 * vfio_mdev 572 * vfio_mdev_device 573 * KVM 574 575 To build the vfio_ap module, the kernel build must be configured with the 576 following Kconfig elements selected: 577 * IOMMU_SUPPORT 578 * S390 579 * AP 580 * VFIO 581 * KVM 582 583 If using make menuconfig select the following to build the vfio_ap module:: 584 585 -> Device Drivers 586 -> IOMMU Hardware Support 587 select S390 AP IOMMU Support 588 -> VFIO Non-Privileged userspace driver framework 589 -> Mediated device driver frramework 590 -> VFIO driver for Mediated devices 591 -> I/O subsystem 592 -> VFIO support for AP devices 593 5942. Secure the AP queues to be used by the three guests so that the host can not 595 access them. To secure them, there are two sysfs files that specify 596 bitmasks marking a subset of the APQN range as usable only by the default AP 597 queue device drivers. All remaining APQNs are available for use by 598 any other device driver. The vfio_ap device driver is currently the only 599 non-default device driver. The location of the sysfs files containing the 600 masks are:: 601 602 /sys/bus/ap/apmask 603 /sys/bus/ap/aqmask 604 605 The 'apmask' is a 256-bit mask that identifies a set of AP adapter IDs 606 (APID). Each bit in the mask, from left to right, corresponds to an APID from 607 0-255. If a bit is set, the APID belongs to the subset of APQNs marked as 608 available only to the default AP queue device drivers. 609 610 The 'aqmask' is a 256-bit mask that identifies a set of AP queue indexes 611 (APQI). Each bit in the mask, from left to right, corresponds to an APQI from 612 0-255. If a bit is set, the APQI belongs to the subset of APQNs marked as 613 available only to the default AP queue device drivers. 614 615 The Cartesian product of the APIDs corresponding to the bits set in the 616 apmask and the APQIs corresponding to the bits set in the aqmask comprise 617 the subset of APQNs that can be used only by the host default device drivers. 618 All other APQNs are available to the non-default device drivers such as the 619 vfio_ap driver. 620 621 Take, for example, the following masks:: 622 623 apmask: 624 0x7d00000000000000000000000000000000000000000000000000000000000000 625 626 aqmask: 627 0x8000000000000000000000000000000000000000000000000000000000000000 628 629 The masks indicate: 630 631 * Adapters 1, 2, 3, 4, 5, and 7 are available for use by the host default 632 device drivers. 633 634 * Domain 0 is available for use by the host default device drivers 635 636 * The subset of APQNs available for use only by the default host device 637 drivers are: 638 639 (1,0), (2,0), (3,0), (4.0), (5,0) and (7,0) 640 641 * All other APQNs are available for use by the non-default device drivers. 642 643 The APQN of each AP queue device assigned to the linux host is checked by the 644 AP bus against the set of APQNs derived from the Cartesian product of APIDs 645 and APQIs marked as available to the default AP queue device drivers. If a 646 match is detected, only the default AP queue device drivers will be probed; 647 otherwise, the vfio_ap device driver will be probed. 648 649 By default, the two masks are set to reserve all APQNs for use by the default 650 AP queue device drivers. There are two ways the default masks can be changed: 651 652 1. The sysfs mask files can be edited by echoing a string into the 653 respective sysfs mask file in one of two formats: 654 655 * An absolute hex string starting with 0x - like "0x12345678" - sets 656 the mask. If the given string is shorter than the mask, it is padded 657 with 0s on the right; for example, specifying a mask value of 0x41 is 658 the same as specifying:: 659 660 0x4100000000000000000000000000000000000000000000000000000000000000 661 662 Keep in mind that the mask reads from left to right, so the mask 663 above identifies device numbers 1 and 7 (01000001). 664 665 If the string is longer than the mask, the operation is terminated with 666 an error (EINVAL). 667 668 * Individual bits in the mask can be switched on and off by specifying 669 each bit number to be switched in a comma separated list. Each bit 670 number string must be prepended with a ('+') or minus ('-') to indicate 671 the corresponding bit is to be switched on ('+') or off ('-'). Some 672 valid values are: 673 674 - "+0" switches bit 0 on 675 - "-13" switches bit 13 off 676 - "+0x41" switches bit 65 on 677 - "-0xff" switches bit 255 off 678 679 The following example: 680 681 +0,-6,+0x47,-0xf0 682 683 Switches bits 0 and 71 (0x47) on 684 685 Switches bits 6 and 240 (0xf0) off 686 687 Note that the bits not specified in the list remain as they were before 688 the operation. 689 690 2. The masks can also be changed at boot time via parameters on the kernel 691 command line like this: 692 693 ap.apmask=0xffff ap.aqmask=0x40 694 695 This would create the following masks:: 696 697 apmask: 698 0xffff000000000000000000000000000000000000000000000000000000000000 699 700 aqmask: 701 0x4000000000000000000000000000000000000000000000000000000000000000 702 703 Resulting in these two pools:: 704 705 default drivers pool: adapter 0-15, domain 1 706 alternate drivers pool: adapter 16-255, domains 0, 2-255 707 708 **Note:** 709 Changing a mask such that one or more APQNs will be taken from a vfio_ap 710 mediated device (see below) will fail with an error (EBUSY). A message 711 is logged to the kernel ring buffer which can be viewed with the 'dmesg' 712 command. The output identifies each APQN flagged as 'in use' and identifies 713 the vfio_ap mediated device to which it is assigned; for example: 714 715 Userspace may not re-assign queue 05.0054 already assigned to 62177883-f1bb-47f0-914d-32a22e3a8804 716 Userspace may not re-assign queue 04.0054 already assigned to cef03c3c-903d-4ecc-9a83-40694cb8aee4 717 718Securing the APQNs for our example 719---------------------------------- 720 To secure the AP queues 05.0004, 05.0047, 05.00ab, 05.00ff, 06.0004, 06.0047, 721 06.00ab, and 06.00ff for use by the vfio_ap device driver, the corresponding 722 APQNs can be removed from the default masks using either of the following 723 commands:: 724 725 echo -5,-6 > /sys/bus/ap/apmask 726 727 echo -4,-0x47,-0xab,-0xff > /sys/bus/ap/aqmask 728 729 Or the masks can be set as follows:: 730 731 echo 0xf9ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff \ 732 > apmask 733 734 echo 0xf7fffffffffffffffeffffffffffffffffffffffffeffffffffffffffffffffe \ 735 > aqmask 736 737 This will result in AP queues 05.0004, 05.0047, 05.00ab, 05.00ff, 06.0004, 738 06.0047, 06.00ab, and 06.00ff getting bound to the vfio_ap device driver. The 739 sysfs directory for the vfio_ap device driver will now contain symbolic links 740 to the AP queue devices bound to it:: 741 742 /sys/bus/ap 743 ... [drivers] 744 ...... [vfio_ap] 745 ......... [05.0004] 746 ......... [05.0047] 747 ......... [05.00ab] 748 ......... [05.00ff] 749 ......... [06.0004] 750 ......... [06.0047] 751 ......... [06.00ab] 752 ......... [06.00ff] 753 754 Keep in mind that only type 10 and newer adapters (i.e., CEX4 and later) 755 can be bound to the vfio_ap device driver. The reason for this is to 756 simplify the implementation by not needlessly complicating the design by 757 supporting older devices that will go out of service in the relatively near 758 future and for which there are few older systems on which to test. 759 760 The administrator, therefore, must take care to secure only AP queues that 761 can be bound to the vfio_ap device driver. The device type for a given AP 762 queue device can be read from the parent card's sysfs directory. For example, 763 to see the hardware type of the queue 05.0004: 764 765 cat /sys/bus/ap/devices/card05/hwtype 766 767 The hwtype must be 10 or higher (CEX4 or newer) in order to be bound to the 768 vfio_ap device driver. 769 7703. Create the mediated devices needed to configure the AP matrixes for the 771 three guests and to provide an interface to the vfio_ap driver for 772 use by the guests:: 773 774 /sys/devices/vfio_ap/matrix/ 775 --- [mdev_supported_types] 776 ------ [vfio_ap-passthrough] (passthrough vfio_ap mediated device type) 777 --------- create 778 --------- [devices] 779 780 To create the mediated devices for the three guests:: 781 782 uuidgen > create 783 uuidgen > create 784 uuidgen > create 785 786 or 787 788 echo $uuid1 > create 789 echo $uuid2 > create 790 echo $uuid3 > create 791 792 This will create three mediated devices in the [devices] subdirectory named 793 after the UUID written to the create attribute file. We call them $uuid1, 794 $uuid2 and $uuid3 and this is the sysfs directory structure after creation:: 795 796 /sys/devices/vfio_ap/matrix/ 797 --- [mdev_supported_types] 798 ------ [vfio_ap-passthrough] 799 --------- [devices] 800 ------------ [$uuid1] 801 --------------- assign_adapter 802 --------------- assign_control_domain 803 --------------- assign_domain 804 --------------- matrix 805 --------------- unassign_adapter 806 --------------- unassign_control_domain 807 --------------- unassign_domain 808 809 ------------ [$uuid2] 810 --------------- assign_adapter 811 --------------- assign_control_domain 812 --------------- assign_domain 813 --------------- matrix 814 --------------- unassign_adapter 815 ----------------unassign_control_domain 816 ----------------unassign_domain 817 818 ------------ [$uuid3] 819 --------------- assign_adapter 820 --------------- assign_control_domain 821 --------------- assign_domain 822 --------------- matrix 823 --------------- unassign_adapter 824 ----------------unassign_control_domain 825 ----------------unassign_domain 826 827 Note *****: The vfio_ap mdevs do not persist across reboots unless the 828 mdevctl tool is used to create and persist them. 829 8304. The administrator now needs to configure the matrixes for the mediated 831 devices $uuid1 (for Guest1), $uuid2 (for Guest2) and $uuid3 (for Guest3). 832 833 This is how the matrix is configured for Guest1:: 834 835 echo 5 > assign_adapter 836 echo 6 > assign_adapter 837 echo 4 > assign_domain 838 echo 0xab > assign_domain 839 840 Control domains can similarly be assigned using the assign_control_domain 841 sysfs file. 842 843 If a mistake is made configuring an adapter, domain or control domain, 844 you can use the unassign_xxx files to unassign the adapter, domain or 845 control domain. 846 847 To display the matrix configuration for Guest1:: 848 849 cat matrix 850 851 To display the matrix that is or will be assigned to Guest1:: 852 853 cat guest_matrix 854 855 This is how the matrix is configured for Guest2:: 856 857 echo 5 > assign_adapter 858 echo 0x47 > assign_domain 859 echo 0xff > assign_domain 860 861 This is how the matrix is configured for Guest3:: 862 863 echo 6 > assign_adapter 864 echo 0x47 > assign_domain 865 echo 0xff > assign_domain 866 867 In order to successfully assign an adapter: 868 869 * The adapter number specified must represent a value from 0 up to the 870 maximum adapter number configured for the system. If an adapter number 871 higher than the maximum is specified, the operation will terminate with 872 an error (ENODEV). 873 874 Note: The maximum adapter number can be obtained via the sysfs 875 /sys/bus/ap/ap_max_adapter_id attribute file. 876 877 * Each APQN derived from the Cartesian product of the APID of the adapter 878 being assigned and the APQIs of the domains previously assigned: 879 880 - Must only be available to the vfio_ap device driver as specified in the 881 sysfs /sys/bus/ap/apmask and /sys/bus/ap/aqmask attribute files. If even 882 one APQN is reserved for use by the host device driver, the operation 883 will terminate with an error (EADDRNOTAVAIL). 884 885 - Must NOT be assigned to another vfio_ap mediated device. If even one APQN 886 is assigned to another vfio_ap mediated device, the operation will 887 terminate with an error (EBUSY). 888 889 - Must NOT be assigned while the sysfs /sys/bus/ap/apmask and 890 sys/bus/ap/aqmask attribute files are being edited or the operation may 891 terminate with an error (EBUSY). 892 893 In order to successfully assign a domain: 894 895 * The domain number specified must represent a value from 0 up to the 896 maximum domain number configured for the system. If a domain number 897 higher than the maximum is specified, the operation will terminate with 898 an error (ENODEV). 899 900 Note: The maximum domain number can be obtained via the sysfs 901 /sys/bus/ap/ap_max_domain_id attribute file. 902 903 * Each APQN derived from the Cartesian product of the APQI of the domain 904 being assigned and the APIDs of the adapters previously assigned: 905 906 - Must only be available to the vfio_ap device driver as specified in the 907 sysfs /sys/bus/ap/apmask and /sys/bus/ap/aqmask attribute files. If even 908 one APQN is reserved for use by the host device driver, the operation 909 will terminate with an error (EADDRNOTAVAIL). 910 911 - Must NOT be assigned to another vfio_ap mediated device. If even one APQN 912 is assigned to another vfio_ap mediated device, the operation will 913 terminate with an error (EBUSY). 914 915 - Must NOT be assigned while the sysfs /sys/bus/ap/apmask and 916 sys/bus/ap/aqmask attribute files are being edited or the operation may 917 terminate with an error (EBUSY). 918 919 In order to successfully assign a control domain: 920 921 * The domain number specified must represent a value from 0 up to the maximum 922 domain number configured for the system. If a control domain number higher 923 than the maximum is specified, the operation will terminate with an 924 error (ENODEV). 925 9265. Start Guest1:: 927 928 /usr/bin/qemu-system-s390x ... -cpu host,ap=on,apqci=on,apft=on,apqi=on \ 929 -device vfio-ap,sysfsdev=/sys/devices/vfio_ap/matrix/$uuid1 ... 930 9317. Start Guest2:: 932 933 /usr/bin/qemu-system-s390x ... -cpu host,ap=on,apqci=on,apft=on,apqi=on \ 934 -device vfio-ap,sysfsdev=/sys/devices/vfio_ap/matrix/$uuid2 ... 935 9367. Start Guest3:: 937 938 /usr/bin/qemu-system-s390x ... -cpu host,ap=on,apqci=on,apft=on,apqi=on \ 939 -device vfio-ap,sysfsdev=/sys/devices/vfio_ap/matrix/$uuid3 ... 940 941When the guest is shut down, the vfio_ap mediated devices may be removed. 942 943Using our example again, to remove the vfio_ap mediated device $uuid1:: 944 945 /sys/devices/vfio_ap/matrix/ 946 --- [mdev_supported_types] 947 ------ [vfio_ap-passthrough] 948 --------- [devices] 949 ------------ [$uuid1] 950 --------------- remove 951 952:: 953 954 echo 1 > remove 955 956This will remove all of the matrix mdev device's sysfs structures including 957the mdev device itself. To recreate and reconfigure the matrix mdev device, 958all of the steps starting with step 3 will have to be performed again. Note 959that the remove will fail if a guest using the vfio_ap mdev is still running. 960 961It is not necessary to remove a vfio_ap mdev, but one may want to 962remove it if no guest will use it during the remaining lifetime of the linux 963host. If the vfio_ap mdev is removed, one may want to also reconfigure 964the pool of adapters and queues reserved for use by the default drivers. 965 966Hot plug/unplug support: 967======================== 968An adapter, domain or control domain may be hot plugged into a running KVM 969guest by assigning it to the vfio_ap mediated device being used by the guest if 970the following conditions are met: 971 972* The adapter, domain or control domain must also be assigned to the host's 973 AP configuration. 974 975* Each APQN derived from the Cartesian product comprised of the APID of the 976 adapter being assigned and the APQIs of the domains assigned must reference a 977 queue device bound to the vfio_ap device driver. 978 979* To hot plug a domain, each APQN derived from the Cartesian product 980 comprised of the APQI of the domain being assigned and the APIDs of the 981 adapters assigned must reference a queue device bound to the vfio_ap device 982 driver. 983 984An adapter, domain or control domain may be hot unplugged from a running KVM 985guest by unassigning it from the vfio_ap mediated device being used by the 986guest. 987 988Over-provisioning of AP queues for a KVM guest: 989=============================================== 990Over-provisioning is defined herein as the assignment of adapters or domains to 991a vfio_ap mediated device that do not reference AP devices in the host's AP 992configuration. The idea here is that when the adapter or domain becomes 993available, it will be automatically hot-plugged into the KVM guest using 994the vfio_ap mediated device to which it is assigned as long as each new APQN 995resulting from plugging it in references a queue device bound to the vfio_ap 996device driver. 997 998Driver Features 999=============== 1000The vfio_ap driver exposes a sysfs file containing supported features. 1001This exists so third party tools (like Libvirt and mdevctl) can query the 1002availability of specific features. 1003 1004The features list can be found here: /sys/bus/matrix/devices/matrix/features 1005 1006Entries are space delimited. Each entry consists of a combination of 1007alphanumeric and underscore characters. 1008 1009Example: 1010cat /sys/bus/matrix/devices/matrix/features 1011guest_matrix dyn ap_config 1012 1013the following features are advertised: 1014 1015---------------+---------------------------------------------------------------+ 1016| Flag | Description | 1017+==============+===============================================================+ 1018| guest_matrix | guest_matrix attribute exists. It reports the matrix of | 1019| | adapters and domains that are or will be passed through to a | 1020| | guest when the mdev is attached to it. | 1021+--------------+---------------------------------------------------------------+ 1022| dyn | Indicates hot plug/unplug of AP adapters, domains and control | 1023| | domains for a guest to which the mdev is attached. | 1024+------------+-----------------------------------------------------------------+ 1025| ap_config | ap_config interface for one-shot modifications to mdev config | 1026+--------------+---------------------------------------------------------------+ 1027 1028Limitations 1029=========== 1030Live guest migration is not supported for guests using AP devices without 1031intervention by a system administrator. Before a KVM guest can be migrated, 1032the vfio_ap mediated device must be removed. Unfortunately, it can not be 1033removed manually (i.e., echo 1 > /sys/devices/vfio_ap/matrix/$UUID/remove) while 1034the mdev is in use by a KVM guest. If the guest is being emulated by QEMU, 1035its mdev can be hot unplugged from the guest in one of two ways: 1036 10371. If the KVM guest was started with libvirt, you can hot unplug the mdev via 1038 the following commands: 1039 1040 virsh detach-device <guestname> <path-to-device-xml> 1041 1042 For example, to hot unplug mdev 62177883-f1bb-47f0-914d-32a22e3a8804 from 1043 the guest named 'my-guest': 1044 1045 virsh detach-device my-guest ~/config/my-guest-hostdev.xml 1046 1047 The contents of my-guest-hostdev.xml: 1048 1049.. code-block:: xml 1050 1051 <hostdev mode='subsystem' type='mdev' managed='no' model='vfio-ap'> 1052 <source> 1053 <address uuid='62177883-f1bb-47f0-914d-32a22e3a8804'/> 1054 </source> 1055 </hostdev> 1056 1057 1058 virsh qemu-monitor-command <guest-name> --hmp "device-del <device-id>" 1059 1060 For example, to hot unplug the vfio_ap mediated device identified on the 1061 qemu command line with 'id=hostdev0' from the guest named 'my-guest': 1062 1063.. code-block:: sh 1064 1065 virsh qemu-monitor-command my-guest --hmp "device_del hostdev0" 1066 10672. A vfio_ap mediated device can be hot unplugged by attaching the qemu monitor 1068 to the guest and using the following qemu monitor command: 1069 1070 (QEMU) device-del id=<device-id> 1071 1072 For example, to hot unplug the vfio_ap mediated device that was specified 1073 on the qemu command line with 'id=hostdev0' when the guest was started: 1074 1075 (QEMU) device-del id=hostdev0 1076 1077After live migration of the KVM guest completes, an AP configuration can be 1078restored to the KVM guest by hot plugging a vfio_ap mediated device on the target 1079system into the guest in one of two ways: 1080 10811. If the KVM guest was started with libvirt, you can hot plug a matrix mediated 1082 device into the guest via the following virsh commands: 1083 1084 virsh attach-device <guestname> <path-to-device-xml> 1085 1086 For example, to hot plug mdev 62177883-f1bb-47f0-914d-32a22e3a8804 into 1087 the guest named 'my-guest': 1088 1089 virsh attach-device my-guest ~/config/my-guest-hostdev.xml 1090 1091 The contents of my-guest-hostdev.xml: 1092 1093.. code-block:: xml 1094 1095 <hostdev mode='subsystem' type='mdev' managed='no' model='vfio-ap'> 1096 <source> 1097 <address uuid='62177883-f1bb-47f0-914d-32a22e3a8804'/> 1098 </source> 1099 </hostdev> 1100 1101 1102 virsh qemu-monitor-command <guest-name> --hmp \ 1103 "device_add vfio-ap,sysfsdev=<path-to-mdev>,id=<device-id>" 1104 1105 For example, to hot plug the vfio_ap mediated device 1106 62177883-f1bb-47f0-914d-32a22e3a8804 into the guest named 'my-guest' with 1107 device-id hostdev0: 1108 1109 virsh qemu-monitor-command my-guest --hmp \ 1110 "device_add vfio-ap,\ 1111 sysfsdev=/sys/devices/vfio_ap/matrix/62177883-f1bb-47f0-914d-32a22e3a8804,\ 1112 id=hostdev0" 1113 11142. A vfio_ap mediated device can be hot plugged by attaching the qemu monitor 1115 to the guest and using the following qemu monitor command: 1116 1117 (qemu) device_add "vfio-ap,sysfsdev=<path-to-mdev>,id=<device-id>" 1118 1119 For example, to plug the vfio_ap mediated device 1120 62177883-f1bb-47f0-914d-32a22e3a8804 into the guest with the device-id 1121 hostdev0: 1122 1123 (QEMU) device-add "vfio-ap,\ 1124 sysfsdev=/sys/devices/vfio_ap/matrix/62177883-f1bb-47f0-914d-32a22e3a8804,\ 1125 id=hostdev0" 1126