1*2dbc0838SMauro Carvalho Chehab======================================================== 2*2dbc0838SMauro Carvalho ChehabOpenCAPI (Open Coherent Accelerator Processor Interface) 3*2dbc0838SMauro Carvalho Chehab======================================================== 4*2dbc0838SMauro Carvalho Chehab 5*2dbc0838SMauro Carvalho ChehabOpenCAPI is an interface between processors and accelerators. It aims 6*2dbc0838SMauro Carvalho Chehabat being low-latency and high-bandwidth. The specification is 7*2dbc0838SMauro Carvalho Chehabdeveloped by the `OpenCAPI Consortium <http://opencapi.org/>`_. 8*2dbc0838SMauro Carvalho Chehab 9*2dbc0838SMauro Carvalho ChehabIt allows an accelerator (which could be a FPGA, ASICs, ...) to access 10*2dbc0838SMauro Carvalho Chehabthe host memory coherently, using virtual addresses. An OpenCAPI 11*2dbc0838SMauro Carvalho Chehabdevice can also host its own memory, that can be accessed from the 12*2dbc0838SMauro Carvalho Chehabhost. 13*2dbc0838SMauro Carvalho Chehab 14*2dbc0838SMauro Carvalho ChehabOpenCAPI is known in linux as 'ocxl', as the open, processor-agnostic 15*2dbc0838SMauro Carvalho Chehabevolution of 'cxl' (the driver for the IBM CAPI interface for 16*2dbc0838SMauro Carvalho Chehabpowerpc), which was named that way to avoid confusion with the ISDN 17*2dbc0838SMauro Carvalho ChehabCAPI subsystem. 18*2dbc0838SMauro Carvalho Chehab 19*2dbc0838SMauro Carvalho Chehab 20*2dbc0838SMauro Carvalho ChehabHigh-level view 21*2dbc0838SMauro Carvalho Chehab=============== 22*2dbc0838SMauro Carvalho Chehab 23*2dbc0838SMauro Carvalho ChehabOpenCAPI defines a Data Link Layer (DL) and Transaction Layer (TL), to 24*2dbc0838SMauro Carvalho Chehabbe implemented on top of a physical link. Any processor or device 25*2dbc0838SMauro Carvalho Chehabimplementing the DL and TL can start sharing memory. 26*2dbc0838SMauro Carvalho Chehab 27*2dbc0838SMauro Carvalho Chehab:: 28*2dbc0838SMauro Carvalho Chehab 29*2dbc0838SMauro Carvalho Chehab +-----------+ +-------------+ 30*2dbc0838SMauro Carvalho Chehab | | | | 31*2dbc0838SMauro Carvalho Chehab | | | Accelerated | 32*2dbc0838SMauro Carvalho Chehab | Processor | | Function | 33*2dbc0838SMauro Carvalho Chehab | | +--------+ | Unit | +--------+ 34*2dbc0838SMauro Carvalho Chehab | |--| Memory | | (AFU) |--| Memory | 35*2dbc0838SMauro Carvalho Chehab | | +--------+ | | +--------+ 36*2dbc0838SMauro Carvalho Chehab +-----------+ +-------------+ 37*2dbc0838SMauro Carvalho Chehab | | 38*2dbc0838SMauro Carvalho Chehab +-----------+ +-------------+ 39*2dbc0838SMauro Carvalho Chehab | TL | | TLX | 40*2dbc0838SMauro Carvalho Chehab +-----------+ +-------------+ 41*2dbc0838SMauro Carvalho Chehab | | 42*2dbc0838SMauro Carvalho Chehab +-----------+ +-------------+ 43*2dbc0838SMauro Carvalho Chehab | DL | | DLX | 44*2dbc0838SMauro Carvalho Chehab +-----------+ +-------------+ 45*2dbc0838SMauro Carvalho Chehab | | 46*2dbc0838SMauro Carvalho Chehab | PHY | 47*2dbc0838SMauro Carvalho Chehab +---------------------------------------+ 48*2dbc0838SMauro Carvalho Chehab 49*2dbc0838SMauro Carvalho Chehab 50*2dbc0838SMauro Carvalho Chehab 51*2dbc0838SMauro Carvalho ChehabDevice discovery 52*2dbc0838SMauro Carvalho Chehab================ 53*2dbc0838SMauro Carvalho Chehab 54*2dbc0838SMauro Carvalho ChehabOpenCAPI relies on a PCI-like configuration space, implemented on the 55*2dbc0838SMauro Carvalho Chehabdevice. So the host can discover AFUs by querying the config space. 56*2dbc0838SMauro Carvalho Chehab 57*2dbc0838SMauro Carvalho ChehabOpenCAPI devices in Linux are treated like PCI devices (with a few 58*2dbc0838SMauro Carvalho Chehabcaveats). The firmware is expected to abstract the hardware as if it 59*2dbc0838SMauro Carvalho Chehabwas a PCI link. A lot of the existing PCI infrastructure is reused: 60*2dbc0838SMauro Carvalho Chehabdevices are scanned and BARs are assigned during the standard PCI 61*2dbc0838SMauro Carvalho Chehabenumeration. Commands like 'lspci' can therefore be used to see what 62*2dbc0838SMauro Carvalho Chehabdevices are available. 63*2dbc0838SMauro Carvalho Chehab 64*2dbc0838SMauro Carvalho ChehabThe configuration space defines the AFU(s) that can be found on the 65*2dbc0838SMauro Carvalho Chehabphysical adapter, such as its name, how many memory contexts it can 66*2dbc0838SMauro Carvalho Chehabwork with, the size of its MMIO areas, ... 67*2dbc0838SMauro Carvalho Chehab 68*2dbc0838SMauro Carvalho Chehab 69*2dbc0838SMauro Carvalho Chehab 70*2dbc0838SMauro Carvalho ChehabMMIO 71*2dbc0838SMauro Carvalho Chehab==== 72*2dbc0838SMauro Carvalho Chehab 73*2dbc0838SMauro Carvalho ChehabOpenCAPI defines two MMIO areas for each AFU: 74*2dbc0838SMauro Carvalho Chehab 75*2dbc0838SMauro Carvalho Chehab* the global MMIO area, with registers pertinent to the whole AFU. 76*2dbc0838SMauro Carvalho Chehab* a per-process MMIO area, which has a fixed size for each context. 77*2dbc0838SMauro Carvalho Chehab 78*2dbc0838SMauro Carvalho Chehab 79*2dbc0838SMauro Carvalho Chehab 80*2dbc0838SMauro Carvalho ChehabAFU interrupts 81*2dbc0838SMauro Carvalho Chehab============== 82*2dbc0838SMauro Carvalho Chehab 83*2dbc0838SMauro Carvalho ChehabOpenCAPI includes the possibility for an AFU to send an interrupt to a 84*2dbc0838SMauro Carvalho Chehabhost process. It is done through a 'intrp_req' defined in the 85*2dbc0838SMauro Carvalho ChehabTransaction Layer, specifying a 64-bit object handle which defines the 86*2dbc0838SMauro Carvalho Chehabinterrupt. 87*2dbc0838SMauro Carvalho Chehab 88*2dbc0838SMauro Carvalho ChehabThe driver allows a process to allocate an interrupt and obtain its 89*2dbc0838SMauro Carvalho Chehab64-bit object handle, that can be passed to the AFU. 90*2dbc0838SMauro Carvalho Chehab 91*2dbc0838SMauro Carvalho Chehab 92*2dbc0838SMauro Carvalho Chehab 93*2dbc0838SMauro Carvalho Chehabchar devices 94*2dbc0838SMauro Carvalho Chehab============ 95*2dbc0838SMauro Carvalho Chehab 96*2dbc0838SMauro Carvalho ChehabThe driver creates one char device per AFU found on the physical 97*2dbc0838SMauro Carvalho Chehabdevice. A physical device may have multiple functions and each 98*2dbc0838SMauro Carvalho Chehabfunction can have multiple AFUs. At the time of this writing though, 99*2dbc0838SMauro Carvalho Chehabit has only been tested with devices exporting only one AFU. 100*2dbc0838SMauro Carvalho Chehab 101*2dbc0838SMauro Carvalho ChehabChar devices can be found in /dev/ocxl/ and are named as: 102*2dbc0838SMauro Carvalho Chehab/dev/ocxl/<AFU name>.<location>.<index> 103*2dbc0838SMauro Carvalho Chehab 104*2dbc0838SMauro Carvalho Chehabwhere <AFU name> is a max 20-character long name, as found in the 105*2dbc0838SMauro Carvalho Chehabconfig space of the AFU. 106*2dbc0838SMauro Carvalho Chehab<location> is added by the driver and can help distinguish devices 107*2dbc0838SMauro Carvalho Chehabwhen a system has more than one instance of the same OpenCAPI device. 108*2dbc0838SMauro Carvalho Chehab<index> is also to help distinguish AFUs in the unlikely case where a 109*2dbc0838SMauro Carvalho Chehabdevice carries multiple copies of the same AFU. 110*2dbc0838SMauro Carvalho Chehab 111*2dbc0838SMauro Carvalho Chehab 112*2dbc0838SMauro Carvalho Chehab 113*2dbc0838SMauro Carvalho ChehabSysfs class 114*2dbc0838SMauro Carvalho Chehab=========== 115*2dbc0838SMauro Carvalho Chehab 116*2dbc0838SMauro Carvalho ChehabAn ocxl class is added for the devices representing the AFUs. See 117*2dbc0838SMauro Carvalho Chehab/sys/class/ocxl. The layout is described in 118*2dbc0838SMauro Carvalho ChehabDocumentation/ABI/testing/sysfs-class-ocxl 119*2dbc0838SMauro Carvalho Chehab 120*2dbc0838SMauro Carvalho Chehab 121*2dbc0838SMauro Carvalho Chehab 122*2dbc0838SMauro Carvalho ChehabUser API 123*2dbc0838SMauro Carvalho Chehab======== 124*2dbc0838SMauro Carvalho Chehab 125*2dbc0838SMauro Carvalho Chehabopen 126*2dbc0838SMauro Carvalho Chehab---- 127*2dbc0838SMauro Carvalho Chehab 128*2dbc0838SMauro Carvalho ChehabBased on the AFU definition found in the config space, an AFU may 129*2dbc0838SMauro Carvalho Chehabsupport working with more than one memory context, in which case the 130*2dbc0838SMauro Carvalho Chehabassociated char device may be opened multiple times by different 131*2dbc0838SMauro Carvalho Chehabprocesses. 132*2dbc0838SMauro Carvalho Chehab 133*2dbc0838SMauro Carvalho Chehab 134*2dbc0838SMauro Carvalho Chehabioctl 135*2dbc0838SMauro Carvalho Chehab----- 136*2dbc0838SMauro Carvalho Chehab 137*2dbc0838SMauro Carvalho ChehabOCXL_IOCTL_ATTACH: 138*2dbc0838SMauro Carvalho Chehab 139*2dbc0838SMauro Carvalho Chehab Attach the memory context of the calling process to the AFU so that 140*2dbc0838SMauro Carvalho Chehab the AFU can access its memory. 141*2dbc0838SMauro Carvalho Chehab 142*2dbc0838SMauro Carvalho ChehabOCXL_IOCTL_IRQ_ALLOC: 143*2dbc0838SMauro Carvalho Chehab 144*2dbc0838SMauro Carvalho Chehab Allocate an AFU interrupt and return an identifier. 145*2dbc0838SMauro Carvalho Chehab 146*2dbc0838SMauro Carvalho ChehabOCXL_IOCTL_IRQ_FREE: 147*2dbc0838SMauro Carvalho Chehab 148*2dbc0838SMauro Carvalho Chehab Free a previously allocated AFU interrupt. 149*2dbc0838SMauro Carvalho Chehab 150*2dbc0838SMauro Carvalho ChehabOCXL_IOCTL_IRQ_SET_FD: 151*2dbc0838SMauro Carvalho Chehab 152*2dbc0838SMauro Carvalho Chehab Associate an event fd to an AFU interrupt so that the user process 153*2dbc0838SMauro Carvalho Chehab can be notified when the AFU sends an interrupt. 154*2dbc0838SMauro Carvalho Chehab 155*2dbc0838SMauro Carvalho ChehabOCXL_IOCTL_GET_METADATA: 156*2dbc0838SMauro Carvalho Chehab 157*2dbc0838SMauro Carvalho Chehab Obtains configuration information from the card, such at the size of 158*2dbc0838SMauro Carvalho Chehab MMIO areas, the AFU version, and the PASID for the current context. 159*2dbc0838SMauro Carvalho Chehab 160*2dbc0838SMauro Carvalho ChehabOCXL_IOCTL_ENABLE_P9_WAIT: 161*2dbc0838SMauro Carvalho Chehab 162*2dbc0838SMauro Carvalho Chehab Allows the AFU to wake a userspace thread executing 'wait'. Returns 163*2dbc0838SMauro Carvalho Chehab information to userspace to allow it to configure the AFU. Note that 164*2dbc0838SMauro Carvalho Chehab this is only available on POWER9. 165*2dbc0838SMauro Carvalho Chehab 166*2dbc0838SMauro Carvalho ChehabOCXL_IOCTL_GET_FEATURES: 167*2dbc0838SMauro Carvalho Chehab 168*2dbc0838SMauro Carvalho Chehab Reports on which CPU features that affect OpenCAPI are usable from 169*2dbc0838SMauro Carvalho Chehab userspace. 170*2dbc0838SMauro Carvalho Chehab 171*2dbc0838SMauro Carvalho Chehab 172*2dbc0838SMauro Carvalho Chehabmmap 173*2dbc0838SMauro Carvalho Chehab---- 174*2dbc0838SMauro Carvalho Chehab 175*2dbc0838SMauro Carvalho ChehabA process can mmap the per-process MMIO area for interactions with the 176*2dbc0838SMauro Carvalho ChehabAFU. 177