1c220a1faSMauro Carvalho Chehab================================================= 2c220a1faSMauro Carvalho ChehabFPGA Device Feature List (DFL) Framework Overview 3c220a1faSMauro Carvalho Chehab================================================= 4c220a1faSMauro Carvalho Chehab 5c220a1faSMauro Carvalho ChehabAuthors: 6c220a1faSMauro Carvalho Chehab 7c220a1faSMauro Carvalho Chehab- Enno Luebbers <enno.luebbers@intel.com> 8c220a1faSMauro Carvalho Chehab- Xiao Guangrong <guangrong.xiao@linux.intel.com> 9c220a1faSMauro Carvalho Chehab- Wu Hao <hao.wu@intel.com> 10d72260ccSXu Yilun- Xu Yilun <yilun.xu@intel.com> 11c220a1faSMauro Carvalho Chehab 122de4ba17SRandy DunlapThe Device Feature List (DFL) FPGA framework (and drivers according to 13580e3137STom Rixthis framework) hides the very details of low layer hardware and provides 14c220a1faSMauro Carvalho Chehabunified interfaces to userspace. Applications could use these interfaces to 15c220a1faSMauro Carvalho Chehabconfigure, enumerate, open and access FPGA accelerators on platforms which 16c220a1faSMauro Carvalho Chehabimplement the DFL in the device memory. Besides this, the DFL framework 17c220a1faSMauro Carvalho Chehabenables system level management functions such as FPGA reconfiguration. 18c220a1faSMauro Carvalho Chehab 19c220a1faSMauro Carvalho Chehab 20c220a1faSMauro Carvalho ChehabDevice Feature List (DFL) Overview 21c220a1faSMauro Carvalho Chehab================================== 22c220a1faSMauro Carvalho ChehabDevice Feature List (DFL) defines a linked list of feature headers within the 23c220a1faSMauro Carvalho Chehabdevice MMIO space to provide an extensible way of adding features. Software can 24c220a1faSMauro Carvalho Chehabwalk through these predefined data structures to enumerate FPGA features: 25c220a1faSMauro Carvalho ChehabFPGA Interface Unit (FIU), Accelerated Function Unit (AFU) and Private Features, 26c220a1faSMauro Carvalho Chehabas illustrated below:: 27c220a1faSMauro Carvalho Chehab 28c220a1faSMauro Carvalho Chehab Header Header Header Header 29c220a1faSMauro Carvalho Chehab +----------+ +-->+----------+ +-->+----------+ +-->+----------+ 30c220a1faSMauro Carvalho Chehab | Type | | | Type | | | Type | | | Type | 31c220a1faSMauro Carvalho Chehab | FIU | | | Private | | | Private | | | Private | 32c220a1faSMauro Carvalho Chehab +----------+ | | Feature | | | Feature | | | Feature | 33c220a1faSMauro Carvalho Chehab | Next_DFH |--+ +----------+ | +----------+ | +----------+ 34c220a1faSMauro Carvalho Chehab +----------+ | Next_DFH |--+ | Next_DFH |--+ | Next_DFH |--> NULL 35c220a1faSMauro Carvalho Chehab | ID | +----------+ +----------+ +----------+ 36c220a1faSMauro Carvalho Chehab +----------+ | ID | | ID | | ID | 37c220a1faSMauro Carvalho Chehab | Next_AFU |--+ +----------+ +----------+ +----------+ 38c220a1faSMauro Carvalho Chehab +----------+ | | Feature | | Feature | | Feature | 39c220a1faSMauro Carvalho Chehab | Header | | | Register | | Register | | Register | 40c220a1faSMauro Carvalho Chehab | Register | | | Set | | Set | | Set | 41c220a1faSMauro Carvalho Chehab | Set | | +----------+ +----------+ +----------+ 42c220a1faSMauro Carvalho Chehab +----------+ | Header 43c220a1faSMauro Carvalho Chehab +-->+----------+ 44c220a1faSMauro Carvalho Chehab | Type | 45c220a1faSMauro Carvalho Chehab | AFU | 46c220a1faSMauro Carvalho Chehab +----------+ 47c220a1faSMauro Carvalho Chehab | Next_DFH |--> NULL 48c220a1faSMauro Carvalho Chehab +----------+ 49c220a1faSMauro Carvalho Chehab | GUID | 50c220a1faSMauro Carvalho Chehab +----------+ 51c220a1faSMauro Carvalho Chehab | Header | 52c220a1faSMauro Carvalho Chehab | Register | 53c220a1faSMauro Carvalho Chehab | Set | 54c220a1faSMauro Carvalho Chehab +----------+ 55c220a1faSMauro Carvalho Chehab 56c220a1faSMauro Carvalho ChehabFPGA Interface Unit (FIU) represents a standalone functional unit for the 57c220a1faSMauro Carvalho Chehabinterface to FPGA, e.g. the FPGA Management Engine (FME) and Port (more 58c220a1faSMauro Carvalho Chehabdescriptions on FME and Port in later sections). 59c220a1faSMauro Carvalho Chehab 60f7c2e792STom RixAccelerated Function Unit (AFU) represents an FPGA programmable region and 61c220a1faSMauro Carvalho Chehabalways connects to a FIU (e.g. a Port) as its child as illustrated above. 62c220a1faSMauro Carvalho Chehab 63c220a1faSMauro Carvalho ChehabPrivate Features represent sub features of the FIU and AFU. They could be 64c220a1faSMauro Carvalho Chehabvarious function blocks with different IDs, but all private features which 65c220a1faSMauro Carvalho Chehabbelong to the same FIU or AFU, must be linked to one list via the Next Device 66c220a1faSMauro Carvalho ChehabFeature Header (Next_DFH) pointer. 67c220a1faSMauro Carvalho Chehab 68c220a1faSMauro Carvalho ChehabEach FIU, AFU and Private Feature could implement its own functional registers. 69c220a1faSMauro Carvalho ChehabThe functional register set for FIU and AFU, is named as Header Register Set, 70c220a1faSMauro Carvalho Chehabe.g. FME Header Register Set, and the one for Private Feature, is named as 71c220a1faSMauro Carvalho ChehabFeature Register Set, e.g. FME Partial Reconfiguration Feature Register Set. 72c220a1faSMauro Carvalho Chehab 73c220a1faSMauro Carvalho ChehabThis Device Feature List provides a way of linking features together, it's 74c220a1faSMauro Carvalho Chehabconvenient for software to locate each feature by walking through this list, 75c220a1faSMauro Carvalho Chehaband can be implemented in register regions of any FPGA device. 76c220a1faSMauro Carvalho Chehab 77c220a1faSMauro Carvalho Chehab 78*46879f71SMatthew GerlachDevice Feature Header - Version 0 79*46879f71SMatthew Gerlach================================= 80*46879f71SMatthew GerlachVersion 0 (DFHv0) is the original version of the Device Feature Header. 81*46879f71SMatthew GerlachAll multi-byte quantities in DFHv0 are little-endian. 82*46879f71SMatthew GerlachThe format of DFHv0 is shown below:: 83*46879f71SMatthew Gerlach 84*46879f71SMatthew Gerlach +-----------------------------------------------------------------------+ 85*46879f71SMatthew Gerlach |63 Type 60|59 DFH VER 52|51 Rsvd 41|40 EOL|39 Next 16|15 REV 12|11 ID 0| 0x00 86*46879f71SMatthew Gerlach +-----------------------------------------------------------------------+ 87*46879f71SMatthew Gerlach |63 GUID_L 0| 0x08 88*46879f71SMatthew Gerlach +-----------------------------------------------------------------------+ 89*46879f71SMatthew Gerlach |63 GUID_H 0| 0x10 90*46879f71SMatthew Gerlach +-----------------------------------------------------------------------+ 91*46879f71SMatthew Gerlach 92*46879f71SMatthew Gerlach- Offset 0x00 93*46879f71SMatthew Gerlach 94*46879f71SMatthew Gerlach * Type - The type of DFH (e.g. FME, AFU, or private feature). 95*46879f71SMatthew Gerlach * DFH VER - The version of the DFH. 96*46879f71SMatthew Gerlach * Rsvd - Currently unused. 97*46879f71SMatthew Gerlach * EOL - Set if the DFH is the end of the Device Feature List (DFL). 98*46879f71SMatthew Gerlach * Next - The offset in bytes of the next DFH in the DFL from the DFH start, 99*46879f71SMatthew Gerlach and the start of a DFH must be aligned to an 8 byte boundary. 100*46879f71SMatthew Gerlach If EOL is set, Next is the size of MMIO of the last feature in the list. 101*46879f71SMatthew Gerlach * REV - The revision of the feature associated with this header. 102*46879f71SMatthew Gerlach * ID - The feature ID if Type is private feature. 103*46879f71SMatthew Gerlach 104*46879f71SMatthew Gerlach- Offset 0x08 105*46879f71SMatthew Gerlach 106*46879f71SMatthew Gerlach * GUID_L - Least significant 64 bits of a 128-bit Globally Unique Identifier 107*46879f71SMatthew Gerlach (present only if Type is FME or AFU). 108*46879f71SMatthew Gerlach 109*46879f71SMatthew Gerlach- Offset 0x10 110*46879f71SMatthew Gerlach 111*46879f71SMatthew Gerlach * GUID_H - Most significant 64 bits of a 128-bit Globally Unique Identifier 112*46879f71SMatthew Gerlach (present only if Type is FME or AFU). 113*46879f71SMatthew Gerlach 114*46879f71SMatthew Gerlach 115*46879f71SMatthew GerlachDevice Feature Header - Version 1 116*46879f71SMatthew Gerlach================================= 117*46879f71SMatthew GerlachVersion 1 (DFHv1) of the Device Feature Header adds the following functionality: 118*46879f71SMatthew Gerlach 119*46879f71SMatthew Gerlach* Provides a standardized mechanism for features to describe 120*46879f71SMatthew Gerlach parameters/capabilities to software. 121*46879f71SMatthew Gerlach* Standardize the use of a GUID for all DFHv1 types. 122*46879f71SMatthew Gerlach* Decouples the DFH location from the register space of the feature itself. 123*46879f71SMatthew Gerlach 124*46879f71SMatthew GerlachAll multi-byte quantities in DFHv1 are little-endian. 125*46879f71SMatthew GerlachThe format of Version 1 of the Device Feature Header (DFH) is shown below:: 126*46879f71SMatthew Gerlach 127*46879f71SMatthew Gerlach +-----------------------------------------------------------------------+ 128*46879f71SMatthew Gerlach |63 Type 60|59 DFH VER 52|51 Rsvd 41|40 EOL|39 Next 16|15 REV 12|11 ID 0| 0x00 129*46879f71SMatthew Gerlach +-----------------------------------------------------------------------+ 130*46879f71SMatthew Gerlach |63 GUID_L 0| 0x08 131*46879f71SMatthew Gerlach +-----------------------------------------------------------------------+ 132*46879f71SMatthew Gerlach |63 GUID_H 0| 0x10 133*46879f71SMatthew Gerlach +-----------------------------------------------------------------------+ 134*46879f71SMatthew Gerlach |63 Reg Address/Offset 1| Rel 0| 0x18 135*46879f71SMatthew Gerlach +-----------------------------------------------------------------------+ 136*46879f71SMatthew Gerlach |63 Reg Size 32|Params 31|30 Group 16|15 Instance 0| 0x20 137*46879f71SMatthew Gerlach +-----------------------------------------------------------------------+ 138*46879f71SMatthew Gerlach |63 Next 35|34RSV33|EOP32|31 Param Version 16|15 Param ID 0| 0x28 139*46879f71SMatthew Gerlach +-----------------------------------------------------------------------+ 140*46879f71SMatthew Gerlach |63 Parameter Data 0| 0x30 141*46879f71SMatthew Gerlach +-----------------------------------------------------------------------+ 142*46879f71SMatthew Gerlach 143*46879f71SMatthew Gerlach ... 144*46879f71SMatthew Gerlach 145*46879f71SMatthew Gerlach +-----------------------------------------------------------------------+ 146*46879f71SMatthew Gerlach |63 Next 35|34RSV33|EOP32|31 Param Version 16|15 Param ID 0| 147*46879f71SMatthew Gerlach +-----------------------------------------------------------------------+ 148*46879f71SMatthew Gerlach |63 Parameter Data 0| 149*46879f71SMatthew Gerlach +-----------------------------------------------------------------------+ 150*46879f71SMatthew Gerlach 151*46879f71SMatthew Gerlach- Offset 0x00 152*46879f71SMatthew Gerlach 153*46879f71SMatthew Gerlach * Type - The type of DFH (e.g. FME, AFU, or private feature). 154*46879f71SMatthew Gerlach * DFH VER - The version of the DFH. 155*46879f71SMatthew Gerlach * Rsvd - Currently unused. 156*46879f71SMatthew Gerlach * EOL - Set if the DFH is the end of the Device Feature List (DFL). 157*46879f71SMatthew Gerlach * Next - The offset in bytes of the next DFH in the DFL from the DFH start, 158*46879f71SMatthew Gerlach and the start of a DFH must be aligned to an 8 byte boundary. 159*46879f71SMatthew Gerlach If EOL is set, Next is the size of MMIO of the last feature in the list. 160*46879f71SMatthew Gerlach * REV - The revision of the feature associated with this header. 161*46879f71SMatthew Gerlach * ID - The feature ID if Type is private feature. 162*46879f71SMatthew Gerlach 163*46879f71SMatthew Gerlach- Offset 0x08 164*46879f71SMatthew Gerlach 165*46879f71SMatthew Gerlach * GUID_L - Least significant 64 bits of a 128-bit Globally Unique Identifier. 166*46879f71SMatthew Gerlach 167*46879f71SMatthew Gerlach- Offset 0x10 168*46879f71SMatthew Gerlach 169*46879f71SMatthew Gerlach * GUID_H - Most significant 64 bits of a 128-bit Globally Unique Identifier. 170*46879f71SMatthew Gerlach 171*46879f71SMatthew Gerlach- Offset 0x18 172*46879f71SMatthew Gerlach 173*46879f71SMatthew Gerlach * Reg Address/Offset - If Rel bit is set, then the value is the high 63 bits 174*46879f71SMatthew Gerlach of a 16-bit aligned absolute address of the feature's registers. Otherwise 175*46879f71SMatthew Gerlach the value is the offset from the start of the DFH of the feature's registers. 176*46879f71SMatthew Gerlach 177*46879f71SMatthew Gerlach- Offset 0x20 178*46879f71SMatthew Gerlach 179*46879f71SMatthew Gerlach * Reg Size - Size of feature's register set in bytes. 180*46879f71SMatthew Gerlach * Params - Set if DFH has a list of parameter blocks. 181*46879f71SMatthew Gerlach * Group - Id of group if feature is part of a group. 182*46879f71SMatthew Gerlach * Instance - Id of feature instance within a group. 183*46879f71SMatthew Gerlach 184*46879f71SMatthew Gerlach- Offset 0x28 if feature has parameters 185*46879f71SMatthew Gerlach 186*46879f71SMatthew Gerlach * Next - Offset to the next parameter block in 8 byte words. If EOP set, 187*46879f71SMatthew Gerlach size in 8 byte words of last parameter. 188*46879f71SMatthew Gerlach * Param Version - Version of Param ID. 189*46879f71SMatthew Gerlach * Param ID - ID of parameter. 190*46879f71SMatthew Gerlach 191*46879f71SMatthew Gerlach- Offset 0x30 192*46879f71SMatthew Gerlach 193*46879f71SMatthew Gerlach * Parameter Data - Parameter data whose size and format is defined by 194*46879f71SMatthew Gerlach version and ID of the parameter. 195*46879f71SMatthew Gerlach 196*46879f71SMatthew Gerlach 197c220a1faSMauro Carvalho ChehabFIU - FME (FPGA Management Engine) 198c220a1faSMauro Carvalho Chehab================================== 199c220a1faSMauro Carvalho ChehabThe FPGA Management Engine performs reconfiguration and other infrastructure 200c220a1faSMauro Carvalho Chehabfunctions. Each FPGA device only has one FME. 201c220a1faSMauro Carvalho Chehab 202c220a1faSMauro Carvalho ChehabUser-space applications can acquire exclusive access to the FME using open(), 203c220a1faSMauro Carvalho Chehaband release it using close(). 204c220a1faSMauro Carvalho Chehab 205c220a1faSMauro Carvalho ChehabThe following functions are exposed through ioctls: 206c220a1faSMauro Carvalho Chehab 207c220a1faSMauro Carvalho Chehab- Get driver API version (DFL_FPGA_GET_API_VERSION) 208c220a1faSMauro Carvalho Chehab- Check for extensions (DFL_FPGA_CHECK_EXTENSION) 209c220a1faSMauro Carvalho Chehab- Program bitstream (DFL_FPGA_FME_PORT_PR) 21077a0ef48SWu Hao- Assign port to PF (DFL_FPGA_FME_PORT_ASSIGN) 21177a0ef48SWu Hao- Release port from PF (DFL_FPGA_FME_PORT_RELEASE) 2128adfb7c6SXu Yilun- Get number of irqs of FME global error (DFL_FPGA_FME_ERR_GET_IRQ_NUM) 2138adfb7c6SXu Yilun- Set interrupt trigger for FME error (DFL_FPGA_FME_ERR_SET_IRQ) 214c220a1faSMauro Carvalho Chehab 215c220a1faSMauro Carvalho ChehabMore functions are exposed through sysfs 216c220a1faSMauro Carvalho Chehab(/sys/class/fpga_region/regionX/dfl-fme.n/): 217c220a1faSMauro Carvalho Chehab 218c220a1faSMauro Carvalho Chehab Read bitstream ID (bitstream_id) 219c220a1faSMauro Carvalho Chehab bitstream_id indicates version of the static FPGA region. 220c220a1faSMauro Carvalho Chehab 221c220a1faSMauro Carvalho Chehab Read bitstream metadata (bitstream_metadata) 222c220a1faSMauro Carvalho Chehab bitstream_metadata includes detailed information of static FPGA region, 223c220a1faSMauro Carvalho Chehab e.g. synthesis date and seed. 224c220a1faSMauro Carvalho Chehab 225c220a1faSMauro Carvalho Chehab Read number of ports (ports_num) 226c220a1faSMauro Carvalho Chehab one FPGA device may have more than one port, this sysfs interface indicates 227c220a1faSMauro Carvalho Chehab how many ports the FPGA device has. 228c220a1faSMauro Carvalho Chehab 22977a0ef48SWu Hao Global error reporting management (errors/) 23077a0ef48SWu Hao error reporting sysfs interfaces allow user to read errors detected by the 23177a0ef48SWu Hao hardware, and clear the logged errors. 23277a0ef48SWu Hao 233214be747SXu Yilun Power management (dfl_fme_power hwmon) 234214be747SXu Yilun power management hwmon sysfs interfaces allow user to read power management 235214be747SXu Yilun information (power consumption, thresholds, threshold status, limits, etc.) 236214be747SXu Yilun and configure power thresholds for different throttling levels. 237214be747SXu Yilun 238214be747SXu Yilun Thermal management (dfl_fme_thermal hwmon) 239214be747SXu Yilun thermal management hwmon sysfs interfaces allow user to read thermal 240214be747SXu Yilun management information (current temperature, thresholds, threshold status, 241214be747SXu Yilun etc.). 242214be747SXu Yilun 2431f03ac94SXu Yilun Performance reporting 2441f03ac94SXu Yilun performance counters are exposed through perf PMU APIs. Standard perf tool 2451f03ac94SXu Yilun can be used to monitor all available perf events. Please see performance 2461f03ac94SXu Yilun counter section below for more detailed information. 2471f03ac94SXu Yilun 248c220a1faSMauro Carvalho Chehab 249c220a1faSMauro Carvalho ChehabFIU - PORT 250c220a1faSMauro Carvalho Chehab========== 251c220a1faSMauro Carvalho ChehabA port represents the interface between the static FPGA fabric and a partially 252c220a1faSMauro Carvalho Chehabreconfigurable region containing an AFU. It controls the communication from SW 253c220a1faSMauro Carvalho Chehabto the accelerator and exposes features such as reset and debug. Each FPGA 254c220a1faSMauro Carvalho Chehabdevice may have more than one port, but always one AFU per port. 255c220a1faSMauro Carvalho Chehab 256c220a1faSMauro Carvalho Chehab 257c220a1faSMauro Carvalho ChehabAFU 258c220a1faSMauro Carvalho Chehab=== 259c220a1faSMauro Carvalho ChehabAn AFU is attached to a port FIU and exposes a fixed length MMIO region to be 260c220a1faSMauro Carvalho Chehabused for accelerator-specific control registers. 261c220a1faSMauro Carvalho Chehab 262c220a1faSMauro Carvalho ChehabUser-space applications can acquire exclusive access to an AFU attached to a 263c220a1faSMauro Carvalho Chehabport by using open() on the port device node and release it using close(). 264c220a1faSMauro Carvalho Chehab 265c220a1faSMauro Carvalho ChehabThe following functions are exposed through ioctls: 266c220a1faSMauro Carvalho Chehab 267c220a1faSMauro Carvalho Chehab- Get driver API version (DFL_FPGA_GET_API_VERSION) 268c220a1faSMauro Carvalho Chehab- Check for extensions (DFL_FPGA_CHECK_EXTENSION) 269c220a1faSMauro Carvalho Chehab- Get port info (DFL_FPGA_PORT_GET_INFO) 270c220a1faSMauro Carvalho Chehab- Get MMIO region info (DFL_FPGA_PORT_GET_REGION_INFO) 271c220a1faSMauro Carvalho Chehab- Map DMA buffer (DFL_FPGA_PORT_DMA_MAP) 272c220a1faSMauro Carvalho Chehab- Unmap DMA buffer (DFL_FPGA_PORT_DMA_UNMAP) 273c220a1faSMauro Carvalho Chehab- Reset AFU (DFL_FPGA_PORT_RESET) 2748adfb7c6SXu Yilun- Get number of irqs of port error (DFL_FPGA_PORT_ERR_GET_IRQ_NUM) 2758adfb7c6SXu Yilun- Set interrupt trigger for port error (DFL_FPGA_PORT_ERR_SET_IRQ) 2768adfb7c6SXu Yilun- Get number of irqs of UINT (DFL_FPGA_PORT_UINT_GET_IRQ_NUM) 2778adfb7c6SXu Yilun- Set interrupt trigger for UINT (DFL_FPGA_PORT_UINT_SET_IRQ) 278c220a1faSMauro Carvalho Chehab 279c220a1faSMauro Carvalho ChehabDFL_FPGA_PORT_RESET: 280c220a1faSMauro Carvalho Chehab reset the FPGA Port and its AFU. Userspace can do Port 281c220a1faSMauro Carvalho Chehab reset at any time, e.g. during DMA or Partial Reconfiguration. But it should 282c220a1faSMauro Carvalho Chehab never cause any system level issue, only functional failure (e.g. DMA or PR 283c220a1faSMauro Carvalho Chehab operation failure) and be recoverable from the failure. 284c220a1faSMauro Carvalho Chehab 285c220a1faSMauro Carvalho ChehabUser-space applications can also mmap() accelerator MMIO regions. 286c220a1faSMauro Carvalho Chehab 287c220a1faSMauro Carvalho ChehabMore functions are exposed through sysfs: 288c220a1faSMauro Carvalho Chehab(/sys/class/fpga_region/<regionX>/<dfl-port.m>/): 289c220a1faSMauro Carvalho Chehab 290c220a1faSMauro Carvalho Chehab Read Accelerator GUID (afu_id) 291c220a1faSMauro Carvalho Chehab afu_id indicates which PR bitstream is programmed to this AFU. 292c220a1faSMauro Carvalho Chehab 29377a0ef48SWu Hao Error reporting (errors/) 29477a0ef48SWu Hao error reporting sysfs interfaces allow user to read port/afu errors 29577a0ef48SWu Hao detected by the hardware, and clear the logged errors. 29677a0ef48SWu Hao 297c220a1faSMauro Carvalho Chehab 298c220a1faSMauro Carvalho ChehabDFL Framework Overview 299c220a1faSMauro Carvalho Chehab====================== 300c220a1faSMauro Carvalho Chehab 301c220a1faSMauro Carvalho Chehab:: 302c220a1faSMauro Carvalho Chehab 303c220a1faSMauro Carvalho Chehab +----------+ +--------+ +--------+ +--------+ 304c220a1faSMauro Carvalho Chehab | FME | | AFU | | AFU | | AFU | 305c220a1faSMauro Carvalho Chehab | Module | | Module | | Module | | Module | 306c220a1faSMauro Carvalho Chehab +----------+ +--------+ +--------+ +--------+ 307c220a1faSMauro Carvalho Chehab +-----------------------+ 308c220a1faSMauro Carvalho Chehab | FPGA Container Device | Device Feature List 309c220a1faSMauro Carvalho Chehab | (FPGA Base Region) | Framework 310c220a1faSMauro Carvalho Chehab +-----------------------+ 311c220a1faSMauro Carvalho Chehab ------------------------------------------------------------------ 312c220a1faSMauro Carvalho Chehab +----------------------------+ 313c220a1faSMauro Carvalho Chehab | FPGA DFL Device Module | 314c220a1faSMauro Carvalho Chehab | (e.g. PCIE/Platform Device)| 315c220a1faSMauro Carvalho Chehab +----------------------------+ 316c220a1faSMauro Carvalho Chehab +------------------------+ 317c220a1faSMauro Carvalho Chehab | FPGA Hardware Device | 318c220a1faSMauro Carvalho Chehab +------------------------+ 319c220a1faSMauro Carvalho Chehab 320c220a1faSMauro Carvalho ChehabDFL framework in kernel provides common interfaces to create container device 321c220a1faSMauro Carvalho Chehab(FPGA base region), discover feature devices and their private features from the 322c220a1faSMauro Carvalho Chehabgiven Device Feature Lists and create platform devices for feature devices 323c220a1faSMauro Carvalho Chehab(e.g. FME, Port and AFU) with related resources under the container device. It 324c220a1faSMauro Carvalho Chehabalso abstracts operations for the private features and exposes common ops to 325c220a1faSMauro Carvalho Chehabfeature device drivers. 326c220a1faSMauro Carvalho Chehab 327580e3137STom RixThe FPGA DFL Device could be different hardware, e.g. PCIe device, platform 328c220a1faSMauro Carvalho Chehabdevice and etc. Its driver module is always loaded first once the device is 329c220a1faSMauro Carvalho Chehabcreated by the system. This driver plays an infrastructural role in the 330c220a1faSMauro Carvalho Chehabdriver architecture. It locates the DFLs in the device memory, handles them 331c220a1faSMauro Carvalho Chehaband related resources to common interfaces from DFL framework for enumeration. 332c220a1faSMauro Carvalho Chehab(Please refer to drivers/fpga/dfl.c for detailed enumeration APIs). 333c220a1faSMauro Carvalho Chehab 334c220a1faSMauro Carvalho ChehabThe FPGA Management Engine (FME) driver is a platform driver which is loaded 335c220a1faSMauro Carvalho Chehabautomatically after FME platform device creation from the DFL device module. It 336c220a1faSMauro Carvalho Chehabprovides the key features for FPGA management, including: 337c220a1faSMauro Carvalho Chehab 338c220a1faSMauro Carvalho Chehab a) Expose static FPGA region information, e.g. version and metadata. 339c220a1faSMauro Carvalho Chehab Users can read related information via sysfs interfaces exposed 340c220a1faSMauro Carvalho Chehab by FME driver. 341c220a1faSMauro Carvalho Chehab 342c220a1faSMauro Carvalho Chehab b) Partial Reconfiguration. The FME driver creates FPGA manager, FPGA 343c220a1faSMauro Carvalho Chehab bridges and FPGA regions during PR sub feature initialization. Once 344c220a1faSMauro Carvalho Chehab it receives a DFL_FPGA_FME_PORT_PR ioctl from user, it invokes the 345c220a1faSMauro Carvalho Chehab common interface function from FPGA Region to complete the partial 346c220a1faSMauro Carvalho Chehab reconfiguration of the PR bitstream to the given port. 347c220a1faSMauro Carvalho Chehab 348c220a1faSMauro Carvalho ChehabSimilar to the FME driver, the FPGA Accelerated Function Unit (AFU) driver is 349c220a1faSMauro Carvalho Chehabprobed once the AFU platform device is created. The main function of this module 350c220a1faSMauro Carvalho Chehabis to provide an interface for userspace applications to access the individual 351c220a1faSMauro Carvalho Chehabaccelerators, including basic reset control on port, AFU MMIO region export, dma 352c220a1faSMauro Carvalho Chehabbuffer mapping service functions. 353c220a1faSMauro Carvalho Chehab 354c220a1faSMauro Carvalho ChehabAfter feature platform devices creation, matched platform drivers will be loaded 355c220a1faSMauro Carvalho Chehabautomatically to handle different functionalities. Please refer to next sections 356c220a1faSMauro Carvalho Chehabfor detailed information on functional units which have been already implemented 357c220a1faSMauro Carvalho Chehabunder this DFL framework. 358c220a1faSMauro Carvalho Chehab 359c220a1faSMauro Carvalho Chehab 360c220a1faSMauro Carvalho ChehabPartial Reconfiguration 361c220a1faSMauro Carvalho Chehab======================= 362c220a1faSMauro Carvalho ChehabAs mentioned above, accelerators can be reconfigured through partial 363c220a1faSMauro Carvalho Chehabreconfiguration of a PR bitstream file. The PR bitstream file must have been 364c220a1faSMauro Carvalho Chehabgenerated for the exact static FPGA region and targeted reconfigurable region 365c220a1faSMauro Carvalho Chehab(port) of the FPGA, otherwise, the reconfiguration operation will fail and 366c220a1faSMauro Carvalho Chehabpossibly cause system instability. This compatibility can be checked by 367c220a1faSMauro Carvalho Chehabcomparing the compatibility ID noted in the header of PR bitstream file against 368c220a1faSMauro Carvalho Chehabthe compat_id exposed by the target FPGA region. This check is usually done by 369c220a1faSMauro Carvalho Chehabuserspace before calling the reconfiguration IOCTL. 370c220a1faSMauro Carvalho Chehab 371c220a1faSMauro Carvalho Chehab 37277a0ef48SWu HaoFPGA virtualization - PCIe SRIOV 37377a0ef48SWu Hao================================ 37477a0ef48SWu HaoThis section describes the virtualization support on DFL based FPGA device to 37577a0ef48SWu Haoenable accessing an accelerator from applications running in a virtual machine 37677a0ef48SWu Hao(VM). This section only describes the PCIe based FPGA device with SRIOV support. 37777a0ef48SWu Hao 37877a0ef48SWu HaoFeatures supported by the particular FPGA device are exposed through Device 37977a0ef48SWu HaoFeature Lists, as illustrated below: 38077a0ef48SWu Hao 38177a0ef48SWu Hao:: 38277a0ef48SWu Hao 38377a0ef48SWu Hao +-------------------------------+ +-------------+ 38477a0ef48SWu Hao | PF | | VF | 38577a0ef48SWu Hao +-------------------------------+ +-------------+ 38677a0ef48SWu Hao ^ ^ ^ ^ 38777a0ef48SWu Hao | | | | 38877a0ef48SWu Hao +-----|------------|---------|--------------|-------+ 38977a0ef48SWu Hao | | | | | | 39077a0ef48SWu Hao | +-----+ +-------+ +-------+ +-------+ | 39177a0ef48SWu Hao | | FME | | Port0 | | Port1 | | Port2 | | 39277a0ef48SWu Hao | +-----+ +-------+ +-------+ +-------+ | 39377a0ef48SWu Hao | ^ ^ ^ | 39477a0ef48SWu Hao | | | | | 39577a0ef48SWu Hao | +-------+ +------+ +-------+ | 39677a0ef48SWu Hao | | AFU | | AFU | | AFU | | 39777a0ef48SWu Hao | +-------+ +------+ +-------+ | 39877a0ef48SWu Hao | | 39977a0ef48SWu Hao | DFL based FPGA PCIe Device | 40077a0ef48SWu Hao +---------------------------------------------------+ 40177a0ef48SWu Hao 40277a0ef48SWu HaoFME is always accessed through the physical function (PF). 40377a0ef48SWu Hao 40477a0ef48SWu HaoPorts (and related AFUs) are accessed via PF by default, but could be exposed 40577a0ef48SWu Haothrough virtual function (VF) devices via PCIe SRIOV. Each VF only contains 40677a0ef48SWu Hao1 Port and 1 AFU for isolation. Users could assign individual VFs (accelerators) 40777a0ef48SWu Haocreated via PCIe SRIOV interface, to virtual machines. 40877a0ef48SWu Hao 40977a0ef48SWu HaoThe driver organization in virtualization case is illustrated below: 41077a0ef48SWu Hao:: 41177a0ef48SWu Hao 41277a0ef48SWu Hao +-------++------++------+ | 41377a0ef48SWu Hao | FME || FME || FME | | 41477a0ef48SWu Hao | FPGA || FPGA || FPGA | | 41577a0ef48SWu Hao |Manager||Bridge||Region| | 41677a0ef48SWu Hao +-------++------++------+ | 41777a0ef48SWu Hao +-----------------------+ +--------+ | +--------+ 41877a0ef48SWu Hao | FME | | AFU | | | AFU | 41977a0ef48SWu Hao | Module | | Module | | | Module | 42077a0ef48SWu Hao +-----------------------+ +--------+ | +--------+ 42177a0ef48SWu Hao +-----------------------+ | +-----------------------+ 42277a0ef48SWu Hao | FPGA Container Device | | | FPGA Container Device | 42377a0ef48SWu Hao | (FPGA Base Region) | | | (FPGA Base Region) | 42477a0ef48SWu Hao +-----------------------+ | +-----------------------+ 42577a0ef48SWu Hao +------------------+ | +------------------+ 42677a0ef48SWu Hao | FPGA PCIE Module | | Virtual | FPGA PCIE Module | 42777a0ef48SWu Hao +------------------+ Host | Machine +------------------+ 42877a0ef48SWu Hao -------------------------------------- | ------------------------------ 42977a0ef48SWu Hao +---------------+ | +---------------+ 43077a0ef48SWu Hao | PCI PF Device | | | PCI VF Device | 43177a0ef48SWu Hao +---------------+ | +---------------+ 43277a0ef48SWu Hao 433f7c2e792STom RixFPGA PCIe device driver is always loaded first once an FPGA PCIe PF or VF device 43477a0ef48SWu Haois detected. It: 43577a0ef48SWu Hao 43677a0ef48SWu Hao* Finishes enumeration on both FPGA PCIe PF and VF device using common 43777a0ef48SWu Hao interfaces from DFL framework. 43877a0ef48SWu Hao* Supports SRIOV. 43977a0ef48SWu Hao 44077a0ef48SWu HaoThe FME device driver plays a management role in this driver architecture, it 44177a0ef48SWu Haoprovides ioctls to release Port from PF and assign Port to PF. After release 44277a0ef48SWu Haoa port from PF, then it's safe to expose this port through a VF via PCIe SRIOV 44377a0ef48SWu Haosysfs interface. 44477a0ef48SWu Hao 44577a0ef48SWu HaoTo enable accessing an accelerator from applications running in a VM, the 44677a0ef48SWu Haorespective AFU's port needs to be assigned to a VF using the following steps: 44777a0ef48SWu Hao 44877a0ef48SWu Hao#. The PF owns all AFU ports by default. Any port that needs to be 44977a0ef48SWu Hao reassigned to a VF must first be released through the 45077a0ef48SWu Hao DFL_FPGA_FME_PORT_RELEASE ioctl on the FME device. 45177a0ef48SWu Hao 45277a0ef48SWu Hao#. Once N ports are released from PF, then user can use command below 45377a0ef48SWu Hao to enable SRIOV and VFs. Each VF owns only one Port with AFU. 45477a0ef48SWu Hao 45577a0ef48SWu Hao :: 45677a0ef48SWu Hao 45777a0ef48SWu Hao echo N > $PCI_DEVICE_PATH/sriov_numvfs 45877a0ef48SWu Hao 45977a0ef48SWu Hao#. Pass through the VFs to VMs 46077a0ef48SWu Hao 46177a0ef48SWu Hao#. The AFU under VF is accessible from applications in VM (using the 46277a0ef48SWu Hao same driver inside the VF). 46377a0ef48SWu Hao 46477a0ef48SWu HaoNote that an FME can't be assigned to a VF, thus PR and other management 46577a0ef48SWu Haofunctions are only available via the PF. 46677a0ef48SWu Hao 467c220a1faSMauro Carvalho ChehabDevice enumeration 468c220a1faSMauro Carvalho Chehab================== 469c220a1faSMauro Carvalho ChehabThis section introduces how applications enumerate the fpga device from 470c220a1faSMauro Carvalho Chehabthe sysfs hierarchy under /sys/class/fpga_region. 471c220a1faSMauro Carvalho Chehab 472c220a1faSMauro Carvalho ChehabIn the example below, two DFL based FPGA devices are installed in the host. Each 473c220a1faSMauro Carvalho Chehabfpga device has one FME and two ports (AFUs). 474c220a1faSMauro Carvalho Chehab 475c220a1faSMauro Carvalho ChehabFPGA regions are created under /sys/class/fpga_region/:: 476c220a1faSMauro Carvalho Chehab 477c220a1faSMauro Carvalho Chehab /sys/class/fpga_region/region0 478c220a1faSMauro Carvalho Chehab /sys/class/fpga_region/region1 479c220a1faSMauro Carvalho Chehab /sys/class/fpga_region/region2 480c220a1faSMauro Carvalho Chehab ... 481c220a1faSMauro Carvalho Chehab 482c220a1faSMauro Carvalho ChehabApplication needs to search each regionX folder, if feature device is found, 483c220a1faSMauro Carvalho Chehab(e.g. "dfl-port.n" or "dfl-fme.m" is found), then it's the base 484c220a1faSMauro Carvalho Chehabfpga region which represents the FPGA device. 485c220a1faSMauro Carvalho Chehab 486c220a1faSMauro Carvalho ChehabEach base region has one FME and two ports (AFUs) as child devices:: 487c220a1faSMauro Carvalho Chehab 488c220a1faSMauro Carvalho Chehab /sys/class/fpga_region/region0/dfl-fme.0 489c220a1faSMauro Carvalho Chehab /sys/class/fpga_region/region0/dfl-port.0 490c220a1faSMauro Carvalho Chehab /sys/class/fpga_region/region0/dfl-port.1 491c220a1faSMauro Carvalho Chehab ... 492c220a1faSMauro Carvalho Chehab 493c220a1faSMauro Carvalho Chehab /sys/class/fpga_region/region3/dfl-fme.1 494c220a1faSMauro Carvalho Chehab /sys/class/fpga_region/region3/dfl-port.2 495c220a1faSMauro Carvalho Chehab /sys/class/fpga_region/region3/dfl-port.3 496c220a1faSMauro Carvalho Chehab ... 497c220a1faSMauro Carvalho Chehab 498c220a1faSMauro Carvalho ChehabIn general, the FME/AFU sysfs interfaces are named as follows:: 499c220a1faSMauro Carvalho Chehab 500c220a1faSMauro Carvalho Chehab /sys/class/fpga_region/<regionX>/<dfl-fme.n>/ 501c220a1faSMauro Carvalho Chehab /sys/class/fpga_region/<regionX>/<dfl-port.m>/ 502c220a1faSMauro Carvalho Chehab 503c220a1faSMauro Carvalho Chehabwith 'n' consecutively numbering all FMEs and 'm' consecutively numbering all 504c220a1faSMauro Carvalho Chehabports. 505c220a1faSMauro Carvalho Chehab 506c220a1faSMauro Carvalho ChehabThe device nodes used for ioctl() or mmap() can be referenced through:: 507c220a1faSMauro Carvalho Chehab 508c220a1faSMauro Carvalho Chehab /sys/class/fpga_region/<regionX>/<dfl-fme.n>/dev 509c220a1faSMauro Carvalho Chehab /sys/class/fpga_region/<regionX>/<dfl-port.n>/dev 510c220a1faSMauro Carvalho Chehab 511c220a1faSMauro Carvalho Chehab 5121f03ac94SXu YilunPerformance Counters 5131f03ac94SXu Yilun==================== 5141f03ac94SXu YilunPerformance reporting is one private feature implemented in FME. It could 5151f03ac94SXu Yilunsupports several independent, system-wide, device counter sets in hardware to 5161f03ac94SXu Yilunmonitor and count for performance events, including "basic", "cache", "fabric", 5171f03ac94SXu Yilun"vtd" and "vtd_sip" counters. Users could use standard perf tool to monitor 5181f03ac94SXu YilunFPGA cache hit/miss rate, transaction number, interface clock counter of AFU 5191f03ac94SXu Yilunand other FPGA performance events. 5201f03ac94SXu Yilun 5211f03ac94SXu YilunDifferent FPGA devices may have different counter sets, depending on hardware 5221f03ac94SXu Yilunimplementation. E.g., some discrete FPGA cards don't have any cache. User could 5231f03ac94SXu Yilunuse "perf list" to check which perf events are supported by target hardware. 5241f03ac94SXu Yilun 5251f03ac94SXu YilunIn order to allow user to use standard perf API to access these performance 5261f03ac94SXu Yiluncounters, driver creates a perf PMU, and related sysfs interfaces in 5271f03ac94SXu Yilun/sys/bus/event_source/devices/dfl_fme* to describe available perf events and 5281f03ac94SXu Yilunconfiguration options. 5291f03ac94SXu Yilun 5301f03ac94SXu YilunThe "format" directory describes the format of the config field of struct 5311f03ac94SXu Yilunperf_event_attr. There are 3 bitfields for config: "evtype" defines which type 5321f03ac94SXu Yilunthe perf event belongs to; "event" is the identity of the event within its 5331f03ac94SXu Yiluncategory; "portid" is introduced to decide counters set to monitor on FPGA 5341f03ac94SXu Yilunoverall data or a specific port. 5351f03ac94SXu Yilun 5361f03ac94SXu YilunThe "events" directory describes the configuration templates for all available 5371f03ac94SXu Yilunevents which can be used with perf tool directly. For example, fab_mmio_read 5381f03ac94SXu Yilunhas the configuration "event=0x06,evtype=0x02,portid=0xff", which shows this 5391f03ac94SXu Yilunevent belongs to fabric type (0x02), the local event id is 0x06 and it is for 5401f03ac94SXu Yilunoverall monitoring (portid=0xff). 5411f03ac94SXu Yilun 5421f03ac94SXu YilunExample usage of perf:: 5431f03ac94SXu Yilun 5441f03ac94SXu Yilun $# perf list |grep dfl_fme 5451f03ac94SXu Yilun 5461f03ac94SXu Yilun dfl_fme0/fab_mmio_read/ [Kernel PMU event] 5471f03ac94SXu Yilun <...> 5481f03ac94SXu Yilun dfl_fme0/fab_port_mmio_read,portid=?/ [Kernel PMU event] 5491f03ac94SXu Yilun <...> 5501f03ac94SXu Yilun 5511f03ac94SXu Yilun $# perf stat -a -e dfl_fme0/fab_mmio_read/ <command> 5521f03ac94SXu Yilun or 5531f03ac94SXu Yilun $# perf stat -a -e dfl_fme0/event=0x06,evtype=0x02,portid=0xff/ <command> 5541f03ac94SXu Yilun or 5551f03ac94SXu Yilun $# perf stat -a -e dfl_fme0/config=0xff2006/ <command> 5561f03ac94SXu Yilun 5571f03ac94SXu YilunAnother example, fab_port_mmio_read monitors mmio read of a specific port. So 5581f03ac94SXu Yilunits configuration template is "event=0x06,evtype=0x01,portid=?". The portid 5591f03ac94SXu Yilunshould be explicitly set. 5601f03ac94SXu Yilun 5611f03ac94SXu YilunIts usage of perf:: 5621f03ac94SXu Yilun 5631f03ac94SXu Yilun $# perf stat -a -e dfl_fme0/fab_port_mmio_read,portid=0x0/ <command> 5641f03ac94SXu Yilun or 5651f03ac94SXu Yilun $# perf stat -a -e dfl_fme0/event=0x06,evtype=0x02,portid=0x0/ <command> 5661f03ac94SXu Yilun or 5671f03ac94SXu Yilun $# perf stat -a -e dfl_fme0/config=0x2006/ <command> 5681f03ac94SXu Yilun 5691f03ac94SXu YilunPlease note for fabric counters, overall perf events (fab_*) and port perf 5701f03ac94SXu Yilunevents (fab_port_*) actually share one set of counters in hardware, so it can't 5711f03ac94SXu Yilunmonitor both at the same time. If this set of counters is configured to monitor 5721f03ac94SXu Yilunoverall data, then per port perf data is not supported. See below example:: 5731f03ac94SXu Yilun 5741f03ac94SXu Yilun $# perf stat -e dfl_fme0/fab_mmio_read/,dfl_fme0/fab_port_mmio_write,\ 5751f03ac94SXu Yilun portid=0/ sleep 1 5761f03ac94SXu Yilun 5771f03ac94SXu Yilun Performance counter stats for 'system wide': 5781f03ac94SXu Yilun 5791f03ac94SXu Yilun 3 dfl_fme0/fab_mmio_read/ 5801f03ac94SXu Yilun <not supported> dfl_fme0/fab_port_mmio_write,portid=0x0/ 5811f03ac94SXu Yilun 5821f03ac94SXu Yilun 1.001750904 seconds time elapsed 5831f03ac94SXu Yilun 5841f03ac94SXu YilunThe driver also provides a "cpumask" sysfs attribute, which contains only one 5851f03ac94SXu YilunCPU id used to access these perf events. Counting on multiple CPU is not allowed 5861f03ac94SXu Yilunsince they are system-wide counters on FPGA device. 5871f03ac94SXu Yilun 5881f03ac94SXu YilunThe current driver does not support sampling. So "perf record" is unsupported. 5891f03ac94SXu Yilun 5901f03ac94SXu Yilun 5918adfb7c6SXu YilunInterrupt support 5928adfb7c6SXu Yilun================= 5938adfb7c6SXu YilunSome FME and AFU private features are able to generate interrupts. As mentioned 5948adfb7c6SXu Yilunabove, users could call ioctl (DFL_FPGA_*_GET_IRQ_NUM) to know whether or how 5958adfb7c6SXu Yilunmany interrupts are supported for this private feature. Drivers also implement 5968adfb7c6SXu Yilunan eventfd based interrupt handling mechanism for users to get notified when 5978adfb7c6SXu Yiluninterrupt happens. Users could set eventfds to driver via 5988adfb7c6SXu Yilunioctl (DFL_FPGA_*_SET_IRQ), and then poll/select on these eventfds waiting for 5998adfb7c6SXu Yilunnotification. 6008adfb7c6SXu YilunIn Current DFL, 3 sub features (Port error, FME global error and AFU interrupt) 6018adfb7c6SXu Yilunsupport interrupts. 6028adfb7c6SXu Yilun 6038adfb7c6SXu Yilun 604c220a1faSMauro Carvalho ChehabAdd new FIUs support 605c220a1faSMauro Carvalho Chehab==================== 606c220a1faSMauro Carvalho ChehabIt's possible that developers made some new function blocks (FIUs) under this 607c220a1faSMauro Carvalho ChehabDFL framework, then new platform device driver needs to be developed for the 608c220a1faSMauro Carvalho Chehabnew feature dev (FIU) following the same way as existing feature dev drivers 609c220a1faSMauro Carvalho Chehab(e.g. FME and Port/AFU platform device driver). Besides that, it requires 610c220a1faSMauro Carvalho Chehabmodification on DFL framework enumeration code too, for new FIU type detection 611c220a1faSMauro Carvalho Chehaband related platform devices creation. 612c220a1faSMauro Carvalho Chehab 613c220a1faSMauro Carvalho Chehab 614c220a1faSMauro Carvalho ChehabAdd new private features support 615c220a1faSMauro Carvalho Chehab================================ 616c220a1faSMauro Carvalho ChehabIn some cases, we may need to add some new private features to existing FIUs 617c220a1faSMauro Carvalho Chehab(e.g. FME or Port). Developers don't need to touch enumeration code in DFL 618c220a1faSMauro Carvalho Chehabframework, as each private feature will be parsed automatically and related 619c220a1faSMauro Carvalho Chehabmmio resources can be found under FIU platform device created by DFL framework. 620c220a1faSMauro Carvalho ChehabDeveloper only needs to provide a sub feature driver with matched feature id. 621c220a1faSMauro Carvalho ChehabFME Partial Reconfiguration Sub Feature driver (see drivers/fpga/dfl-fme-pr.c) 622c220a1faSMauro Carvalho Chehabcould be a reference. 623c220a1faSMauro Carvalho Chehab 6242b28c9e0STianfei zhangPlease refer to below link to existing feature id table and guide for new feature 6252b28c9e0STianfei zhangids application. 6262b28c9e0STianfei zhanghttps://github.com/OPAE/dfl-feature-id 6272b28c9e0STianfei zhang 6282b28c9e0STianfei zhang 629fa41d105SMatthew GerlachLocation of DFLs on a PCI Device 63020612d24SLukas Bulwahn================================ 631fa41d105SMatthew GerlachThe original method for finding a DFL on a PCI device assumed the start of the 632fa41d105SMatthew Gerlachfirst DFL to offset 0 of bar 0. If the first node of the DFL is an FME, 633fa41d105SMatthew Gerlachthen further DFLs in the port(s) are specified in FME header registers. 634fa41d105SMatthew GerlachAlternatively, a PCIe vendor specific capability structure can be used to 635fa41d105SMatthew Gerlachspecify the location of all the DFLs on the device, providing flexibility 636fa41d105SMatthew Gerlachfor the type of starting node in the DFL. Intel has reserved the 637fa41d105SMatthew GerlachVSEC ID of 0x43 for this purpose. The vendor specific 638fa41d105SMatthew Gerlachdata begins with a 4 byte vendor specific register for the number of DFLs followed 4 byte 639fa41d105SMatthew GerlachOffset/BIR vendor specific registers for each DFL. Bits 2:0 of Offset/BIR register 640fa41d105SMatthew Gerlachindicates the BAR, and bits 31:3 form the 8 byte aligned offset where bits 2:0 are 641fa41d105SMatthew Gerlachzero. 64220612d24SLukas Bulwahn:: 643fa41d105SMatthew Gerlach 644fa41d105SMatthew Gerlach +----------------------------+ 645fa41d105SMatthew Gerlach |31 Number of DFLS 0| 646fa41d105SMatthew Gerlach +----------------------------+ 647fa41d105SMatthew Gerlach |31 Offset 3|2 BIR 0| 648fa41d105SMatthew Gerlach +----------------------------+ 649fa41d105SMatthew Gerlach . . . 650fa41d105SMatthew Gerlach +----------------------------+ 651fa41d105SMatthew Gerlach |31 Offset 3|2 BIR 0| 652fa41d105SMatthew Gerlach +----------------------------+ 653fa41d105SMatthew Gerlach 654fa41d105SMatthew GerlachBeing able to specify more than one DFL per BAR has been considered, but it 655fa41d105SMatthew Gerlachwas determined the use case did not provide value. Specifying a single DFL 656fa41d105SMatthew Gerlachper BAR simplifies the implementation and allows for extra error checking. 657c220a1faSMauro Carvalho Chehab 658d72260ccSXu Yilun 659d72260ccSXu YilunUserspace driver support for DFL devices 660d72260ccSXu Yilun======================================== 661d72260ccSXu YilunThe purpose of an FPGA is to be reprogrammed with newly developed hardware 662d72260ccSXu Yiluncomponents. New hardware can instantiate a new private feature in the DFL, and 663d72260ccSXu Yilunthen present a DFL device in the system. In some cases users may need a 664d72260ccSXu Yilunuserspace driver for the DFL device: 665d72260ccSXu Yilun 666d72260ccSXu Yilun* Users may need to run some diagnostic test for their hardware. 667d72260ccSXu Yilun* Users may prototype the kernel driver in user space. 668d72260ccSXu Yilun* Some hardware is designed for specific purposes and does not fit into one of 669d72260ccSXu Yilun the standard kernel subsystems. 670d72260ccSXu Yilun 671d72260ccSXu YilunThis requires direct access to MMIO space and interrupt handling from 672d72260ccSXu Yilunuserspace. The uio_dfl module exposes the UIO device interfaces for this 673d72260ccSXu Yilunpurpose. 674d72260ccSXu Yilun 675d72260ccSXu YilunCurrently the uio_dfl driver only supports the Ether Group sub feature, which 676d72260ccSXu Yilunhas no irq in hardware. So the interrupt handling is not added in this driver. 677d72260ccSXu Yilun 678d72260ccSXu YilunUIO_DFL should be selected to enable the uio_dfl module driver. To support a 679d72260ccSXu Yilunnew DFL feature via UIO direct access, its feature id should be added to the 680d72260ccSXu Yilundriver's id_table. 681d72260ccSXu Yilun 682d72260ccSXu Yilun 683c220a1faSMauro Carvalho ChehabOpen discussion 684c220a1faSMauro Carvalho Chehab=============== 685c220a1faSMauro Carvalho ChehabFME driver exports one ioctl (DFL_FPGA_FME_PORT_PR) for partial reconfiguration 686c220a1faSMauro Carvalho Chehabto user now. In the future, if unified user interfaces for reconfiguration are 687c220a1faSMauro Carvalho Chehabadded, FME driver should switch to them from ioctl interface. 688