1.. SPDX-License-Identifier: GPL-2.0 2 3================= 4PCI NTB Function 5================= 6 7:Author: Kishon Vijay Abraham I <kishon@ti.com> 8 9PCI Non-Transparent Bridges (NTB) allow two host systems to communicate 10with each other by exposing each host as a device to the other host. 11NTBs typically support the ability to generate interrupts on the remote 12machine, expose memory ranges as BARs, and perform DMA. They also support 13scratchpads, which are areas of memory within the NTB that are accessible 14from both machines. 15 16PCI NTB Function allows two different systems (or hosts) to communicate 17with each other by configuring the endpoint instances in such a way that 18transactions from one system are routed to the other system. 19 20In the below diagram, PCI NTB function configures the SoC with multiple 21PCI Endpoint (EP) instances in such a way that transactions from one EP 22controller are routed to the other EP controller. Once PCI NTB function 23configures the SoC with multiple EP instances, HOST1 and HOST2 can 24communicate with each other using SoC as a bridge. 25 26.. code-block:: text 27 28 +-------------+ +-------------+ 29 | | | | 30 | HOST1 | | HOST2 | 31 | | | | 32 +------^------+ +------^------+ 33 | | 34 | | 35 +---------|-------------------------------------------------|---------+ 36 | +------v------+ +------v------+ | 37 | | | | | | 38 | | EP | | EP | | 39 | | CONTROLLER1 | | CONTROLLER2 | | 40 | | <-----------------------------------> | | 41 | | | | | | 42 | | | | | | 43 | | | SoC With Multiple EP Instances | | | 44 | | | (Configured using NTB Function) | | | 45 | +-------------+ +-------------+ | 46 +---------------------------------------------------------------------+ 47 48Constructs used for Implementing NTB 49==================================== 50 51 1) Config Region 52 2) Self Scratchpad Registers 53 3) Peer Scratchpad Registers 54 4) Doorbell (DB) Registers 55 5) Memory Window (MW) 56 57 58Config Region: 59-------------- 60 61Config Region is a construct that is specific to NTB implemented using NTB 62Endpoint Function Driver. The host and endpoint side NTB function driver will 63exchange information with each other using this region. Config Region has 64Control/Status Registers for configuring the Endpoint Controller. Host can 65write into this region for configuring the outbound Address Translation Unit 66(ATU) and to indicate the link status. Endpoint can indicate the status of 67commands issued by host in this region. Endpoint can also indicate the 68scratchpad offset and number of memory windows to the host using this region. 69 70The format of Config Region is given below. All the fields here are 32 bits. 71 72.. code-block:: text 73 74 +------------------------+ 75 | COMMAND | 76 +------------------------+ 77 | ARGUMENT | 78 +------------------------+ 79 | STATUS | 80 +------------------------+ 81 | TOPOLOGY | 82 +------------------------+ 83 | ADDRESS (LOWER 32) | 84 +------------------------+ 85 | ADDRESS (UPPER 32) | 86 +------------------------+ 87 | SIZE | 88 +------------------------+ 89 | NO OF MEMORY WINDOW | 90 +------------------------+ 91 | MEMORY WINDOW1 OFFSET | 92 +------------------------+ 93 | SPAD OFFSET | 94 +------------------------+ 95 | SPAD COUNT | 96 +------------------------+ 97 | DB ENTRY SIZE | 98 +------------------------+ 99 | DB DATA | 100 +------------------------+ 101 | : | 102 +------------------------+ 103 | : | 104 +------------------------+ 105 | DB DATA | 106 +------------------------+ 107 108 109 COMMAND: 110 111 NTB function supports three commands: 112 113 CMD_CONFIGURE_DOORBELL (0x1): Command to configure doorbell. Before 114 invoking this command, the host should allocate and initialize 115 MSI/MSI-X vectors (i.e., initialize the MSI/MSI-X Capability in the 116 Endpoint). The endpoint on receiving this command will configure 117 the outbound ATU such that transactions to Doorbell BAR will be routed 118 to the MSI/MSI-X address programmed by the host. The ARGUMENT 119 register should be populated with number of DBs to configure (in the 120 lower 16 bits) and if MSI or MSI-X should be configured (BIT 16). 121 122 CMD_CONFIGURE_MW (0x2): Command to configure memory window (MW). The 123 host invokes this command after allocating a buffer that can be 124 accessed by remote host. The allocated address should be programmed 125 in the ADDRESS register (64 bit), the size should be programmed in 126 the SIZE register and the memory window index should be programmed 127 in the ARGUMENT register. The endpoint on receiving this command 128 will configure the outbound ATU such that transactions to MW BAR 129 are routed to the address provided by the host. 130 131 CMD_LINK_UP (0x3): Command to indicate an NTB application is 132 bound to the EP device on the host side. Once the endpoint 133 receives this command from both the hosts, the endpoint will 134 raise a LINK_UP event to both the hosts to indicate the host 135 NTB applications can start communicating with each other. 136 137 ARGUMENT: 138 139 The value of this register is based on the commands issued in 140 command register. See COMMAND section for more information. 141 142 TOPOLOGY: 143 144 Set to NTB_TOPO_B2B_USD for Primary interface 145 Set to NTB_TOPO_B2B_DSD for Secondary interface 146 147 ADDRESS/SIZE: 148 149 Address and Size to be used while configuring the memory window. 150 See "CMD_CONFIGURE_MW" for more info. 151 152 MEMORY WINDOW1 OFFSET: 153 154 Memory Window 1 and Doorbell registers are packed together in the 155 same BAR. The initial portion of the region will have doorbell 156 registers and the latter portion of the region is for memory window 1. 157 This register will specify the offset of the memory window 1. 158 159 NO OF MEMORY WINDOW: 160 161 Specifies the number of memory windows supported by the NTB device. 162 163 SPAD OFFSET: 164 165 Self scratchpad region and config region are packed together in the 166 same BAR. The initial portion of the region will have config region 167 and the latter portion of the region is for self scratchpad. This 168 register will specify the offset of the self scratchpad registers. 169 170 SPAD COUNT: 171 172 Specifies the number of scratchpad registers supported by the NTB 173 device. 174 175 DB ENTRY SIZE: 176 177 Used to determine the offset within the DB BAR that should be written 178 in order to raise doorbell. EPF NTB can use either MSI or MSI-X to 179 ring doorbell (MSI-X support will be added later). MSI uses same 180 address for all the interrupts and MSI-X can provide different 181 addresses for different interrupts. The MSI/MSI-X address is provided 182 by the host and the address it gives is based on the MSI/MSI-X 183 implementation supported by the host. For instance, ARM platform 184 using GIC ITS will have the same MSI-X address for all the interrupts. 185 In order to support all the combinations and use the same mechanism 186 for both MSI and MSI-X, EPF NTB allocates a separate region in the 187 Outbound Address Space for each of the interrupts. This region will 188 be mapped to the MSI/MSI-X address provided by the host. If a host 189 provides the same address for all the interrupts, all the regions 190 will be translated to the same address. If a host provides different 191 addresses, the regions will be translated to different addresses. This 192 will ensure there is no difference while raising the doorbell. 193 194 DB DATA: 195 196 EPF NTB supports 32 interrupts, so there are 32 DB DATA registers. 197 This holds the MSI/MSI-X data that has to be written to MSI address 198 for raising doorbell interrupt. This will be populated by EPF NTB 199 while invoking CMD_CONFIGURE_DOORBELL. 200 201Scratchpad Registers: 202--------------------- 203 204 Each host has its own register space allocated in the memory of NTB endpoint 205 controller. They are both readable and writable from both sides of the bridge. 206 They are used by applications built over NTB and can be used to pass control 207 and status information between both sides of a device. 208 209 Scratchpad registers has 2 parts 210 1) Self Scratchpad: Host's own register space 211 2) Peer Scratchpad: Remote host's register space. 212 213Doorbell Registers: 214------------------- 215 216 Doorbell Registers are used by the hosts to interrupt each other. 217 218Memory Window: 219-------------- 220 221 Actual transfer of data between the two hosts will happen using the 222 memory window. 223 224Modeling Constructs: 225==================== 226 227There are 5 or more distinct regions (config, self scratchpad, peer 228scratchpad, doorbell, one or more memory windows) to be modeled to achieve 229NTB functionality. At least one memory window is required while more than 230one is permitted. All these regions should be mapped to BARs for hosts to 231access these regions. 232 233If one 32-bit BAR is allocated for each of these regions, the scheme would 234look like this: 235 236====== =============== 237BAR NO CONSTRUCTS USED 238====== =============== 239BAR0 Config Region 240BAR1 Self Scratchpad 241BAR2 Peer Scratchpad 242BAR3 Doorbell 243BAR4 Memory Window 1 244BAR5 Memory Window 2 245====== =============== 246 247However if we allocate a separate BAR for each of the regions, there would not 248be enough BARs for all the regions in a platform that supports only 64-bit 249BARs. 250 251In order to be supported by most of the platforms, the regions should be 252packed and mapped to BARs in a way that provides NTB functionality and 253also makes sure the host doesn't access any region that it is not supposed 254to. 255 256The following scheme is used in EPF NTB Function: 257 258====== =============================== 259BAR NO CONSTRUCTS USED 260====== =============================== 261BAR0 Config Region + Self Scratchpad 262BAR1 Peer Scratchpad 263BAR2 Doorbell + Memory Window 1 264BAR3 Memory Window 2 265BAR4 Memory Window 3 266BAR5 Memory Window 4 267====== =============================== 268 269With this scheme, for the basic NTB functionality 3 BARs should be sufficient. 270 271Modeling Config/Scratchpad Region: 272---------------------------------- 273 274.. code-block:: text 275 276 +-----------------+------->+------------------+ +-----------------+ 277 | BAR0 | | CONFIG REGION | | BAR0 | 278 +-----------------+----+ +------------------+<-------+-----------------+ 279 | BAR1 | | |SCRATCHPAD REGION | | BAR1 | 280 +-----------------+ +-->+------------------+<-------+-----------------+ 281 | BAR2 | Local Memory | BAR2 | 282 +-----------------+ +-----------------+ 283 | BAR3 | | BAR3 | 284 +-----------------+ +-----------------+ 285 | BAR4 | | BAR4 | 286 +-----------------+ +-----------------+ 287 | BAR5 | | BAR5 | 288 +-----------------+ +-----------------+ 289 EP CONTROLLER 1 EP CONTROLLER 2 290 291Above diagram shows Config region + Scratchpad region for HOST1 (connected to 292EP controller 1) allocated in local memory. The HOST1 can access the config 293region and scratchpad region (self scratchpad) using BAR0 of EP controller 1. 294The peer host (HOST2 connected to EP controller 2) can also access this 295scratchpad region (peer scratchpad) using BAR1 of EP controller 2. This 296diagram shows the case where Config region and Scratchpad regions are allocated 297for HOST1, however the same is applicable for HOST2. 298 299Modeling Doorbell/Memory Window 1: 300---------------------------------- 301 302.. code-block:: text 303 304 +-----------------+ +----->+----------------+-----------+-----------------+ 305 | BAR0 | | | Doorbell 1 +-----------> MSI-X ADDRESS 1 | 306 +-----------------+ | +----------------+ +-----------------+ 307 | BAR1 | | | Doorbell 2 +---------+ | | 308 +-----------------+----+ +----------------+ | | | 309 | BAR2 | | Doorbell 3 +-------+ | +-----------------+ 310 +-----------------+----+ +----------------+ | +-> MSI-X ADDRESS 2 | 311 | BAR3 | | | Doorbell 4 +-----+ | +-----------------+ 312 +-----------------+ | |----------------+ | | | | 313 | BAR4 | | | | | | +-----------------+ 314 +-----------------+ | | MW1 +---+ | +-->+ MSI-X ADDRESS 3|| 315 | BAR5 | | | | | | +-----------------+ 316 +-----------------+ +----->-----------------+ | | | | 317 EP CONTROLLER 1 | | | | +-----------------+ 318 | | | +---->+ MSI-X ADDRESS 4 | 319 +----------------+ | +-----------------+ 320 EP CONTROLLER 2 | | | 321 (OB SPACE) | | | 322 +-------> MW1 | 323 | | 324 | | 325 +-----------------+ 326 | | 327 | | 328 | | 329 | | 330 | | 331 +-----------------+ 332 PCI Address Space 333 (Managed by HOST2) 334 335Above diagram shows how the doorbell and memory window 1 is mapped so that 336HOST1 can raise doorbell interrupt on HOST2 and also how HOST1 can access 337buffers exposed by HOST2 using memory window1 (MW1). Here doorbell and 338memory window 1 regions are allocated in EP controller 2 outbound (OB) address 339space. Allocating and configuring BARs for doorbell and memory window1 340is done during the initialization phase of NTB endpoint function driver. 341Mapping from EP controller 2 OB space to PCI address space is done when HOST2 342sends CMD_CONFIGURE_MW/CMD_CONFIGURE_DOORBELL. 343 344Modeling Optional Memory Windows: 345--------------------------------- 346 347This is modeled the same was as MW1 but each of the additional memory windows 348is mapped to separate BARs. 349