1.. include:: <isonum.txt> 2 3========================================================= 4DPAA2 (Data Path Acceleration Architecture Gen2) Overview 5========================================================= 6 7:Copyright: |copy| 2015 Freescale Semiconductor Inc. 8:Copyright: |copy| 2018 NXP 9 10This document provides an overview of the Freescale DPAA2 architecture 11and how it is integrated into the Linux kernel. 12 13Introduction 14============ 15 16DPAA2 is a hardware architecture designed for high-speeed network 17packet processing. DPAA2 consists of sophisticated mechanisms for 18processing Ethernet packets, queue management, buffer management, 19autonomous L2 switching, virtual Ethernet bridging, and accelerator 20(e.g. crypto) sharing. 21 22A DPAA2 hardware component called the Management Complex (or MC) manages the 23DPAA2 hardware resources. The MC provides an object-based abstraction for 24software drivers to use the DPAA2 hardware. 25The MC uses DPAA2 hardware resources such as queues, buffer pools, and 26network ports to create functional objects/devices such as network 27interfaces, an L2 switch, or accelerator instances. 28The MC provides memory-mapped I/O command interfaces (MC portals) 29which DPAA2 software drivers use to operate on DPAA2 objects. 30 31The diagram below shows an overview of the DPAA2 resource management 32architecture:: 33 34 +--------------------------------------+ 35 | OS | 36 | DPAA2 drivers | 37 | | | 38 +-----------------------------|--------+ 39 | 40 | (create,discover,connect 41 | config,use,destroy) 42 | 43 DPAA2 | 44 +------------------------| mc portal |-+ 45 | | | 46 | +- - - - - - - - - - - - -V- - -+ | 47 | | | | 48 | | Management Complex (MC) | | 49 | | | | 50 | +- - - - - - - - - - - - - - - -+ | 51 | | 52 | Hardware Hardware | 53 | Resources Objects | 54 | --------- ------- | 55 | -queues -DPRC | 56 | -buffer pools -DPMCP | 57 | -Eth MACs/ports -DPIO | 58 | -network interface -DPNI | 59 | profiles -DPMAC | 60 | -queue portals -DPBP | 61 | -MC portals ... | 62 | ... | 63 | | 64 +--------------------------------------+ 65 66 67The MC mediates operations such as create, discover, 68connect, configuration, and destroy. Fast-path operations 69on data, such as packet transmit/receive, are not mediated by 70the MC and are done directly using memory mapped regions in 71DPIO objects. 72 73Overview of DPAA2 Objects 74========================= 75 76The section provides a brief overview of some key DPAA2 objects. 77A simple scenario is described illustrating the objects involved 78in creating a network interfaces. 79 80DPRC (Datapath Resource Container) 81---------------------------------- 82 83A DPRC is a container object that holds all the other 84types of DPAA2 objects. In the example diagram below there 85are 8 objects of 5 types (DPMCP, DPIO, DPBP, DPNI, and DPMAC) 86in the container. 87 88:: 89 90 +---------------------------------------------------------+ 91 | DPRC | 92 | | 93 | +-------+ +-------+ +-------+ +-------+ +-------+ | 94 | | DPMCP | | DPIO | | DPBP | | DPNI | | DPMAC | | 95 | +-------+ +-------+ +-------+ +---+---+ +---+---+ | 96 | | DPMCP | | DPIO | | 97 | +-------+ +-------+ | 98 | | DPMCP | | 99 | +-------+ | 100 | | 101 +---------------------------------------------------------+ 102 103From the point of view of an OS, a DPRC behaves similar to a plug and 104play bus, like PCI. DPRC commands can be used to enumerate the contents 105of the DPRC, discover the hardware objects present (including mappable 106regions and interrupts). 107 108:: 109 110 DPRC.1 (bus) 111 | 112 +--+--------+-------+-------+-------+ 113 | | | | | 114 DPMCP.1 DPIO.1 DPBP.1 DPNI.1 DPMAC.1 115 DPMCP.2 DPIO.2 116 DPMCP.3 117 118Hardware objects can be created and destroyed dynamically, providing 119the ability to hot plug/unplug objects in and out of the DPRC. 120 121A DPRC has a mappable MMIO region (an MC portal) that can be used 122to send MC commands. It has an interrupt for status events (like 123hotplug). 124All objects in a container share the same hardware "isolation context". 125This means that with respect to an IOMMU the isolation granularity 126is at the DPRC (container) level, not at the individual object 127level. 128 129DPRCs can be defined statically and populated with objects 130via a config file passed to the MC when firmware starts it. 131 132DPAA2 Objects for an Ethernet Network Interface 133----------------------------------------------- 134 135A typical Ethernet NIC is monolithic-- the NIC device contains TX/RX 136queuing mechanisms, configuration mechanisms, buffer management, 137physical ports, and interrupts. DPAA2 uses a more granular approach 138utilizing multiple hardware objects. Each object provides specialized 139functions. Groups of these objects are used by software to provide 140Ethernet network interface functionality. This approach provides 141efficient use of finite hardware resources, flexibility, and 142performance advantages. 143 144The diagram below shows the objects needed for a simple 145network interface configuration on a system with 2 CPUs. 146 147:: 148 149 +---+---+ +---+---+ 150 CPU0 CPU1 151 +---+---+ +---+---+ 152 | | 153 +---+---+ +---+---+ 154 DPIO DPIO 155 +---+---+ +---+---+ 156 \ / 157 \ / 158 \ / 159 +---+---+ 160 DPNI --- DPBP,DPMCP 161 +---+---+ 162 | 163 | 164 +---+---+ 165 DPMAC 166 +---+---+ 167 | 168 port/PHY 169 170Below the objects are described. For each object a brief description 171is provided along with a summary of the kinds of operations the object 172supports and a summary of key resources of the object (MMIO regions 173and IRQs). 174 175DPMAC (Datapath Ethernet MAC) 176~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 177Represents an Ethernet MAC, a hardware device that connects to an Ethernet 178PHY and allows physical transmission and reception of Ethernet frames. 179 180- MMIO regions: none 181- IRQs: DPNI link change 182- commands: set link up/down, link config, get stats, 183 IRQ config, enable, reset 184 185DPNI (Datapath Network Interface) 186Contains TX/RX queues, network interface configuration, and RX buffer pool 187configuration mechanisms. The TX/RX queues are in memory and are identified 188by queue number. 189 190- MMIO regions: none 191- IRQs: link state 192- commands: port config, offload config, queue config, 193 parse/classify config, IRQ config, enable, reset 194 195DPIO (Datapath I/O) 196~~~~~~~~~~~~~~~~~~~ 197Provides interfaces to enqueue and dequeue 198packets and do hardware buffer pool management operations. The DPAA2 199architecture separates the mechanism to access queues (the DPIO object) 200from the queues themselves. The DPIO provides an MMIO interface to 201enqueue/dequeue packets. To enqueue something a descriptor is written 202to the DPIO MMIO region, which includes the target queue number. 203There will typically be one DPIO assigned to each CPU. This allows all 204CPUs to simultaneously perform enqueue/dequeued operations. DPIOs are 205expected to be shared by different DPAA2 drivers. 206 207- MMIO regions: queue operations, buffer management 208- IRQs: data availability, congestion notification, buffer 209 pool depletion 210- commands: IRQ config, enable, reset 211 212DPBP (Datapath Buffer Pool) 213~~~~~~~~~~~~~~~~~~~~~~~~~~~ 214Represents a hardware buffer pool. 215 216- MMIO regions: none 217- IRQs: none 218- commands: enable, reset 219 220DPMCP (Datapath MC Portal) 221~~~~~~~~~~~~~~~~~~~~~~~~~~ 222Provides an MC command portal. 223Used by drivers to send commands to the MC to manage 224objects. 225 226- MMIO regions: MC command portal 227- IRQs: command completion 228- commands: IRQ config, enable, reset 229 230Object Connections 231================== 232Some objects have explicit relationships that must 233be configured: 234 235- DPNI <--> DPMAC 236- DPNI <--> DPNI 237- DPNI <--> L2-switch-port 238 239 A DPNI must be connected to something such as a DPMAC, 240 another DPNI, or L2 switch port. The DPNI connection 241 is made via a DPRC command. 242 243:: 244 245 +-------+ +-------+ 246 | DPNI | | DPMAC | 247 +---+---+ +---+---+ 248 | | 249 +==========+ 250 251- DPNI <--> DPBP 252 253 A network interface requires a 'buffer pool' (DPBP 254 object) which provides a list of pointers to memory 255 where received Ethernet data is to be copied. The 256 Ethernet driver configures the DPBPs associated with 257 the network interface. 258 259Interrupts 260========== 261All interrupts generated by DPAA2 objects are message 262interrupts. At the hardware level message interrupts 263generated by devices will normally have 3 components-- 2641) a non-spoofable 'device-id' expressed on the hardware 265bus, 2) an address, 3) a data value. 266 267In the case of DPAA2 devices/objects, all objects in the 268same container/DPRC share the same 'device-id'. 269For ARM-based SoC this is the same as the stream ID. 270 271 272DPAA2 Linux Drivers Overview 273============================ 274 275This section provides an overview of the Linux kernel drivers for 276DPAA2-- 1) the bus driver and associated "DPAA2 infrastructure" 277drivers and 2) functional object drivers (such as Ethernet). 278 279As described previously, a DPRC is a container that holds the other 280types of DPAA2 objects. It is functionally similar to a plug-and-play 281bus controller. 282Each object in the DPRC is a Linux "device" and is bound to a driver. 283The diagram below shows the Linux drivers involved in a networking 284scenario and the objects bound to each driver. A brief description 285of each driver follows. 286 287:: 288 289 +------------+ 290 | OS Network | 291 | Stack | 292 +------------+ +------------+ 293 | Allocator |. . . . . . . | Ethernet | 294 |(DPMCP,DPBP)| | (DPNI) | 295 +-.----------+ +---+---+----+ 296 . . ^ | 297 . . <data avail, | | <enqueue, 298 . . tx confirm> | | dequeue> 299 +-------------+ . | | 300 | DPRC driver | . +---+---V----+ +---------+ 301 | (DPRC) | . . . . . .| DPIO driver| | MAC | 302 +----------+--+ | (DPIO) | | (DPMAC) | 303 | +------+-----+ +-----+---+ 304 |<dev add/remove> | | 305 | | | 306 +--------+----------+ | +--+---+ 307 | MC-bus driver | | | PHY | 308 | | | |driver| 309 | /bus/fsl-mc | | +--+---+ 310 +-------------------+ | | 311 | | 312 ========================= HARDWARE =========|=================|====== 313 DPIO | 314 | | 315 DPNI---DPBP | 316 | | 317 DPMAC | 318 | | 319 PHY ---------------+ 320 ============================================|======================== 321 322A brief description of each driver is provided below. 323 324MC-bus driver 325------------- 326The MC-bus driver is a platform driver and is probed from a 327node in the device tree (compatible "fsl,qoriq-mc") passed in by boot 328firmware. It is responsible for bootstrapping the DPAA2 kernel 329infrastructure. 330Key functions include: 331 332- registering a new bus type named "fsl-mc" with the kernel, 333 and implementing bus call-backs (e.g. match/uevent/dev_groups) 334- implementing APIs for DPAA2 driver registration and for device 335 add/remove 336- creates an MSI IRQ domain 337- doing a 'device add' to expose the 'root' DPRC, in turn triggering 338 a bind of the root DPRC to the DPRC driver 339 340The binding for the MC-bus device-tree node can be consulted at 341*Documentation/devicetree/bindings/misc/fsl,qoriq-mc.txt*. 342The sysfs bind/unbind interfaces for the MC-bus can be consulted at 343*Documentation/ABI/testing/sysfs-bus-fsl-mc*. 344 345DPRC driver 346----------- 347The DPRC driver is bound to DPRC objects and does runtime management 348of a bus instance. It performs the initial bus scan of the DPRC 349and handles interrupts for container events such as hot plug by 350re-scanning the DPRC. 351 352Allocator 353--------- 354Certain objects such as DPMCP and DPBP are generic and fungible, 355and are intended to be used by other drivers. For example, 356the DPAA2 Ethernet driver needs: 357 358- DPMCPs to send MC commands, to configure network interfaces 359- DPBPs for network buffer pools 360 361The allocator driver registers for these allocatable object types 362and those objects are bound to the allocator when the bus is probed. 363The allocator maintains a pool of objects that are available for 364allocation by other DPAA2 drivers. 365 366DPIO driver 367----------- 368The DPIO driver is bound to DPIO objects and provides services that allow 369other drivers such as the Ethernet driver to enqueue and dequeue data for 370their respective objects. 371Key services include: 372 373- data availability notifications 374- hardware queuing operations (enqueue and dequeue of data) 375- hardware buffer pool management 376 377To transmit a packet the Ethernet driver puts data on a queue and 378invokes a DPIO API. For receive, the Ethernet driver registers 379a data availability notification callback. To dequeue a packet 380a DPIO API is used. 381There is typically one DPIO object per physical CPU for optimum 382performance, allowing different CPUs to simultaneously enqueue 383and dequeue data. 384 385The DPIO driver operates on behalf of all DPAA2 drivers 386active in the kernel-- Ethernet, crypto, compression, 387etc. 388 389Ethernet driver 390--------------- 391The Ethernet driver is bound to a DPNI and implements the kernel 392interfaces needed to connect the DPAA2 network interface to 393the network stack. 394Each DPNI corresponds to a Linux network interface. 395 396MAC driver 397---------- 398An Ethernet PHY is an off-chip, board specific component and is managed 399by the appropriate PHY driver via an mdio bus. The MAC driver 400plays a role of being a proxy between the PHY driver and the 401MC. It does this proxy via the MC commands to a DPMAC object. 402If the PHY driver signals a link change, the MAC driver notifies 403the MC via a DPMAC command. If a network interface is brought 404up or down, the MC notifies the DPMAC driver via an interrupt and 405the driver can take appropriate action. 406