1.. SPDX-License-Identifier: GPL-2.0 2 3.. _devlink_port: 4 5============ 6Devlink Port 7============ 8 9``devlink-port`` is a port that exists on the device. It has a logically 10separate ingress/egress point of the device. A devlink port can be any one 11of many flavours. A devlink port flavour along with port attributes 12describe what a port represents. 13 14A device driver that intends to publish a devlink port sets the 15devlink port attributes and registers the devlink port. 16 17Devlink port flavours are described below. 18 19.. list-table:: List of devlink port flavours 20 :widths: 33 90 21 22 * - Flavour 23 - Description 24 * - ``DEVLINK_PORT_FLAVOUR_PHYSICAL`` 25 - Any kind of physical port. This can be an eswitch physical port or any 26 other physical port on the device. 27 * - ``DEVLINK_PORT_FLAVOUR_DSA`` 28 - This indicates a DSA interconnect port. 29 * - ``DEVLINK_PORT_FLAVOUR_CPU`` 30 - This indicates a CPU port applicable only to DSA. 31 * - ``DEVLINK_PORT_FLAVOUR_PCI_PF`` 32 - This indicates an eswitch port representing a port of PCI 33 physical function (PF). 34 * - ``DEVLINK_PORT_FLAVOUR_PCI_VF`` 35 - This indicates an eswitch port representing a port of PCI 36 virtual function (VF). 37 * - ``DEVLINK_PORT_FLAVOUR_PCI_SF`` 38 - This indicates an eswitch port representing a port of PCI 39 subfunction (SF). 40 * - ``DEVLINK_PORT_FLAVOUR_VIRTUAL`` 41 - Any virtual port facing the user. 42 43Devlink port can have a different type based on the link layer described below. 44 45.. list-table:: List of devlink port types 46 :widths: 23 90 47 48 * - Type 49 - Description 50 * - ``DEVLINK_PORT_TYPE_ETH`` 51 - Driver should set this port type when a link layer of the port is 52 Ethernet. 53 * - ``DEVLINK_PORT_TYPE_IB`` 54 - Driver should set this port type when a link layer of the port is 55 InfiniBand. 56 * - ``DEVLINK_PORT_TYPE_AUTO`` 57 - This type is indicated by the user when driver should detect the port 58 type automatically. 59 60PCI controllers 61--------------- 62In most cases a PCI device has only one controller. A controller consists of 63potentially multiple physical, virtual functions and subfunctions. A function 64consists of one or more ports. This port is represented by the devlink eswitch 65port. 66 67A PCI device connected to multiple CPUs or multiple PCI root complexes or a 68SmartNIC, however, may have multiple controllers. For a device with multiple 69controllers, each controller is distinguished by a unique controller number. 70An eswitch is on the PCI device which supports ports of multiple controllers. 71 72An example view of a system with two controllers:: 73 74 --------------------------------------------------------- 75 | | 76 | --------- --------- ------- ------- | 77 ----------- | | vf(s) | | sf(s) | |vf(s)| |sf(s)| | 78 | server | | ------- ----/---- ---/----- ------- ---/--- ---/--- | 79 | pci rc |=== | pf0 |______/________/ | pf1 |___/_______/ | 80 | connect | | ------- ------- | 81 ----------- | | controller_num=1 (no eswitch) | 82 ------|-------------------------------------------------- 83 (internal wire) 84 | 85 --------------------------------------------------------- 86 | devlink eswitch ports and reps | 87 | ----------------------------------------------------- | 88 | |ctrl-0 | ctrl-0 | ctrl-0 | ctrl-0 | ctrl-0 |ctrl-0 | | 89 | |pf0 | pf0vfN | pf0sfN | pf1 | pf1vfN |pf1sfN | | 90 | ----------------------------------------------------- | 91 | |ctrl-1 | ctrl-1 | ctrl-1 | ctrl-1 | ctrl-1 |ctrl-1 | | 92 | |pf0 | pf0vfN | pf0sfN | pf1 | pf1vfN |pf1sfN | | 93 | ----------------------------------------------------- | 94 | | 95 | | 96 ----------- | --------- --------- ------- ------- | 97 | smartNIC| | | vf(s) | | sf(s) | |vf(s)| |sf(s)| | 98 | pci rc |==| ------- ----/---- ---/----- ------- ---/--- ---/--- | 99 | connect | | | pf0 |______/________/ | pf1 |___/_______/ | 100 ----------- | ------- ------- | 101 | | 102 | local controller_num=0 (eswitch) | 103 --------------------------------------------------------- 104 105In the above example, the external controller (identified by controller number = 1) 106doesn't have the eswitch. Local controller (identified by controller number = 0) 107has the eswitch. The Devlink instance on the local controller has eswitch 108devlink ports for both the controllers. 109 110A non-zero controller number may also be used for ports that are not external. 111For example, a SmartNIC may have additional local PCI physical functions 112that are managed by the eswitch but are not on an external host. These 113ports use a non-zero controller number to distinguish them from the eswitch 114manager's own functions, while the external flag remains unset. 115 116The ``phys_port_name`` includes the controller prefix (``c<controller_num>``) 117whenever the controller number is non-zero, regardless of the external flag. 118 119Function configuration 120====================== 121 122Users can configure one or more function attributes before enumerating the PCI 123function. Usually it means, user should configure function attribute 124before a bus specific device for the function is created. However, when 125SRIOV is enabled, virtual function devices are created on the PCI bus. 126Hence, function attribute should be configured before binding virtual 127function device to the driver. For subfunctions, this means user should 128configure port function attribute before activating the port function. 129 130A user may set the hardware address of the function using 131`devlink port function set hw_addr` command. For Ethernet port function 132this means a MAC address. 133 134Users may also set the RoCE capability of the function using 135`devlink port function set roce` command. 136 137Users may also set the function as migratable using 138`devlink port function set migratable` command. 139 140Users may also set the IPsec crypto capability of the function using 141`devlink port function set ipsec_crypto` command. 142 143Users may also set the IPsec packet capability of the function using 144`devlink port function set ipsec_packet` command. 145 146The ``migratable`` attribute may be set only on ports with 147``DEVLINK_PORT_FLAVOUR_PCI_VF``. 148 149Users may also set the maximum IO event queues of the function 150using `devlink port function set max_io_eqs` command. 151 152Function attributes 153=================== 154 155MAC address setup 156----------------- 157The configured MAC address of the PCI VF/SF will be used by netdevice and rdma 158device created for the PCI VF/SF. 159 160- Get the MAC address of the VF identified by its unique devlink port index:: 161 162 $ devlink port show pci/0000:06:00.0/2 163 pci/0000:06:00.0/2: type eth netdev enp6s0pf0vf1 flavour pcivf pfnum 0 vfnum 1 164 function: 165 hw_addr 00:00:00:00:00:00 166 167- Set the MAC address of the VF identified by its unique devlink port index:: 168 169 $ devlink port function set pci/0000:06:00.0/2 hw_addr 00:11:22:33:44:55 170 171 $ devlink port show pci/0000:06:00.0/2 172 pci/0000:06:00.0/2: type eth netdev enp6s0pf0vf1 flavour pcivf pfnum 0 vfnum 1 173 function: 174 hw_addr 00:11:22:33:44:55 175 176- Get the MAC address of the SF identified by its unique devlink port index:: 177 178 $ devlink port show pci/0000:06:00.0/32768 179 pci/0000:06:00.0/32768: type eth netdev enp6s0pf0sf88 flavour pcisf pfnum 0 sfnum 88 180 function: 181 hw_addr 00:00:00:00:00:00 182 183- Set the MAC address of the SF identified by its unique devlink port index:: 184 185 $ devlink port function set pci/0000:06:00.0/32768 hw_addr 00:00:00:00:88:88 186 187 $ devlink port show pci/0000:06:00.0/32768 188 pci/0000:06:00.0/32768: type eth netdev enp6s0pf0sf88 flavour pcisf pfnum 0 sfnum 88 189 function: 190 hw_addr 00:00:00:00:88:88 191 192RoCE capability setup 193--------------------- 194Not all PCI VFs/SFs require RoCE capability. 195 196When RoCE capability is disabled, it saves system memory per PCI VF/SF. 197 198When user disables RoCE capability for a VF/SF, user application cannot send or 199receive any RoCE packets through this VF/SF and RoCE GID table for this PCI 200will be empty. 201 202When RoCE capability is disabled in the device using port function attribute, 203VF/SF driver cannot override it. 204 205- Get RoCE capability of the VF device:: 206 207 $ devlink port show pci/0000:06:00.0/2 208 pci/0000:06:00.0/2: type eth netdev enp6s0pf0vf1 flavour pcivf pfnum 0 vfnum 1 209 function: 210 hw_addr 00:00:00:00:00:00 roce enable 211 212- Set RoCE capability of the VF device:: 213 214 $ devlink port function set pci/0000:06:00.0/2 roce disable 215 216 $ devlink port show pci/0000:06:00.0/2 217 pci/0000:06:00.0/2: type eth netdev enp6s0pf0vf1 flavour pcivf pfnum 0 vfnum 1 218 function: 219 hw_addr 00:00:00:00:00:00 roce disable 220 221migratable capability setup 222--------------------------- 223Live migration is the process of transferring a live virtual machine 224from one physical host to another without disrupting its normal 225operation. 226 227User who want PCI VFs to be able to perform live migration need to 228explicitly enable the VF migratable capability. 229 230When user enables migratable capability for a VF, and the HV binds the VF to VFIO driver 231with migration support, the user can migrate the VM with this VF from one HV to a 232different one. 233 234However, when migratable capability is enable, device will disable features which cannot 235be migrated. Thus migratable cap can impose limitations on a VF so let the user decide. 236 237Example of LM with migratable function configuration: 238- Get migratable capability of the VF device:: 239 240 $ devlink port show pci/0000:06:00.0/2 241 pci/0000:06:00.0/2: type eth netdev enp6s0pf0vf1 flavour pcivf pfnum 0 vfnum 1 242 function: 243 hw_addr 00:00:00:00:00:00 migratable disable 244 245- Set migratable capability of the VF device:: 246 247 $ devlink port function set pci/0000:06:00.0/2 migratable enable 248 249 $ devlink port show pci/0000:06:00.0/2 250 pci/0000:06:00.0/2: type eth netdev enp6s0pf0vf1 flavour pcivf pfnum 0 vfnum 1 251 function: 252 hw_addr 00:00:00:00:00:00 migratable enable 253 254- Bind VF to VFIO driver with migration support:: 255 256 $ echo <pci_id> > /sys/bus/pci/devices/0000:08:00.0/driver/unbind 257 $ echo mlx5_vfio_pci > /sys/bus/pci/devices/0000:08:00.0/driver_override 258 $ echo <pci_id> > /sys/bus/pci/devices/0000:08:00.0/driver/bind 259 260Attach VF to the VM. 261Start the VM. 262Perform live migration. 263 264IPsec crypto capability setup 265----------------------------- 266When user enables IPsec crypto capability for a VF, user application can offload 267XFRM state crypto operation (Encrypt/Decrypt) to this VF. 268 269When IPsec crypto capability is disabled (default) for a VF, the XFRM state is 270processed in software by the kernel. 271 272- Get IPsec crypto capability of the VF device:: 273 274 $ devlink port show pci/0000:06:00.0/2 275 pci/0000:06:00.0/2: type eth netdev enp6s0pf0vf1 flavour pcivf pfnum 0 vfnum 1 276 function: 277 hw_addr 00:00:00:00:00:00 ipsec_crypto disabled 278 279- Set IPsec crypto capability of the VF device:: 280 281 $ devlink port function set pci/0000:06:00.0/2 ipsec_crypto enable 282 283 $ devlink port show pci/0000:06:00.0/2 284 pci/0000:06:00.0/2: type eth netdev enp6s0pf0vf1 flavour pcivf pfnum 0 vfnum 1 285 function: 286 hw_addr 00:00:00:00:00:00 ipsec_crypto enabled 287 288IPsec packet capability setup 289----------------------------- 290When user enables IPsec packet capability for a VF, user application can offload 291XFRM state and policy crypto operation (Encrypt/Decrypt) to this VF, as well as 292IPsec encapsulation. 293 294When IPsec packet capability is disabled (default) for a VF, the XFRM state and 295policy is processed in software by the kernel. 296 297- Get IPsec packet capability of the VF device:: 298 299 $ devlink port show pci/0000:06:00.0/2 300 pci/0000:06:00.0/2: type eth netdev enp6s0pf0vf1 flavour pcivf pfnum 0 vfnum 1 301 function: 302 hw_addr 00:00:00:00:00:00 ipsec_packet disabled 303 304- Set IPsec packet capability of the VF device:: 305 306 $ devlink port function set pci/0000:06:00.0/2 ipsec_packet enable 307 308 $ devlink port show pci/0000:06:00.0/2 309 pci/0000:06:00.0/2: type eth netdev enp6s0pf0vf1 flavour pcivf pfnum 0 vfnum 1 310 function: 311 hw_addr 00:00:00:00:00:00 ipsec_packet enabled 312 313Maximum IO events queues setup 314------------------------------ 315When user sets maximum number of IO event queues for a SF or 316a VF, such function driver is limited to consume only enforced 317number of IO event queues. 318 319IO event queues deliver events related to IO queues, including network 320device transmit and receive queues (txq and rxq) and RDMA Queue Pairs (QPs). 321For example, the number of netdevice channels and RDMA device completion 322vectors are derived from the function's IO event queues. Usually, the number 323of interrupt vectors consumed by the driver is limited by the number of IO 324event queues per device, as each of the IO event queues is connected to an 325interrupt vector. 326 327- Get maximum IO event queues of the VF device:: 328 329 $ devlink port show pci/0000:06:00.0/2 330 pci/0000:06:00.0/2: type eth netdev enp6s0pf0vf1 flavour pcivf pfnum 0 vfnum 1 331 function: 332 hw_addr 00:00:00:00:00:00 ipsec_packet disabled max_io_eqs 10 333 334- Set maximum IO event queues of the VF device:: 335 336 $ devlink port function set pci/0000:06:00.0/2 max_io_eqs 32 337 338 $ devlink port show pci/0000:06:00.0/2 339 pci/0000:06:00.0/2: type eth netdev enp6s0pf0vf1 flavour pcivf pfnum 0 vfnum 1 340 function: 341 hw_addr 00:00:00:00:00:00 ipsec_packet disabled max_io_eqs 32 342 343Subfunction 344============ 345 346Subfunction is a lightweight function that has a parent PCI function on which 347it is deployed. Subfunction is created and deployed in unit of 1. Unlike 348SRIOV VFs, a subfunction doesn't require its own PCI virtual function. 349A subfunction communicates with the hardware through the parent PCI function. 350 351To use a subfunction, 3 steps setup sequence is followed: 352 3531) create - create a subfunction; 3542) configure - configure subfunction attributes; 3553) deploy - deploy the subfunction; 356 357Subfunction management is done using devlink port user interface. 358User performs setup on the subfunction management device. 359 360(1) Create 361---------- 362A subfunction is created using a devlink port interface. A user adds the 363subfunction by adding a devlink port of subfunction flavour. The devlink 364kernel code calls down to subfunction management driver (devlink ops) and asks 365it to create a subfunction devlink port. Driver then instantiates the 366subfunction port and any associated objects such as health reporters and 367representor netdevice. 368 369(2) Configure 370------------- 371A subfunction devlink port is created but it is not active yet. That means the 372entities are created on devlink side, the e-switch port representor is created, 373but the subfunction device itself is not created. A user might use e-switch port 374representor to do settings, putting it into bridge, adding TC rules, etc. A user 375might as well configure the hardware address (such as MAC address) of the 376subfunction while subfunction is inactive. 377 378(3) Deploy 379---------- 380Once a subfunction is configured, user must activate it to use it. Upon 381activation, subfunction management driver asks the subfunction management 382device to instantiate the subfunction device on particular PCI function. 383A subfunction device is created on the :ref:`Documentation/driver-api/auxiliary_bus.rst <auxiliary_bus>`. 384At this point a matching subfunction driver binds to the subfunction's auxiliary device. 385 386Rate object management 387====================== 388 389Devlink provides API to manage tx rates of single devlink port or a group. 390This is done through rate objects, which can be one of the two types: 391 392``leaf`` 393 Represents a single devlink port; created/destroyed by the driver. Since leaf 394 have 1to1 mapping to its devlink port, in user space it is referred as 395 ``pci/<bus_addr>/<port_index>``; 396 397``node`` 398 Represents a group of rate objects (leafs and/or nodes); created/deleted by 399 request from the userspace; initially empty (no rate objects added). In 400 userspace it is referred as ``pci/<bus_addr>/<node_name>``, where 401 ``node_name`` can be any identifier, except decimal number, to avoid 402 collisions with leafs. 403 404API allows to configure following rate object's parameters: 405 406``tx_share`` 407 Minimum TX rate value shared among all other rate objects, or rate objects 408 that parts of the parent group, if it is a part of the same group. 409 410``tx_max`` 411 Maximum TX rate value. 412 413``tx_priority`` 414 Allows for usage of strict priority arbiter among siblings. This 415 arbitration scheme attempts to schedule nodes based on their priority 416 as long as the nodes remain within their bandwidth limit. The higher the 417 priority the higher the probability that the node will get selected for 418 scheduling. 419 420``tx_weight`` 421 Allows for usage of Weighted Fair Queuing arbitration scheme among 422 siblings. This arbitration scheme can be used simultaneously with the 423 strict priority. As a node is configured with a higher rate it gets more 424 BW relative to its siblings. Values are relative like a percentage 425 points, they basically tell how much BW should node take relative to 426 its siblings. 427 428``parent`` 429 Parent node name. Parent node rate limits are considered as additional limits 430 to all node children limits. ``tx_max`` is an upper limit for children. 431 ``tx_share`` is a total bandwidth distributed among children. 432 If the device supports cross-function scheduling, the parent can be from a 433 different function of the same underlying device. 434 435``tc_bw`` 436 Allow users to set the bandwidth allocation per traffic class on rate 437 objects. This enables fine-grained QoS configurations by assigning a relative 438 share value to each traffic class. The bandwidth is distributed in proportion 439 to the share value for each class, relative to the sum of all shares. 440 When applied to a non-leaf node, tc_bw determines how bandwidth is shared 441 among its child elements. 442 443``tx_priority`` and ``tx_weight`` can be used simultaneously. In that case 444nodes with the same priority form a WFQ subgroup in the sibling group 445and arbitration among them is based on assigned weights. 446 447Arbitration flow from the high level: 448 449#. Choose a node, or group of nodes with the highest priority that stays 450 within the BW limit and are not blocked. Use ``tx_priority`` as a 451 parameter for this arbitration. 452 453#. If group of nodes have the same priority perform WFQ arbitration on 454 that subgroup. Use ``tx_weight`` as a parameter for this arbitration. 455 456#. Select the winner node, and continue arbitration flow among its children, 457 until leaf node is reached, and the winner is established. 458 459#. If all the nodes from the highest priority sub-group are satisfied, or 460 overused their assigned BW, move to the lower priority nodes. 461 462Driver implementations are allowed to support both or either rate object types 463and setting methods of their parameters. Additionally driver implementation 464may export nodes/leafs and their child-parent relationships. 465 466Terms and Definitions 467===================== 468 469.. list-table:: Terms and Definitions 470 :widths: 22 90 471 472 * - Term 473 - Definitions 474 * - ``PCI device`` 475 - A physical PCI device having one or more PCI buses consists of one or 476 more PCI controllers. 477 * - ``PCI controller`` 478 - A controller consists of potentially multiple physical functions, 479 virtual functions and subfunctions. 480 * - ``Port function`` 481 - An object to manage the function of a port. 482 * - ``Subfunction`` 483 - A lightweight function that has parent PCI function on which it is 484 deployed. 485 * - ``Subfunction device`` 486 - A bus device of the subfunction, usually on a auxiliary bus. 487 * - ``Subfunction driver`` 488 - A device driver for the subfunction auxiliary device. 489 * - ``Subfunction management device`` 490 - A PCI physical function that supports subfunction management. 491 * - ``Subfunction management driver`` 492 - A device driver for PCI physical function that supports 493 subfunction management using devlink port interface. 494 * - ``Subfunction host driver`` 495 - A device driver for PCI physical function that hosts subfunction 496 devices. In most cases it is same as subfunction management driver. When 497 subfunction is used on external controller, subfunction management and 498 host drivers are different. 499