1.. SPDX-License-Identifier: GPL-2.0 2 3.. _devlink_port: 4 5============ 6Devlink Port 7============ 8 9``devlink-port`` is a port that exists on the device. It has a logically 10separate ingress/egress point of the device. A devlink port can be any one 11of many flavours. A devlink port flavour along with port attributes 12describe what a port represents. 13 14A device driver that intends to publish a devlink port sets the 15devlink port attributes and registers the devlink port. 16 17Devlink port flavours are described below. 18 19.. list-table:: List of devlink port flavours 20 :widths: 33 90 21 22 * - Flavour 23 - Description 24 * - ``DEVLINK_PORT_FLAVOUR_PHYSICAL`` 25 - Any kind of physical port. This can be an eswitch physical port or any 26 other physical port on the device. 27 * - ``DEVLINK_PORT_FLAVOUR_DSA`` 28 - This indicates a DSA interconnect port. 29 * - ``DEVLINK_PORT_FLAVOUR_CPU`` 30 - This indicates a CPU port applicable only to DSA. 31 * - ``DEVLINK_PORT_FLAVOUR_PCI_PF`` 32 - This indicates an eswitch port representing a port of PCI 33 physical function (PF). 34 * - ``DEVLINK_PORT_FLAVOUR_PCI_VF`` 35 - This indicates an eswitch port representing a port of PCI 36 virtual function (VF). 37 * - ``DEVLINK_PORT_FLAVOUR_PCI_SF`` 38 - This indicates an eswitch port representing a port of PCI 39 subfunction (SF). 40 * - ``DEVLINK_PORT_FLAVOUR_VIRTUAL`` 41 - This indicates a virtual port for the PCI virtual function. 42 43Devlink port can have a different type based on the link layer described below. 44 45.. list-table:: List of devlink port types 46 :widths: 23 90 47 48 * - Type 49 - Description 50 * - ``DEVLINK_PORT_TYPE_ETH`` 51 - Driver should set this port type when a link layer of the port is 52 Ethernet. 53 * - ``DEVLINK_PORT_TYPE_IB`` 54 - Driver should set this port type when a link layer of the port is 55 InfiniBand. 56 * - ``DEVLINK_PORT_TYPE_AUTO`` 57 - This type is indicated by the user when driver should detect the port 58 type automatically. 59 60PCI controllers 61--------------- 62In most cases a PCI device has only one controller. A controller consists of 63potentially multiple physical, virtual functions and subfunctions. A function 64consists of one or more ports. This port is represented by the devlink eswitch 65port. 66 67A PCI device connected to multiple CPUs or multiple PCI root complexes or a 68SmartNIC, however, may have multiple controllers. For a device with multiple 69controllers, each controller is distinguished by a unique controller number. 70An eswitch is on the PCI device which supports ports of multiple controllers. 71 72An example view of a system with two controllers:: 73 74 --------------------------------------------------------- 75 | | 76 | --------- --------- ------- ------- | 77 ----------- | | vf(s) | | sf(s) | |vf(s)| |sf(s)| | 78 | server | | ------- ----/---- ---/----- ------- ---/--- ---/--- | 79 | pci rc |=== | pf0 |______/________/ | pf1 |___/_______/ | 80 | connect | | ------- ------- | 81 ----------- | | controller_num=1 (no eswitch) | 82 ------|-------------------------------------------------- 83 (internal wire) 84 | 85 --------------------------------------------------------- 86 | devlink eswitch ports and reps | 87 | ----------------------------------------------------- | 88 | |ctrl-0 | ctrl-0 | ctrl-0 | ctrl-0 | ctrl-0 |ctrl-0 | | 89 | |pf0 | pf0vfN | pf0sfN | pf1 | pf1vfN |pf1sfN | | 90 | ----------------------------------------------------- | 91 | |ctrl-1 | ctrl-1 | ctrl-1 | ctrl-1 | ctrl-1 |ctrl-1 | | 92 | |pf0 | pf0vfN | pf0sfN | pf1 | pf1vfN |pf1sfN | | 93 | ----------------------------------------------------- | 94 | | 95 | | 96 ----------- | --------- --------- ------- ------- | 97 | smartNIC| | | vf(s) | | sf(s) | |vf(s)| |sf(s)| | 98 | pci rc |==| ------- ----/---- ---/----- ------- ---/--- ---/--- | 99 | connect | | | pf0 |______/________/ | pf1 |___/_______/ | 100 ----------- | ------- ------- | 101 | | 102 | local controller_num=0 (eswitch) | 103 --------------------------------------------------------- 104 105In the above example, the external controller (identified by controller number = 1) 106doesn't have the eswitch. Local controller (identified by controller number = 0) 107has the eswitch. The Devlink instance on the local controller has eswitch 108devlink ports for both the controllers. 109 110Function configuration 111====================== 112 113Users can configure one or more function attributes before enumerating the PCI 114function. Usually it means, user should configure function attribute 115before a bus specific device for the function is created. However, when 116SRIOV is enabled, virtual function devices are created on the PCI bus. 117Hence, function attribute should be configured before binding virtual 118function device to the driver. For subfunctions, this means user should 119configure port function attribute before activating the port function. 120 121A user may set the hardware address of the function using 122`devlink port function set hw_addr` command. For Ethernet port function 123this means a MAC address. 124 125Users may also set the RoCE capability of the function using 126`devlink port function set roce` command. 127 128Users may also set the function as migratable using 129`devlink port function set migratable` command. 130 131Users may also set the IPsec crypto capability of the function using 132`devlink port function set ipsec_crypto` command. 133 134Users may also set the IPsec packet capability of the function using 135`devlink port function set ipsec_packet` command. 136 137Function attributes 138=================== 139 140MAC address setup 141----------------- 142The configured MAC address of the PCI VF/SF will be used by netdevice and rdma 143device created for the PCI VF/SF. 144 145- Get the MAC address of the VF identified by its unique devlink port index:: 146 147 $ devlink port show pci/0000:06:00.0/2 148 pci/0000:06:00.0/2: type eth netdev enp6s0pf0vf1 flavour pcivf pfnum 0 vfnum 1 149 function: 150 hw_addr 00:00:00:00:00:00 151 152- Set the MAC address of the VF identified by its unique devlink port index:: 153 154 $ devlink port function set pci/0000:06:00.0/2 hw_addr 00:11:22:33:44:55 155 156 $ devlink port show pci/0000:06:00.0/2 157 pci/0000:06:00.0/2: type eth netdev enp6s0pf0vf1 flavour pcivf pfnum 0 vfnum 1 158 function: 159 hw_addr 00:11:22:33:44:55 160 161- Get the MAC address of the SF identified by its unique devlink port index:: 162 163 $ devlink port show pci/0000:06:00.0/32768 164 pci/0000:06:00.0/32768: type eth netdev enp6s0pf0sf88 flavour pcisf pfnum 0 sfnum 88 165 function: 166 hw_addr 00:00:00:00:00:00 167 168- Set the MAC address of the SF identified by its unique devlink port index:: 169 170 $ devlink port function set pci/0000:06:00.0/32768 hw_addr 00:00:00:00:88:88 171 172 $ devlink port show pci/0000:06:00.0/32768 173 pci/0000:06:00.0/32768: type eth netdev enp6s0pf0sf88 flavour pcisf pfnum 0 sfnum 88 174 function: 175 hw_addr 00:00:00:00:88:88 176 177RoCE capability setup 178--------------------- 179Not all PCI VFs/SFs require RoCE capability. 180 181When RoCE capability is disabled, it saves system memory per PCI VF/SF. 182 183When user disables RoCE capability for a VF/SF, user application cannot send or 184receive any RoCE packets through this VF/SF and RoCE GID table for this PCI 185will be empty. 186 187When RoCE capability is disabled in the device using port function attribute, 188VF/SF driver cannot override it. 189 190- Get RoCE capability of the VF device:: 191 192 $ devlink port show pci/0000:06:00.0/2 193 pci/0000:06:00.0/2: type eth netdev enp6s0pf0vf1 flavour pcivf pfnum 0 vfnum 1 194 function: 195 hw_addr 00:00:00:00:00:00 roce enable 196 197- Set RoCE capability of the VF device:: 198 199 $ devlink port function set pci/0000:06:00.0/2 roce disable 200 201 $ devlink port show pci/0000:06:00.0/2 202 pci/0000:06:00.0/2: type eth netdev enp6s0pf0vf1 flavour pcivf pfnum 0 vfnum 1 203 function: 204 hw_addr 00:00:00:00:00:00 roce disable 205 206migratable capability setup 207--------------------------- 208Live migration is the process of transferring a live virtual machine 209from one physical host to another without disrupting its normal 210operation. 211 212User who want PCI VFs to be able to perform live migration need to 213explicitly enable the VF migratable capability. 214 215When user enables migratable capability for a VF, and the HV binds the VF to VFIO driver 216with migration support, the user can migrate the VM with this VF from one HV to a 217different one. 218 219However, when migratable capability is enable, device will disable features which cannot 220be migrated. Thus migratable cap can impose limitations on a VF so let the user decide. 221 222Example of LM with migratable function configuration: 223- Get migratable capability of the VF device:: 224 225 $ devlink port show pci/0000:06:00.0/2 226 pci/0000:06:00.0/2: type eth netdev enp6s0pf0vf1 flavour pcivf pfnum 0 vfnum 1 227 function: 228 hw_addr 00:00:00:00:00:00 migratable disable 229 230- Set migratable capability of the VF device:: 231 232 $ devlink port function set pci/0000:06:00.0/2 migratable enable 233 234 $ devlink port show pci/0000:06:00.0/2 235 pci/0000:06:00.0/2: type eth netdev enp6s0pf0vf1 flavour pcivf pfnum 0 vfnum 1 236 function: 237 hw_addr 00:00:00:00:00:00 migratable enable 238 239- Bind VF to VFIO driver with migration support:: 240 241 $ echo <pci_id> > /sys/bus/pci/devices/0000:08:00.0/driver/unbind 242 $ echo mlx5_vfio_pci > /sys/bus/pci/devices/0000:08:00.0/driver_override 243 $ echo <pci_id> > /sys/bus/pci/devices/0000:08:00.0/driver/bind 244 245Attach VF to the VM. 246Start the VM. 247Perform live migration. 248 249IPsec crypto capability setup 250----------------------------- 251When user enables IPsec crypto capability for a VF, user application can offload 252XFRM state crypto operation (Encrypt/Decrypt) to this VF. 253 254When IPsec crypto capability is disabled (default) for a VF, the XFRM state is 255processed in software by the kernel. 256 257- Get IPsec crypto capability of the VF device:: 258 259 $ devlink port show pci/0000:06:00.0/2 260 pci/0000:06:00.0/2: type eth netdev enp6s0pf0vf1 flavour pcivf pfnum 0 vfnum 1 261 function: 262 hw_addr 00:00:00:00:00:00 ipsec_crypto disabled 263 264- Set IPsec crypto capability of the VF device:: 265 266 $ devlink port function set pci/0000:06:00.0/2 ipsec_crypto enable 267 268 $ devlink port show pci/0000:06:00.0/2 269 pci/0000:06:00.0/2: type eth netdev enp6s0pf0vf1 flavour pcivf pfnum 0 vfnum 1 270 function: 271 hw_addr 00:00:00:00:00:00 ipsec_crypto enabled 272 273IPsec packet capability setup 274----------------------------- 275When user enables IPsec packet capability for a VF, user application can offload 276XFRM state and policy crypto operation (Encrypt/Decrypt) to this VF, as well as 277IPsec encapsulation. 278 279When IPsec packet capability is disabled (default) for a VF, the XFRM state and 280policy is processed in software by the kernel. 281 282- Get IPsec packet capability of the VF device:: 283 284 $ devlink port show pci/0000:06:00.0/2 285 pci/0000:06:00.0/2: type eth netdev enp6s0pf0vf1 flavour pcivf pfnum 0 vfnum 1 286 function: 287 hw_addr 00:00:00:00:00:00 ipsec_packet disabled 288 289- Set IPsec packet capability of the VF device:: 290 291 $ devlink port function set pci/0000:06:00.0/2 ipsec_packet enable 292 293 $ devlink port show pci/0000:06:00.0/2 294 pci/0000:06:00.0/2: type eth netdev enp6s0pf0vf1 flavour pcivf pfnum 0 vfnum 1 295 function: 296 hw_addr 00:00:00:00:00:00 ipsec_packet enabled 297 298Subfunction 299============ 300 301Subfunction is a lightweight function that has a parent PCI function on which 302it is deployed. Subfunction is created and deployed in unit of 1. Unlike 303SRIOV VFs, a subfunction doesn't require its own PCI virtual function. 304A subfunction communicates with the hardware through the parent PCI function. 305 306To use a subfunction, 3 steps setup sequence is followed: 307 3081) create - create a subfunction; 3092) configure - configure subfunction attributes; 3103) deploy - deploy the subfunction; 311 312Subfunction management is done using devlink port user interface. 313User performs setup on the subfunction management device. 314 315(1) Create 316---------- 317A subfunction is created using a devlink port interface. A user adds the 318subfunction by adding a devlink port of subfunction flavour. The devlink 319kernel code calls down to subfunction management driver (devlink ops) and asks 320it to create a subfunction devlink port. Driver then instantiates the 321subfunction port and any associated objects such as health reporters and 322representor netdevice. 323 324(2) Configure 325------------- 326A subfunction devlink port is created but it is not active yet. That means the 327entities are created on devlink side, the e-switch port representor is created, 328but the subfunction device itself is not created. A user might use e-switch port 329representor to do settings, putting it into bridge, adding TC rules, etc. A user 330might as well configure the hardware address (such as MAC address) of the 331subfunction while subfunction is inactive. 332 333(3) Deploy 334---------- 335Once a subfunction is configured, user must activate it to use it. Upon 336activation, subfunction management driver asks the subfunction management 337device to instantiate the subfunction device on particular PCI function. 338A subfunction device is created on the :ref:`Documentation/driver-api/auxiliary_bus.rst <auxiliary_bus>`. 339At this point a matching subfunction driver binds to the subfunction's auxiliary device. 340 341Rate object management 342====================== 343 344Devlink provides API to manage tx rates of single devlink port or a group. 345This is done through rate objects, which can be one of the two types: 346 347``leaf`` 348 Represents a single devlink port; created/destroyed by the driver. Since leaf 349 have 1to1 mapping to its devlink port, in user space it is referred as 350 ``pci/<bus_addr>/<port_index>``; 351 352``node`` 353 Represents a group of rate objects (leafs and/or nodes); created/deleted by 354 request from the userspace; initially empty (no rate objects added). In 355 userspace it is referred as ``pci/<bus_addr>/<node_name>``, where 356 ``node_name`` can be any identifier, except decimal number, to avoid 357 collisions with leafs. 358 359API allows to configure following rate object's parameters: 360 361``tx_share`` 362 Minimum TX rate value shared among all other rate objects, or rate objects 363 that parts of the parent group, if it is a part of the same group. 364 365``tx_max`` 366 Maximum TX rate value. 367 368``tx_priority`` 369 Allows for usage of strict priority arbiter among siblings. This 370 arbitration scheme attempts to schedule nodes based on their priority 371 as long as the nodes remain within their bandwidth limit. The higher the 372 priority the higher the probability that the node will get selected for 373 scheduling. 374 375``tx_weight`` 376 Allows for usage of Weighted Fair Queuing arbitration scheme among 377 siblings. This arbitration scheme can be used simultaneously with the 378 strict priority. As a node is configured with a higher rate it gets more 379 BW relative to its siblings. Values are relative like a percentage 380 points, they basically tell how much BW should node take relative to 381 its siblings. 382 383``parent`` 384 Parent node name. Parent node rate limits are considered as additional limits 385 to all node children limits. ``tx_max`` is an upper limit for children. 386 ``tx_share`` is a total bandwidth distributed among children. 387 388``tx_priority`` and ``tx_weight`` can be used simultaneously. In that case 389nodes with the same priority form a WFQ subgroup in the sibling group 390and arbitration among them is based on assigned weights. 391 392Arbitration flow from the high level: 393 394#. Choose a node, or group of nodes with the highest priority that stays 395 within the BW limit and are not blocked. Use ``tx_priority`` as a 396 parameter for this arbitration. 397 398#. If group of nodes have the same priority perform WFQ arbitration on 399 that subgroup. Use ``tx_weight`` as a parameter for this arbitration. 400 401#. Select the winner node, and continue arbitration flow among its children, 402 until leaf node is reached, and the winner is established. 403 404#. If all the nodes from the highest priority sub-group are satisfied, or 405 overused their assigned BW, move to the lower priority nodes. 406 407Driver implementations are allowed to support both or either rate object types 408and setting methods of their parameters. Additionally driver implementation 409may export nodes/leafs and their child-parent relationships. 410 411Terms and Definitions 412===================== 413 414.. list-table:: Terms and Definitions 415 :widths: 22 90 416 417 * - Term 418 - Definitions 419 * - ``PCI device`` 420 - A physical PCI device having one or more PCI buses consists of one or 421 more PCI controllers. 422 * - ``PCI controller`` 423 - A controller consists of potentially multiple physical functions, 424 virtual functions and subfunctions. 425 * - ``Port function`` 426 - An object to manage the function of a port. 427 * - ``Subfunction`` 428 - A lightweight function that has parent PCI function on which it is 429 deployed. 430 * - ``Subfunction device`` 431 - A bus device of the subfunction, usually on a auxiliary bus. 432 * - ``Subfunction driver`` 433 - A device driver for the subfunction auxiliary device. 434 * - ``Subfunction management device`` 435 - A PCI physical function that supports subfunction management. 436 * - ``Subfunction management driver`` 437 - A device driver for PCI physical function that supports 438 subfunction management using devlink port interface. 439 * - ``Subfunction host driver`` 440 - A device driver for PCI physical function that hosts subfunction 441 devices. In most cases it is same as subfunction management driver. When 442 subfunction is used on external controller, subfunction management and 443 host drivers are different. 444