1.. SPDX-License-Identifier: GPL-2.0 2 3==================== 4mlx5 devlink support 5==================== 6 7This document describes the devlink features implemented by the ``mlx5`` 8device driver. 9 10Parameters 11========== 12 13.. list-table:: Generic parameters implemented 14 15 * - Name 16 - Mode 17 - Validation 18 - Notes 19 * - ``enable_roce`` 20 - driverinit 21 - Boolean 22 - If the device supports RoCE disablement, RoCE enablement state controls 23 device support for RoCE capability. Otherwise, the control occurs in the 24 driver stack. When RoCE is disabled at the driver level, only raw 25 ethernet QPs are supported. 26 * - ``io_eq_size`` 27 - driverinit 28 - The range is between 64 and 4096. 29 - 30 * - ``event_eq_size`` 31 - driverinit 32 - The range is between 64 and 4096. 33 - 34 * - ``max_macs`` 35 - driverinit 36 - The range is between 1 and 2^31. Only power of 2 values are supported. 37 - 38 * - ``enable_sriov`` 39 - permanent 40 - Boolean 41 - Applies to each physical function (PF) independently, if the device 42 supports it. Otherwise, it applies symmetrically to all PFs. 43 * - ``total_vfs`` 44 - permanent 45 - The range is between 1 and a device-specific max. 46 - Applies to each physical function (PF) independently, if the device 47 supports it. Otherwise, it applies symmetrically to all PFs. 48 * - ``max_sfs`` 49 - permanent 50 - The range is between 0 and a device-specific max. 51 - Applies to each physical function (PF) independently. 52 53Note: permanent parameters such as ``enable_sriov``, ``total_vfs`` and ``max_sfs`` 54 require FW reset to take effect 55 56.. code-block:: bash 57 58 # setup parameters 59 devlink dev param set pci/0000:01:00.0 name enable_sriov value true cmode permanent 60 devlink dev param set pci/0000:01:00.0 name total_vfs value 8 cmode permanent 61 62 # Fw reset 63 devlink dev reload pci/0000:01:00.0 action fw_activate 64 65 # for PCI related config such as sriov PCI reset/rescan is required: 66 echo 1 >/sys/bus/pci/devices/0000:01:00.0/remove 67 echo 1 >/sys/bus/pci/rescan 68 grep ^ /sys/bus/pci/devices/0000:01:00.0/sriov_* 69 70 * - ``num_doorbells`` 71 - driverinit 72 - This controls the number of channel doorbells used by the netdev. In all 73 cases, an additional doorbell is allocated and used for non-channel 74 communication (e.g. for PTP, HWS, etc.). Supported values are: 75 76 - 0: No channel-specific doorbells, use the global one for everything. 77 - [1, max_num_channels]: Spread netdev channels equally across these 78 doorbells. 79 80The ``mlx5`` driver also implements the following driver-specific 81parameters. 82 83.. list-table:: Driver-specific parameters implemented 84 :widths: 5 5 5 85 85 86 * - Name 87 - Type 88 - Mode 89 - Description 90 * - ``flow_steering_mode`` 91 - string 92 - runtime 93 - Controls the flow steering mode of the driver 94 95 * ``dmfs`` Device managed flow steering. In DMFS mode, the HW 96 steering entities are created and managed through firmware. 97 * ``smfs`` Software managed flow steering. In SMFS mode, the HW 98 steering entities are created and manage through the driver without 99 firmware intervention. 100 * ``hmfs`` Hardware managed flow steering. In HMFS mode, the driver 101 is configuring steering rules directly to the HW using Work Queues with 102 a special new type of WQE (Work Queue Element). 103 104 SMFS mode is faster and provides better rule insertion rate compared to 105 default DMFS mode. 106 * - ``fdb_large_groups`` 107 - u32 108 - driverinit 109 - Control the number of large groups (size > 1) in the FDB table. 110 111 * The default value is 15, and the range is between 1 and 1024. 112 * - ``esw_multiport`` 113 - Boolean 114 - runtime 115 - Control MultiPort E-Switch shared fdb mode. 116 117 An experimental mode where a single E-Switch is used and all the vports 118 and physical ports on the NIC are connected to it. 119 120 An example is to send traffic from a VF that is created on PF0 to an 121 uplink that is natively associated with the uplink of PF1 122 123 Note: Future devices, ConnectX-8 and onward, will eventually have this 124 as the default to allow forwarding between all NIC ports in a single 125 E-switch environment and the dual E-switch mode will likely get 126 deprecated. 127 128 Default: disabled 129 * - ``esw_port_metadata`` 130 - Boolean 131 - runtime 132 - When applicable, disabling eswitch metadata can increase packet rate up 133 to 20% depending on the use case and packet sizes. 134 135 Eswitch port metadata state controls whether to internally tag packets 136 with metadata. Metadata tagging must be enabled for multi-port RoCE, 137 failover between representors and stacked devices. By default metadata is 138 enabled on the supported devices in E-switch. Metadata is applicable only 139 for E-switch in switchdev mode and users may disable it when NONE of the 140 below use cases will be in use: 141 1. HCA is in Dual/multi-port RoCE mode. 142 2. VF/SF representor bonding (Usually used for Live migration) 143 3. Stacked devices 144 145 When metadata is disabled, the above use cases will fail to initialize if 146 users try to enable them. 147 148 Note: Setting this parameter does not take effect immediately. Setting 149 must happen in legacy mode and eswitch port metadata takes effect after 150 enabling switchdev mode. 151 * - ``hairpin_num_queues`` 152 - u32 153 - driverinit 154 - We refer to a TC NIC rule that involves forwarding as "hairpin". 155 Hairpin queues are mlx5 hardware specific implementation for hardware 156 forwarding of such packets. 157 158 Control the number of hairpin queues. 159 * - ``hairpin_queue_size`` 160 - u32 161 - driverinit 162 - Control the size (in packets) of the hairpin queues. 163 * - ``pcie_cong_inbound_high`` 164 - u16 165 - driverinit 166 - High threshold configuration for PCIe congestion events. The firmware 167 will send an event once device side inbound PCIe traffic went 168 above the configured high threshold for a long enough period (at least 169 200ms). 170 171 See pci_bw_inbound_high ethtool stat. 172 173 Units are 0.01 %. Accepted values are in range [0, 10000]. 174 pcie_cong_inbound_low < pcie_cong_inbound_high. 175 Default value: 9000 (Corresponds to 90%). 176 * - ``pcie_cong_inbound_low`` 177 - u16 178 - driverinit 179 - Low threshold configuration for PCIe congestion events. The firmware 180 will send an event once device side inbound PCIe traffic went 181 below the configured low threshold, only after having been previously in 182 a congested state. 183 184 See pci_bw_inbound_low ethtool stat. 185 186 Units are 0.01 %. Accepted values are in range [0, 10000]. 187 pcie_cong_inbound_low < pcie_cong_inbound_high. 188 Default value: 7500. 189 * - ``pcie_cong_outbound_high`` 190 - u16 191 - driverinit 192 - High threshold configuration for PCIe congestion events. The firmware 193 will send an event once device side outbound PCIe traffic went 194 above the configured high threshold for a long enough period (at least 195 200ms). 196 197 See pci_bw_outbound_high ethtool stat. 198 199 Units are 0.01 %. Accepted values are in range [0, 10000]. 200 pcie_cong_outbound_low < pcie_cong_outbound_high. 201 Default value: 9000 (Corresponds to 90%). 202 * - ``pcie_cong_outbound_low`` 203 - u16 204 - driverinit 205 - Low threshold configuration for PCIe congestion events. The firmware 206 will send an event once device side outbound PCIe traffic went 207 below the configured low threshold, only after having been previously in 208 a congested state. 209 210 See pci_bw_outbound_low ethtool stat. 211 212 Units are 0.01 %. Accepted values are in range [0, 10000]. 213 pcie_cong_outbound_low < pcie_cong_outbound_high. 214 Default value: 7500. 215 216 * - ``cqe_compress_type`` 217 - string 218 - permanent 219 - Configure which mechanism/algorithm should be used by the NIC that will 220 affect the rate (aggressiveness) of compressed CQEs depending on PCIe bus 221 conditions and other internal NIC factors. This mode affects all queues 222 that enable compression. 223 * ``balanced`` : Merges fewer CQEs, resulting in a moderate compression ratio but maintaining a balance between bandwidth savings and performance 224 * ``aggressive`` : Merges more CQEs into a single entry, achieving a higher compression rate and maximizing performance, particularly under high traffic loads 225 226 * - ``swp_l4_csum_mode`` 227 - string 228 - permanent 229 - Configure how the L4 checksum is calculated by the device when using 230 Software Parser (SWP) hints for header locations. 231 232 * ``default`` : Use the device's default checksum calculation 233 mode. The driver will discover during init whether or 234 full_csum or l4_only is in use. Setting this value explicitly 235 from userspace is not allowed, but some firmware versions may 236 return this value on param read. 237 * ``full_csum`` : Calculate full checksum including the pseudo-header 238 * ``l4_only`` : Calculate L4-only checksum, excluding the pseudo-header 239 240The ``mlx5`` driver supports reloading via ``DEVLINK_CMD_RELOAD`` 241 242Info versions 243============= 244 245The ``mlx5`` driver reports the following versions 246 247.. list-table:: devlink info versions implemented 248 :widths: 5 5 90 249 250 * - Name 251 - Type 252 - Description 253 * - ``fw.psid`` 254 - fixed 255 - Used to represent the board id of the device. 256 * - ``fw.version`` 257 - stored, running 258 - Three digit major.minor.subminor firmware version number. 259 260Health reporters 261================ 262 263tx reporter 264----------- 265The tx reporter is responsible for reporting and recovering of the following three error scenarios: 266 267- tx timeout 268 Report on kernel tx timeout detection. 269 Recover by searching lost interrupts. 270- tx error completion 271 Report on error tx completion. 272 Recover by flushing the tx queue and reset it. 273- tx PTP port timestamping CQ unhealthy 274 Report too many CQEs never delivered on port ts CQ. 275 Recover by flushing and re-creating all PTP channels. 276 277tx reporter also support on demand diagnose callback, on which it provides 278real time information of its send queues status. 279 280User commands examples: 281 282- Diagnose send queues status:: 283 284 $ devlink health diagnose pci/0000:82:00.0 reporter tx 285 286.. note:: 287 This command has valid output only when interface is up, otherwise the command has empty output. 288 289- Show number of tx errors indicated, number of recover flows ended successfully, 290 is autorecover enabled and graceful period from last recover:: 291 292 $ devlink health show pci/0000:82:00.0 reporter tx 293 294rx reporter 295----------- 296The rx reporter is responsible for reporting and recovering of the following two error scenarios: 297 298- rx queues' initialization (population) timeout 299 Population of rx queues' descriptors on ring initialization is done 300 in napi context via triggering an irq. In case of a failure to get 301 the minimum amount of descriptors, a timeout would occur, and 302 descriptors could be recovered by polling the EQ (Event Queue). 303- rx completions with errors (reported by HW on interrupt context) 304 Report on rx completion error. 305 Recover (if needed) by flushing the related queue and reset it. 306 307rx reporter also supports on demand diagnose callback, on which it 308provides real time information of its receive queues' status. 309 310- Diagnose rx queues' status and corresponding completion queue:: 311 312 $ devlink health diagnose pci/0000:82:00.0 reporter rx 313 314.. note:: 315 This command has valid output only when interface is up. Otherwise, the command has empty output. 316 317- Show number of rx errors indicated, number of recover flows ended successfully, 318 is autorecover enabled, and graceful period from last recover:: 319 320 $ devlink health show pci/0000:82:00.0 reporter rx 321 322fw reporter 323----------- 324The fw reporter implements `diagnose` and `dump` callbacks. 325It follows symptoms of fw error such as fw syndrome by triggering 326fw core dump and storing it into the dump buffer. 327The fw reporter diagnose command can be triggered any time by the user to check 328current fw status. 329 330User commands examples: 331 332- Check fw heath status:: 333 334 $ devlink health diagnose pci/0000:82:00.0 reporter fw 335 336- Read FW core dump if already stored or trigger new one:: 337 338 $ devlink health dump show pci/0000:82:00.0 reporter fw 339 340.. note:: 341 This command can run only on the PF which has fw tracer ownership, 342 running it on other PF or any VF will return "Operation not permitted". 343 344fw fatal reporter 345----------------- 346The fw fatal reporter implements `dump` and `recover` callbacks. 347It follows fatal errors indications by CR-space dump and recover flow. 348The CR-space dump uses vsc interface which is valid even if the FW command 349interface is not functional, which is the case in most FW fatal errors. 350The recover function runs recover flow which reloads the driver and triggers fw 351reset if needed. 352On firmware error, the health buffer is dumped into the dmesg. The log 353level is derived from the error's severity (given in health buffer). 354 355User commands examples: 356 357- Run fw recover flow manually:: 358 359 $ devlink health recover pci/0000:82:00.0 reporter fw_fatal 360 361- Read FW CR-space dump if already stored or trigger new one:: 362 363 $ devlink health dump show pci/0000:82:00.1 reporter fw_fatal 364 365.. note:: 366 This command can run only on PF. 367 368vnic reporter 369------------- 370The vnic reporter implements only the `diagnose` callback. 371It is responsible for querying the vnic diagnostic counters from fw and displaying 372them in realtime. 373 374Description of the vnic counters: 375 376- total_error_queues 377 number of queues in an error state due to 378 an async error or errored command. 379- send_queue_priority_update_flow 380 number of QP/SQ priority/SL update events. 381- cq_overrun 382 number of times CQ entered an error state due to an overflow. 383- async_eq_overrun 384 number of times an EQ mapped to async events was overrun. 385- comp_eq_overrun 386 number of times an EQ mapped to completion events was 387 overrun. 388- quota_exceeded_command 389 number of commands issued and failed due to quota exceeded. 390- invalid_command 391 number of commands issued and failed dues to any reason other than quota 392 exceeded. 393- nic_receive_steering_discard 394 number of packets that completed RX flow 395 steering but were discarded due to a mismatch in flow table. 396- generated_pkt_steering_fail 397 number of packets generated by the VNIC experiencing unexpected steering 398 failure (at any point in steering flow). 399- handled_pkt_steering_fail 400 number of packets handled by the VNIC experiencing unexpected steering 401 failure (at any point in steering flow owned by the VNIC, including the FDB 402 for the eswitch owner). 403- icm_consumption 404 amount of Interconnect Host Memory (ICM) consumed by the vnic in 405 granularity of 4KB. ICM is host memory allocated by SW upon HCA request 406 and is used for storing data structures that control HCA operation. 407- bar_uar_access 408 number of WRITE or READ access operations to the UAR on the PCIe BAR. 409- odp_local_triggered_page_fault 410 number of locally-triggered page-faults due to ODP. 411- odp_remote_triggered_page_fault 412 number of remotly-triggered page-faults due to ODP. 413 414User commands examples: 415 416- Diagnose PF/VF vnic counters:: 417 418 $ devlink health diagnose pci/0000:82:00.1 reporter vnic 419 420- Diagnose representor vnic counters (performed by supplying devlink port of the 421 representor, which can be obtained via devlink port command):: 422 423 $ devlink health diagnose pci/0000:82:00.1/65537 reporter vnic 424 425.. note:: 426 This command can run over all interfaces such as PF/VF and representor ports. 427 428Rates 429===== 430 431mlx5 devices can limit transmission of individual VFs or a group of them via 432the devlink-rate API in switchdev mode. 433 434User commands examples: 435 436- Print the existing rates:: 437 438 $ devlink port function rate show 439 440- Set a max tx limit on traffic from VF0:: 441 442 $ devlink port function rate set pci/0000:82:00.0/1 tx_max 10Gbit 443 444- Create a rate group with a max tx limit and add two VFs to it:: 445 446 $ devlink port function rate add pci/0000:82:00.0/group1 tx_max 10Gbit 447 $ devlink port function rate set pci/0000:82:00.0/1 parent group1 448 $ devlink port function rate set pci/0000:82:00.0/2 parent group1 449 450- Same scenario, with a min guarantee of 20% of the bandwidth for the first VF:: 451 452 $ devlink port function rate add pci/0000:82:00.0/group1 tx_max 10Gbit 453 $ devlink port function rate set pci/0000:82:00.0/1 parent group1 tx_share 2Gbit 454 $ devlink port function rate set pci/0000:82:00.0/2 parent group1 455 456- Cross-device scheduling:: 457 458 $ devlink port function rate add pci/0000:82:00.0/group1 tx_max 10Gbit 459 $ devlink port function rate set pci/0000:82:00.1/32769 parent pci/0000:82:00.0/group1 460