1.\" 2.\" This file and its contents are supplied under the terms of the 3.\" Common Development and Distribution License ("CDDL"), version 1.0. 4.\" You may only use this file in accordance with the terms of version 5.\" 1.0 of the CDDL. 6.\" 7.\" A full copy of the text of the CDDL should have accompanied this 8.\" source. A copy of the CDDL is also available via the Internet at 9.\" http://www.illumos.org/license/CDDL. 10.\" 11.\" 12.\" Copyright 2023 the University of Queensland 13.\" 14.Dd August 27, 2020 15.Dt MLXCX 4D 16.Os 17.Sh NAME 18.Nm mlxcx 19.Nd Mellanox ConnectX-4/5/6 Ethernet controller driver 20.Sh SYNOPSIS 21.Pa /dev/net/mlxcx* 22.Sh DESCRIPTION 23The 24.Sy mlxcx 25driver is a GLDv3 NIC driver for the ConnectX-4, ConnectX-4 Lx, ConnectX-5 and 26ConnectX-6 families of ethernet controllers from Mellanox. 27It supports the Data Link Provider Interface, 28.Xr dlpi 4P . 29.Pp 30This driver supports: 31.Bl -dash -offset indent 32.It 33Jumbo frames up to 9000 bytes. 34.It 35Checksum offload for TCP, UDP, IPv4 and IPv6. 36.It 37Group support with VLAN and MAC steering to avoid software classification 38when using VNICs. 39.It 40Promiscuous access via 41.Xr snoop 8 and 42.Xr dlpi 4P 43.It 44LED control 45.It 46Transceiver information 47.It 48Internal temperature sensors 49.El 50.Pp 51At this time, the driver does not support Large Send Offload (LSO), energy 52efficient Ethernet (EEE), or the use of flow control through hardware pause 53frames. 54.Sh CONFIGURATION 55The 56.Sy mlxcx.conf 57file contains user configurable parameters, including the ability to set the 58number of rings and groups advertised to MAC, the sizes of rings and groups, 59and the maximum number of MAC address filters available. 60.Sh PROPERTIES 61The driver supports the following device properties which may be tuned through 62its driver.conf file, 63.Pa /kernel/drv/mlxcx.conf . 64These properties cannot be changed after the driver has been attached. 65.Pp 66These properties are not considered stable at this time, and may change. 67.Bl -hang -width Ds 68.It Sy eq_size_shift 69.Bd -filled -compact 70Minimum: 71.Sy 2 | 72Maximum: 73.Sy device dependent (up to 255) 74.Ed 75.Bd -filled 76The 77.Sy eq_size_shift 78property determines the number of entries on Event Queues for the device. 79The number of entries is calculated as 80.Dv (1 << eq_size_shift) , 81so a value of 9 would mean 512 entries are created on each Event Queue. 82The default value is 83.Sy 9 . 84.Ed 85.It Sy cq_size_shift 86.Bd -filled -compact 87Minimum: 88.Sy 2 | 89Maximum: 90.Sy device dependent (up to 255) 91.Ed 92.Bd -filled 93The 94.Sy cq_size_shift 95property determines the number of entries on Completion Queues for the device. 96The number of entries is calculated as 97.Li (1 << cq_size_shift) , 98so a value of 9 would mean 512 entries are created on each Event Queue. 99The default value is device dependent, 100.Sy 10 101for devices with maximum supported speed of 10Gb/s or less and 102.Sy 12 103for devices with higher supported speeds. 104This should be kept very close to the value set for 105.Sy rq_size_shift 106and 107.Sy sq_size_shift . 108.Ed 109.It Sy rq_size_shift 110.Bd -filled -compact 111Minimum: 112.Sy 2 | 113Maximum: 114.Sy device dependent (up to 255) 115.Ed 116.Bd -filled 117The 118.Sy rq_size_shift 119property determines the number of descriptors on Receive Queues for the device. 120The number of descriptors is calculated as 121.Dv (1 << rq_size_shift) , 122so a value of 9 would mean 512 descriptors are created on each Receive Queue. 123This sets the number of packets on RX rings advertised to MAC. 124The default value is device dependent, 125.Sy 10 126for devices with maximum supported speed of 10Gb/s or less and 127.Sy 12 128for devices with higher supported speeds. 129.Ed 130.It Sy sq_size_shift 131.Bd -filled -compact 132Minimum: 133.Sy 2 | 134Maximum: 135.Sy device dependent (up to 255) 136.Ed 137.Bd -filled 138The 139.Sy sq_size_shift 140property determines the number of descriptors on Send Queues for the device. 141The number of descriptors is calculated as 142.Dv (1 << sq_size_shift) , 143so a value of 9 would mean 512 descriptors are created on each Send Queue. 144This sets the number of packets on RX rings advertised to MAC. 145The default value is device dependent, 146.Sy 11 147for devices with maximum supported speed of 10Gb/s or less and 148.Sy 13 149for devices with higher supported speeds. 150Note that large packets often occupy more than one descriptor slot on the SQ, 151so it is sometimes a good idea to increase this if using a large MTU. 152.Ed 153.It Sy tx_ngroups 154.Bd -filled -compact 155Minimum: 156.Sy 1 | 157Maximum: 158.Sy device dependent 159.Ed 160.Bd -filled 161The 162.Sy tx_ngroups 163property determines the number of TX groups advertised to MAC. 164The default value is 165.Sy 1 . 166.Ed 167.It Sy tx_nrings_per_group 168.Bd -filled -compact 169Minimum: 170.Sy 1 | 171Maximum: 172.Sy device dependent 173.Ed 174.Bd -filled 175The 176.Sy tx_nrings_per_group 177property determines the number of rings in each TX group advertised to MAC. 178The default value is 179.Sy 64 . 180.Ed 181.It Sy rx_ngroups_large 182.Bd -filled -compact 183Minimum: 184.Sy 1 | 185Maximum: 186.Sy device dependent 187.Ed 188.Bd -filled 189The 190.Sy rx_ngroups_large 191property determines the number of "large" RX groups advertised to MAC. 192The size of "large" RX groups is set by the 193.Sy rx_nrings_per_large_group 194property. 195The default value is 196.Sy 2 . 197.Ed 198.It Sy rx_nrings_per_large_group 199.Bd -filled -compact 200Minimum: 201.Sy 1 | 202Maximum: 203.Sy device dependent 204.Ed 205.Bd -filled 206The 207.Sy rx_nrings_per_large_group 208property determines the number of rings in each "large" RX group advertised to 209MAC. 210The number of such groups is determined by the 211.Sy rx_ngroups_large 212property. 213The default value is 214.Sy 16 . 215.Ed 216.It Sy rx_ngroups_small 217.Bd -filled -compact 218Minimum: 219.Sy 1 | 220Maximum: 221.Sy device dependent 222.Ed 223.Bd -filled 224The 225.Sy rx_ngroups_small 226property determines the number of "small" RX groups advertised to MAC. 227The size of "small" RX groups is set by the 228.Sy rx_nrings_per_small_group 229property. 230It is recommended to use many small groups when using a large number of 231VNICs on top of the NIC (e.g. on a system with many zones). 232The default value is 233.Sy 256 . 234.Ed 235.It Sy rx_nrings_per_small_group 236.Bd -filled -compact 237Minimum: 238.Sy 1 | 239Maximum: 240.Sy device dependent 241.Ed 242.Bd -filled 243The 244.Sy rx_nrings_per_small_group 245property determines the number of rings in each "small" RX group advertised to 246MAC. 247The number of such groups is determined by the 248.Sy rx_ngroups_small 249property. 250The default value is 251.Sy 4 . 252.Ed 253.It Sy ftbl_root_size_shift 254.Bd -filled -compact 255Minimum: 256.Sy 4 | 257Maximum: 258.Sy device dependent 259.Ed 260.Bd -filled 261The 262.Sy ftbl_root_size_shift 263property determines the number of flow table entries on the root flow table, 264and therefore how many MAC addresses can be filtered into groups across the 265entire NIC. 266The number of flow entries is calculated as 267.Dv (1 << ftbl_root_size_shift) , 268so a value of 9 would mean 512 entries are created in the root flow table. 269The default value is 270.Sy 12 . 271.Ed 272.It Sy cqemod_period_usec 273.Bd -filled -compact 274Minimum: 275.Sy 1 | 276Maximum: 277.Sy 65535 278.Ed 279.Bd -filled 280The 281.Sy cqemod_period_usec 282property determines the maximum delay after a completion event has occurred 283before an event queue entry (and thus an interrupt) is generated. 284The delay is measured in microseconds. 285The default value is 286.Sy 50 . 287.Ed 288.It Sy cqemod_count 289.Bd -filled -compact 290Minimum: 291.Sy 1 | 292Maximum: 293.Sy 65535 294.Ed 295.Bd -filled 296The 297.Sy cqemod_count 298property determines the maximum number of completion events that can have 299occurred before an event queue entry (and thus an interrupt) is generated. 300The default value is 301.Sy 80% of the CQ size . 302.Ed 303.It Sy intrmod_period_usec 304.Bd -filled -compact 305Minimum: 306.Sy 1 | 307Maximum: 308.Sy 65535 309.Ed 310.Bd -filled 311The 312.Sy intrmod_period_usec 313property determines the maximum delay after an event queue entry has been 314generated before an interrupt is raised. 315The delay is measured in microseconds. 316The default value is 317.Sy 10 . 318.Ed 319.It Sy tx_bind_threshold 320.Bd -filled -compact 321Minimum: 322.Sy 1 | 323Maximum: 324.Sy 65535 325.Ed 326.Bd -filled 327The 328.Sy tx_bind_threshold 329property determines the minimum number of bytes in a packet before the driver 330uses 331.Xr ddi_dma_addr_bind_handle 9F 332to bind the packet memory for DMA, rather than copying the memory as it does 333for small packets. 334DMA binds are expensive and involve taking locks in the PCI nexus driver, so it 335is seldom worth using them for small packets. 336The default value is 337.Sy 2048 . 338.Ed 339.It Sy rx_limit_per_completion 340.Bd -filled -compact 341Minimum: 342.Sy 16 | 343Maximum: 344.Sy 4096 345.Ed 346.Bd -filled 347The 348.Sy rx_limit_per_completion 349property determines the maximum number of packets that 350will be processed on a given completion ring during a single interrupt. 351This is done to try and guarantee some amount of liveness in the system. 352The default value is 353.Sy 256 . 354.Ed 355.It Sy rx_p50_loan_min_size 356.Bd -filled -compact 357Minimum: 358.Sy 0 | 359Maximum: 360.Sy MTU 361.Ed 362.Bd -filled 363The 364.Sy rx_p50_loan_min_size 365property determines the minimum size of packet buffers allowed to be loaned 366to MAC when the ring has reached >=50% of its buffers already on loan. 367Packet buffers larger than this value will be copied. 368At <50% of ring buffers on loan, all buffers will be loaned. 369At >=75% of buffers on loan, all packets will be copied instead to ensure 370ring availability. 371The default value is 372.Sy 256 . 373.Ed 374.El 375.Sh FILES 376.Bl -tag -width Pa 377.It Pa /kernel/drv/amd64/mlxcx 378Device driver (x86) 379.It Pa /kernel/drv/mlxcx.conf 380Driver configuration file containing user-configurable options 381.El 382.Sh SEE ALSO 383.Xr dlpi 4P , 384.Xr driver.conf 5 , 385.Xr dladm 8 , 386.Xr snoop 8 387