1.\" Copyright (c) 2011-2016, Chelsio Inc 2.\" All rights reserved. 3.\" 4.\" Redistribution and use in source and binary forms, with or without 5.\" modification, are permitted provided that the following conditions are met: 6.\" 7.\" 1. Redistributions of source code must retain the above copyright notice, 8.\" this list of conditions and the following disclaimer. 9.\" 10.\" 2. Redistributions in binary form must reproduce the above copyright 11.\" notice, this list of conditions and the following disclaimer in the 12.\" documentation and/or other materials provided with the distribution. 13.\" 14.\" 3. Neither the name of the Chelsio Inc nor the names of its 15.\" contributors may be used to endorse or promote products derived from 16.\" this software without specific prior written permission. 17.\" 18.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" 19.\" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 20.\" IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE 21.\" ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE 22.\" LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR 23.\" CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF 24.\" SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS 25.\" INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN 26.\" CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) 27.\" ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE 28.\" POSSIBILITY OF SUCH DAMAGE. 29.\" 30.\" * Other names and brands may be claimed as the property of others. 31.\" 32.Dd September 29, 2025 33.Dt CXGBE 4 34.Os 35.Sh NAME 36.Nm cxgbe 37.Nd "Chelsio T4-, T5-, and T6-based 100Gb, 40Gb, 25Gb, 10Gb, and 1Gb Ethernet adapter driver" 38.Sh SYNOPSIS 39To compile this driver into the kernel, 40place the following lines in your 41kernel configuration file: 42.Bd -ragged -offset indent 43.Cd "device cxgbe" 44.Ed 45.Pp 46To load the driver as a 47module at boot time, place the following lines in 48.Xr loader.conf 5 : 49.Bd -literal -offset indent 50t4fw_cfg_load="YES" 51t5fw_cfg_load="YES" 52t6fw_cfg_load="YES" 53if_cxgbe_load="YES" 54.Ed 55.Sh DESCRIPTION 56The 57.Nm 58driver provides support for PCI Express Ethernet adapters based on 59the Chelsio Terminator 4, Terminator 5, and Terminator 6 ASICs (T4, T5, and T6). 60The driver supports Jumbo Frames, Transmit/Receive checksum offload, 61TCP segmentation offload (TSO), Large Receive Offload (LRO), VLAN 62tag insertion/extraction, VLAN checksum offload, VLAN TSO, VXLAN checksum 63offload, VXLAN TSO, and Receive Side Steering (RSS). 64For further hardware information and questions related to hardware 65requirements, see 66.Pa http://www.chelsio.com/ . 67.Pp 68The 69.Nm 70driver uses different names for devices based on the associated ASIC: 71.Bl -column -offset indent "ASIC" "Port Name" "Parent Device" 72.It Sy ASIC Ta Sy Port Name Ta Sy Parent Device Ta Sy Virtual Interface 73.It T4 Ta cxgbe Ta t4nex Ta vcxgbe 74.It T5 Ta cxl Ta t5nex Ta vcxl 75.It T6 Ta cc Ta t6nex Ta vcc 76.It T7 Ta che Ta chnex Ta vche 77.El 78.Pp 79Loader tunables with the hw.cxgbe prefix apply to all cards. 80The driver provides sysctl MIBs for both ports and parent devices using 81the names above. 82For example, a T5 adapter provides port MIBs under dev.cxl and 83adapter-wide MIBs under dev.t5nex. 84References to sysctl MIBs in the remainder of this page use 85dev.<port> for port MIBs and dev.<nexus> for adapter-wide MIBs. 86.Pp 87For more information on configuring this device, see 88.Xr ifconfig 8 . 89.Sh HARDWARE 90The 91.Nm 92driver supports 100Gb and 25Gb Ethernet adapters based on the T6 ASIC: 93.Pp 94.Bl -bullet -compact 95.It 96Chelsio T6225-CR 97.It 98Chelsio T6225-SO-CR 99.It 100Chelsio T62100-LP-CR 101.It 102Chelsio T62100-SO-CR 103.It 104Chelsio T62100-CR 105.El 106.Pp 107The 108.Nm 109driver supports 40Gb, 10Gb and 1Gb Ethernet adapters based on the T5 ASIC: 110.Pp 111.Bl -bullet -compact 112.It 113Chelsio T580-CR 114.It 115Chelsio T580-LP-CR 116.It 117Chelsio T580-LP-SO-CR 118.It 119Chelsio T560-CR 120.It 121Chelsio T540-CR 122.It 123Chelsio T540-LP-CR 124.It 125Chelsio T522-CR 126.It 127Chelsio T520-LL-CR 128.It 129Chelsio T520-CR 130.It 131Chelsio T520-SO 132.It 133Chelsio T520-BT 134.It 135Chelsio T504-BT 136.El 137.Pp 138The 139.Nm 140driver supports 10Gb and 1Gb Ethernet adapters based on the T4 ASIC: 141.Pp 142.Bl -bullet -compact 143.It 144Chelsio T420-CR 145.It 146Chelsio T422-CR 147.It 148Chelsio T440-CR 149.It 150Chelsio T420-BCH 151.It 152Chelsio T440-BCH 153.It 154Chelsio T440-CH 155.It 156Chelsio T420-SO 157.It 158Chelsio T420-CX 159.It 160Chelsio T420-BT 161.It 162Chelsio T404-BT 163.El 164.Sh LOADER TUNABLES 165Tunables can be set at the 166.Xr loader 8 167prompt before booting the kernel or stored in 168.Xr loader.conf 5 . 169There are multiple tunables that control the number of queues of various 170types. 171A negative value for such a tunable instructs the driver to create 172up to that many queues if there are enough CPU cores available. 173.Bl -tag -width indent 174.It Va hw.cxgbe.ntxq 175Number of NIC tx queues used for a port. 176The default is 16 or the number 177of CPU cores in the system, whichever is less. 178.It Va hw.cxgbe.nrxq 179Number of NIC rx queues used for a port. 180The default is 8 or the number 181of CPU cores in the system, whichever is less. 182.It Va hw.cxgbe.nofldtxq 183Number of TOE tx queues used for a port. 184The default is 8 or the 185number of CPU cores in the system, whichever is less. 186.It Va hw.cxgbe.nofldrxq 187Number of TOE rx queues used for a port. 188The default is 2 or the 189number of CPU cores in the system, whichever is less. 190.It Va hw.cxgbe.num_vis 191Number of virtual interfaces (VIs) created for each port. 192Each virtual interface creates a separate network interface. 193The first virtual interface on each port is required and represents 194the primary network interface on the port. 195Additional virtual interfaces on a port are named using the Virtual Interface 196name from the table above. 197Additional virtual interfaces use a single pair of queues 198for rx and tx as well an additional pair of queues for TOE rx and tx. 199The default is 1. 200.It Va hw.cxgbe.holdoff_timer_idx 201.It Va hw.cxgbe.holdoff_timer_idx_ofld 202Timer index value used to delay interrupts. 203The holdoff timer list has the values 1, 5, 10, 50, 100, and 200 204by default (all values are in microseconds) and the index selects a 205value from this list. 206holdoff_timer_idx_ofld applies to queues used for TOE rx. 207The default value is 1 which means the timer value is 5us. 208Different interfaces can be assigned different values at any time via the 209dev.<port>.X.holdoff_tmr_idx and dev.<port>.X.holdoff_tmr_idx_ofld sysctls. 210.It Va hw.cxgbe.holdoff_pktc_idx 211.It Va hw.cxgbe.holdoff_pktc_idx_ofld 212Packet-count index value used to delay interrupts. 213The packet-count list has the values 1, 8, 16, and 32 by default, 214and the index selects a value from this list. 215holdoff_pktc_idx_ofld applies to queues used for TOE rx. 216The default value is -1 which means packet counting is disabled and interrupts 217are generated based solely on the holdoff timer value. 218Different interfaces can be assigned different values via the 219dev.<port>.X.holdoff_pktc_idx and dev.<port>.X.holdoff_pktc_idx_ofld sysctls. 220These sysctls work only when the interface has never been marked up (as done by 221ifconfig up). 222.It Va hw.cxgbe.qsize_txq 223Number of entries in a transmit queue's descriptor ring. 224A buf_ring of the same size is also allocated for additional 225software queuing. 226See 227.Xr ifnet 9 . 228The default value is 1024. 229Different interfaces can be assigned different values via the 230dev.<port>.X.qsize_txq sysctl. 231This sysctl works only when the interface has never been marked up (as done by 232ifconfig up). 233.It Va hw.cxgbe.qsize_rxq 234Number of entries in a receive queue's descriptor ring. 235The default value is 1024. 236Different interfaces can be assigned different values via the 237dev.<port>.X.qsize_rxq sysctl. 238This sysctl works only when the interface has never been marked up (as done by 239ifconfig up). 240.It Va hw.cxgbe.interrupt_types 241Permitted interrupt types. 242Bit 0 represents INTx (line interrupts), bit 1 MSI, and bit 2 MSI-X. 243The default is 7 (all allowed). 244The driver selects the best possible type out of the allowed types. 245.It Va hw.cxgbe.pcie_relaxed_ordering 246PCIe Relaxed Ordering. 247-1 indicates the driver should determine whether to enable or disable PCIe RO. 2480 disables PCIe RO. 2491 enables PCIe RO. 2502 indicates the driver should not modify the PCIe RO setting. 251The default is -1. 252.It Va hw.cxgbe.fw_install 2530 prohibits the driver from installing a firmware on the card. 2541 allows the driver to install a new firmware if internal driver 255heuristics indicate that the new firmware is preferable to the one 256already on the card. 2572 instructs the driver to always install the new firmware on the card as 258long as it is compatible with the driver and is a different version than 259the one already on the card. 260The default is 1. 261.It Va hw.cxgbe.fl_pktshift 262Number of padding bytes inserted before the beginning of an Ethernet 263frame in the receive buffer. 264The default value is 0. 265A value of 2 would ensure that the Ethernet payload (usually the IP header) 266is at a 4 byte aligned address. 2670-7 are all valid values. 268.It Va hw.cxgbe.fl_pad 269A non-zero value ensures that writes from the hardware to a receive buffer are 270padded up to the specified boundary. 271The default is -1 which lets the driver pick a pad boundary. 2720 disables trailer padding completely. 273.It Va hw.cxgbe.cong_drop 274Controls the hardware response to congestion. 275-1 disables congestion feedback and is not recommended. 2760 instructs the hardware to backpressure its pipeline on congestion. 277This usually results in the port emitting PAUSE frames. 2781 instructs the hardware to drop frames destined for congested queues. 2792 instructs the hardware to both backpressure the pipeline and drop frames. 280.It Va hw.cxgbe.pause_settings 281PAUSE frame settings. 282Bit 0 is rx_pause, bit 1 is tx_pause, bit 2 is pause_autoneg. 283rx_pause = 1 instructs the hardware to heed incoming PAUSE frames, 0 instructs 284it to ignore them. 285tx_pause = 1 allows the hardware to emit PAUSE frames when its receive FIFO 286reaches a high threshold, 0 prohibits the hardware from emitting PAUSE frames. 287pause_autoneg = 1 overrides the rx_pause and tx_pause bits and instructs the 288hardware to negotiate PAUSE settings with the link peer. 289The default is 7 (all three = 1). 290This tunable establishes the default PAUSE settings for all ports. 291Settings can be displayed and controlled on a per-port basis via the 292dev.<port>.X.pause_settings sysctl. 293.It Va hw.cxgbe.fec 294Forward Error Correction settings. 295-1 (default) means driver should automatically pick a value. 2960 disables FEC. 297Finer grained control can be achieved by setting individual bits. 298Bit 0 enables RS FEC, bit 1 enables BASE-R FEC (aka Firecode FEC), bit 2992 enables NO FEC, and bit 6 enables the FEC that is recommended by the 300transceiver/cable that is plugged in. 301These bits can be set together in any combination. 302This tunable establishes the default FEC settings for all ports. 303Settings can be controlled on a per-port basis via the 304dev.<port>.X.requested_fec sysctl. 305The FEC in use on the link is available in dev.<port>.X.link_fec when 306the link is up. 307.It Va hw.cxgbe.autoneg 308Link autonegotiation settings. 309This tunable establishes the default autonegotiation settings for all ports. 310Settings can be displayed and controlled on a per-port basis via the 311dev.<port>.X.autoneg sysctl. 3120 disables autonegotiation. 3131 enables autonegotiation. 314The default is -1 which lets the driver pick a value. 315dev.<port>.X.autoneg is -1 for port and module combinations that do not support 316autonegotiation. 317.It Va hw.cxgbe.buffer_packing 318Allow the hardware to deliver multiple frames in the same receive buffer 319opportunistically. 320The default is -1 which lets the driver decide. 3210 or 1 explicitly disable or enable this feature. 322.It Va hw.cxgbe.largest_rx_cluster 323.It Va hw.cxgbe.safest_rx_cluster 324Sizes of rx clusters. 325Each of these must be set to one of the sizes available 326(usually 2048, 4096, 9216, and 16384) and largest_rx_cluster must be greater 327than or equal to safest_rx_cluster. 328The defaults are 16384 and 4096 respectively. 329The driver never attempts to allocate a receive buffer larger than 330largest_rx_cluster and falls back to allocating buffers of 331safest_rx_cluster size if an allocation larger than safest_rx_cluster fails. 332Note that largest_rx_cluster merely establishes a ceiling -- the driver is 333allowed to allocate buffers of smaller sizes. 334.It Va hw.cxgbe.config_file 335Select a pre-packaged device configuration file. 336A configuration file contains a recipe for partitioning and configuring the 337hardware resources on the card. 338This tunable is for specialized applications only and should not be used in 339normal operation. 340The configuration profile currently in use is available in the dev.<nexus>.X.cf 341and dev.<nexus>.X.cfcsum sysctls. 342.It Va hw.cxgbe.linkcaps_allowed 343.It Va hw.cxgbe.niccaps_allowed 344.It Va hw.cxgbe.toecaps_allowed 345.It Va hw.cxgbe.rdmacaps_allowed 346.It Va hw.cxgbe.iscsicaps_allowed 347.It Va hw.cxgbe.fcoecaps_allowed 348Disallowing capabilities provides a hint to the driver and firmware to not 349reserve hardware resources for that feature. 350Each of these is a bit field with a bit for each sub-capability within the 351capability. 352This tunable is for specialized applications only and should not be used in 353normal operation. 354The capabilities for which hardware resources have been reserved are listed in 355dev.<nexus>.X.*caps sysctls. 356.It Va hw.cxgbe.tx_vm_wr 357Setting this to 1 instructs the driver to use VM work requests to transmit data. 358This lets PF interfaces transmit frames to VF interfaces over the internal 359switch in the ASIC. 360Note that the 361.Xr cxgbev 4 362VF driver always uses VM work requests and is not affected by this tunable. 363The default value is 0 and should be changed only if PF and VF interfaces need 364to communicate with each other. 365Different interfaces can be assigned different values using the 366dev.<port>.X.tx_vm_wr sysctl when the interface is administratively down. 367.It Va hw.cxgbe.attack_filter 368Set to 1 to enable the "attack filter". 369Default is 0. 370The attack filter will drop an incoming frame if any of these conditions is 371true: src ip/ip6 == dst ip/ip6; tcp and src/dst ip is not unicast; src/dst ip is 372loopback (127.x.y.z); src ip6 is not unicast; src/dst ip6 is loopback (::1/128) 373or unspecified (::/128); tcp and src/dst ip6 is mcast (ff00::/8). 374This facility is available on T4 and T5 based cards only. 375.It Va hw.cxgbe.drop_ip_fragments 376Set to 1 to drop all incoming IP fragments. 377Default is 0. 378Note that this drops valid frames. 379.It Va hw.cxgbe.drop_pkts_with_l2_errors 380Set to 1 to drop incoming frames with Layer 2 length or checksum errors. 381Default is 1. 382.It Va hw.cxgbe.drop_pkts_with_l3_errors 383Set to 1 to drop incoming frames with IP version, length, or checksum errors. 384The IP checksum is validated for TCP or UDP packets only. 385Default is 0. 386.It Va hw.cxgbe.drop_pkts_with_l4_errors 387Set to 1 to drop incoming frames with Layer 4 (TCP or UDP) length, 388checksum, or other errors. 389Default is 0. 390.El 391.Sh SUPPORT 392For general information and support, 393go to the Chelsio support website at: 394.Pa http://www.chelsio.com/ . 395.Pp 396If an issue is identified with this driver with a supported adapter, 397email all the specific information related to the issue to 398.Aq Mt support@chelsio.com . 399.Sh SEE ALSO 400.Xr arp 4 , 401.Xr ccr 4 , 402.Xr cxgb 4 , 403.Xr cxgbev 4 , 404.Xr netintro 4 , 405.Xr ng_ether 4 , 406.Xr ifconfig 8 407.Sh HISTORY 408The 409.Nm 410device driver first appeared in 411.Fx 9.0 . 412Support for T5 cards first appeared in 413.Fx 9.2 414and 415.Fx 10.0 . 416Support for T6 cards first appeared in 417.Fx 11.1 418and 419.Fx 12.0 . 420Support for T7 cards first appeared in 421.Fx 15.0 . 422.Sh AUTHORS 423.An -nosplit 424The 425.Nm 426driver was written by 427.An Navdeep Parhar Aq Mt np@FreeBSD.org . 428