1.\" Copyright (c) 2015, Chelsio Inc 2.\" All rights reserved. 3.\" 4.\" Redistribution and use in source and binary forms, with or without 5.\" modification, are permitted provided that the following conditions are met: 6.\" 7.\" 1. Redistributions of source code must retain the above copyright notice, 8.\" this list of conditions and the following disclaimer. 9.\" 10.\" 2. Redistributions in binary form must reproduce the above copyright 11.\" notice, this list of conditions and the following disclaimer in the 12.\" documentation and/or other materials provided with the distribution. 13.\" 14.\" 3. Neither the name of the Chelsio Inc nor the names of its 15.\" contributors may be used to endorse or promote products derived from 16.\" this software without specific prior written permission. 17.\" 18.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" 19.\" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 20.\" IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE 21.\" ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE 22.\" LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR 23.\" CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF 24.\" SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS 25.\" INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN 26.\" CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) 27.\" ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE 28.\" POSSIBILITY OF SUCH DAMAGE. 29.\" 30.\" * Other names and brands may be claimed as the property of others. 31.\" 32.\" $FreeBSD$ 33.\" 34.Dd April 13, 2018 35.Dt CXGBETOOL 8 36.Os 37.Sh NAME 38.Nm cxgbetool 39.Nd Userspace companion to 40.Xr cxgbe 4 41.Sh SYNOPSIS 42.Bl -item -compact 43.It 44.Nm Ar nexus command Op Ar parameter ... 45.Pp 46.It 47.Nm Ar nexus Cm clearstats Ar port_id 48.It 49.Nm Ar nexus Cm context Bro Cm ingress | egress | fl | cong Brc Ar cntxt_id 50.It 51.Nm Ar nexus Cm filter mode Op Ar match-criteria ... 52.It 53.Nm Ar nexus Cm filter Ar idx Bro Ar filter-specification | Cm delete Brc 54.It 55.Nm Ar nexus Cm filter list 56.It 57.Nm Ar nexus Cm i2c Ar port_id devaddr addr Op Ar len 58.It 59.Nm Ar nexus Cm loadcfg Ar fw-config.txt 60.It 61.Nm Ar nexus Cm loadcfg clear 62.It 63.Nm Ar nexus Cm loadfw Ar fw-image.bin 64.It 65.Nm Ar nexus Cm memdump Ar addr len 66.It 67.Nm Ar nexus Cm policy Ar cop.txt 68.It 69.Nm Ar nexus Cm policy clear 70.It 71.Nm Ar nexus Bro Cm reg | reg64 Brc Ar addr Ns Op Ar =val 72.It 73.Nm Ar nexus Cm regdump Op Ar register-block ... 74.It 75.Nm Ar nexus Cm sched-class Ar sub-command Op Ar param Ar value 76.It 77.Nm Ar nexus Cm sched-queue Ar port Ar queue Ar class 78.It 79.Nm Ar nexus Cm stdio 80.It 81.Nm Ar nexus Cm tcb Ar tid 82.El 83.Sh DESCRIPTION 84.Nm 85provides command-line access to features and debug facilities exported by 86.Xr cxgbe 4 87via private ioctls. 88The target nexus device, 89.Va t4nex%d 90, is always the first argument. 91(The parent nexus for an Ethernet port 92.Va cxgbe%d 93is listed in 94.Va dev.cxgbe.%d.%parent 95in the 96.Xr sysctl 8 97MIB). 98The rest consists of a command and any parameters required by that command. 99.Ss Commands 100.Bl -ohang 101.It Cm clearstats Ar port_id 102Clear all transmit, receive, and error statistics of all queues associated 103with a port. 104The total number of ports attached to a nexus is listed in 105.Va dev.t4nex.%d.nports 106and the 0 based 107.Ar port_id 108identifies a port within this range. 109.Pp 110.Bl -item -compact 111.It 112.Cm context ingress Ar ingress_cntxt_id 113.It 114.Cm context cong Ar ingress_cntxt_id 115.It 116.Cm context egress Ar egress_cntxt_id 117.It 118.Cm context fl Ar flm_cntxt_id 119.El 120Display hardware context for an ingress queue, congestion manager, egress 121queue, or freelist manager. 122.Bl -tag -width ingress_cntxt_id -compact 123.It Ar ingress_cntxt_id 124context id of an ingress queue -- the value listed in one of 125.Va dev.t4nex.%d.fwq.cntxt_id Ns , 126.Va dev.cxgbe.%d.rxq.%d.cntxt_id Ns , 127or 128.Va dev.cxgbe.%d.ofld_rxq.%d.cntxt_id Ns . 129.It Ar egress_cntxt_id 130context id of an egress queue -- the value listed in one of 131.Va dev.t4nex.%d.mgmtq.cntxt_id Ns , 132.Va dev.cxgbe.%d.txq.%d.cntxt_id Ns , 133.Va dev.cxgbe.%d.ctrlq.%d.cntxt_id Ns , 134.Va dev.cxgbe.%d.ofld_txq.%d.cntxt_id Ns , 135.Va dev.cxgbe.%d.rxq.%d.fl.cntxt_id Ns , 136or 137.Va dev.cxgbe.%d.ofld_rxq.%d.fl.cntxt_id Ns . 138Note that freelists are egress queues too. 139.It Ar flm_cntxt_id 140context id of a freelist manager. 141The FLM context id is displayed in the 142egress context dump of a freelist as FLMcontextID. 143.El 144.It Cm filter mode Op Ar match-criteria ... 145Display or set the nexus's filter mode. 146.Ar match-criteria 147is a whitespace separated list of criteria from the table below. 148Each criteria has an associated budget which is also listed in the table. 149The total budget allowed is 36 and attempts to set a filter mode that 150exceeds this will be rejected. 151Every filter must conform to the filter mode -- multiple match criteria 152per filter are allowed but only from among those in the current setting 153of the filter mode. 154The filter mode can only be changed when there are no existing filters. 155Its default value is 156.Cm ipv4 ipv6 sip dip sport dport matchtype proto vlan iport fcoe 157.Pp 158(Note that 159.Ar mask 160defaults to all 1s when not provided explicitly. 161Also note that many of the items being matched are discrete numeric 162values rather than bit fields and should be masked with caution.) 163.TS 164center expand; 165cb cb cb cbw(40m) 166cb c l l. 167Criteria Budget Usage Matches if ... 168_ 169ipv4 0 T{ 170.Cm type ipv4 171T} T{ 172incoming packet is an IPv4 datagram. 173T} 174_ 175ipv6 0 T{ 176.Cm type ipv6 177T} T{ 178incoming packet is an IPv6 datagram. 179T} 180_ 181sip 0 T{ 182.Cm sip Ar addr Ns Op / Ns Ar mask 183T} T{ 184bitwise and of the source address in an incoming IP datagram with 185.Ar mask 186equals 187.Ar addr Ns . 188.Ar addr 189can be an IPv4 or IPv6 address. 190T} 191_ 192dip 0 T{ 193.Cm dip Ar addr Ns Op / Ns Ar mask 194T} T{ 195bitwise and of the destination address in an incoming IP datagram with 196.Ar mask 197equals 198.Ar addr Ns . 199.Ar addr 200can be an IPv4 or IPv6 address. 201T} 202_ 203sport 0 T{ 204.Cm sport Ar port Ns Op : Ns Ar mask 205T} T{ 206bitwise and of the source port in an incoming TCP or UDP datagram with 207.Ar mask 208equals 209.Ar port Ns . 210T} 211_ 212dport 0 T{ 213.Cm dport Ar port Ns Op : Ns Ar mask 214T} T{ 215bitwise and of the destination port in an incoming TCP or UDP datagram with 216.Ar mask 217equals 218.Ar port Ns . 219T} 220_ 221fcoe 1 T{ 222.Cm fcoe Brq 0 | 1 223T} T{ 224incoming frame is Fibre Channel over Ethernet(1) or not(0). 225T} 226_ 227iport 3 T{ 228.Cm iport Ar val Ns Op : Ns Ar mask 229T} T{ 230bitwise and of the ingress port with 231.Ar mask 232equals 233.Ar val Ns . 234The ingress port is a 3 bit number that identifies the port on which a 235frame arrived. 236Physical ports are numbered 0-3 and 4-7 are internal loopback paths 237within the chip. 238Note that ingress port is not a bit field so it is not always possible 239to match an arbitrary subset of ingress ports with a single filter rule. 240T} 241_ 242ovlan 17 T{ 243.Cm ovlan Ar tag Ns Op : Ns Ar mask 244T} T{ 245bitwise and of the 16-bit outer VLAN tag of an incoming frame with 246.Ar mask 247equals 248.Ar tag Ns . 249T} 250_ 251vlan 17 T{ 252.Cm vlan Ar tag Ns Op : Ns Ar mask 253T} T{ 254bitwise and of the 16-bit VLAN tag of an incoming QinQ frame with 255.Ar mask 256equals 257.Ar tag Ns . 258The inner VLAN tag is used if the incoming frame is QinQ. 259T} 260_ 261tos 8 T{ 262.Cm tos Ar val Ns Op : Ns Ar mask 263T} T{ 264bitwise and of the 8-bit IP Type of Service/IPv6 Traffic Class in an 265incoming packet with 266.Ar mask 267equals 268.Ar val Ns . 269T} 270_ 271proto 8 T{ 272.Cm proto Ar ipproto Ns Op : Ns Ar mask 273T} T{ 274bitwise and of the 8-bit IP protocol in an incoming packet with 275.Ar mask 276equals 277.Ar ipproto Ns . 278T} 279_ 280ethtype 16 T{ 281.Cm ethtype Ar type Ns Op : Ns Ar mask 282T} T{ 283bitwise and of the 16-bit Ethernet type field of an incoming frame with 284.Ar mask 285equals 286.Ar type Ns . 287T} 288_ 289macidx 9 T{ 290.Cm macidx Ar idx Ns Op : Ns Ar mask 291T} T{ 292bitwise and of the MAC Address Match Index of an incoming frame with 293.Ar mask 294equals 295.Ar idx Ns . 296The MAC Address Match Index refers to an entry in the MPS TCAM or in the 297MPS hash. See 298.Cm matchtype 299for more information. 300T} 301_ 302matchtype 3 T{ 303.Cm matchtype Ar type Ns Op : Ns Ar mask 304T} T{ 305bitwise and of the Match Type of an incoming frame with 306.Ar mask 307equals 308.Ar idx Ns . 309Match Type is one of the following: 310.Bl -tag -width "n" -compact 311.It 0 312destination MAC in incoming frame is a unicast L2 address that is 313programmed in the MPS TCAM. 314.Cm macidx 315can be used to match the index (and thus the MAC address) of the match 316in the TCAM. 317.It 1 318destination MAC in incoming frame is a unicast L2 address that "hit" a 319hash entry in the MPS hash table. 320.Cm macidx 321can be used to match the index of the entry in the MPS hash table. 322.It 2 323destination MAC in incoming frame is a multicast L2 address that is 324programmed in the MPS TCAM. 325.Cm macidx 326can be used to match the index (and thus the MAC address) of the match 327in the TCAM. 328.It 3 329destination MAC in incoming frame is a multicast L2 address that "hit" 330an entry in the MPS hash table. 331.It 4 332interface on which incoming frame was received is in promiscuous mode 333and the destination MAC in the frame is not a broadcast address, and 334does not match in the MPS TCAM or the MPS hash either. (The frame would 335have been discarded if the interface wasn't in promiscuous mode.) 336.It 5 337interface on which incoming frame was received is in promiscuous mode 338and the destination MAC in the frame is not a broadcast address; it 339wasn't looked up in the MPS TCAM or the MPS hash because the chip is 340configured to give precedence to promiscuous mode classification. 341.It 6 342destination MAC in incoming frame is a broadcast address. 343.It 7 344Not documented. Do not use. 345.El 346T} 347_ 348frag 1 T{ 349.Cm frag Brq 0 | 1 350T} T{ 351incoming frame is part of a fragmented IP datagram(1) or not(0). 352T} 353.TE 354.It Cm filter Ar idx Ar filter-specification 355Program a filter at the index specified by 356.Ar idx Ns . 357.Ar filter-specification 358consists of one or more matches to try against an incoming frame and an 359action to perform when all matches succeed. 360.It Cm filter Ar idx Cm delete 361Delete filter that is at the given index. 362.It Cm filter Cm list 363List all filters programmed into the hardware. 364.It Cm i2c Ar port_id devaddr addr Op Ar len 365.It Cm loadcfg Ar fw-config.txt 366Install the firmware configuration file contained in 367.Ar fw-config.txt 368to the card. 369Set hw.cxgbe.config_file="flash" in loader.conf to get 370.Xr cxgbe 4 371to use the on-flash configuration. 372.It Cm loadcfg Cm clear 373Erase configuration file from the card. 374.It Cm loadfw Ar fw-image.bin 375Install the firmware contained in 376.Ar fw-image.bin 377to the card. 378.It Cm memdump Ar addr len 379Display 380.Ar len 381bytes of data of the card's memory starting at 382.Ar addr Ns . 383The card's memory map is available in 384.Va dev.t4nex.%d.misc.meminfo Ns . 385.It Cm policy Ar cop.txt 386Install the Connection Offload Policy (COP) in 387.Ar cop.txt Ns . 388A COP offers fine-grained control over which connections get offloaded and with 389what parameters. 390Set 391.Cm hw.cxgbe.cop_managed_offloading="1" 392in loader.conf to ensure that t4_tom will not offload any connection before a 393COP is installed. 394Note that t4_tom must be loaded and operational (IFCAP_TOE enabled) as always 395for any kind of offload based on the hardware TOE. 396.Bl -column -offset indent "COP installed" "cop_managed_offloading" "Behavior" 397.It Sy COP installed Ta Sy cop_managed_offloading Ta Sy Behavior 398.It NO Ta 0 Ta offload all [Default] 399.It NO Ta 1 Ta no offload 400.It YES Ta Don't Care Ta Rule based offload 401.El 402.Pp 403The policy file consists of empty lines, comments (lines begining with #) and 404any number of rules. 405Rules are applied in the order they appear in the file and processing stops at 406the first match. 407There is an implicit rule that disables offload for connections that do not 408match anything in the policy. 409.Pp 410Each rule consists of a filter part, which determines what connections the 411rule applies to, and a settings part, which determines whether whether matching 412connections will be offloaded and, if so, with what settings. 413The general form of a rule is 414.Bl -ohang -offset indent 415.It Cm \&[ Ar socket-type Cm \&] Ar pcap-filter Cm => Ar settings 416.Pp 417.Ar socket-type 418is one of the following. 419.Bl -tag -width "X" -compact 420.It Sy A 421Active open. 422Connection is being opened by this host. 423.It Sy P 424Passive open. 425Connection was requested by a peer. 426.It Sy L 427Listen called on a socket. 428Disabling offload in such a rule will prevent a hardware listener from being started. 429.It Sy D 430Don't care. 431Matches all of the above. 432.El 433.Pp 434.Ar pcap-filter 435is an expression that follows the 436.Xr pcap-filter 7 437syntax, or it is the keyword 438.Cm all 439that matches everything. 440.Pp 441.Ar settings 442determine whether connections matching 443.Ar socket-type 444and 445.Ar pcap-filter 446are offloaded and optionally sets some per-connection properties if they are. 447A combination of the following is allowed. 448.Bl -tag -width "timestamp" -compact 449.It Cm offload 450Connection should be offloaded. 451Use 452.Cm !offload 453or 454.Cm not offload 455to disable offload instead. 456.It Cm coalesce 457Enable rx payload coalescing. 458Negate to disable. 459.It Cm timestamp 460Enable TCP timestamp option. 461Negate to disable. 462.It Cm sack 463Enable TCP Selective Acknowledgements (SACK). 464Negate to disable. 465.It Cm nagle 466Enable Nagle's algorithm. 467Negate to disable. 468.It Cm ecn 469Enable Explicit Congestion Notification (ECN). 470Negate to disable. 471.It Cm ddp 472Use Direct Data Placement (zero copy receive) and zero copy transmit on the 473connection to service AIO requests on the socket. 474Negate to disable. 475.It Cm tls 476Set ULP mode to ULP_MODE_TLS. 477.It Cm cong Ar algo 478Use the specified congestion control algorithm. 479.Ar algo 480must be one of 481.Cm reno Ns , Cm tahoe Ns , Cm newreno Ns , or Cm highspeed Ns . 482.It Cm class Ar sc 483Bind the connection to the specified tx scheduling class. 484Valid range is 0 to 14 (for T4) and 0 to 15 (T5 onwards). 485.It Cm rxq Ar qid 486Use the specified offload rx queue. 487.Ar qid 488should be between 0 and nofldrxq for the ifnet. 489.It Cm txq Ar qnum 490Use the specified offload tx queue. 491.Ar qid 492should be between 0 and nofldtxq for the ifnet. 493.It Cm bind Ar qnum 494Shorthand for 495.Cm rxq Ar qnum Cm txq Ar qnum Ns . 496Use only when nofldrxq is the same as nofldtxq. 497.It Cm mss Ar val 498Set the advertised TCP MSS in the SYN for this connection to 499.Ar val 500(in bytes). 501The hardware MTU table must already have an entry that is suitable for the MSS. 502.El 503.Pp 504.It Example of a COP. 505Note that hardware listener for port 22 will be IPv4 only because the rule 506before it will prevent any IPv6 servers other than the first two. Also note 507that outgoing connections to 192.168/16 are the only outgoing connections that 508will get offloaded. 509.Bd -literal 510[L] port 80 => offload 511[L] port 443 => offload 512[L] ip6 => !offload 513[L] port 22 => offload 514[P] dst port 80 => offload cong highspeed !sack !ecn 515[P] dst port 443 => offload tls 516[A] dst net 192.168/16 => offload 517[A] all => !offload 518[D] port 22 => offload !nagle 519.Ed 520.El 521.It Cm policy clear 522Remove the Connection Offload Policy (COP) if one is in use. 523.It Bro Cm reg | reg64 Brc Ar addr Ns Op Ar =val 524.It Cm regdump Op Ar register-block ... 525Display contents of device registers. 526One or more 527.Ar register-block 528can be specified to limit the registers displayed. 529The default is to display registers for all blocks. 530Registers with read side effects are not read during a 531.Cm regdump 532operation. 533.Ar register-block 534can be 535.Cm sge pci dbg mc ma edc0 edc1 cim tp ulp_rx ulp_tx pmrx pmtx mps cplsw 536.Cm smb i2c mi uart pmu sf pl le ncsi xgmac Ns . 537.It Cm sched-class config Op Ar param Ar value 538Configure optional feature capabilities for the TX scheduler. 539.Bl -ohang -offset indent 540.It Sy type Ar scheduler-type 541Use packet for the packet scheduler. 542.It Sy minmax Ar value 543A non-zero value will enable "minmax" mode; a zero value will disable "minmax" mode. 544.Pp 545NOTE: Many (most) of the parameters and constraints are adapter-specific 546- for instance the number of channels and classes which are available 547whether various modes are implemented, etc. 548Consult the adapter documentation for specific information on any limitations. 549.El 550.It Cm sched-class params Op Ar param Ar value 551Configure parameters for a scheduling class. 552.Bl -ohang -offset indent 553.It Sy type Ar scheduler-type 554Use packet for packet scheduler. 555.It Sy level Ar scheduler-hierarchy-level 556The "level" within the scheduling hierarchy which is being programed: 557.Pp 558.Bl -tag -width "cl-wrr" -compact -offset indent 559.It Sy cl-rl 560Class Rate Limiting. 561.Pp 562.It Sy cl-wrr 563Class Weighted Round Robin. 564.Pp 565.It Sy ch-rl 566Channel Rate Limiting. 567.El 568.It Sy mode Ar scheduler-mode 569The mode in which the scheduling class is going to operate: 570.Pp 571.Bl -tag -width "class" -compact -offset indent 572.It Sy class 573All of the "flows" bound to the scheduling class will be held to aggregate scheduling constraints. 574.Pp 575.It Sy flow 576Each of the "flows" bound to the scheduling class will be held to the scheduling constraints. 577.El 578.Pp 579E.g. if the scheduling class has a TX bandwidth of 10Mb/s, in 580.Cm class 581mode, all of the "flows" bound to the class would be limited to an 582aggregate bandwidth of 10Mb/s; but in 583.Cm flow 584mode, each of the "flows" bound to the scheduling class would be limited to 10Mb/s. 585.It Sy rate-unit Ar scheduler-rate-unit 586The units of the scheduler rate constraints: 587.Pp 588.Bl -tag -width "bits" -compact -offset indent 589.It Sy bits 590bit rate in Kb/s. 591.Pp 592.It Sy pkts 593packets/s. 594.El 595.It Sy rate-mode Ar scheduler-rate-mode 596The mode of the scheduler rate constraints: 597.Pp 598.Bl -tag -width "relative" -compact -offset indent 599.It Sy relative 600percent of port rate. 601.Pp 602.It Sy absolute 603Kb/s. 604.El 605.It Sy channel Ar scheduler-channel-index 606The scheduling channel to which the scheduling class will be bound. 607.It Sy class Ar scheduler-class-index 608The scheduling class being programmed. 609.It Sy min-rate Ar minimum-rate 610The minimum guaranteed rate to which a rate-limiting scheduling class hierarchy will have access. 611.It Sy max-rate Ar maximum-rate 612The maximum rate for a rate-limiting scheduling class hierarchy. 613.It Sy weight Ar round-robin-weight 614The weight to be used for a weighted-round-robin scheduling hierarchy. 615.It Sy pkt-size Ar average-packet-size 616The average packet size will be used to compute scheduler constraints for a rate-limited scheduler class hierarchy. 617.Pp 618NOTE: Many (most) of the parameters and constraints are adapter-specific - for instance the number of channels and classes which are available, 619whether various modes are implemented, etc. 620Consult the adapter documentation for specific information on any limitations. 621.El 622.It Cm sched-queue Ar port queue class 623Bind the indicated port's NIC TX 624.Ar queue 625to the specified TX Scheduler 626.Ar class. 627If the TX 628.Ar queue 629is 630.Cm all, * 631or any negative value, the binding will apply to 632all of the TX queues associated with the 633.Ar interface. 634If the class is 635.Cm unbind, clear 636or any negative value, the TX queue(s) will be unbound from 637any current TX Scheduler Class binding. 638.It Cm stdio 639Switch to interactive mode. 640.It Cm tcb Ar tid 641Display contents of the hardware TCB (TCP Control Block) for the 642connection identfied by 643.Ar tid Ns . 644.El 645.Sh FILES 646/sys/dev/cxgbe/t4_ioctl.h 647.Sh AUTHORS 648This manual page was written by 649.An Navdeep Parhar Aq np@FreeBSD.org . 650