1.\" Copyright(c) 2016 - 2022 Intel Corporation 2.\" All rights reserved. 3.\" 4.\" This software is available to you under a choice of one of two 5.\" licenses. You may choose to be licensed under the terms of the GNU 6.\" General Public License (GPL) Version 2, available from the file 7.\" COPYING in the main directory of this source tree, or the 8.\" OpenFabrics.org BSD license below: 9.\" 10.\" Redistribution and use in source and binary forms, with or 11.\" without modification, are permitted provided that the following 12.\" conditions are met: 13.\" 14.\" - Redistributions of source code must retain the above 15.\" copyright notice, this list of conditions and the following 16.\" disclaimer. 17.\" 18.\" - Redistributions in binary form must reproduce the above 19.\" copyright notice, this list of conditions and the following 20.\" disclaimer in the documentation and/or other materials 21.\" provided with the distribution. 22.\" 23.\" THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, 24.\" EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF 25.\" MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND 26.\" NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS 27.\" BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN 28.\" ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN 29.\" CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 30.\" SOFTWARE. 31.\" 32.\" $FreeBSD$ 33.\" 34.Dd March 30, 2022 35.Dt IRDMA 4 36.Os 37.Sh NAME 38.Nm irdma 39.Nd RDMA FreeBSD driver for Intel(R) Ethernet Controller E810 40.Sh SYNOPSIS 41This module relies on 42.Xr ice 4 43.Bl -tag -width indent 44.It The following kernel options should be included in the configuration: 45.Cd options OFED 46.Cd options OFED_DEBUG_INIT 47.Cd options COMPAT_LINUXKPI 48.Cd options SDP 49.Cd options IPOIB_CM 50.El 51.Sh DESCRIPTION 52.Ss Features 53The 54.Nm 55driver provides RDMA protocol support on RDMA-capable Intel Ethernet 800 Series 56NICs which are supported by 57.Xr ice 4 58. 59.Pp 60The driver supports both iWARP and RoCEv2 protocols. 61.Sh CONFIGURATION 62.Ss TUNABLES 63Tunables can be set at the 64.Xr loader 8 65prompt before booting the kernel or stored in 66.Xr loader.conf 5 . 67.Bl -tag -width indent 68.It Va dev.irdma<interface_number>.roce_enable 69enables RoCEv2 protocol usage on <interface_numer> interface. 70.Pp 71By default RoCEv2 protocol is used. 72.It Va dev.irdma<interface_number>.dcqcn_cc_cfg_valid 73indicates that all DCQCN parameters are valid and should be updated in 74registers or QP context. 75.Pp 76Setting this parameter to 1 means that settings in 77.Em dcqcn_min_dec_factor , dcqcn_min_rate_MBps , dcqcn_F , dcqcn_T , 78.Em dcqcn_B, dcqcn_rai_factor, dcqcn_hai_factor, dcqcn_rreduce_mperiod 79are taken into account. 80Otherwise default values are used. 81.Pp 82Note: "roce_enable" must also be set for this tunable to take effect. 83.It Va dev.irdma<interface_number>.dcqcn_min_dec_factor 84The minimum factor by which the current transmit rate can be changed when 85processing a CNP. 86Value is given as a percentage (1-100). 87.Pp 88Note: "roce_enable" and "dcqcn_cc_cfg_valid" must also be set for this tunable 89to take effect. 90.It Va dev.irdma<interface_number>.dcqcn_min_rate_MBps 91The minimum value, in Mbits per second, for rate to limit. 92.Pp 93Note: "roce_enable" and "dcqcn_cc_cfg_valid" must also be set for this tunable 94to take effect. 95.It Va dev.irdma<interface_number>.dcqcn_F 96The number of times to stay in each stage of bandwidth recovery. 97.Pp 98Note: "roce_enable" and "dcqcn_cc_cfg_valid" must also be set for this tunable 99to take effect. 100.It Va dev.irdma<interface_number>.dcqcn_T 101The number of microseconds that should elapse before increasing the CWND 102in DCQCN mode. 103.Pp 104Note: "roce_enable" and "dcqcn_cc_cfg_valid" must also be set for this tunable 105to take effect. 106.It Va dev.irdma<interface_number>.dcqcn_B 107The number of bytes to transmit before updating CWND in DCQCN mode. 108.Pp 109Note: "roce_enable" and "dcqcn_cc_cfg_valid" must also be set for this tunable 110to take effect. 111.It Va dev.irdma<interface_number>.dcqcn_rai_factor 112The number of MSS to add to the congestion window in additive increase mode. 113.Pp 114Note: "roce_enable" and "dcqcn_cc_cfg_valid" must also be set for this tunable 115to take effect. 116.It Va dev.irdma<interface_number>.dcqcn_hai_factor 117The number of MSS to add to the congestion window in hyperactive increase mode. 118.Pp 119Note: "roce_enable" and "dcqcn_cc_cfg_valid" must also be set for this tunable 120to take effect. 121.It Va dev.irdma<interface_number>.dcqcn_rreduce_mperiod 122The minimum time between 2 consecutive rate reductions for a single flow. 123Rate reduction will occur only if a CNP is received during the relevant time 124interval. 125.Pp 126Note: "roce_enable" and "dcqcn_cc_cfg_valid" must also be set for this tunable 127to take effect. 128.El 129.Ss SYSCTL PROCEDURES 130Sysctl controls are available for runtime adjustments. 131.Bl -tag -width indent 132.It Va dev.irdma<interface_number>.debug 133defines level of debug messages. 134.Pp 135Typical value: 1 for errors only, 0x7fffffff for full debug. 136.It Va dev.irdma<interface_number>.dcqcn_enable 137enables the DCQCN algorithm for RoCEv2. 138.Pp 139Note: "roce_enable" must also be set for this sysctl to take effect. 140.Pp 141Note: The change may be set at any time, but it will be applied only to newly 142created QPs. 143.El 144.Ss TESTING 145.Bl -enum 146.It 147To load the irdma driver, run: 148.Bd -literal -offset indent 149kldload irdma 150.Ed 151If if_ice is not already loaded, the system will load it on its own. 152Please check whether the value of sysctl 153.Va hw.ice.irdma 154is 1, if the irdma driver is not loading. 155To change the value put: 156.Bd -literal -offset indent 157hw.ice.irdma=1 158.Ed 159in 160.Pa /boot/loader.conf 161and reboot. 162.It 163To check that the driver was loaded, run: 164.Bd -literal -offset indent 165sysctl -a | grep infiniband 166.Ed 167Typically, if everything goes well, around 190 entries per PF will appear. 168.It 169Each interface of the card may work in either iWARP or RoCEv2 mode. 170To enable RoCEv2 compatibility, add: 171.Bd -literal -offset indent 172dev.irdma<interface_number>.roce_enable=1 173.Ed 174where <interface_number> is a desired ice interface number on which 175RoCEv2 protocol needs to be enabled, into: 176.Pa /boot/loader.conf 177, for instance: 178.Bl -tag -width indent 179.It dev.irdma0.roce_enable=0 180.It dev.irdma1.roce_enable=1 181.El 182will keep iWARP mode on ice0 and enable RoCEv2 mode on interface ice1. 183The RoCEv2 mode is the default. 184.Pp 185To check irdma roce_enable status, run: 186.Bd -literal -offset indent 187sysctl dev.irdma<interface_number>.roce_enable 188.Ed 189for instance: 190.Bd -literal -offset indent 191sysctl dev.irdma2.roce_enable 192.Ed 193with returned value of '0' indicate the iWARP mode, and the value of '1' 194indicate the RoCEv2 mode. 195.Pp 196Note: An interface configured in one mode will not be able to connect 197to a node configured in another mode. 198.Pp 199Note: RoCEv2 has currently limited support, for functional testing only. 200DCB and Priority Flow Controller (PFC) are not currently supported which 201may lead to significant performance loss or connectivity issues. 202.It 203Enable flow control in the ice driver: 204.Bd -literal -offset indent 205sysctl dev.ice.<interface_number>.fc=3 206.Ed 207Enable flow control on the switch your system is connected to. 208See your switch documentation for details. 209.It 210The source code for krping software is provided with the kernel in 211/usr/src/sys/contrib/rdma/krping/. 212To compile the software, change directory to 213/usr/src/sys/modules/rdma/krping/ and invoke the following: 214.Bl -tag -width indent 215.It make clean 216.It make 217.It make install 218.It kldload krping 219.El 220.It 221Start a krping server on one machine: 222.Bd -literal -offset indent 223echo size=64,count=1,port=6601,addr=100.0.0.189,server > /dev/krping 224.Ed 225.It 226Connect a client from another machine: 227.Bd -literal -offset indent 228echo size=64,count=1,port=6601,addr=100.0.0.189,client > /dev/krping 229.Ed 230.El 231.Sh SUPPORT 232For general information and support, go to the Intel support website at: 233.Lk http://support.intel.com/ . 234.Pp 235If an issue is identified with this driver with a supported adapter, email all 236the specific information related to the issue to 237.Mt freebsd@intel.com . 238.Sh SEE ALSO 239.Xr ice 4 240.Sh AUTHORS 241.An -nosplit 242The 243.Nm 244driver was prepared by 245.An Bartosz Sobczak Aq Mt bartosz.sobczak@intel.com . 246