1.\" Copyright(c) 2016 - 2022 Intel Corporation 2.\" All rights reserved. 3.\" 4.\" This software is available to you under a choice of one of two 5.\" licenses. You may choose to be licensed under the terms of the GNU 6.\" General Public License (GPL) Version 2, available from the file 7.\" COPYING in the main directory of this source tree, or the 8.\" OpenFabrics.org BSD license below: 9.\" 10.\" Redistribution and use in source and binary forms, with or 11.\" without modification, are permitted provided that the following 12.\" conditions are met: 13.\" 14.\" - Redistributions of source code must retain the above 15.\" copyright notice, this list of conditions and the following 16.\" disclaimer. 17.\" 18.\" - Redistributions in binary form must reproduce the above 19.\" copyright notice, this list of conditions and the following 20.\" disclaimer in the documentation and/or other materials 21.\" provided with the distribution. 22.\" 23.\" THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, 24.\" EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF 25.\" MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND 26.\" NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS 27.\" BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN 28.\" ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN 29.\" CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 30.\" SOFTWARE. 31.\" 32.Dd March 30, 2022 33.Dt IRDMA 4 34.Os 35.Sh NAME 36.Nm irdma 37.Nd RDMA FreeBSD driver for Intel(R) Ethernet Controller E810 38.Sh SYNOPSIS 39This module relies on 40.Xr ice 4 41.Bl -tag -width indent 42.It The following kernel options should be included in the configuration: 43.Cd options OFED 44.Cd options OFED_DEBUG_INIT 45.Cd options COMPAT_LINUXKPI 46.Cd options SDP 47.Cd options IPOIB_CM 48.El 49.Sh DESCRIPTION 50.Ss Features 51The 52.Nm 53driver provides RDMA protocol support on RDMA-capable Intel Ethernet 800 Series 54NICs which are supported by 55.Xr ice 4 56. 57.Pp 58The driver supports both iWARP and RoCEv2 protocols. 59.Sh CONFIGURATION 60.Ss TUNABLES 61Tunables can be set at the 62.Xr loader 8 63prompt before booting the kernel or stored in 64.Xr loader.conf 5 . 65.Bl -tag -width indent 66.It Va dev.irdma<interface_number>.roce_enable 67enables RoCEv2 protocol usage on <interface_numer> interface. 68.Pp 69By default RoCEv2 protocol is used. 70.It Va dev.irdma<interface_number>.dcqcn_cc_cfg_valid 71indicates that all DCQCN parameters are valid and should be updated in 72registers or QP context. 73.Pp 74Setting this parameter to 1 means that settings in 75.Em dcqcn_min_dec_factor , dcqcn_min_rate_MBps , dcqcn_F , dcqcn_T , 76.Em dcqcn_B, dcqcn_rai_factor, dcqcn_hai_factor, dcqcn_rreduce_mperiod 77are taken into account. 78Otherwise default values are used. 79.Pp 80Note: "roce_enable" must also be set for this tunable to take effect. 81.It Va dev.irdma<interface_number>.dcqcn_min_dec_factor 82The minimum factor by which the current transmit rate can be changed when 83processing a CNP. 84Value is given as a percentage (1-100). 85.Pp 86Note: "roce_enable" and "dcqcn_cc_cfg_valid" must also be set for this tunable 87to take effect. 88.It Va dev.irdma<interface_number>.dcqcn_min_rate_MBps 89The minimum value, in Mbits per second, for rate to limit. 90.Pp 91Note: "roce_enable" and "dcqcn_cc_cfg_valid" must also be set for this tunable 92to take effect. 93.It Va dev.irdma<interface_number>.dcqcn_F 94The number of times to stay in each stage of bandwidth recovery. 95.Pp 96Note: "roce_enable" and "dcqcn_cc_cfg_valid" must also be set for this tunable 97to take effect. 98.It Va dev.irdma<interface_number>.dcqcn_T 99The number of microseconds that should elapse before increasing the CWND 100in DCQCN mode. 101.Pp 102Note: "roce_enable" and "dcqcn_cc_cfg_valid" must also be set for this tunable 103to take effect. 104.It Va dev.irdma<interface_number>.dcqcn_B 105The number of bytes to transmit before updating CWND in DCQCN mode. 106.Pp 107Note: "roce_enable" and "dcqcn_cc_cfg_valid" must also be set for this tunable 108to take effect. 109.It Va dev.irdma<interface_number>.dcqcn_rai_factor 110The number of MSS to add to the congestion window in additive increase mode. 111.Pp 112Note: "roce_enable" and "dcqcn_cc_cfg_valid" must also be set for this tunable 113to take effect. 114.It Va dev.irdma<interface_number>.dcqcn_hai_factor 115The number of MSS to add to the congestion window in hyperactive increase mode. 116.Pp 117Note: "roce_enable" and "dcqcn_cc_cfg_valid" must also be set for this tunable 118to take effect. 119.It Va dev.irdma<interface_number>.dcqcn_rreduce_mperiod 120The minimum time between 2 consecutive rate reductions for a single flow. 121Rate reduction will occur only if a CNP is received during the relevant time 122interval. 123.Pp 124Note: "roce_enable" and "dcqcn_cc_cfg_valid" must also be set for this tunable 125to take effect. 126.El 127.Ss SYSCTL PROCEDURES 128Sysctl controls are available for runtime adjustments. 129.Bl -tag -width indent 130.It Va dev.irdma<interface_number>.debug 131defines level of debug messages. 132.Pp 133Typical value: 1 for errors only, 0x7fffffff for full debug. 134.It Va dev.irdma<interface_number>.dcqcn_enable 135enables the DCQCN algorithm for RoCEv2. 136.Pp 137Note: "roce_enable" must also be set for this sysctl to take effect. 138.Pp 139Note: The change may be set at any time, but it will be applied only to newly 140created QPs. 141.El 142.Ss TESTING 143.Bl -enum 144.It 145To load the irdma driver, run: 146.Bd -literal -offset indent 147kldload irdma 148.Ed 149If if_ice is not already loaded, the system will load it on its own. 150Please check whether the value of sysctl 151.Va hw.ice.irdma 152is 1, if the irdma driver is not loading. 153To change the value put: 154.Bd -literal -offset indent 155hw.ice.irdma=1 156.Ed 157in 158.Pa /boot/loader.conf 159and reboot. 160.It 161To check that the driver was loaded, run: 162.Bd -literal -offset indent 163sysctl -a | grep infiniband 164.Ed 165Typically, if everything goes well, around 190 entries per PF will appear. 166.It 167Each interface of the card may work in either iWARP or RoCEv2 mode. 168To enable RoCEv2 compatibility, add: 169.Bd -literal -offset indent 170dev.irdma<interface_number>.roce_enable=1 171.Ed 172where <interface_number> is a desired ice interface number on which 173RoCEv2 protocol needs to be enabled, into: 174.Pa /boot/loader.conf 175, for instance: 176.Bl -tag -width indent 177.It dev.irdma0.roce_enable=0 178.It dev.irdma1.roce_enable=1 179.El 180will keep iWARP mode on ice0 and enable RoCEv2 mode on interface ice1. 181The RoCEv2 mode is the default. 182.Pp 183To check irdma roce_enable status, run: 184.Bd -literal -offset indent 185sysctl dev.irdma<interface_number>.roce_enable 186.Ed 187for instance: 188.Bd -literal -offset indent 189sysctl dev.irdma2.roce_enable 190.Ed 191with returned value of '0' indicate the iWARP mode, and the value of '1' 192indicate the RoCEv2 mode. 193.Pp 194Note: An interface configured in one mode will not be able to connect 195to a node configured in another mode. 196.Pp 197Note: RoCEv2 has currently limited support, for functional testing only. 198DCB and Priority Flow Controller (PFC) are not currently supported which 199may lead to significant performance loss or connectivity issues. 200.It 201Enable flow control in the ice driver: 202.Bd -literal -offset indent 203sysctl dev.ice.<interface_number>.fc=3 204.Ed 205Enable flow control on the switch your system is connected to. 206See your switch documentation for details. 207.It 208The source code for krping software is provided with the kernel in 209/usr/src/sys/contrib/rdma/krping/. 210To compile the software, change directory to 211/usr/src/sys/modules/rdma/krping/ and invoke the following: 212.Bl -tag -width indent 213.It make clean 214.It make 215.It make install 216.It kldload krping 217.El 218.It 219Start a krping server on one machine: 220.Bd -literal -offset indent 221echo size=64,count=1,port=6601,addr=100.0.0.189,server > /dev/krping 222.Ed 223.It 224Connect a client from another machine: 225.Bd -literal -offset indent 226echo size=64,count=1,port=6601,addr=100.0.0.189,client > /dev/krping 227.Ed 228.El 229.Sh SUPPORT 230For general information and support, go to the Intel support website at: 231.Lk http://support.intel.com/ . 232.Pp 233If an issue is identified with this driver with a supported adapter, email all 234the specific information related to the issue to 235.Mt freebsd@intel.com . 236.Sh SEE ALSO 237.Xr ice 4 238.Sh AUTHORS 239.An -nosplit 240The 241.Nm 242driver was prepared by 243.An Bartosz Sobczak Aq Mt bartosz.sobczak@intel.com . 244