1*132db935SJakub Kicinski.. SPDX-License-Identifier: GPL-2.0 2*132db935SJakub Kicinski.. include:: <isonum.txt> 3*132db935SJakub Kicinski 4*132db935SJakub Kicinski============================================= 5*132db935SJakub KicinskiChelsio N210 10Gb Ethernet Network Controller 6*132db935SJakub Kicinski============================================= 7*132db935SJakub Kicinski 8*132db935SJakub KicinskiDriver Release Notes for Linux 9*132db935SJakub Kicinski 10*132db935SJakub KicinskiVersion 2.1.1 11*132db935SJakub Kicinski 12*132db935SJakub KicinskiJune 20, 2005 13*132db935SJakub Kicinski 14*132db935SJakub Kicinski.. Contents 15*132db935SJakub Kicinski 16*132db935SJakub Kicinski INTRODUCTION 17*132db935SJakub Kicinski FEATURES 18*132db935SJakub Kicinski PERFORMANCE 19*132db935SJakub Kicinski DRIVER MESSAGES 20*132db935SJakub Kicinski KNOWN ISSUES 21*132db935SJakub Kicinski SUPPORT 22*132db935SJakub Kicinski 23*132db935SJakub Kicinski 24*132db935SJakub KicinskiIntroduction 25*132db935SJakub Kicinski============ 26*132db935SJakub Kicinski 27*132db935SJakub Kicinski This document describes the Linux driver for Chelsio 10Gb Ethernet Network 28*132db935SJakub Kicinski Controller. This driver supports the Chelsio N210 NIC and is backward 29*132db935SJakub Kicinski compatible with the Chelsio N110 model 10Gb NICs. 30*132db935SJakub Kicinski 31*132db935SJakub Kicinski 32*132db935SJakub KicinskiFeatures 33*132db935SJakub Kicinski======== 34*132db935SJakub Kicinski 35*132db935SJakub KicinskiAdaptive Interrupts (adaptive-rx) 36*132db935SJakub Kicinski--------------------------------- 37*132db935SJakub Kicinski 38*132db935SJakub Kicinski This feature provides an adaptive algorithm that adjusts the interrupt 39*132db935SJakub Kicinski coalescing parameters, allowing the driver to dynamically adapt the latency 40*132db935SJakub Kicinski settings to achieve the highest performance during various types of network 41*132db935SJakub Kicinski load. 42*132db935SJakub Kicinski 43*132db935SJakub Kicinski The interface used to control this feature is ethtool. Please see the 44*132db935SJakub Kicinski ethtool manpage for additional usage information. 45*132db935SJakub Kicinski 46*132db935SJakub Kicinski By default, adaptive-rx is disabled. 47*132db935SJakub Kicinski To enable adaptive-rx:: 48*132db935SJakub Kicinski 49*132db935SJakub Kicinski ethtool -C <interface> adaptive-rx on 50*132db935SJakub Kicinski 51*132db935SJakub Kicinski To disable adaptive-rx, use ethtool:: 52*132db935SJakub Kicinski 53*132db935SJakub Kicinski ethtool -C <interface> adaptive-rx off 54*132db935SJakub Kicinski 55*132db935SJakub Kicinski After disabling adaptive-rx, the timer latency value will be set to 50us. 56*132db935SJakub Kicinski You may set the timer latency after disabling adaptive-rx:: 57*132db935SJakub Kicinski 58*132db935SJakub Kicinski ethtool -C <interface> rx-usecs <microseconds> 59*132db935SJakub Kicinski 60*132db935SJakub Kicinski An example to set the timer latency value to 100us on eth0:: 61*132db935SJakub Kicinski 62*132db935SJakub Kicinski ethtool -C eth0 rx-usecs 100 63*132db935SJakub Kicinski 64*132db935SJakub Kicinski You may also provide a timer latency value while disabling adaptive-rx:: 65*132db935SJakub Kicinski 66*132db935SJakub Kicinski ethtool -C <interface> adaptive-rx off rx-usecs <microseconds> 67*132db935SJakub Kicinski 68*132db935SJakub Kicinski If adaptive-rx is disabled and a timer latency value is specified, the timer 69*132db935SJakub Kicinski will be set to the specified value until changed by the user or until 70*132db935SJakub Kicinski adaptive-rx is enabled. 71*132db935SJakub Kicinski 72*132db935SJakub Kicinski To view the status of the adaptive-rx and timer latency values:: 73*132db935SJakub Kicinski 74*132db935SJakub Kicinski ethtool -c <interface> 75*132db935SJakub Kicinski 76*132db935SJakub Kicinski 77*132db935SJakub KicinskiTCP Segmentation Offloading (TSO) Support 78*132db935SJakub Kicinski----------------------------------------- 79*132db935SJakub Kicinski 80*132db935SJakub Kicinski This feature, also known as "large send", enables a system's protocol stack 81*132db935SJakub Kicinski to offload portions of outbound TCP processing to a network interface card 82*132db935SJakub Kicinski thereby reducing system CPU utilization and enhancing performance. 83*132db935SJakub Kicinski 84*132db935SJakub Kicinski The interface used to control this feature is ethtool version 1.8 or higher. 85*132db935SJakub Kicinski Please see the ethtool manpage for additional usage information. 86*132db935SJakub Kicinski 87*132db935SJakub Kicinski By default, TSO is enabled. 88*132db935SJakub Kicinski To disable TSO:: 89*132db935SJakub Kicinski 90*132db935SJakub Kicinski ethtool -K <interface> tso off 91*132db935SJakub Kicinski 92*132db935SJakub Kicinski To enable TSO:: 93*132db935SJakub Kicinski 94*132db935SJakub Kicinski ethtool -K <interface> tso on 95*132db935SJakub Kicinski 96*132db935SJakub Kicinski To view the status of TSO:: 97*132db935SJakub Kicinski 98*132db935SJakub Kicinski ethtool -k <interface> 99*132db935SJakub Kicinski 100*132db935SJakub Kicinski 101*132db935SJakub KicinskiPerformance 102*132db935SJakub Kicinski=========== 103*132db935SJakub Kicinski 104*132db935SJakub Kicinski The following information is provided as an example of how to change system 105*132db935SJakub Kicinski parameters for "performance tuning" an what value to use. You may or may not 106*132db935SJakub Kicinski want to change these system parameters, depending on your server/workstation 107*132db935SJakub Kicinski application. Doing so is not warranted in any way by Chelsio Communications, 108*132db935SJakub Kicinski and is done at "YOUR OWN RISK". Chelsio will not be held responsible for loss 109*132db935SJakub Kicinski of data or damage to equipment. 110*132db935SJakub Kicinski 111*132db935SJakub Kicinski Your distribution may have a different way of doing things, or you may prefer 112*132db935SJakub Kicinski a different method. These commands are shown only to provide an example of 113*132db935SJakub Kicinski what to do and are by no means definitive. 114*132db935SJakub Kicinski 115*132db935SJakub Kicinski Making any of the following system changes will only last until you reboot 116*132db935SJakub Kicinski your system. You may want to write a script that runs at boot-up which 117*132db935SJakub Kicinski includes the optimal settings for your system. 118*132db935SJakub Kicinski 119*132db935SJakub Kicinski Setting PCI Latency Timer:: 120*132db935SJakub Kicinski 121*132db935SJakub Kicinski setpci -d 1425:: 122*132db935SJakub Kicinski 123*132db935SJakub Kicinski* 0x0c.l=0x0000F800 124*132db935SJakub Kicinski 125*132db935SJakub Kicinski Disabling TCP timestamp:: 126*132db935SJakub Kicinski 127*132db935SJakub Kicinski sysctl -w net.ipv4.tcp_timestamps=0 128*132db935SJakub Kicinski 129*132db935SJakub Kicinski Disabling SACK:: 130*132db935SJakub Kicinski 131*132db935SJakub Kicinski sysctl -w net.ipv4.tcp_sack=0 132*132db935SJakub Kicinski 133*132db935SJakub Kicinski Setting large number of incoming connection requests:: 134*132db935SJakub Kicinski 135*132db935SJakub Kicinski sysctl -w net.ipv4.tcp_max_syn_backlog=3000 136*132db935SJakub Kicinski 137*132db935SJakub Kicinski Setting maximum receive socket buffer size:: 138*132db935SJakub Kicinski 139*132db935SJakub Kicinski sysctl -w net.core.rmem_max=1024000 140*132db935SJakub Kicinski 141*132db935SJakub Kicinski Setting maximum send socket buffer size:: 142*132db935SJakub Kicinski 143*132db935SJakub Kicinski sysctl -w net.core.wmem_max=1024000 144*132db935SJakub Kicinski 145*132db935SJakub Kicinski Set smp_affinity (on a multiprocessor system) to a single CPU:: 146*132db935SJakub Kicinski 147*132db935SJakub Kicinski echo 1 > /proc/irq/<interrupt_number>/smp_affinity 148*132db935SJakub Kicinski 149*132db935SJakub Kicinski Setting default receive socket buffer size:: 150*132db935SJakub Kicinski 151*132db935SJakub Kicinski sysctl -w net.core.rmem_default=524287 152*132db935SJakub Kicinski 153*132db935SJakub Kicinski Setting default send socket buffer size:: 154*132db935SJakub Kicinski 155*132db935SJakub Kicinski sysctl -w net.core.wmem_default=524287 156*132db935SJakub Kicinski 157*132db935SJakub Kicinski Setting maximum option memory buffers:: 158*132db935SJakub Kicinski 159*132db935SJakub Kicinski sysctl -w net.core.optmem_max=524287 160*132db935SJakub Kicinski 161*132db935SJakub Kicinski Setting maximum backlog (# of unprocessed packets before kernel drops):: 162*132db935SJakub Kicinski 163*132db935SJakub Kicinski sysctl -w net.core.netdev_max_backlog=300000 164*132db935SJakub Kicinski 165*132db935SJakub Kicinski Setting TCP read buffers (min/default/max):: 166*132db935SJakub Kicinski 167*132db935SJakub Kicinski sysctl -w net.ipv4.tcp_rmem="10000000 10000000 10000000" 168*132db935SJakub Kicinski 169*132db935SJakub Kicinski Setting TCP write buffers (min/pressure/max):: 170*132db935SJakub Kicinski 171*132db935SJakub Kicinski sysctl -w net.ipv4.tcp_wmem="10000000 10000000 10000000" 172*132db935SJakub Kicinski 173*132db935SJakub Kicinski Setting TCP buffer space (min/pressure/max):: 174*132db935SJakub Kicinski 175*132db935SJakub Kicinski sysctl -w net.ipv4.tcp_mem="10000000 10000000 10000000" 176*132db935SJakub Kicinski 177*132db935SJakub Kicinski TCP window size for single connections: 178*132db935SJakub Kicinski 179*132db935SJakub Kicinski The receive buffer (RX_WINDOW) size must be at least as large as the 180*132db935SJakub Kicinski Bandwidth-Delay Product of the communication link between the sender and 181*132db935SJakub Kicinski receiver. Due to the variations of RTT, you may want to increase the buffer 182*132db935SJakub Kicinski size up to 2 times the Bandwidth-Delay Product. Reference page 289 of 183*132db935SJakub Kicinski "TCP/IP Illustrated, Volume 1, The Protocols" by W. Richard Stevens. 184*132db935SJakub Kicinski 185*132db935SJakub Kicinski At 10Gb speeds, use the following formula:: 186*132db935SJakub Kicinski 187*132db935SJakub Kicinski RX_WINDOW >= 1.25MBytes * RTT(in milliseconds) 188*132db935SJakub Kicinski Example for RTT with 100us: RX_WINDOW = (1,250,000 * 0.1) = 125,000 189*132db935SJakub Kicinski 190*132db935SJakub Kicinski RX_WINDOW sizes of 256KB - 512KB should be sufficient. 191*132db935SJakub Kicinski 192*132db935SJakub Kicinski Setting the min, max, and default receive buffer (RX_WINDOW) size:: 193*132db935SJakub Kicinski 194*132db935SJakub Kicinski sysctl -w net.ipv4.tcp_rmem="<min> <default> <max>" 195*132db935SJakub Kicinski 196*132db935SJakub Kicinski TCP window size for multiple connections: 197*132db935SJakub Kicinski The receive buffer (RX_WINDOW) size may be calculated the same as single 198*132db935SJakub Kicinski connections, but should be divided by the number of connections. The 199*132db935SJakub Kicinski smaller window prevents congestion and facilitates better pacing, 200*132db935SJakub Kicinski especially if/when MAC level flow control does not work well or when it is 201*132db935SJakub Kicinski not supported on the machine. Experimentation may be necessary to attain 202*132db935SJakub Kicinski the correct value. This method is provided as a starting point for the 203*132db935SJakub Kicinski correct receive buffer size. 204*132db935SJakub Kicinski 205*132db935SJakub Kicinski Setting the min, max, and default receive buffer (RX_WINDOW) size is 206*132db935SJakub Kicinski performed in the same manner as single connection. 207*132db935SJakub Kicinski 208*132db935SJakub Kicinski 209*132db935SJakub KicinskiDriver Messages 210*132db935SJakub Kicinski=============== 211*132db935SJakub Kicinski 212*132db935SJakub Kicinski The following messages are the most common messages logged by syslog. These 213*132db935SJakub Kicinski may be found in /var/log/messages. 214*132db935SJakub Kicinski 215*132db935SJakub Kicinski Driver up:: 216*132db935SJakub Kicinski 217*132db935SJakub Kicinski Chelsio Network Driver - version 2.1.1 218*132db935SJakub Kicinski 219*132db935SJakub Kicinski NIC detected:: 220*132db935SJakub Kicinski 221*132db935SJakub Kicinski eth#: Chelsio N210 1x10GBaseX NIC (rev #), PCIX 133MHz/64-bit 222*132db935SJakub Kicinski 223*132db935SJakub Kicinski Link up:: 224*132db935SJakub Kicinski 225*132db935SJakub Kicinski eth#: link is up at 10 Gbps, full duplex 226*132db935SJakub Kicinski 227*132db935SJakub Kicinski Link down:: 228*132db935SJakub Kicinski 229*132db935SJakub Kicinski eth#: link is down 230*132db935SJakub Kicinski 231*132db935SJakub Kicinski 232*132db935SJakub KicinskiKnown Issues 233*132db935SJakub Kicinski============ 234*132db935SJakub Kicinski 235*132db935SJakub Kicinski These issues have been identified during testing. The following information 236*132db935SJakub Kicinski is provided as a workaround to the problem. In some cases, this problem is 237*132db935SJakub Kicinski inherent to Linux or to a particular Linux Distribution and/or hardware 238*132db935SJakub Kicinski platform. 239*132db935SJakub Kicinski 240*132db935SJakub Kicinski 1. Large number of TCP retransmits on a multiprocessor (SMP) system. 241*132db935SJakub Kicinski 242*132db935SJakub Kicinski On a system with multiple CPUs, the interrupt (IRQ) for the network 243*132db935SJakub Kicinski controller may be bound to more than one CPU. This will cause TCP 244*132db935SJakub Kicinski retransmits if the packet data were to be split across different CPUs 245*132db935SJakub Kicinski and re-assembled in a different order than expected. 246*132db935SJakub Kicinski 247*132db935SJakub Kicinski To eliminate the TCP retransmits, set smp_affinity on the particular 248*132db935SJakub Kicinski interrupt to a single CPU. You can locate the interrupt (IRQ) used on 249*132db935SJakub Kicinski the N110/N210 by using ifconfig:: 250*132db935SJakub Kicinski 251*132db935SJakub Kicinski ifconfig <dev_name> | grep Interrupt 252*132db935SJakub Kicinski 253*132db935SJakub Kicinski Set the smp_affinity to a single CPU:: 254*132db935SJakub Kicinski 255*132db935SJakub Kicinski echo 1 > /proc/irq/<interrupt_number>/smp_affinity 256*132db935SJakub Kicinski 257*132db935SJakub Kicinski It is highly suggested that you do not run the irqbalance daemon on your 258*132db935SJakub Kicinski system, as this will change any smp_affinity setting you have applied. 259*132db935SJakub Kicinski The irqbalance daemon runs on a 10 second interval and binds interrupts 260*132db935SJakub Kicinski to the least loaded CPU determined by the daemon. To disable this daemon:: 261*132db935SJakub Kicinski 262*132db935SJakub Kicinski chkconfig --level 2345 irqbalance off 263*132db935SJakub Kicinski 264*132db935SJakub Kicinski By default, some Linux distributions enable the kernel feature, 265*132db935SJakub Kicinski irqbalance, which performs the same function as the daemon. To disable 266*132db935SJakub Kicinski this feature, add the following line to your bootloader:: 267*132db935SJakub Kicinski 268*132db935SJakub Kicinski noirqbalance 269*132db935SJakub Kicinski 270*132db935SJakub Kicinski Example using the Grub bootloader:: 271*132db935SJakub Kicinski 272*132db935SJakub Kicinski title Red Hat Enterprise Linux AS (2.4.21-27.ELsmp) 273*132db935SJakub Kicinski root (hd0,0) 274*132db935SJakub Kicinski kernel /vmlinuz-2.4.21-27.ELsmp ro root=/dev/hda3 noirqbalance 275*132db935SJakub Kicinski initrd /initrd-2.4.21-27.ELsmp.img 276*132db935SJakub Kicinski 277*132db935SJakub Kicinski 2. After running insmod, the driver is loaded and the incorrect network 278*132db935SJakub Kicinski interface is brought up without running ifup. 279*132db935SJakub Kicinski 280*132db935SJakub Kicinski When using 2.4.x kernels, including RHEL kernels, the Linux kernel 281*132db935SJakub Kicinski invokes a script named "hotplug". This script is primarily used to 282*132db935SJakub Kicinski automatically bring up USB devices when they are plugged in, however, 283*132db935SJakub Kicinski the script also attempts to automatically bring up a network interface 284*132db935SJakub Kicinski after loading the kernel module. The hotplug script does this by scanning 285*132db935SJakub Kicinski the ifcfg-eth# config files in /etc/sysconfig/network-scripts, looking 286*132db935SJakub Kicinski for HWADDR=<mac_address>. 287*132db935SJakub Kicinski 288*132db935SJakub Kicinski If the hotplug script does not find the HWADDRR within any of the 289*132db935SJakub Kicinski ifcfg-eth# files, it will bring up the device with the next available 290*132db935SJakub Kicinski interface name. If this interface is already configured for a different 291*132db935SJakub Kicinski network card, your new interface will have incorrect IP address and 292*132db935SJakub Kicinski network settings. 293*132db935SJakub Kicinski 294*132db935SJakub Kicinski To solve this issue, you can add the HWADDR=<mac_address> key to the 295*132db935SJakub Kicinski interface config file of your network controller. 296*132db935SJakub Kicinski 297*132db935SJakub Kicinski To disable this "hotplug" feature, you may add the driver (module name) 298*132db935SJakub Kicinski to the "blacklist" file located in /etc/hotplug. It has been noted that 299*132db935SJakub Kicinski this does not work for network devices because the net.agent script 300*132db935SJakub Kicinski does not use the blacklist file. Simply remove, or rename, the net.agent 301*132db935SJakub Kicinski script located in /etc/hotplug to disable this feature. 302*132db935SJakub Kicinski 303*132db935SJakub Kicinski 3. Transport Protocol (TP) hangs when running heavy multi-connection traffic 304*132db935SJakub Kicinski on an AMD Opteron system with HyperTransport PCI-X Tunnel chipset. 305*132db935SJakub Kicinski 306*132db935SJakub Kicinski If your AMD Opteron system uses the AMD-8131 HyperTransport PCI-X Tunnel 307*132db935SJakub Kicinski chipset, you may experience the "133-Mhz Mode Split Completion Data 308*132db935SJakub Kicinski Corruption" bug identified by AMD while using a 133Mhz PCI-X card on the 309*132db935SJakub Kicinski bus PCI-X bus. 310*132db935SJakub Kicinski 311*132db935SJakub Kicinski AMD states, "Under highly specific conditions, the AMD-8131 PCI-X Tunnel 312*132db935SJakub Kicinski can provide stale data via split completion cycles to a PCI-X card that 313*132db935SJakub Kicinski is operating at 133 Mhz", causing data corruption. 314*132db935SJakub Kicinski 315*132db935SJakub Kicinski AMD's provides three workarounds for this problem, however, Chelsio 316*132db935SJakub Kicinski recommends the first option for best performance with this bug: 317*132db935SJakub Kicinski 318*132db935SJakub Kicinski For 133Mhz secondary bus operation, limit the transaction length and 319*132db935SJakub Kicinski the number of outstanding transactions, via BIOS configuration 320*132db935SJakub Kicinski programming of the PCI-X card, to the following: 321*132db935SJakub Kicinski 322*132db935SJakub Kicinski Data Length (bytes): 1k 323*132db935SJakub Kicinski 324*132db935SJakub Kicinski Total allowed outstanding transactions: 2 325*132db935SJakub Kicinski 326*132db935SJakub Kicinski Please refer to AMD 8131-HT/PCI-X Errata 26310 Rev 3.08 August 2004, 327*132db935SJakub Kicinski section 56, "133-MHz Mode Split Completion Data Corruption" for more 328*132db935SJakub Kicinski details with this bug and workarounds suggested by AMD. 329*132db935SJakub Kicinski 330*132db935SJakub Kicinski It may be possible to work outside AMD's recommended PCI-X settings, try 331*132db935SJakub Kicinski increasing the Data Length to 2k bytes for increased performance. If you 332*132db935SJakub Kicinski have issues with these settings, please revert to the "safe" settings 333*132db935SJakub Kicinski and duplicate the problem before submitting a bug or asking for support. 334*132db935SJakub Kicinski 335*132db935SJakub Kicinski .. note:: 336*132db935SJakub Kicinski 337*132db935SJakub Kicinski The default setting on most systems is 8 outstanding transactions 338*132db935SJakub Kicinski and 2k bytes data length. 339*132db935SJakub Kicinski 340*132db935SJakub Kicinski 4. On multiprocessor systems, it has been noted that an application which 341*132db935SJakub Kicinski is handling 10Gb networking can switch between CPUs causing degraded 342*132db935SJakub Kicinski and/or unstable performance. 343*132db935SJakub Kicinski 344*132db935SJakub Kicinski If running on an SMP system and taking performance measurements, it 345*132db935SJakub Kicinski is suggested you either run the latest netperf-2.4.0+ or use a binding 346*132db935SJakub Kicinski tool such as Tim Hockin's procstate utilities (runon) 347*132db935SJakub Kicinski <http://www.hockin.org/~thockin/procstate/>. 348*132db935SJakub Kicinski 349*132db935SJakub Kicinski Binding netserver and netperf (or other applications) to particular 350*132db935SJakub Kicinski CPUs will have a significant difference in performance measurements. 351*132db935SJakub Kicinski You may need to experiment which CPU to bind the application to in 352*132db935SJakub Kicinski order to achieve the best performance for your system. 353*132db935SJakub Kicinski 354*132db935SJakub Kicinski If you are developing an application designed for 10Gb networking, 355*132db935SJakub Kicinski please keep in mind you may want to look at kernel functions 356*132db935SJakub Kicinski sched_setaffinity & sched_getaffinity to bind your application. 357*132db935SJakub Kicinski 358*132db935SJakub Kicinski If you are just running user-space applications such as ftp, telnet, 359*132db935SJakub Kicinski etc., you may want to try the runon tool provided by Tim Hockin's 360*132db935SJakub Kicinski procstate utility. You could also try binding the interface to a 361*132db935SJakub Kicinski particular CPU: runon 0 ifup eth0 362*132db935SJakub Kicinski 363*132db935SJakub Kicinski 364*132db935SJakub KicinskiSupport 365*132db935SJakub Kicinski======= 366*132db935SJakub Kicinski 367*132db935SJakub Kicinski If you have problems with the software or hardware, please contact our 368*132db935SJakub Kicinski customer support team via email at support@chelsio.com or check our website 369*132db935SJakub Kicinski at http://www.chelsio.com 370*132db935SJakub Kicinski 371*132db935SJakub Kicinski------------------------------------------------------------------------------- 372*132db935SJakub Kicinski 373*132db935SJakub Kicinski:: 374*132db935SJakub Kicinski 375*132db935SJakub Kicinski Chelsio Communications 376*132db935SJakub Kicinski 370 San Aleso Ave. 377*132db935SJakub Kicinski Suite 100 378*132db935SJakub Kicinski Sunnyvale, CA 94085 379*132db935SJakub Kicinski http://www.chelsio.com 380*132db935SJakub Kicinski 381*132db935SJakub KicinskiThis program is free software; you can redistribute it and/or modify 382*132db935SJakub Kicinskiit under the terms of the GNU General Public License, version 2, as 383*132db935SJakub Kicinskipublished by the Free Software Foundation. 384*132db935SJakub Kicinski 385*132db935SJakub KicinskiYou should have received a copy of the GNU General Public License along 386*132db935SJakub Kicinskiwith this program; if not, write to the Free Software Foundation, Inc., 387*132db935SJakub Kicinski59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. 388*132db935SJakub Kicinski 389*132db935SJakub KicinskiTHIS SOFTWARE IS PROVIDED ``AS IS`` AND WITHOUT ANY EXPRESS OR IMPLIED 390*132db935SJakub KicinskiWARRANTIES, INCLUDING, WITHOUT LIMITATION, THE IMPLIED WARRANTIES OF 391*132db935SJakub KicinskiMERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. 392*132db935SJakub Kicinski 393*132db935SJakub KicinskiCopyright |copy| 2003-2005 Chelsio Communications. All rights reserved. 394