1.. SPDX-License-Identifier: GPL-2.0 2 3============================================================ 4Linux kernel driver for Elastic Network Adapter (ENA) family 5============================================================ 6 7Overview 8======== 9 10ENA is a networking interface designed to make good use of modern CPU 11features and system architectures. 12 13The ENA device exposes a lightweight management interface with a 14minimal set of memory mapped registers and extendable command set 15through an Admin Queue. 16 17The driver supports a range of ENA devices, is link-speed independent 18(i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has 19a negotiated and extendable feature set. 20 21Some ENA devices support SR-IOV. This driver is used for both the 22SR-IOV Physical Function (PF) and Virtual Function (VF) devices. 23 24ENA devices enable high speed and low overhead network traffic 25processing by providing multiple Tx/Rx queue pairs (the maximum number 26is advertised by the device via the Admin Queue), a dedicated MSI-X 27interrupt vector per Tx/Rx queue pair, adaptive interrupt moderation, 28and CPU cacheline optimized data placement. 29 30The ENA driver supports industry standard TCP/IP offload features such 31as checksum offload and TCP transmit segmentation offload (TSO). 32Receive-side scaling (RSS) is supported for multi-core scaling. 33 34The ENA driver and its corresponding devices implement health 35monitoring mechanisms such as watchdog, enabling the device and driver 36to recover in a manner transparent to the application, as well as 37debug logs. 38 39Some of the ENA devices support a working mode called Low-latency 40Queue (LLQ), which saves several more microseconds. 41 42Supported PCI vendor ID/device IDs 43================================== 44 45========= ======================= 461d0f:0ec2 ENA PF 471d0f:1ec2 ENA PF with LLQ support 481d0f:ec20 ENA VF 491d0f:ec21 ENA VF with LLQ support 50========= ======================= 51 52ENA Source Code Directory Structure 53=================================== 54 55================= ====================================================== 56ena_com.[ch] Management communication layer. This layer is 57 responsible for the handling all the management 58 (admin) communication between the device and the 59 driver. 60ena_eth_com.[ch] Tx/Rx data path. 61ena_admin_defs.h Definition of ENA management interface. 62ena_eth_io_defs.h Definition of ENA data path interface. 63ena_common_defs.h Common definitions for ena_com layer. 64ena_regs_defs.h Definition of ENA PCI memory-mapped (MMIO) registers. 65ena_netdev.[ch] Main Linux kernel driver. 66ena_syfsfs.[ch] Sysfs files. 67ena_ethtool.c ethtool callbacks. 68ena_pci_id_tbl.h Supported device IDs. 69================= ====================================================== 70 71Management Interface: 72===================== 73 74ENA management interface is exposed by means of: 75 76- PCIe Configuration Space 77- Device Registers 78- Admin Queue (AQ) and Admin Completion Queue (ACQ) 79- Asynchronous Event Notification Queue (AENQ) 80 81ENA device MMIO Registers are accessed only during driver 82initialization and are not involved in further normal device 83operation. 84 85AQ is used for submitting management commands, and the 86results/responses are reported asynchronously through ACQ. 87 88ENA introduces a small set of management commands with room for 89vendor-specific extensions. Most of the management operations are 90framed in a generic Get/Set feature command. 91 92The following admin queue commands are supported: 93 94- Create I/O submission queue 95- Create I/O completion queue 96- Destroy I/O submission queue 97- Destroy I/O completion queue 98- Get feature 99- Set feature 100- Configure AENQ 101- Get statistics 102 103Refer to ena_admin_defs.h for the list of supported Get/Set Feature 104properties. 105 106The Asynchronous Event Notification Queue (AENQ) is a uni-directional 107queue used by the ENA device to send to the driver events that cannot 108be reported using ACQ. AENQ events are subdivided into groups. Each 109group may have multiple syndromes, as shown below 110 111The events are: 112 113 ==================== =============== 114 Group Syndrome 115 ==================== =============== 116 Link state change **X** 117 Fatal error **X** 118 Notification Suspend traffic 119 Notification Resume traffic 120 Keep-Alive **X** 121 ==================== =============== 122 123ACQ and AENQ share the same MSI-X vector. 124 125Keep-Alive is a special mechanism that allows monitoring of the 126device's health. The driver maintains a watchdog (WD) handler which, 127if fired, logs the current state and statistics then resets and 128restarts the ENA device and driver. A Keep-Alive event is delivered by 129the device every second. The driver re-arms the WD upon reception of a 130Keep-Alive event. A missed Keep-Alive event causes the WD handler to 131fire. 132 133Data Path Interface 134=================== 135I/O operations are based on Tx and Rx Submission Queues (Tx SQ and Rx 136SQ correspondingly). Each SQ has a completion queue (CQ) associated 137with it. 138 139The SQs and CQs are implemented as descriptor rings in contiguous 140physical memory. 141 142The ENA driver supports two Queue Operation modes for Tx SQs: 143 144- Regular mode 145 146 * In this mode the Tx SQs reside in the host's memory. The ENA 147 device fetches the ENA Tx descriptors and packet data from host 148 memory. 149 150- Low Latency Queue (LLQ) mode or "push-mode". 151 152 * In this mode the driver pushes the transmit descriptors and the 153 first 128 bytes of the packet directly to the ENA device memory 154 space. The rest of the packet payload is fetched by the 155 device. For this operation mode, the driver uses a dedicated PCI 156 device memory BAR, which is mapped with write-combine capability. 157 158The Rx SQs support only the regular mode. 159 160Note: Not all ENA devices support LLQ, and this feature is negotiated 161 with the device upon initialization. If the ENA device does not 162 support LLQ mode, the driver falls back to the regular mode. 163 164The driver supports multi-queue for both Tx and Rx. This has various 165benefits: 166 167- Reduced CPU/thread/process contention on a given Ethernet interface. 168- Cache miss rate on completion is reduced, particularly for data 169 cache lines that hold the sk_buff structures. 170- Increased process-level parallelism when handling received packets. 171- Increased data cache hit rate, by steering kernel processing of 172 packets to the CPU, where the application thread consuming the 173 packet is running. 174- In hardware interrupt re-direction. 175 176Interrupt Modes 177=============== 178The driver assigns a single MSI-X vector per queue pair (for both Tx 179and Rx directions). The driver assigns an additional dedicated MSI-X vector 180for management (for ACQ and AENQ). 181 182Management interrupt registration is performed when the Linux kernel 183probes the adapter, and it is de-registered when the adapter is 184removed. I/O queue interrupt registration is performed when the Linux 185interface of the adapter is opened, and it is de-registered when the 186interface is closed. 187 188The management interrupt is named:: 189 190 ena-mgmnt@pci:<PCI domain:bus:slot.function> 191 192and for each queue pair, an interrupt is named:: 193 194 <interface name>-Tx-Rx-<queue index> 195 196The ENA device operates in auto-mask and auto-clear interrupt 197modes. That is, once MSI-X is delivered to the host, its Cause bit is 198automatically cleared and the interrupt is masked. The interrupt is 199unmasked by the driver after NAPI processing is complete. 200 201Interrupt Moderation 202==================== 203ENA driver and device can operate in conventional or adaptive interrupt 204moderation mode. 205 206In conventional mode the driver instructs device to postpone interrupt 207posting according to static interrupt delay value. The interrupt delay 208value can be configured through ethtool(8). The following ethtool 209parameters are supported by the driver: tx-usecs, rx-usecs 210 211In adaptive interrupt moderation mode the interrupt delay value is 212updated by the driver dynamically and adjusted every NAPI cycle 213according to the traffic nature. 214 215By default ENA driver applies adaptive coalescing on Rx traffic and 216conventional coalescing on Tx traffic. 217 218Adaptive coalescing can be switched on/off through ethtool(8) 219adaptive_rx on|off parameter. 220 221The driver chooses interrupt delay value according to the number of 222bytes and packets received between interrupt unmasking and interrupt 223posting. The driver uses interrupt delay table that subdivides the 224range of received bytes/packets into 5 levels and assigns interrupt 225delay value to each level. 226 227The user can enable/disable adaptive moderation, modify the interrupt 228delay table and restore its default values through sysfs. 229 230RX copybreak 231============ 232The rx_copybreak is initialized by default to ENA_DEFAULT_RX_COPYBREAK 233and can be configured by the ETHTOOL_STUNABLE command of the 234SIOCETHTOOL ioctl. 235 236SKB 237=== 238The driver-allocated SKB for frames received from Rx handling using 239NAPI context. The allocation method depends on the size of the packet. 240If the frame length is larger than rx_copybreak, napi_get_frags() 241is used, otherwise netdev_alloc_skb_ip_align() is used, the buffer 242content is copied (by CPU) to the SKB, and the buffer is recycled. 243 244Statistics 245========== 246The user can obtain ENA device and driver statistics using ethtool. 247The driver can collect regular or extended statistics (including 248per-queue stats) from the device. 249 250In addition the driver logs the stats to syslog upon device reset. 251 252MTU 253=== 254The driver supports an arbitrarily large MTU with a maximum that is 255negotiated with the device. The driver configures MTU using the 256SetFeature command (ENA_ADMIN_MTU property). The user can change MTU 257via ip(8) and similar legacy tools. 258 259Stateless Offloads 260================== 261The ENA driver supports: 262 263- TSO over IPv4/IPv6 264- TSO with ECN 265- IPv4 header checksum offload 266- TCP/UDP over IPv4/IPv6 checksum offloads 267 268RSS 269=== 270- The ENA device supports RSS that allows flexible Rx traffic 271 steering. 272- Toeplitz and CRC32 hash functions are supported. 273- Different combinations of L2/L3/L4 fields can be configured as 274 inputs for hash functions. 275- The driver configures RSS settings using the AQ SetFeature command 276 (ENA_ADMIN_RSS_HASH_FUNCTION, ENA_ADMIN_RSS_HASH_INPUT and 277 ENA_ADMIN_RSS_REDIRECTION_TABLE_CONFIG properties). 278- If the NETIF_F_RXHASH flag is set, the 32-bit result of the hash 279 function delivered in the Rx CQ descriptor is set in the received 280 SKB. 281- The user can provide a hash key, hash function, and configure the 282 indirection table through ethtool(8). 283 284DATA PATH 285========= 286Tx 287-- 288 289end_start_xmit() is called by the stack. This function does the following: 290 291- Maps data buffers (skb->data and frags). 292- Populates ena_buf for the push buffer (if the driver and device are 293 in push mode.) 294- Prepares ENA bufs for the remaining frags. 295- Allocates a new request ID from the empty req_id ring. The request 296 ID is the index of the packet in the Tx info. This is used for 297 out-of-order TX completions. 298- Adds the packet to the proper place in the Tx ring. 299- Calls ena_com_prepare_tx(), an ENA communication layer that converts 300 the ena_bufs to ENA descriptors (and adds meta ENA descriptors as 301 needed.) 302 303 * This function also copies the ENA descriptors and the push buffer 304 to the Device memory space (if in push mode.) 305 306- Writes doorbell to the ENA device. 307- When the ENA device finishes sending the packet, a completion 308 interrupt is raised. 309- The interrupt handler schedules NAPI. 310- The ena_clean_tx_irq() function is called. This function handles the 311 completion descriptors generated by the ENA, with a single 312 completion descriptor per completed packet. 313 314 * req_id is retrieved from the completion descriptor. The tx_info of 315 the packet is retrieved via the req_id. The data buffers are 316 unmapped and req_id is returned to the empty req_id ring. 317 * The function stops when the completion descriptors are completed or 318 the budget is reached. 319 320Rx 321-- 322 323- When a packet is received from the ENA device. 324- The interrupt handler schedules NAPI. 325- The ena_clean_rx_irq() function is called. This function calls 326 ena_rx_pkt(), an ENA communication layer function, which returns the 327 number of descriptors used for a new unhandled packet, and zero if 328 no new packet is found. 329- Then it calls the ena_clean_rx_irq() function. 330- ena_eth_rx_skb() checks packet length: 331 332 * If the packet is small (len < rx_copybreak), the driver allocates 333 a SKB for the new packet, and copies the packet payload into the 334 SKB data buffer. 335 336 - In this way the original data buffer is not passed to the stack 337 and is reused for future Rx packets. 338 339 * Otherwise the function unmaps the Rx buffer, then allocates the 340 new SKB structure and hooks the Rx buffer to the SKB frags. 341 342- The new SKB is updated with the necessary information (protocol, 343 checksum hw verify result, etc.), and then passed to the network 344 stack, using the NAPI interface function napi_gro_receive(). 345