1.. SPDX-License-Identifier: GPL-2.0 2 3============================================================ 4Linux kernel driver for Elastic Network Adapter (ENA) family 5============================================================ 6 7Overview 8======== 9 10ENA is a networking interface designed to make good use of modern CPU 11features and system architectures. 12 13The ENA device exposes a lightweight management interface with a 14minimal set of memory mapped registers and extendible command set 15through an Admin Queue. 16 17The driver supports a range of ENA devices, is link-speed independent 18(i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc), and has 19a negotiated and extendible feature set. 20 21Some ENA devices support SR-IOV. This driver is used for both the 22SR-IOV Physical Function (PF) and Virtual Function (VF) devices. 23 24ENA devices enable high speed and low overhead network traffic 25processing by providing multiple Tx/Rx queue pairs (the maximum number 26is advertised by the device via the Admin Queue), a dedicated MSI-X 27interrupt vector per Tx/Rx queue pair, adaptive interrupt moderation, 28and CPU cacheline optimized data placement. 29 30The ENA driver supports industry standard TCP/IP offload features such as 31checksum offload. Receive-side scaling (RSS) is supported for multi-core 32scaling. 33 34The ENA driver and its corresponding devices implement health 35monitoring mechanisms such as watchdog, enabling the device and driver 36to recover in a manner transparent to the application, as well as 37debug logs. 38 39Some of the ENA devices support a working mode called Low-latency 40Queue (LLQ), which saves several more microseconds. 41ENA Source Code Directory Structure 42=================================== 43 44================= ====================================================== 45ena_com.[ch] Management communication layer. This layer is 46 responsible for the handling all the management 47 (admin) communication between the device and the 48 driver. 49ena_eth_com.[ch] Tx/Rx data path. 50ena_admin_defs.h Definition of ENA management interface. 51ena_eth_io_defs.h Definition of ENA data path interface. 52ena_common_defs.h Common definitions for ena_com layer. 53ena_regs_defs.h Definition of ENA PCI memory-mapped (MMIO) registers. 54ena_netdev.[ch] Main Linux kernel driver. 55ena_ethtool.c ethtool callbacks. 56ena_pci_id_tbl.h Supported device IDs. 57================= ====================================================== 58 59Management Interface: 60===================== 61 62ENA management interface is exposed by means of: 63 64- PCIe Configuration Space 65- Device Registers 66- Admin Queue (AQ) and Admin Completion Queue (ACQ) 67- Asynchronous Event Notification Queue (AENQ) 68 69ENA device MMIO Registers are accessed only during driver 70initialization and are not used during further normal device 71operation. 72 73AQ is used for submitting management commands, and the 74results/responses are reported asynchronously through ACQ. 75 76ENA introduces a small set of management commands with room for 77vendor-specific extensions. Most of the management operations are 78framed in a generic Get/Set feature command. 79 80The following admin queue commands are supported: 81 82- Create I/O submission queue 83- Create I/O completion queue 84- Destroy I/O submission queue 85- Destroy I/O completion queue 86- Get feature 87- Set feature 88- Configure AENQ 89- Get statistics 90 91Refer to ena_admin_defs.h for the list of supported Get/Set Feature 92properties. 93 94The Asynchronous Event Notification Queue (AENQ) is a uni-directional 95queue used by the ENA device to send to the driver events that cannot 96be reported using ACQ. AENQ events are subdivided into groups. Each 97group may have multiple syndromes, as shown below 98 99The events are: 100 101==================== =============== 102Group Syndrome 103==================== =============== 104Link state change **X** 105Fatal error **X** 106Notification Suspend traffic 107Notification Resume traffic 108Keep-Alive **X** 109==================== =============== 110 111ACQ and AENQ share the same MSI-X vector. 112 113Keep-Alive is a special mechanism that allows monitoring the device's health. 114A Keep-Alive event is delivered by the device every second. 115The driver maintains a watchdog (WD) handler which logs the current state and 116statistics. If the keep-alive events aren't delivered as expected the WD resets 117the device and the driver. 118 119Data Path Interface 120=================== 121 122I/O operations are based on Tx and Rx Submission Queues (Tx SQ and Rx 123SQ correspondingly). Each SQ has a completion queue (CQ) associated 124with it. 125 126The SQs and CQs are implemented as descriptor rings in contiguous 127physical memory. 128 129The ENA driver supports two Queue Operation modes for Tx SQs: 130 131- **Regular mode:** 132 In this mode the Tx SQs reside in the host's memory. The ENA 133 device fetches the ENA Tx descriptors and packet data from host 134 memory. 135 136- **Low Latency Queue (LLQ) mode or "push-mode":** 137 In this mode the driver pushes the transmit descriptors and the 138 first 96 bytes of the packet directly to the ENA device memory 139 space. The rest of the packet payload is fetched by the 140 device. For this operation mode, the driver uses a dedicated PCI 141 device memory BAR, which is mapped with write-combine capability. 142 143 **Note that** not all ENA devices support LLQ, and this feature is negotiated 144 with the device upon initialization. If the ENA device does not 145 support LLQ mode, the driver falls back to the regular mode. 146 147The Rx SQs support only the regular mode. 148 149The driver supports multi-queue for both Tx and Rx. This has various 150benefits: 151 152- Reduced CPU/thread/process contention on a given Ethernet interface. 153- Cache miss rate on completion is reduced, particularly for data 154 cache lines that hold the sk_buff structures. 155- Increased process-level parallelism when handling received packets. 156- Increased data cache hit rate, by steering kernel processing of 157 packets to the CPU, where the application thread consuming the 158 packet is running. 159- In hardware interrupt re-direction. 160 161Interrupt Modes 162=============== 163 164The driver assigns a single MSI-X vector per queue pair (for both Tx 165and Rx directions). The driver assigns an additional dedicated MSI-X vector 166for management (for ACQ and AENQ). 167 168Management interrupt registration is performed when the Linux kernel 169probes the adapter, and it is de-registered when the adapter is 170removed. I/O queue interrupt registration is performed when the Linux 171interface of the adapter is opened, and it is de-registered when the 172interface is closed. 173 174The management interrupt is named:: 175 176 ena-mgmnt@pci:<PCI domain:bus:slot.function> 177 178and for each queue pair, an interrupt is named:: 179 180 <interface name>-Tx-Rx-<queue index> 181 182The ENA device operates in auto-mask and auto-clear interrupt 183modes. That is, once MSI-X is delivered to the host, its Cause bit is 184automatically cleared and the interrupt is masked. The interrupt is 185unmasked by the driver after NAPI processing is complete. 186 187Interrupt Moderation 188==================== 189 190ENA driver and device can operate in conventional or adaptive interrupt 191moderation mode. 192 193**In conventional mode** the driver instructs device to postpone interrupt 194posting according to static interrupt delay value. The interrupt delay 195value can be configured through `ethtool(8)`. The following `ethtool` 196parameters are supported by the driver: ``tx-usecs``, ``rx-usecs`` 197 198**In adaptive interrupt** moderation mode the interrupt delay value is 199updated by the driver dynamically and adjusted every NAPI cycle 200according to the traffic nature. 201 202Adaptive coalescing can be switched on/off through `ethtool(8)`'s 203:code:`adaptive_rx on|off` parameter. 204 205More information about Adaptive Interrupt Moderation (DIM) can be found in 206Documentation/networking/net_dim.rst 207 208RX copybreak 209============ 210The rx_copybreak is initialized by default to ENA_DEFAULT_RX_COPYBREAK 211and can be configured by the ETHTOOL_STUNABLE command of the 212SIOCETHTOOL ioctl. 213 214Statistics 215========== 216 217The user can obtain ENA device and driver statistics using `ethtool`. 218The driver can collect regular or extended statistics (including 219per-queue stats) from the device. 220 221In addition the driver logs the stats to syslog upon device reset. 222 223MTU 224=== 225 226The driver supports an arbitrarily large MTU with a maximum that is 227negotiated with the device. The driver configures MTU using the 228SetFeature command (ENA_ADMIN_MTU property). The user can change MTU 229via `ip(8)` and similar legacy tools. 230 231Stateless Offloads 232================== 233 234The ENA driver supports: 235 236- IPv4 header checksum offload 237- TCP/UDP over IPv4/IPv6 checksum offloads 238 239RSS 240=== 241 242- The ENA device supports RSS that allows flexible Rx traffic 243 steering. 244- Toeplitz and CRC32 hash functions are supported. 245- Different combinations of L2/L3/L4 fields can be configured as 246 inputs for hash functions. 247- The driver configures RSS settings using the AQ SetFeature command 248 (ENA_ADMIN_RSS_HASH_FUNCTION, ENA_ADMIN_RSS_HASH_INPUT and 249 ENA_ADMIN_RSS_INDIRECTION_TABLE_CONFIG properties). 250- If the NETIF_F_RXHASH flag is set, the 32-bit result of the hash 251 function delivered in the Rx CQ descriptor is set in the received 252 SKB. 253- The user can provide a hash key, hash function, and configure the 254 indirection table through `ethtool(8)`. 255 256DATA PATH 257========= 258 259Tx 260-- 261 262:code:`ena_start_xmit()` is called by the stack. This function does the following: 263 264- Maps data buffers (``skb->data`` and frags). 265- Populates ``ena_buf`` for the push buffer (if the driver and device are 266 in push mode). 267- Prepares ENA bufs for the remaining frags. 268- Allocates a new request ID from the empty ``req_id`` ring. The request 269 ID is the index of the packet in the Tx info. This is used for 270 out-of-order Tx completions. 271- Adds the packet to the proper place in the Tx ring. 272- Calls :code:`ena_com_prepare_tx()`, an ENA communication layer that converts 273 the ``ena_bufs`` to ENA descriptors (and adds meta ENA descriptors as 274 needed). 275 276 * This function also copies the ENA descriptors and the push buffer 277 to the Device memory space (if in push mode). 278 279- Writes a doorbell to the ENA device. 280- When the ENA device finishes sending the packet, a completion 281 interrupt is raised. 282- The interrupt handler schedules NAPI. 283- The :code:`ena_clean_tx_irq()` function is called. This function handles the 284 completion descriptors generated by the ENA, with a single 285 completion descriptor per completed packet. 286 287 * ``req_id`` is retrieved from the completion descriptor. The ``tx_info`` of 288 the packet is retrieved via the ``req_id``. The data buffers are 289 unmapped and ``req_id`` is returned to the empty ``req_id`` ring. 290 * The function stops when the completion descriptors are completed or 291 the budget is reached. 292 293Rx 294-- 295 296- When a packet is received from the ENA device. 297- The interrupt handler schedules NAPI. 298- The :code:`ena_clean_rx_irq()` function is called. This function calls 299 :code:`ena_com_rx_pkt()`, an ENA communication layer function, which returns the 300 number of descriptors used for a new packet, and zero if 301 no new packet is found. 302- :code:`ena_rx_skb()` checks packet length: 303 304 * If the packet is small (len < rx_copybreak), the driver allocates 305 a SKB for the new packet, and copies the packet payload into the 306 SKB data buffer. 307 308 - In this way the original data buffer is not passed to the stack 309 and is reused for future Rx packets. 310 311 * Otherwise the function unmaps the Rx buffer, sets the first 312 descriptor as `skb`'s linear part and the other descriptors as the 313 `skb`'s frags. 314 315- The new SKB is updated with the necessary information (protocol, 316 checksum hw verify result, etc), and then passed to the network 317 stack, using the NAPI interface function :code:`napi_gro_receive()`. 318