1.. SPDX-License-Identifier: GPL-2.0+ 2 3============================================================== 4Linux kernel driver for Compute Engine Virtual Ethernet (gve): 5============================================================== 6 7Supported Hardware 8=================== 9The GVE driver binds to a single PCI device id used by the virtual 10Ethernet device found in some Compute Engine VMs. 11 12+--------------+----------+---------+ 13|Field | Value | Comments| 14+==============+==========+=========+ 15|Vendor ID | `0x1AE0` | Google | 16+--------------+----------+---------+ 17|Device ID | `0x0042` | | 18+--------------+----------+---------+ 19|Sub-vendor ID | `0x1AE0` | Google | 20+--------------+----------+---------+ 21|Sub-device ID | `0x0058` | | 22+--------------+----------+---------+ 23|Revision ID | `0x0` | | 24+--------------+----------+---------+ 25|Device Class | `0x200` | Ethernet| 26+--------------+----------+---------+ 27 28PCI Bars 29======== 30The gVNIC PCI device exposes three 32-bit memory BARS: 31- Bar0 - Device configuration and status registers. 32- Bar1 - MSI-X vector table 33- Bar2 - IRQ, RX and TX doorbells 34 35Device Interactions 36=================== 37The driver interacts with the device in the following ways: 38 - Registers 39 - A block of MMIO registers 40 - See gve_register.h for more detail 41 - Admin Queue 42 - See description below 43 - Reset 44 - At any time the device can be reset 45 - Interrupts 46 - See supported interrupts below 47 - Transmit and Receive Queues 48 - See description below 49 50Descriptor Formats 51------------------ 52GVE supports two descriptor formats: GQI and DQO. These two formats have 53entirely different descriptors, which will be described below. 54 55Registers 56--------- 57All registers are MMIO. 58 59The registers are used for initializing and configuring the device as well as 60querying device status in response to management interrupts. 61 62Endianness 63---------- 64- Admin Queue messages and registers are all Big Endian. 65- GQI descriptors and datapath registers are Big Endian. 66- DQO descriptors and datapath registers are Little Endian. 67 68Admin Queue (AQ) 69---------------- 70The Admin Queue is a PAGE_SIZE memory block, treated as an array of AQ 71commands, used by the driver to issue commands to the device and set up 72resources.The driver and the device maintain a count of how many commands 73have been submitted and executed. To issue AQ commands, the driver must do 74the following (with proper locking): 75 761) Copy new commands into next available slots in the AQ array 772) Increment its counter by he number of new commands 783) Write the counter into the GVE_ADMIN_QUEUE_DOORBELL register 794) Poll the ADMIN_QUEUE_EVENT_COUNTER register until it equals 80 the value written to the doorbell, or until a timeout. 81 82The device will update the status field in each AQ command reported as 83executed through the ADMIN_QUEUE_EVENT_COUNTER register. 84 85Device Resets 86------------- 87A device reset is triggered by writing 0x0 to the AQ PFN register. 88This causes the device to release all resources allocated by the 89driver, including the AQ itself. 90 91Interrupts 92---------- 93The following interrupts are supported by the driver: 94 95Management Interrupt 96~~~~~~~~~~~~~~~~~~~~ 97The management interrupt is used by the device to tell the driver to 98look at the GVE_DEVICE_STATUS register. 99 100The handler for the management irq simply queues the service task in 101the workqueue to check the register and acks the irq. 102 103Notification Block Interrupts 104~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 105The notification block interrupts are used to tell the driver to poll 106the queues associated with that interrupt. 107 108The handler for these irqs schedule the napi for that block to run 109and poll the queues. 110 111GQI Traffic Queues 112------------------ 113GQI queues are composed of a descriptor ring and a buffer and are assigned to a 114notification block. 115 116The descriptor rings are power-of-two-sized ring buffers consisting of 117fixed-size descriptors. They advance their head pointer using a __be32 118doorbell located in Bar2. The tail pointers are advanced by consuming 119descriptors in-order and updating a __be32 counter. Both the doorbell 120and the counter overflow to zero. 121 122Each queue's buffers must be registered in advance with the device as a 123queue page list, and packet data can only be put in those pages. 124 125Transmit 126~~~~~~~~ 127gve maps the buffers for transmit rings into a FIFO and copies the packets 128into the FIFO before sending them to the NIC. 129 130Receive 131~~~~~~~ 132The buffers for receive rings are put into a data ring that is the same 133length as the descriptor ring and the head and tail pointers advance over 134the rings together. 135 136DQO Traffic Queues 137------------------ 138- Every TX and RX queue is assigned a notification block. 139 140- TX and RX buffers queues, which send descriptors to the device, use MMIO 141 doorbells to notify the device of new descriptors. 142 143- RX and TX completion queues, which receive descriptors from the device, use a 144 "generation bit" to know when a descriptor was populated by the device. The 145 driver initializes all bits with the "current generation". The device will 146 populate received descriptors with the "next generation" which is inverted 147 from the current generation. When the ring wraps, the current/next generation 148 are swapped. 149 150- It's the driver's responsibility to ensure that the RX and TX completion 151 queues are not overrun. This can be accomplished by limiting the number of 152 descriptors posted to HW. 153 154- TX packets have a 16 bit completion_tag and RX buffers have a 16 bit 155 buffer_id. These will be returned on the TX completion and RX queues 156 respectively to let the driver know which packet/buffer was completed. 157 158Transmit 159~~~~~~~~ 160A packet's buffers are DMA mapped for the device to access before transmission. 161After the packet was successfully transmitted, the buffers are unmapped. 162 163Receive 164~~~~~~~ 165The driver posts fixed sized buffers to HW on the RX buffer queue. The packet 166received on the associated RX queue may span multiple descriptors. 167