1.\" SPDX-License-Identifier: BSD-3-Clause 2.\" 3.\" Copyright (c) 2023-2024 Google LLC 4.\" 5.\" Redistribution and use in source and binary forms, with or without modification, 6.\" are permitted provided that the following conditions are met: 7.\" 8.\" 1. Redistributions of source code must retain the above copyright notice, this 9.\" list of conditions and the following disclaimer. 10.\" 11.\" 2. Redistributions in binary form must reproduce the above copyright notice, 12.\" this list of conditions and the following disclaimer in the documentation 13.\" and/or other materials provided with the distribution. 14.\" 15.\" 3. Neither the name of the copyright holder nor the names of its contributors 16.\" may be used to endorse or promote products derived from this software without 17.\" specific prior written permission. 18.\" 19.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND 20.\" ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED 21.\" WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE 22.\" DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR 23.\" ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES 24.\" (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; 25.\" LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON 26.\" ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT 27.\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS 28.\" SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 29.Dd October 14, 2024 30.Dt GVE 4 31.Os 32.Sh NAME 33.Nm gve 34.Nd "Ethernet driver for Google Virtual NIC (gVNIC)" 35.Sh SYNOPSIS 36To compile this driver into the kernel, 37place the following lines in your 38kernel configuration file: 39.Bd -ragged -offset indent 40.Cd "device gve" 41.Ed 42.Pp 43Alternatively, to load the driver as a 44module at boot time, place the following line in 45.Xr loader.conf 5 : 46.Bd -literal -offset indent 47if_gve_load="YES" 48.Ed 49.Sh DESCRIPTION 50gVNIC is a virtual network interface designed specifically for Google Compute Engine (GCE). 51It is required to support per-VM Tier-1 networking performance, and for using certain VM shapes on GCE. 52.Pp 53.Nm 54is the driver for gVNIC. 55It supports the following features: 56.Pp 57.Bl -bullet -compact 58.It 59RX checksum offload 60.It 61TX chesksum offload 62.It 63TCP Segmentation Offload (TSO) 64.It 65Large Receive Offload (LRO) in software 66.It 67Jumbo frames 68.It 69Receive Side Scaling (RSS) 70.El 71.Pp 72For more information on configuring this device, see 73.Xr ifconfig 8 . 74.Sh HARDWARE 75.Nm 76binds to a single PCI device ID presented by gVNIC: 77.Pp 78.Bl -bullet -compact 79.It 800x1AE0:0x0042 81.El 82.Sh EXAMPLES 83.Pp 84Change the TX queue count to 4 for the gve0 interface: 85.D1 sysctl dev.gve.0.num_tx_queues=4 86.Pp 87Change the RX queue count to 4 for the gve0 interface: 88.D1 sysctl dev.gve.0.num_rx_queues=4 89.Pp 90Change the TX ring size to 512 for the gve0 interface: 91.D1 sysctl dev.gve.0.tx_ring_size=512 92.Pp 93Change the RX ring size to 512 for the gve0 interface: 94.D1 sysctl dev.gve.0.rx_ring_size=512 95.Sh DIAGNOSTICS 96The following messages are recorded during driver initialization: 97.Bl -diag 98.It "Enabled MSIX with %d vectors" 99.It "Configured device resources" 100.It "Successfully attached %s" 101.It "Deconfigured device resources" 102.El 103.Pp 104These messages are seen if driver initialization fails. 105Global (across-queues) allocation failures: 106.Bl -diag 107.It "Failed to configure device resources: err=%d" 108.It "No compatible queue formats" 109.It "Failed to allocate ifnet struct" 110.It "Failed to allocate admin queue mem" 111.It "Failed to alloc DMA mem for DescribeDevice" 112.It "Failed to allocate QPL page" 113.El 114.Pp 115irq and BAR allocation failures: 116.Bl -diag 117.It "Failed to acquire any msix vectors" 118.It "Tried to acquire %d msix vectors, got only %d" 119.It "Failed to setup irq %d for Tx queue %d " 120.It "Failed to setup irq %d for Rx queue %d " 121.It "Failed to allocate irq %d for mgmnt queue" 122.It "Failed to setup irq %d for mgmnt queue, err: %d" 123.It "Failed to allocate BAR0" 124.It "Failed to allocate BAR2" 125.It "Failed to allocate msix table" 126.El 127.Pp 128Rx queue-specific allocation failures: 129.Bl -diag 130.It "No QPL left for rx ring %d" 131.It "Failed to alloc queue resources for rx ring %d" 132.It "Failed to alloc desc ring for rx ring %d" 133.It "Failed to alloc data ring for rx ring %d" 134.El 135.Pp 136Tx queue-specific allocation failures: 137.Bl -diag 138.It "No QPL left for tx ring %d" 139.It "Failed to alloc queue resources for tx ring %d" 140.It "Failed to alloc desc ring for tx ring %d" 141.It "Failed to vmap fifo, qpl_id = %d" 142.El 143.Pp 144The following messages are recorded when the interface detach fails: 145.Bl -diag 146.It "Failed to deconfigure device resources: err=%d" 147.El 148.Pp 149If bootverbose is on, the following messages are recorded when the interface is being brought up: 150.Bl -diag 151.It "Created %d rx queues" 152.It "Created %d tx queues" 153.It "MTU set to %d" 154.El 155.Pp 156The following messages are recorded when the interface is being brought down: 157.Bl -diag 158.It "Destroyed %d rx queues" 159.It "Destroyed %d tx queues" 160.El 161.Pp 162These messages are seen if errors are encountered when bringing the interface up or down: 163.Bl -diag 164.It "Failed to destroy rxq %d, err: %d" 165.It "Failed to destroy txq %d, err: %d" 166.It "Failed to create rxq %d, err: %d" 167.It "Failed to create txq %d, err: %d" 168.It "Failed to set MTU to %d" 169.It "Invalid new MTU setting. new mtu: %d max mtu: %d min mtu: %d" 170.It "Cannot bring the iface up when detached" 171.It "Reached max number of registered pages %lu > %lu" 172.It "Failed to init lro for rx ring %d" 173.El 174.Pp 175These messages are seen if any admin queue command fails: 176.Bl -diag 177.It "AQ command(%u): failed with status %d" 178.It "AQ command(%u): unknown status code %d" 179.It "AQ commands timed out, need to reset AQ" 180.It "Unknown AQ command opcode %d" 181.El 182.Pp 183These messages are recorded when the device is being reset due to an error: 184.Bl -diag 185.It "Scheduling reset task!" 186.It "Waiting until admin queue is released." 187.It "Admin queue released" 188.El 189.Pp 190If it was the NIC that requested the reset, this message is recorded: 191.Bl -diag 192.It "Device requested reset" 193.El 194.Pp 195If the reset fails during the reinitialization phase, this message is recorded: 196.Bl -diag 197.It "Restore failed!" 198.El 199.Pp 200These two messages correspond to the NIC alerting the driver to link state changes: 201.Bl -diag 202.It "Device link is up." 203.It "Device link is down." 204.El 205.Pp 206Apart from these messages, the driver exposes per-queue packet and error counters as sysctl nodes. 207Global (across queues) counters can be read using 208.Xr netstat 1 . 209.Sh SYSCTL VARIABLES 210.Nm 211exposes the following 212.Xr sysctl 8 213variables: 214.Bl -tag -width indent 215.It Va hw.gve.driver_version 216The driver version. 217This is read-only. 218.It Va hw.gve.queue_format 219The queue format in use. 220This is read-only. 221.It Va hw.gve.disable_hw_lro 222Setting this boot-time tunable to 1 disables Large Receive Offload (LRO) in the NIC. 223The default value is 0, which means hardware LRO is enabled by default. 224The software LRO stack in the kernel is always used. 225This sysctl variable needs to be set before loading the driver, using 226.Xr loader.conf 5 . 227.It Va dev.gve.X.num_rx_queues and dev.gve.X.num_tx_queues 228Run-time tunables that represent the number of currently used RX/TX queues. 229The default value is the max number of RX/TX queues the device can support. 230.Pp 231This call turns down the interface while setting up the new queues, 232which may potentially cause any new packets to be dropped. 233This call can fail if the system is not able to provide the driver with enough resources. 234In that situation, the driver will revert to the previous number of RX/TX queues. 235If this also fails, a device reset will be triggered. 236.Pp 237Note: sysctl nodes for queue stats remain available even if a queue is removed. 238.Pp 239.It Va dev.gve.X.rx_ring_size and dev.gve.X.tx_ring_size 240Run-time tunables that represent the current ring size for RX/TX queues. 241The default value is set to device defaults for ring size. 242.Pp 243This call turns down the interface while setting up the queues with the new ring size, 244which may potentially cause any new packets to be dropped. 245This call can fail if the system is not able to provide the driver with enough resources. 246In that situation, the driver will try to revert to the previous ring size for RX/TX queues. 247If this also fails, the device will be in an unhealthy state and will need to be reloaded. 248This value must be a power of 2 and within the defined range. 249.Pp 250.El 251.Sh LIMITATIONS 252.Nm 253does not support the transmission of VLAN-tagged packets. 254All VLAN-tagged traffic is dropped. 255.Sh QUEUE FORMATS 256.Nm 257features different datapath modes called queue formats: 258.Pp 259.Bl -bullet -compact 260.It 261GQI_QPL: "QPL" stands for "Queue Page List" and refers to the fact that 262hardware expects a fixed bounce buffer and cannot access arbitrary memory. 263GQI is the older descriptor format. 264The G in "GQI" refers to an older generation of hardware, and the "QI" 265stands for "Queue In-order" referring to the fact that the NIC sends 266Tx and Rx completions in the same order as the one in which the corresponding 267descriptors were posted by the driver. 268.It 269DQO_RDA: DQO is the descriptor format required to take full advantage of 270next generation VM shapes. 271"RDA" stands for "Raw DMA Addressing" and refers to the fact that hardware 272can work with DMA-ed packets and does not expect them to be copied into or 273out of a fixed bounce buffer. 274The D in "DQO" refers to a newer generation of hardware, and the "QO" 275stands for "Queue Out-of-order" referring to the fact that the NIC might 276send Tx and Rx completions in an order different from the one in which 277the corresponding descriptors were posted by the driver. 278.It 279DQO_QPL: The next generation descriptor format in the "QPL" mode. 280.El 281.Sh SUPPORT 282Please email gvnic-drivers@google.com with the specifics of the issue encountered. 283.Sh SEE ALSO 284.Xr netstat 1 , 285.Xr loader.conf 5 , 286.Xr ifconfig 8 , 287.Xr sysctl 8 288.Sh HISTORY 289The 290.Nm 291device driver first appeared in 292.Fx 13.3 . 293.Sh AUTHORS 294The 295.Nm 296driver was written by Google. 297