154dfc97bSShailend Chand.\" SPDX-License-Identifier: BSD-3-Clause 254dfc97bSShailend Chand.\" 354dfc97bSShailend Chand.\" Copyright (c) 2023 Google LLC 454dfc97bSShailend Chand.\" 554dfc97bSShailend Chand.\" Redistribution and use in source and binary forms, with or without modification, 654dfc97bSShailend Chand.\" are permitted provided that the following conditions are met: 754dfc97bSShailend Chand.\" 854dfc97bSShailend Chand.\" 1. Redistributions of source code must retain the above copyright notice, this 954dfc97bSShailend Chand.\" list of conditions and the following disclaimer. 1054dfc97bSShailend Chand.\" 1154dfc97bSShailend Chand.\" 2. Redistributions in binary form must reproduce the above copyright notice, 1254dfc97bSShailend Chand.\" this list of conditions and the following disclaimer in the documentation 1354dfc97bSShailend Chand.\" and/or other materials provided with the distribution. 1454dfc97bSShailend Chand.\" 1554dfc97bSShailend Chand.\" 3. Neither the name of the copyright holder nor the names of its contributors 1654dfc97bSShailend Chand.\" may be used to endorse or promote products derived from this software without 1754dfc97bSShailend Chand.\" specific prior written permission. 1854dfc97bSShailend Chand.\" 1954dfc97bSShailend Chand.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND 2054dfc97bSShailend Chand.\" ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED 2154dfc97bSShailend Chand.\" WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE 2254dfc97bSShailend Chand.\" DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR 2354dfc97bSShailend Chand.\" ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES 2454dfc97bSShailend Chand.\" (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; 2554dfc97bSShailend Chand.\" LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON 2654dfc97bSShailend Chand.\" ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT 2754dfc97bSShailend Chand.\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS 2854dfc97bSShailend Chand.\" SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 2954dfc97bSShailend Chand.Dd April 26, 2023 3054dfc97bSShailend Chand.Dt GVE 4 3154dfc97bSShailend Chand.Os 3254dfc97bSShailend Chand.Sh NAME 3354dfc97bSShailend Chand.Nm gve 3454dfc97bSShailend Chand.Nd "Ethernet driver for Google Virtual NIC (gVNIC)" 3554dfc97bSShailend Chand.Sh SYNOPSIS 3654dfc97bSShailend ChandTo compile this driver into the kernel, 3754dfc97bSShailend Chandplace the following lines in your 3854dfc97bSShailend Chandkernel configuration file: 3954dfc97bSShailend Chand.Bd -ragged -offset indent 4054dfc97bSShailend Chand.Cd "device gve" 4154dfc97bSShailend Chand.Ed 4254dfc97bSShailend Chand.Pp 4354dfc97bSShailend ChandAlternatively, to load the driver as a 4454dfc97bSShailend Chandmodule at boot time, place the following line in 4554dfc97bSShailend Chand.Xr loader.conf 5 : 4654dfc97bSShailend Chand.Bd -literal -offset indent 4754dfc97bSShailend Chandif_gve_load="YES" 4854dfc97bSShailend Chand.Ed 4954dfc97bSShailend Chand.Sh DESCRIPTION 5054dfc97bSShailend ChandgVNIC is a virtual network interface designed specifically for Google Compute Engine (GCE). 5154dfc97bSShailend ChandIt is required to support per-VM Tier-1 networking performance, and for using certain VM shapes on GCE. 5254dfc97bSShailend Chand.Pp 5354dfc97bSShailend Chand.Nm 5454dfc97bSShailend Chandis the driver for gVNIC. 5554dfc97bSShailend ChandIt supports the following features: 5654dfc97bSShailend Chand.Pp 5754dfc97bSShailend Chand.Bl -bullet -compact 5854dfc97bSShailend Chand.It 5954dfc97bSShailend ChandRX checksum offload 6054dfc97bSShailend Chand.It 6154dfc97bSShailend ChandTX chesksum offload 6254dfc97bSShailend Chand.It 6354dfc97bSShailend ChandTCP Segmentation Offload (TSO) 6454dfc97bSShailend Chand.It 6554dfc97bSShailend ChandLarge Receive Offload (LRO) in software 6654dfc97bSShailend Chand.It 6754dfc97bSShailend ChandJumbo frames 6854dfc97bSShailend Chand.It 6954dfc97bSShailend ChandReceive Side Scaling (RSS) 7054dfc97bSShailend Chand.El 7154dfc97bSShailend Chand.Pp 7254dfc97bSShailend ChandFor more information on configuring this device, see 7354dfc97bSShailend Chand.Xr ifconfig 8 . 7454dfc97bSShailend Chand.Sh HARDWARE 7554dfc97bSShailend Chand.Nm 7654dfc97bSShailend Chandbinds to a single PCI device ID presented by gVNIC: 7754dfc97bSShailend Chand.Pp 7854dfc97bSShailend Chand.Bl -bullet -compact 7954dfc97bSShailend Chand.It 8054dfc97bSShailend Chand0x1AE0:0x0042 8154dfc97bSShailend Chand.El 8254dfc97bSShailend Chand.Sh DIAGNOSTICS 8354dfc97bSShailend ChandThe following messages are recorded during driver initialization: 8454dfc97bSShailend Chand.Bl -diag 8554dfc97bSShailend Chand.It "Enabled MSIX with %d vectors" 8654dfc97bSShailend Chand.It "Configured device resources" 8754dfc97bSShailend Chand.It "Successfully attached %s" 8854dfc97bSShailend Chand.It "Deconfigured device resources" 8954dfc97bSShailend Chand.El 9054dfc97bSShailend Chand.Pp 9154dfc97bSShailend ChandThese messages are seen if driver initialization fails. 9254dfc97bSShailend ChandGlobal (across-queues) allocation failures: 9354dfc97bSShailend Chand.Bl -diag 9454dfc97bSShailend Chand.It "Failed to configure device resources: err=%d" 9554dfc97bSShailend Chand.It "No compatible queue formats" 9654dfc97bSShailend Chand.It "Failed to allocate ifnet struct" 9754dfc97bSShailend Chand.It "Failed to allocate admin queue mem" 9854dfc97bSShailend Chand.It "Failed to alloc DMA mem for DescribeDevice" 9954dfc97bSShailend Chand.It "Failed to allocate QPL page" 10054dfc97bSShailend Chand.El 10154dfc97bSShailend Chand.Pp 10254dfc97bSShailend Chandirq and BAR allocation failures: 10354dfc97bSShailend Chand.Bl -diag 10454dfc97bSShailend Chand.It "Failed to acquire any msix vectors" 10554dfc97bSShailend Chand.It "Tried to acquire %d msix vectors, got only %d" 10654dfc97bSShailend Chand.It "Failed to setup irq %d for Tx queue %d " 10754dfc97bSShailend Chand.It "Failed to setup irq %d for Rx queue %d " 10854dfc97bSShailend Chand.It "Failed to allocate irq %d for mgmnt queue" 10954dfc97bSShailend Chand.It "Failed to setup irq %d for mgmnt queue, err: %d" 11054dfc97bSShailend Chand.It "Failed to allocate BAR0" 11154dfc97bSShailend Chand.It "Failed to allocate BAR2" 11254dfc97bSShailend Chand.It "Failed to allocate msix table" 11354dfc97bSShailend Chand.El 11454dfc97bSShailend Chand.Pp 11554dfc97bSShailend ChandRx queue-specific allocation failures: 11654dfc97bSShailend Chand.Bl -diag 11754dfc97bSShailend Chand.It "No QPL left for rx ring %d" 11854dfc97bSShailend Chand.It "Failed to alloc queue resources for rx ring %d" 11954dfc97bSShailend Chand.It "Failed to alloc desc ring for rx ring %d" 12054dfc97bSShailend Chand.It "Failed to alloc data ring for rx ring %d" 12154dfc97bSShailend Chand.El 12254dfc97bSShailend Chand.Pp 12354dfc97bSShailend ChandTx queue-specific allocation failures: 12454dfc97bSShailend Chand.Bl -diag 12554dfc97bSShailend Chand.It "No QPL left for tx ring %d" 12654dfc97bSShailend Chand.It "Failed to alloc queue resources for tx ring %d" 12754dfc97bSShailend Chand.It "Failed to alloc desc ring for tx ring %d" 12854dfc97bSShailend Chand.It "Failed to vmap fifo, qpl_id = %d" 12954dfc97bSShailend Chand.El 13054dfc97bSShailend Chand.El 13154dfc97bSShailend Chand.Pp 13254dfc97bSShailend ChandThe following messages are recorded when the interface detach fails: 13354dfc97bSShailend Chand.Bl -diag 13454dfc97bSShailend Chand.It "Failed to deconfigure device resources: err=%d" 13554dfc97bSShailend Chand.El 13654dfc97bSShailend Chand.Pp 13754dfc97bSShailend ChandIf bootverbose is on, the following messages are recorded when the interface is being brought up: 13854dfc97bSShailend Chand.Bl -diag 13954dfc97bSShailend Chand.It "Created %d rx queues" 14054dfc97bSShailend Chand.It "Created %d tx queues" 14154dfc97bSShailend Chand.It "MTU set to %d" 14254dfc97bSShailend Chand.El 14354dfc97bSShailend Chand.Pp 14454dfc97bSShailend ChandThe following messages are recorded when the interface is being brought down: 14554dfc97bSShailend Chand.Bl -diag 14654dfc97bSShailend Chand.It "Destroyed %d rx queues" 14754dfc97bSShailend Chand.It "Destroyed %d tx queues" 14854dfc97bSShailend Chand.El 14954dfc97bSShailend Chand.Pp 15054dfc97bSShailend ChandThese messages are seen if errors are encountered when bringing the interface up or down: 15154dfc97bSShailend Chand.Bl -diag 15254dfc97bSShailend Chand.It "Failed to destroy rxq %d, err: %d" 15354dfc97bSShailend Chand.It "Failed to destroy txq %d, err: %d" 15454dfc97bSShailend Chand.It "Failed to create rxq %d, err: %d" 15554dfc97bSShailend Chand.It "Failed to create txq %d, err: %d" 15654dfc97bSShailend Chand.It "Failed to set MTU to %d" 15754dfc97bSShailend Chand.It "Invalid new MTU setting. new mtu: %d max mtu: %d min mtu: %d" 15854dfc97bSShailend Chand.It "Cannot bring the iface up when detached" 15954dfc97bSShailend Chand.It "Reached max number of registered pages %lu > %lu" 16054dfc97bSShailend Chand.It "Failed to init lro for rx ring %d" 16154dfc97bSShailend Chand.El 16254dfc97bSShailend Chand.Pp 16354dfc97bSShailend ChandThese messages are seen if any admin queue command fails: 16454dfc97bSShailend Chand.Bl -diag 16554dfc97bSShailend Chand.It "AQ command(%u): failed with status %d" 16654dfc97bSShailend Chand.It "AQ command(%u): unknown status code %d" 16754dfc97bSShailend Chand.It "AQ commands timed out, need to reset AQ" 16854dfc97bSShailend Chand.It "Unknown AQ command opcode %d" 16954dfc97bSShailend Chand.El 17054dfc97bSShailend Chand.Pp 17154dfc97bSShailend ChandThese messages are recorded when the device is being reset due to an error: 17254dfc97bSShailend Chand.Bl -diag 17354dfc97bSShailend Chand.It "Scheduling reset task!" 17454dfc97bSShailend Chand.It "Waiting until admin queue is released." 17554dfc97bSShailend Chand.It "Admin queue released" 17654dfc97bSShailend Chand.El 17754dfc97bSShailend Chand.Pp 17854dfc97bSShailend ChandIf it was the NIC that requested the reset, this message is recorded: 17954dfc97bSShailend Chand.Bl -diag 18054dfc97bSShailend Chand.It "Device requested reset" 18154dfc97bSShailend Chand.El 18254dfc97bSShailend Chand.Pp 18354dfc97bSShailend ChandIf the reset fails during the reinitialization phase, this message is recorded: 18454dfc97bSShailend Chand.Bl -diag 18554dfc97bSShailend Chand.It "Restore failed!" 18654dfc97bSShailend Chand.El 18754dfc97bSShailend Chand.Pp 188ffc4f93eSBenedict ReuschlingThese two messages correspond to the NIC alerting the driver to link state changes: 18954dfc97bSShailend Chand.Bl -diag 19054dfc97bSShailend Chand.It "Device link is up." 19154dfc97bSShailend Chand.It "Device link is down." 19254dfc97bSShailend Chand.El 19354dfc97bSShailend Chand.Pp 19454dfc97bSShailend ChandApart from these messages, the driver exposes per-queue packet and error counters as sysctl nodes. 19554dfc97bSShailend ChandGlobal (across queues) counters can be read using 19654dfc97bSShailend Chand.Xr netstat 8 . 19754dfc97bSShailend Chand.Sh LIMITATIONS 19854dfc97bSShailend Chand.Nm 19954dfc97bSShailend Chanddoes not support the transmission of VLAN-tagged packets. 20054dfc97bSShailend ChandAll VLAN-tagged traffic is dropped. 20154dfc97bSShailend Chand.Sh SUPPORT 20254dfc97bSShailend ChandPlease email gvnic-drivers@google.com with the specifics of the issue encountered. 20354dfc97bSShailend Chand.El 20454dfc97bSShailend Chand.Sh SEE ALSO 20554dfc97bSShailend Chand.Xr ifconfig 8 , 20654dfc97bSShailend Chand.Xr netstat 8 20754dfc97bSShailend Chand.Sh HISTORY 20854dfc97bSShailend ChandThe 20954dfc97bSShailend Chand.Nm 21054dfc97bSShailend Chanddevice driver first appeared in 211*5e6bef46STom Hukins.Fx 13.3 . 21254dfc97bSShailend Chand.Sh AUTHORS 21354dfc97bSShailend ChandThe 21454dfc97bSShailend Chand.Nm 21554dfc97bSShailend Chanddriver was written by Google. 216