1*5ec879e2SJoel Fernandes.. SPDX-License-Identifier: (GPL-2.0+ OR MIT) 2*5ec879e2SJoel Fernandes 3*5ec879e2SJoel Fernandes========== 4*5ec879e2SJoel FernandesVBIOS 5*5ec879e2SJoel Fernandes========== 6*5ec879e2SJoel FernandesThis document describes the layout of the VBIOS image which is a series of concatenated 7*5ec879e2SJoel Fernandesimages in the ROM of the GPU. The VBIOS is mirrored onto the BAR 0 space and is read 8*5ec879e2SJoel Fernandesby both Boot ROM firmware (also known as IFR or init-from-rom firmware) on the GPU to 9*5ec879e2SJoel Fernandesbootstrap various microcontrollers (PMU, SEC, GSP) with critical initialization before 10*5ec879e2SJoel Fernandesthe driver loads, as well as by the nova-core driver in the kernel to boot the GSP. 11*5ec879e2SJoel Fernandes 12*5ec879e2SJoel FernandesThe format of the images in the ROM follow the "BIOS Specification" part of the 13*5ec879e2SJoel FernandesPCI specification, with Nvidia-specific extensions. The ROM images of type FwSec 14*5ec879e2SJoel Fernandesare the ones that contain Falcon ucode and what we are mainly looking for. 15*5ec879e2SJoel Fernandes 16*5ec879e2SJoel FernandesAs an example, the following are the different image types that can be found in the 17*5ec879e2SJoel FernandesVBIOS of an Ampere GA102 GPU which is supported by the nova-core driver. 18*5ec879e2SJoel Fernandes 19*5ec879e2SJoel Fernandes- PciAt Image (Type 0x00) - This is the standard PCI BIOS image, whose name 20*5ec879e2SJoel Fernandes likely comes from the "IBM PC/AT" architecture. 21*5ec879e2SJoel Fernandes 22*5ec879e2SJoel Fernandes- EFI Image (Type 0x03) - This is the EFI BIOS image. It contains the UEFI GOP 23*5ec879e2SJoel Fernandes driver that is used to display UEFI graphics output. 24*5ec879e2SJoel Fernandes 25*5ec879e2SJoel Fernandes- First FwSec Image (Type 0xE0) - The first FwSec image (Secure Firmware) 26*5ec879e2SJoel Fernandes 27*5ec879e2SJoel Fernandes- Second FwSec Image (Type 0xE0) - The second FwSec image (Secure Firmware) 28*5ec879e2SJoel Fernandes contains various microcodes (also known as an applications) that do a range 29*5ec879e2SJoel Fernandes of different functions. The FWSEC ucode is run in heavy-secure mode and 30*5ec879e2SJoel Fernandes typically runs directly on the GSP (it could be running on a different 31*5ec879e2SJoel Fernandes designated processor in future generations but as of Ampere, it is the GSP). 32*5ec879e2SJoel Fernandes This firmware then loads other firmware ucodes onto the PMU and SEC2 33*5ec879e2SJoel Fernandes microcontrollers for gfw initialization after GPU reset and before the driver 34*5ec879e2SJoel Fernandes loads (see devinit.rst). The DEVINIT ucode is itself another ucode that is 35*5ec879e2SJoel Fernandes stored in this ROM partition. 36*5ec879e2SJoel Fernandes 37*5ec879e2SJoel FernandesOnce located, the Falcon ucodes have "Application Interfaces" in their data 38*5ec879e2SJoel Fernandesmemory (DMEM). For FWSEC, the application interface we use for FWSEC is the 39*5ec879e2SJoel Fernandes"DMEM mapper" interface which is configured to run the "FRTS" command. This 40*5ec879e2SJoel Fernandescommand carves out the WPR2 (Write-Protected Region) in VRAM. It then places 41*5ec879e2SJoel Fernandesimportant power-management data, called 'FRTS', into this region. The WPR2 42*5ec879e2SJoel Fernandesregion is only accessible to heavy-secure ucode. 43*5ec879e2SJoel Fernandes 44*5ec879e2SJoel Fernandes.. note:: 45*5ec879e2SJoel Fernandes It is not clear why FwSec has 2 different partitions in the ROM, but they both 46*5ec879e2SJoel Fernandes are of type 0xE0 and can be identified as such. This could be subject to change 47*5ec879e2SJoel Fernandes in future generations. 48*5ec879e2SJoel Fernandes 49*5ec879e2SJoel FernandesVBIOS ROM Layout 50*5ec879e2SJoel Fernandes---------------- 51*5ec879e2SJoel FernandesThe VBIOS layout is roughly a series of concatenated images laid out as follows:: 52*5ec879e2SJoel Fernandes 53*5ec879e2SJoel Fernandes +----------------------------------------------------------------------------+ 54*5ec879e2SJoel Fernandes | VBIOS (Starting at ROM_OFFSET: 0x300000) | 55*5ec879e2SJoel Fernandes +----------------------------------------------------------------------------+ 56*5ec879e2SJoel Fernandes | +-----------------------------------------------+ | 57*5ec879e2SJoel Fernandes | | PciAt Image (Type 0x00) | | 58*5ec879e2SJoel Fernandes | +-----------------------------------------------+ | 59*5ec879e2SJoel Fernandes | | +-------------------+ | | 60*5ec879e2SJoel Fernandes | | | ROM Header | | | 61*5ec879e2SJoel Fernandes | | | (Signature 0xAA55)| | | 62*5ec879e2SJoel Fernandes | | +-------------------+ | | 63*5ec879e2SJoel Fernandes | | | rom header's pci_data_struct_offset | | 64*5ec879e2SJoel Fernandes | | | points to the PCIR structure | | 65*5ec879e2SJoel Fernandes | | V | | 66*5ec879e2SJoel Fernandes | | +-------------------+ | | 67*5ec879e2SJoel Fernandes | | | PCIR Structure | | | 68*5ec879e2SJoel Fernandes | | | (Signature "PCIR")| | | 69*5ec879e2SJoel Fernandes | | | last_image: 0x80 | | | 70*5ec879e2SJoel Fernandes | | | image_len: size | | | 71*5ec879e2SJoel Fernandes | | | in 512-byte units | | | 72*5ec879e2SJoel Fernandes | | +-------------------+ | | 73*5ec879e2SJoel Fernandes | | | | | 74*5ec879e2SJoel Fernandes | | | NPDE immediately follows PCIR | | 75*5ec879e2SJoel Fernandes | | V | | 76*5ec879e2SJoel Fernandes | | +-------------------+ | | 77*5ec879e2SJoel Fernandes | | | NPDE Structure | | | 78*5ec879e2SJoel Fernandes | | | (Signature "NPDE")| | | 79*5ec879e2SJoel Fernandes | | | last_image: 0x00 | | | 80*5ec879e2SJoel Fernandes | | +-------------------+ | | 81*5ec879e2SJoel Fernandes | | | | 82*5ec879e2SJoel Fernandes | | +-------------------+ | | 83*5ec879e2SJoel Fernandes | | | BIT Header | (Signature scanning | | 84*5ec879e2SJoel Fernandes | | | (Signature "BIT") | provides the location | | 85*5ec879e2SJoel Fernandes | | +-------------------+ of the BIT table) | | 86*5ec879e2SJoel Fernandes | | | header is | | 87*5ec879e2SJoel Fernandes | | | followed by a table of tokens | | 88*5ec879e2SJoel Fernandes | | V one of which is for falcon data. | | 89*5ec879e2SJoel Fernandes | | +-------------------+ | | 90*5ec879e2SJoel Fernandes | | | BIT Tokens | | | 91*5ec879e2SJoel Fernandes | | | ______________ | | | 92*5ec879e2SJoel Fernandes | | | | Falcon Data | | | | 93*5ec879e2SJoel Fernandes | | | | Token (0x70)|---+------------>------------+--+ | 94*5ec879e2SJoel Fernandes | | | +-------------+ | falcon_data_ptr() | | | 95*5ec879e2SJoel Fernandes | | +-------------------+ | V | 96*5ec879e2SJoel Fernandes | +-----------------------------------------------+ | | 97*5ec879e2SJoel Fernandes | (no gap between images) | | 98*5ec879e2SJoel Fernandes | +-----------------------------------------------+ | | 99*5ec879e2SJoel Fernandes | | EFI Image (Type 0x03) | | | 100*5ec879e2SJoel Fernandes | +-----------------------------------------------+ | | 101*5ec879e2SJoel Fernandes | | Contains the UEFI GOP driver (Graphics Output)| | | 102*5ec879e2SJoel Fernandes | | +-------------------+ | | | 103*5ec879e2SJoel Fernandes | | | ROM Header | | | | 104*5ec879e2SJoel Fernandes | | +-------------------+ | | | 105*5ec879e2SJoel Fernandes | | | PCIR Structure | | | | 106*5ec879e2SJoel Fernandes | | +-------------------+ | | | 107*5ec879e2SJoel Fernandes | | | NPDE Structure | | | | 108*5ec879e2SJoel Fernandes | | +-------------------+ | | | 109*5ec879e2SJoel Fernandes | | | Image data | | | | 110*5ec879e2SJoel Fernandes | | +-------------------+ | | | 111*5ec879e2SJoel Fernandes | +-----------------------------------------------+ | | 112*5ec879e2SJoel Fernandes | (no gap between images) | | 113*5ec879e2SJoel Fernandes | +-----------------------------------------------+ | | 114*5ec879e2SJoel Fernandes | | First FwSec Image (Type 0xE0) | | | 115*5ec879e2SJoel Fernandes | +-----------------------------------------------+ | | 116*5ec879e2SJoel Fernandes | | +-------------------+ | | | 117*5ec879e2SJoel Fernandes | | | ROM Header | | | | 118*5ec879e2SJoel Fernandes | | +-------------------+ | | | 119*5ec879e2SJoel Fernandes | | | PCIR Structure | | | | 120*5ec879e2SJoel Fernandes | | +-------------------+ | | | 121*5ec879e2SJoel Fernandes | | | NPDE Structure | | | | 122*5ec879e2SJoel Fernandes | | +-------------------+ | | | 123*5ec879e2SJoel Fernandes | | | Image data | | | | 124*5ec879e2SJoel Fernandes | | +-------------------+ | | | 125*5ec879e2SJoel Fernandes | +-----------------------------------------------+ | | 126*5ec879e2SJoel Fernandes | (no gap between images) | | 127*5ec879e2SJoel Fernandes | +-----------------------------------------------+ | | 128*5ec879e2SJoel Fernandes | | Second FwSec Image (Type 0xE0) | | | 129*5ec879e2SJoel Fernandes | +-----------------------------------------------+ | | 130*5ec879e2SJoel Fernandes | | +-------------------+ | | | 131*5ec879e2SJoel Fernandes | | | ROM Header | | | | 132*5ec879e2SJoel Fernandes | | +-------------------+ | | | 133*5ec879e2SJoel Fernandes | | | PCIR Structure | | | | 134*5ec879e2SJoel Fernandes | | +-------------------+ | | | 135*5ec879e2SJoel Fernandes | | | NPDE Structure | | | | 136*5ec879e2SJoel Fernandes | | +-------------------+ | | | 137*5ec879e2SJoel Fernandes | | | | | 138*5ec879e2SJoel Fernandes | | +-------------------+ | | | 139*5ec879e2SJoel Fernandes | | | PMU Lookup Table | <- falcon_data_offset <----+ | 140*5ec879e2SJoel Fernandes | | | +-------------+ | pmu_lookup_table | | 141*5ec879e2SJoel Fernandes | | | | Entry 0x85 | | | | 142*5ec879e2SJoel Fernandes | | | | FWSEC_PROD | | | | 143*5ec879e2SJoel Fernandes | | | +-------------+ | | | 144*5ec879e2SJoel Fernandes | | +-------------------+ | | 145*5ec879e2SJoel Fernandes | | | | | 146*5ec879e2SJoel Fernandes | | | points to | | 147*5ec879e2SJoel Fernandes | | V | | 148*5ec879e2SJoel Fernandes | | +-------------------+ | | 149*5ec879e2SJoel Fernandes | | | FalconUCodeDescV3 | <- falcon_ucode_offset | | 150*5ec879e2SJoel Fernandes | | | (FWSEC Firmware) | fwsec_header() | | 151*5ec879e2SJoel Fernandes | | +-------------------+ | | 152*5ec879e2SJoel Fernandes | | | immediately followed by... | | 153*5ec879e2SJoel Fernandes | | V | | 154*5ec879e2SJoel Fernandes | | +----------------------------+ | | 155*5ec879e2SJoel Fernandes | | | Signatures + FWSEC Ucode | | | 156*5ec879e2SJoel Fernandes | | | fwsec_sigs(), fwsec_ucode()| | | 157*5ec879e2SJoel Fernandes | | +----------------------------+ | | 158*5ec879e2SJoel Fernandes | +-----------------------------------------------+ | 159*5ec879e2SJoel Fernandes | | 160*5ec879e2SJoel Fernandes +----------------------------------------------------------------------------+ 161*5ec879e2SJoel Fernandes 162*5ec879e2SJoel Fernandes.. note:: 163*5ec879e2SJoel Fernandes This diagram is created based on an GA-102 Ampere GPU as an example and could 164*5ec879e2SJoel Fernandes vary for future or other GPUs. 165*5ec879e2SJoel Fernandes 166*5ec879e2SJoel Fernandes.. note:: 167*5ec879e2SJoel Fernandes For more explanations of acronyms, see the detailed descriptions in `vbios.rs`. 168*5ec879e2SJoel Fernandes 169*5ec879e2SJoel FernandesFalcon data Lookup 170*5ec879e2SJoel Fernandes------------------ 171*5ec879e2SJoel FernandesA key part of the VBIOS extraction code (vbios.rs) is to find the location of the 172*5ec879e2SJoel FernandesFalcon data in the VBIOS which contains the PMU lookup table. This lookup table is 173*5ec879e2SJoel Fernandesused to find the required Falcon ucode based on an application ID. 174*5ec879e2SJoel Fernandes 175*5ec879e2SJoel FernandesThe location of the PMU lookup table is found by scanning the BIT (`BIOS Information Table`_) 176*5ec879e2SJoel Fernandestokens for a token with the id `BIT_TOKEN_ID_FALCON_DATA` (0x70) which indicates the 177*5ec879e2SJoel Fernandesoffset of the same from the start of the VBIOS image. Unfortunately, the offset 178*5ec879e2SJoel Fernandesdoes not account for the EFI image located between the PciAt and FwSec images. 179*5ec879e2SJoel FernandesThe `vbios.rs` code compensates for this with appropriate arithmetic. 180*5ec879e2SJoel Fernandes 181*5ec879e2SJoel Fernandes.. _`BIOS Information Table`: https://download.nvidia.com/open-gpu-doc/BIOS-Information-Table/1/BIOS-Information-Table.html 182