1.. SPDX-License-Identifier: (GPL-2.0+ OR MIT) 2 3========== 4VBIOS 5========== 6This document describes the layout of the VBIOS image which is a series of concatenated 7images in the ROM of the GPU. The VBIOS is mirrored onto the BAR 0 space and is read 8by both Boot ROM firmware (also known as IFR or init-from-rom firmware) on the GPU to 9bootstrap various microcontrollers (PMU, SEC, GSP) with critical initialization before 10the driver loads, as well as by the nova-core driver in the kernel to boot the GSP. 11 12The format of the images in the ROM follow the "BIOS Specification" part of the 13PCI specification, with Nvidia-specific extensions. The ROM images of type FwSec 14are the ones that contain Falcon ucode and what we are mainly looking for. 15 16As an example, the following are the different image types that can be found in the 17VBIOS of an Ampere GA102 GPU which is supported by the nova-core driver. 18 19- PciAt Image (Type 0x00) - This is the standard PCI BIOS image, whose name 20 likely comes from the "IBM PC/AT" architecture. 21 22- EFI Image (Type 0x03) - This is the EFI BIOS image. It contains the UEFI GOP 23 driver that is used to display UEFI graphics output. 24 25- First FwSec Image (Type 0xE0) - The first FwSec image (Secure Firmware) 26 27- Second FwSec Image (Type 0xE0) - The second FwSec image (Secure Firmware) 28 contains various microcodes (also known as an applications) that do a range 29 of different functions. The FWSEC ucode is run in heavy-secure mode and 30 typically runs directly on the GSP (it could be running on a different 31 designated processor in future generations but as of Ampere, it is the GSP). 32 This firmware then loads other firmware ucodes onto the PMU and SEC2 33 microcontrollers for gfw initialization after GPU reset and before the driver 34 loads (see devinit.rst). The DEVINIT ucode is itself another ucode that is 35 stored in this ROM partition. 36 37Once located, the Falcon ucodes have "Application Interfaces" in their data 38memory (DMEM). For FWSEC, the application interface we use for FWSEC is the 39"DMEM mapper" interface which is configured to run the "FRTS" command. This 40command carves out the WPR2 (Write-Protected Region) in VRAM. It then places 41important power-management data, called 'FRTS', into this region. The WPR2 42region is only accessible to heavy-secure ucode. 43 44.. note:: 45 It is not clear why FwSec has 2 different partitions in the ROM, but they both 46 are of type 0xE0 and can be identified as such. This could be subject to change 47 in future generations. 48 49IFR Header 50---------- 51On Kepler and later GPUs, the ROM begins with an Init-from-ROM (IFR) header 52rather than a standard PCI ROM signature (0xAA55). The driver must parse the 53IFR header to find where the PCI ROM images actually start. 54 55Init-from-ROM (IFR) is a special GPU feature used for power management 56on some Nvidia GPUs. It references data in the VBIOS for its operation, 57but for drivers the important piece is a header that precedes the 58VBIOS PCI Expansion ROM. 59 60Most such GPUs do not need to parse the IFR header in order to find the 61VBIOS, but the Nvidia GA100 is the exception. GA100 lacks a display engine, 62so the PRAMIN method (which reads the VBIOS from VRAM via display hardware) 63is unavailable, forcing the driver to read the ROM directly via PROM. 64On other similar GPUs, either PRAMIN succeeds before PROM is tried, or the 65IFR hardware has already applied the ROM offset so that PROM reads 66transparently skip the IFR header. 67 68The driver should first check for the standard 0xAA55 signature at offset 0. 69If found, there is no IFR header and the PCI ROM images start at 70offset 0. If not found, check for the IFR signature and parse the header to 71determine the PCI ROM image offset. 72 73Fixed Header Format 74~~~~~~~~~~~~~~~~~~~ 75 76The IFR header begins with four 32-bit words at fixed offsets:: 77 78 Offset Name Fields 79 ------ ------- ------ 80 0x00 FIXED0 bits 31:0 - Signature (must be 0x4947564E, ASCII "NVGI") 81 0x04 FIXED1 bit 31 - Reserved 82 bits 30:16 - FIXED_DATA_SIZE Fixed data size (offset to extended section) 83 bits 15:8 - VERSIONSW Software version 84 bits 7:0 - Reserved 85 0x08 FIXED2 bit 31 - Reserved 86 bits 30:20 - Reserved (zero) 87 bits 19:0 - TOTAL_DATA_SIZE Total data size 88 89Finding the PCI ROM Image Offset 90~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 91 92The method to find this offset depends on `VERSIONSW`. 93 94- **Version 1 and 2**: Read `FIXED_DATA_SIZE` from `FIXED1` to get the extended 95 section offset. The PCI ROM image is the 32-bit value at `FIXED_DATA_SIZE + 4`. 96 97- **Version 3**: Read `TOTAL_DATA_SIZE` from `FIXED2`. The 32-bit value at that 98 offset is a flash status offset. Add 4096 to get the ROM directory offset, 99 `ROM_DIRECTORY_OFFSET`. The ROM directory must have signature 0x44524652 100 (ASCII "RFRD"). The PCI ROM image offset is the 32-bit value at 101 `ROM_DIRECTORY_OFFSET + 8`. 102 103The PCI ROM image offset must be 4-byte aligned. All offsets are relative to the 104start of ROM (BAR0 + 0x300000). 105 106VBIOS ROM Layout 107---------------- 108The VBIOS (PCI Expansion ROM) is a series of concatenated images laid out as 109follows. On GPUs with an IFR header, this layout begins at the image offset 110determined by parsing the IFR header. On older GPUs, it begins at offset 0:: 111 112 +----------------------------------------------------------------------------+ 113 | VBIOS (Starting at ROM_OFFSET: 0x300000 + IFR image offset) | 114 +----------------------------------------------------------------------------+ 115 | +-----------------------------------------------+ | 116 | | PciAt Image (Type 0x00) | | 117 | +-----------------------------------------------+ | 118 | | +-------------------+ | | 119 | | | ROM Header | | | 120 | | | (Signature 0xAA55)| | | 121 | | +-------------------+ | | 122 | | | rom header's pci_data_struct_offset | | 123 | | | points to the PCIR structure | | 124 | | V | | 125 | | +-------------------+ | | 126 | | | PCIR Structure | | | 127 | | | (Signature "PCIR")| | | 128 | | | last_image: 0x80 | | | 129 | | | image_len: size | | | 130 | | | in 512-byte units | | | 131 | | +-------------------+ | | 132 | | | | | 133 | | | NPDE immediately follows PCIR | | 134 | | V | | 135 | | +-------------------+ | | 136 | | | NPDE Structure | | | 137 | | | (Signature "NPDE")| | | 138 | | | last_image: 0x00 | | | 139 | | +-------------------+ | | 140 | | | | 141 | | +-------------------+ | | 142 | | | BIT Header | (Signature scanning | | 143 | | | (Signature "BIT") | provides the location | | 144 | | +-------------------+ of the BIT table) | | 145 | | | header is | | 146 | | | followed by a table of tokens | | 147 | | V one of which is for falcon data. | | 148 | | +-------------------+ | | 149 | | | BIT Tokens | | | 150 | | | ______________ | | | 151 | | | | Falcon Data | | | | 152 | | | | Token (0x70)|---+------------>------------+--+ | 153 | | | +-------------+ | falcon_data_ptr() | | | 154 | | +-------------------+ | V | 155 | +-----------------------------------------------+ | | 156 | (no gap between images) | | 157 | +-----------------------------------------------+ | | 158 | | EFI Image (Type 0x03) | | | 159 | +-----------------------------------------------+ | | 160 | | Contains the UEFI GOP driver (Graphics Output)| | | 161 | | +-------------------+ | | | 162 | | | ROM Header | | | | 163 | | +-------------------+ | | | 164 | | | PCIR Structure | | | | 165 | | +-------------------+ | | | 166 | | | NPDE Structure | | | | 167 | | +-------------------+ | | | 168 | | | Image data | | | | 169 | | +-------------------+ | | | 170 | +-----------------------------------------------+ | | 171 | (no gap between images) | | 172 | +-----------------------------------------------+ | | 173 | | First FwSec Image (Type 0xE0) | | | 174 | +-----------------------------------------------+ | | 175 | | +-------------------+ | | | 176 | | | ROM Header | | | | 177 | | +-------------------+ | | | 178 | | | PCIR Structure | | | | 179 | | +-------------------+ | | | 180 | | | NPDE Structure | | | | 181 | | +-------------------+ | | | 182 | | | Image data | | | | 183 | | +-------------------+ | | | 184 | +-----------------------------------------------+ | | 185 | (no gap between images) | | 186 | +-----------------------------------------------+ | | 187 | | Second FwSec Image (Type 0xE0) | | | 188 | +-----------------------------------------------+ | | 189 | | +-------------------+ | | | 190 | | | ROM Header | | | | 191 | | +-------------------+ | | | 192 | | | PCIR Structure | | | | 193 | | +-------------------+ | | | 194 | | | NPDE Structure | | | | 195 | | +-------------------+ | | | 196 | | | | | 197 | | +-------------------+ | | | 198 | | | PMU Lookup Table | <- falcon_data_offset <----+ | 199 | | | +-------------+ | pmu_lookup_table | | 200 | | | | Entry 0x85 | | | | 201 | | | | FWSEC_PROD | | | | 202 | | | +-------------+ | | | 203 | | +-------------------+ | | 204 | | | | | 205 | | | points to | | 206 | | V | | 207 | | +-------------------+ | | 208 | | | FalconUCodeDescV3 | <- falcon_ucode_offset | | 209 | | | (FWSEC Firmware) | fwsec_header() | | 210 | | +-------------------+ | | 211 | | | immediately followed by... | | 212 | | V | | 213 | | +----------------------------+ | | 214 | | | Signatures + FWSEC Ucode | | | 215 | | | fwsec_sigs(), fwsec_ucode()| | | 216 | | +----------------------------+ | | 217 | +-----------------------------------------------+ | 218 | | 219 +----------------------------------------------------------------------------+ 220 221.. note:: 222 This diagram is created based on an GA-102 Ampere GPU as an example and could 223 vary for future or other GPUs. 224 225.. note:: 226 For more explanations of acronyms, see the detailed descriptions in `vbios.rs`. 227 228Falcon data Lookup 229------------------ 230A key part of the VBIOS extraction code (vbios.rs) is to find the location of the 231Falcon data in the VBIOS which contains the PMU lookup table. This lookup table is 232used to find the required Falcon ucode based on an application ID. 233 234The location of the PMU lookup table is found by scanning the BIT (`BIOS Information Table`_) 235tokens for a token with the Falcon data token id (0x70) which indicates the 236offset of the same from the start of the VBIOS image. Unfortunately, the offset 237does not account for the EFI image located between the PciAt and FwSec images. 238The `vbios.rs` code compensates for this with appropriate arithmetic. 239 240.. _`BIOS Information Table`: https://download.nvidia.com/open-gpu-doc/BIOS-Information-Table/1/BIOS-Information-Table.html 241