1*e5e716dbSJoel Fernandes.. SPDX-License-Identifier: (GPL-2.0+ OR MIT) 2*e5e716dbSJoel Fernandes 3*e5e716dbSJoel Fernandes========================= 4*e5e716dbSJoel FernandesFWSEC (Firmware Security) 5*e5e716dbSJoel Fernandes========================= 6*e5e716dbSJoel FernandesThis document briefly/conceptually describes the FWSEC (Firmware Security) image 7*e5e716dbSJoel Fernandesand its role in the GPU boot sequence. As such, this information is subject to 8*e5e716dbSJoel Fernandeschange in the future and is only current as of the Ampere GPU family. However, 9*e5e716dbSJoel Fernandeshopefully the concepts described will be useful for understanding the kernel code 10*e5e716dbSJoel Fernandesthat deals with it. All the information is derived from publicly available 11*e5e716dbSJoel Fernandessources such as public drivers and documentation. 12*e5e716dbSJoel Fernandes 13*e5e716dbSJoel FernandesThe role of FWSEC is to provide a secure boot process. It runs in 14*e5e716dbSJoel Fernandes'Heavy-secure' mode, and performs firmware verification after a GPU reset 15*e5e716dbSJoel Fernandesbefore loading various ucode images onto other microcontrollers on the GPU, 16*e5e716dbSJoel Fernandessuch as the PMU and GSP. 17*e5e716dbSJoel Fernandes 18*e5e716dbSJoel FernandesFWSEC itself is an application stored in the VBIOS ROM in the FWSEC partition of 19*e5e716dbSJoel FernandesROM (see vbios.rst for more details). It contains different commands like FRTS 20*e5e716dbSJoel Fernandes(Firmware Runtime Services) and SB (Secure Booting other microcontrollers after 21*e5e716dbSJoel Fernandesreset and loading them with other non-FWSEC ucode). The kernel driver only needs 22*e5e716dbSJoel Fernandesto perform FRTS, since Secure Boot (SB) has already completed by the time the driver 23*e5e716dbSJoel Fernandesis loaded. 24*e5e716dbSJoel Fernandes 25*e5e716dbSJoel FernandesThe FRTS command carves out the WPR2 region (Write protected region) which contains 26*e5e716dbSJoel Fernandesdata required for power management. Once setup, only HS mode ucode can access it 27*e5e716dbSJoel Fernandes(see falcon.rst for privilege levels). 28*e5e716dbSJoel Fernandes 29*e5e716dbSJoel FernandesThe FWSEC image is located in the VBIOS ROM in the partition of the ROM that contains 30*e5e716dbSJoel Fernandesvarious ucode images (also known as applications) -- one of them being FWSEC. For how 31*e5e716dbSJoel Fernandesit is extracted, see vbios.rst and the vbios.rs source code. 32*e5e716dbSJoel Fernandes 33*e5e716dbSJoel FernandesThe Falcon data for each ucode images (including the FWSEC image) is a combination 34*e5e716dbSJoel Fernandesof headers, data sections (DMEM) and instruction code sections (IMEM). All these 35*e5e716dbSJoel Fernandesucode images are stored in the same ROM partition and the PMU table is used to look 36*e5e716dbSJoel Fernandesup the application to load it based on its application ID (see vbios.rs). 37*e5e716dbSJoel Fernandes 38*e5e716dbSJoel FernandesFor the nova-core driver, the FWSEC contains an 'application interface' called 39*e5e716dbSJoel FernandesDMEMMAPPER. This interface is used to execute the 'FWSEC-FRTS' command, among others. 40*e5e716dbSJoel FernandesFor Ampere, FWSEC is running on the GSP in Heavy-secure mode and runs FRTS. 41*e5e716dbSJoel Fernandes 42*e5e716dbSJoel FernandesFWSEC Memory Layout 43*e5e716dbSJoel Fernandes------------------- 44*e5e716dbSJoel FernandesThe memory layout of the FWSEC image is as follows:: 45*e5e716dbSJoel Fernandes 46*e5e716dbSJoel Fernandes +---------------------------------------------------------------+ 47*e5e716dbSJoel Fernandes | FWSEC ROM image (type 0xE0) | 48*e5e716dbSJoel Fernandes | | 49*e5e716dbSJoel Fernandes | +---------------------------------+ | 50*e5e716dbSJoel Fernandes | | PMU Falcon Ucode Table | | 51*e5e716dbSJoel Fernandes | | (PmuLookupTable) | | 52*e5e716dbSJoel Fernandes | | +-------------------------+ | | 53*e5e716dbSJoel Fernandes | | | Table Header | | | 54*e5e716dbSJoel Fernandes | | | - version: 0x01 | | | 55*e5e716dbSJoel Fernandes | | | - header_size: 6 | | | 56*e5e716dbSJoel Fernandes | | | - entry_size: 6 | | | 57*e5e716dbSJoel Fernandes | | | - entry_count: N | | | 58*e5e716dbSJoel Fernandes | | | - desc_version:3(unused)| | | 59*e5e716dbSJoel Fernandes | | +-------------------------+ | | 60*e5e716dbSJoel Fernandes | | ... | | 61*e5e716dbSJoel Fernandes | | +-------------------------+ | | 62*e5e716dbSJoel Fernandes | | | Entry for FWSEC (0x85) | | | 63*e5e716dbSJoel Fernandes | | | (PmuLookupTableEntry) | | | 64*e5e716dbSJoel Fernandes | | | - app_id: 0x85 (FWSEC) |----|----+ | 65*e5e716dbSJoel Fernandes | | | - target_id: 0x01 (PMU) | | | | 66*e5e716dbSJoel Fernandes | | | - data: offset ---------|----|----|---+ look up FWSEC | 67*e5e716dbSJoel Fernandes | | +-------------------------+ | | | | 68*e5e716dbSJoel Fernandes | +---------------------------------+ | | | 69*e5e716dbSJoel Fernandes | | | | 70*e5e716dbSJoel Fernandes | | | | 71*e5e716dbSJoel Fernandes | +---------------------------------+ | | | 72*e5e716dbSJoel Fernandes | | FWSEC Ucode Component |<---+ | | 73*e5e716dbSJoel Fernandes | | (aka Falcon data) | | | 74*e5e716dbSJoel Fernandes | | +-------------------------+ | | | 75*e5e716dbSJoel Fernandes | | | FalconUCodeDescV3 |<---|--------+ | 76*e5e716dbSJoel Fernandes | | | - hdr | | | 77*e5e716dbSJoel Fernandes | | | - stored_size | | | 78*e5e716dbSJoel Fernandes | | | - pkc_data_offset | | | 79*e5e716dbSJoel Fernandes | | | - interface_offset -----|----|----------------+ | 80*e5e716dbSJoel Fernandes | | | - imem_phys_base | | | | 81*e5e716dbSJoel Fernandes | | | - imem_load_size | | | | 82*e5e716dbSJoel Fernandes | | | - imem_virt_base | | | | 83*e5e716dbSJoel Fernandes | | | - dmem_phys_base | | | | 84*e5e716dbSJoel Fernandes | | | - dmem_load_size | | | | 85*e5e716dbSJoel Fernandes | | | - engine_id_mask | | | | 86*e5e716dbSJoel Fernandes | | | - ucode_id | | | | 87*e5e716dbSJoel Fernandes | | | - signature_count | | look up sig | | 88*e5e716dbSJoel Fernandes | | | - signature_versions --------------+ | | 89*e5e716dbSJoel Fernandes | | +-------------------------+ | | | | 90*e5e716dbSJoel Fernandes | | (no gap) | | | | 91*e5e716dbSJoel Fernandes | | +-------------------------+ | | | | 92*e5e716dbSJoel Fernandes | | | Signatures Section |<---|-----+ | | 93*e5e716dbSJoel Fernandes | | | (384 bytes per sig) | | | | 94*e5e716dbSJoel Fernandes | | | - RSA-3K Signature 1 | | | | 95*e5e716dbSJoel Fernandes | | | - RSA-3K Signature 2 | | | | 96*e5e716dbSJoel Fernandes | | | ... | | | | 97*e5e716dbSJoel Fernandes | | +-------------------------+ | | | 98*e5e716dbSJoel Fernandes | | | | | 99*e5e716dbSJoel Fernandes | | +-------------------------+ | | | 100*e5e716dbSJoel Fernandes | | | IMEM Section (Code) | | | | 101*e5e716dbSJoel Fernandes | | | | | | | 102*e5e716dbSJoel Fernandes | | | Contains instruction | | | | 103*e5e716dbSJoel Fernandes | | | code etc. | | | | 104*e5e716dbSJoel Fernandes | | +-------------------------+ | | | 105*e5e716dbSJoel Fernandes | | | | | 106*e5e716dbSJoel Fernandes | | +-------------------------+ | | | 107*e5e716dbSJoel Fernandes | | | DMEM Section (Data) | | | | 108*e5e716dbSJoel Fernandes | | | | | | | 109*e5e716dbSJoel Fernandes | | | +---------------------+ | | | | 110*e5e716dbSJoel Fernandes | | | | Application | |<---|----------------+ | 111*e5e716dbSJoel Fernandes | | | | Interface Table | | | | 112*e5e716dbSJoel Fernandes | | | | (FalconAppifHdrV1) | | | | 113*e5e716dbSJoel Fernandes | | | | Header: | | | | 114*e5e716dbSJoel Fernandes | | | | - version: 0x01 | | | | 115*e5e716dbSJoel Fernandes | | | | - header_size: 4 | | | | 116*e5e716dbSJoel Fernandes | | | | - entry_size: 8 | | | | 117*e5e716dbSJoel Fernandes | | | | - entry_count: N | | | | 118*e5e716dbSJoel Fernandes | | | | | | | | 119*e5e716dbSJoel Fernandes | | | | Entries: | | | | 120*e5e716dbSJoel Fernandes | | | | +-----------------+ | | | | 121*e5e716dbSJoel Fernandes | | | | | DEVINIT (ID 1) | | | | | 122*e5e716dbSJoel Fernandes | | | | | - id: 0x01 | | | | | 123*e5e716dbSJoel Fernandes | | | | | - dmemOffset X -|-|-|----+ | 124*e5e716dbSJoel Fernandes | | | | +-----------------+ | | | | 125*e5e716dbSJoel Fernandes | | | | +-----------------+ | | | | 126*e5e716dbSJoel Fernandes | | | | | DMEMMAPPER(ID 4)| | | | | 127*e5e716dbSJoel Fernandes | | | | | - id: 0x04 | | | | Used only for DevInit | 128*e5e716dbSJoel Fernandes | | | | | (NVFW_FALCON_ | | | | application (not FWSEC) | 129*e5e716dbSJoel Fernandes | | | | | APPIF_ID_DMEMMAPPER) | | 130*e5e716dbSJoel Fernandes | | | | | - dmemOffset Y -|-|-|----|-----+ | 131*e5e716dbSJoel Fernandes | | | | +-----------------+ | | | | | 132*e5e716dbSJoel Fernandes | | | +---------------------+ | | | | 133*e5e716dbSJoel Fernandes | | | | | | | 134*e5e716dbSJoel Fernandes | | | +---------------------+ | | | | 135*e5e716dbSJoel Fernandes | | | | DEVINIT Engine |<|----+ | Used by FWSEC | 136*e5e716dbSJoel Fernandes | | | | Interface | | | | app. | 137*e5e716dbSJoel Fernandes | | | +---------------------+ | | | | 138*e5e716dbSJoel Fernandes | | | | | | | 139*e5e716dbSJoel Fernandes | | | +---------------------+ | | | | 140*e5e716dbSJoel Fernandes | | | | DMEM Mapper (ID 4) |<|----+-----+ | 141*e5e716dbSJoel Fernandes | | | | (FalconAppifDmemmapperV3) | | 142*e5e716dbSJoel Fernandes | | | | - signature: "DMAP" | | | | 143*e5e716dbSJoel Fernandes | | | | - version: 0x0003 | | | | 144*e5e716dbSJoel Fernandes | | | | - Size: 64 bytes | | | | 145*e5e716dbSJoel Fernandes | | | | - cmd_in_buffer_off | |----|------------+ | 146*e5e716dbSJoel Fernandes | | | | - cmd_in_buffer_size| | | | | 147*e5e716dbSJoel Fernandes | | | | - cmd_out_buffer_off| |----|------------|-----+ | 148*e5e716dbSJoel Fernandes | | | | - cmd_out_buffer_sz | | | | | | 149*e5e716dbSJoel Fernandes | | | | - init_cmd | | | | | | 150*e5e716dbSJoel Fernandes | | | | - features | | | | | | 151*e5e716dbSJoel Fernandes | | | | - cmd_mask0/1 | | | | | | 152*e5e716dbSJoel Fernandes | | | +---------------------+ | | | | | 153*e5e716dbSJoel Fernandes | | | | | | | | 154*e5e716dbSJoel Fernandes | | | +---------------------+ | | | | | 155*e5e716dbSJoel Fernandes | | | | Command Input Buffer|<|----|------------+ | | 156*e5e716dbSJoel Fernandes | | | | - Command data | | | | | 157*e5e716dbSJoel Fernandes | | | | - Arguments | | | | | 158*e5e716dbSJoel Fernandes | | | +---------------------+ | | | | 159*e5e716dbSJoel Fernandes | | | | | | | 160*e5e716dbSJoel Fernandes | | | +---------------------+ | | | | 161*e5e716dbSJoel Fernandes | | | | Command Output |<|----|------------------+ | 162*e5e716dbSJoel Fernandes | | | | Buffer | | | | 163*e5e716dbSJoel Fernandes | | | | - Results | | | | 164*e5e716dbSJoel Fernandes | | | | - Status | | | | 165*e5e716dbSJoel Fernandes | | | +---------------------+ | | | 166*e5e716dbSJoel Fernandes | | +-------------------------+ | | 167*e5e716dbSJoel Fernandes | +---------------------------------+ | 168*e5e716dbSJoel Fernandes | | 169*e5e716dbSJoel Fernandes +---------------------------------------------------------------+ 170*e5e716dbSJoel Fernandes 171*e5e716dbSJoel Fernandes.. note:: 172*e5e716dbSJoel Fernandes This is using an GA-102 Ampere GPU as an example and could vary for future GPUs. 173*e5e716dbSJoel Fernandes 174*e5e716dbSJoel Fernandes.. note:: 175*e5e716dbSJoel Fernandes The FWSEC image also plays a role in memory scrubbing (ECC initialization) and VPR 176*e5e716dbSJoel Fernandes (Video Protected Region) initialization as well. Before the nova-core driver is even 177*e5e716dbSJoel Fernandes loaded, the FWSEC image is running on the GSP in heavy-secure mode. After the devinit 178*e5e716dbSJoel Fernandes sequence completes, it does VRAM memory scrubbing (ECC initialization). On consumer 179*e5e716dbSJoel Fernandes GPUs, it scrubs only part of memory and then initiates 'async scrubbing'. Before this 180*e5e716dbSJoel Fernandes async scrubbing completes, the unscrubbed VRAM cannot be used for allocation (thus DRM 181*e5e716dbSJoel Fernandes memory allocators need to wait for this scrubbing to complete). 182