1.. SPDX-License-Identifier: GPL-2.0 2 3==================================== 4Nested KVM on POWER 5==================================== 6 7Introduction 8============ 9 10This document explains how a guest operating system can act as a 11hypervisor and run nested guests through the use of hypercalls, if the 12hypervisor has implemented them. The terms L0, L1, and L2 are used to 13refer to different software entities. L0 is the hypervisor mode entity 14that would normally be called the "host" or "hypervisor". L1 is a 15guest virtual machine that is directly run under L0 and is initiated 16and controlled by L0. L2 is a guest virtual machine that is initiated 17and controlled by L1 acting as a hypervisor. 18 19Existing API 20============ 21 22Linux/KVM has had support for Nesting as an L0 or L1 since 2018 23 24The L0 code was added:: 25 26 commit 8e3f5fc1045dc49fd175b978c5457f5f51e7a2ce 27 Author: Paul Mackerras <paulus@ozlabs.org> 28 Date: Mon Oct 8 16:31:03 2018 +1100 29 KVM: PPC: Book3S HV: Framework and hcall stubs for nested virtualization 30 31The L1 code was added:: 32 33 commit 360cae313702cdd0b90f82c261a8302fecef030a 34 Author: Paul Mackerras <paulus@ozlabs.org> 35 Date: Mon Oct 8 16:31:04 2018 +1100 36 KVM: PPC: Book3S HV: Nested guest entry via hypercall 37 38This API works primarily using a single hcall h_enter_nested(). This 39call made by the L1 to tell the L0 to start an L2 vCPU with the given 40state. The L0 then starts this L2 and runs until an L2 exit condition 41is reached. Once the L2 exits, the state of the L2 is given back to 42the L1 by the L0. The full L2 vCPU state is always transferred from 43and to L1 when the L2 is run. The L0 doesn't keep any state on the L2 44vCPU (except in the short sequence in the L0 on L1 -> L2 entry and L2 45-> L1 exit). 46 47The only state kept by the L0 is the partition table. The L1 registers 48it's partition table using the h_set_partition_table() hcall. All 49other state held by the L0 about the L2s is cached state (such as 50shadow page tables). 51 52The L1 may run any L2 or vCPU without first informing the L0. It 53simply starts the vCPU using h_enter_nested(). The creation of L2s and 54vCPUs is done implicitly whenever h_enter_nested() is called. 55 56In this document, we call this existing API the v1 API. 57 58New PAPR API 59=============== 60 61The new PAPR API changes from the v1 API such that the creating L2 and 62associated vCPUs is explicit. In this document, we call this the v2 63API. 64 65h_enter_nested() is replaced with H_GUEST_VCPU_RUN(). Before this can 66be called the L1 must explicitly create the L2 using h_guest_create() 67and any associated vCPUs() created with h_guest_create_vCPU(). Getting 68and setting vCPU state can also be performed using h_guest_{g|s}et 69hcall. 70 71The basic execution flow is for an L1 to create an L2, run it, and 72delete it is: 73 74- L1 and L0 negotiate capabilities with H_GUEST_{G,S}ET_CAPABILITIES() 75 (normally at L1 boot time). 76 77- L1 requests the L0 create an L2 with H_GUEST_CREATE() and receives a token 78 79- L1 requests the L0 create an L2 vCPU with H_GUEST_CREATE_VCPU() 80 81- L1 and L0 communicate the vCPU state using the H_GUEST_{G,S}ET() hcall 82 83- L1 requests the L0 runs the vCPU running H_GUEST_VCPU_RUN() hcall 84 85- L1 deletes L2 with H_GUEST_DELETE() 86 87More details of the individual hcalls follows: 88 89HCALL Details 90============= 91 92This documentation is provided to give an overall understating of the 93API. It doesn't aim to provide all the details required to implement 94an L1 or L0. Latest version of PAPR can be referred to for more details. 95 96All these HCALLs are made by the L1 to the L0. 97 98H_GUEST_GET_CAPABILITIES() 99-------------------------- 100 101This is called to get the capabilities of the L0 nested 102hypervisor. This includes capabilities such the CPU versions (eg 103POWER9, POWER10) that are supported as L2s:: 104 105 H_GUEST_GET_CAPABILITIES(uint64 flags) 106 107 Parameters: 108 Input: 109 flags: Reserved 110 Output: 111 R3: Return code 112 R4: Hypervisor Supported Capabilities bitmap 1 113 114H_GUEST_SET_CAPABILITIES() 115-------------------------- 116 117This is called to inform the L0 of the capabilities of the L1 118hypervisor. The set of flags passed here are the same as 119H_GUEST_GET_CAPABILITIES() 120 121Typically, GET will be called first and then SET will be called with a 122subset of the flags returned from GET. This process allows the L0 and 123L1 to negotiate an agreed set of capabilities:: 124 125 H_GUEST_SET_CAPABILITIES(uint64 flags, 126 uint64 capabilitiesBitmap1) 127 Parameters: 128 Input: 129 flags: Reserved 130 capabilitiesBitmap1: Only capabilities advertised through 131 H_GUEST_GET_CAPABILITIES 132 Output: 133 R3: Return code 134 R4: If R3 = H_P2: The number of invalid bitmaps 135 R5: If R3 = H_P2: The index of first invalid bitmap 136 137H_GUEST_CREATE() 138---------------- 139 140This is called to create an L2. A unique ID of the L2 created 141(similar to an LPID) is returned, which can be used on subsequent HCALLs to 142identify the L2:: 143 144 H_GUEST_CREATE(uint64 flags, 145 uint64 continueToken); 146 Parameters: 147 Input: 148 flags: Reserved 149 continueToken: Initial call set to -1. Subsequent calls, 150 after H_Busy or H_LongBusyOrder has been 151 returned, value that was returned in R4. 152 Output: 153 R3: Return code. Notable: 154 H_Not_Enough_Resources: Unable to create Guest VCPU due to not 155 enough Hypervisor memory. See H_GUEST_CREATE_GET_STATE(flags = 156 takeOwnershipOfVcpuState) 157 R4: If R3 = H_Busy or_H_LongBusyOrder -> continueToken 158 159H_GUEST_CREATE_VCPU() 160--------------------- 161 162This is called to create a vCPU associated with an L2. The L2 id 163(returned from H_GUEST_CREATE()) should be passed it. Also passed in 164is a unique (for this L2) vCPUid. This vCPUid is allocated by the 165L1:: 166 167 H_GUEST_CREATE_VCPU(uint64 flags, 168 uint64 guestId, 169 uint64 vcpuId); 170 Parameters: 171 Input: 172 flags: Reserved 173 guestId: ID obtained from H_GUEST_CREATE 174 vcpuId: ID of the vCPU to be created. This must be within the 175 range of 0 to 2047 176 Output: 177 R3: Return code. Notable: 178 H_Not_Enough_Resources: Unable to create Guest VCPU due to not 179 enough Hypervisor memory. See H_GUEST_CREATE_GET_STATE(flags = 180 takeOwnershipOfVcpuState) 181 182H_GUEST_GET_STATE() 183------------------- 184 185This is called to get state associated with an L2 (Guest-wide or vCPU specific). 186This info is passed via the Guest State Buffer (GSB), a standard format as 187explained later in this doc, necessary details below: 188 189This can get either L2 wide or vcpu specific information. Examples of 190L2 wide is the timebase offset or process scoped page table 191info. Examples of vCPU specific are GPRs or VSRs. A bit in the flags 192parameter specifies if this call is L2 wide or vCPU specific and the 193IDs in the GSB must match this. 194 195The L1 provides a pointer to the GSB as a parameter to this call. Also 196provided is the L2 and vCPU IDs associated with the state to set. 197 198The L1 writes only the IDs and sizes in the GSB. L0 writes the 199associated values for each ID in the GSB:: 200 201 H_GUEST_GET_STATE(uint64 flags, 202 uint64 guestId, 203 uint64 vcpuId, 204 uint64 dataBuffer, 205 uint64 dataBufferSizeInBytes); 206 Parameters: 207 Input: 208 flags: 209 Bit 0: getGuestWideState: Request state of the Guest instead 210 of an individual VCPU. 211 Bit 1: takeOwnershipOfVcpuState Indicate the L1 is taking 212 over ownership of the VCPU state and that the L0 can free 213 the storage holding the state. The VCPU state will need to 214 be returned to the Hypervisor via H_GUEST_SET_STATE prior 215 to H_GUEST_RUN_VCPU being called for this VCPU. The data 216 returned in the dataBuffer is in a Hypervisor internal 217 format. 218 Bits 2-63: Reserved 219 guestId: ID obtained from H_GUEST_CREATE 220 vcpuId: ID of the vCPU pass to H_GUEST_CREATE_VCPU 221 dataBuffer: A L1 real address of the GSB. 222 If takeOwnershipOfVcpuState, size must be at least the size 223 returned by ID=0x0001 224 dataBufferSizeInBytes: Size of dataBuffer 225 Output: 226 R3: Return code 227 R4: If R3 = H_Invalid_Element_Id: The array index of the bad 228 element ID. 229 If R3 = H_Invalid_Element_Size: The array index of the bad 230 element size. 231 If R3 = H_Invalid_Element_Value: The array index of the bad 232 element value. 233 234H_GUEST_SET_STATE() 235------------------- 236 237This is called to set L2 wide or vCPU specific L2 state. This info is 238passed via the Guest State Buffer (GSB), necessary details below: 239 240This can set either L2 wide or vcpu specific information. Examples of 241L2 wide is the timebase offset or process scoped page table 242info. Examples of vCPU specific are GPRs or VSRs. A bit in the flags 243parameter specifies if this call is L2 wide or vCPU specific and the 244IDs in the GSB must match this. 245 246The L1 provides a pointer to the GSB as a parameter to this call. Also 247provided is the L2 and vCPU IDs associated with the state to set. 248 249The L1 writes all values in the GSB and the L0 only reads the GSB for 250this call:: 251 252 H_GUEST_SET_STATE(uint64 flags, 253 uint64 guestId, 254 uint64 vcpuId, 255 uint64 dataBuffer, 256 uint64 dataBufferSizeInBytes); 257 Parameters: 258 Input: 259 flags: 260 Bit 0: getGuestWideState: Request state of the Guest instead 261 of an individual VCPU. 262 Bit 1: returnOwnershipOfVcpuState Return Guest VCPU state. See 263 GET_STATE takeOwnershipOfVcpuState 264 Bits 2-63: Reserved 265 guestId: ID obtained from H_GUEST_CREATE 266 vcpuId: ID of the vCPU pass to H_GUEST_CREATE_VCPU 267 dataBuffer: A L1 real address of the GSB. 268 If takeOwnershipOfVcpuState, size must be at least the size 269 returned by ID=0x0001 270 dataBufferSizeInBytes: Size of dataBuffer 271 Output: 272 R3: Return code 273 R4: If R3 = H_Invalid_Element_Id: The array index of the bad 274 element ID. 275 If R3 = H_Invalid_Element_Size: The array index of the bad 276 element size. 277 If R3 = H_Invalid_Element_Value: The array index of the bad 278 element value. 279 280H_GUEST_RUN_VCPU() 281------------------ 282 283This is called to run an L2 vCPU. The L2 and vCPU IDs are passed in as 284parameters. The vCPU runs with the state set previously using 285H_GUEST_SET_STATE(). When the L2 exits, the L1 will resume from this 286hcall. 287 288This hcall also has associated input and output GSBs. Unlike 289H_GUEST_{S,G}ET_STATE(), these GSB pointers are not passed in as 290parameters to the hcall (This was done in the interest of 291performance). The locations of these GSBs must be preregistered using 292the H_GUEST_SET_STATE() call with ID 0x0c00 and 0x0c01 (see table 293below). 294 295The input GSB may contain only VCPU specific elements to be set. This 296GSB may also contain zero elements (ie 0 in the first 4 bytes of the 297GSB) if nothing needs to be set. 298 299On exit from the hcall, the output buffer is filled with elements 300determined by the L0. The reason for the exit is contained in GPR4 (ie 301NIP is put in GPR4). The elements returned depend on the exit 302type. For example, if the exit reason is the L2 doing a hcall (GPR4 = 3030xc00), then GPR3-12 are provided in the output GSB as this is the 304state likely needed to service the hcall. If additional state is 305needed, H_GUEST_GET_STATE() may be called by the L1. 306 307To synthesize interrupts in the L2, when calling H_GUEST_RUN_VCPU() 308the L1 may set a flag (as a hcall parameter) and the L0 will 309synthesize the interrupt in the L2. Alternatively, the L1 may 310synthesize the interrupt itself using H_GUEST_SET_STATE() or the 311H_GUEST_RUN_VCPU() input GSB to set the state appropriately:: 312 313 H_GUEST_RUN_VCPU(uint64 flags, 314 uint64 guestId, 315 uint64 vcpuId, 316 uint64 dataBuffer, 317 uint64 dataBufferSizeInBytes); 318 Parameters: 319 Input: 320 flags: 321 Bit 0: generateExternalInterrupt: Generate an external interrupt 322 Bit 1: generatePrivilegedDoorbell: Generate a Privileged Doorbell 323 Bit 2: sendToSystemReset”: Generate a System Reset Interrupt 324 Bits 3-63: Reserved 325 guestId: ID obtained from H_GUEST_CREATE 326 vcpuId: ID of the vCPU pass to H_GUEST_CREATE_VCPU 327 Output: 328 R3: Return code 329 R4: If R3 = H_Success: The reason L1 VCPU exited (ie. NIA) 330 0x000: The VCPU stopped running for an unspecified reason. An 331 example of this is the Hypervisor stopping a VCPU running 332 due to an outstanding interrupt for the Host Partition. 333 0x980: HDEC 334 0xC00: HCALL 335 0xE00: HDSI 336 0xE20: HISI 337 0xE40: HEA 338 0xF80: HV Fac Unavail 339 If R3 = H_Invalid_Element_Id, H_Invalid_Element_Size, or 340 H_Invalid_Element_Value: R4 is offset of the invalid element 341 in the input buffer. 342 343H_GUEST_DELETE() 344---------------- 345 346This is called to delete an L2. All associated vCPUs are also 347deleted. No specific vCPU delete call is provided. 348 349A flag may be provided to delete all guests. This is used to reset the 350L0 in the case of kdump/kexec:: 351 352 H_GUEST_DELETE(uint64 flags, 353 uint64 guestId) 354 Parameters: 355 Input: 356 flags: 357 Bit 0: deleteAllGuests: deletes all guests 358 Bits 1-63: Reserved 359 guestId: ID obtained from H_GUEST_CREATE 360 Output: 361 R3: Return code 362 363Guest State Buffer 364================== 365 366The Guest State Buffer (GSB) is the main method of communicating state 367about the L2 between the L1 and L0 via H_GUEST_{G,S}ET() and 368H_GUEST_VCPU_RUN() calls. 369 370State may be associated with a whole L2 (eg timebase offset) or a 371specific L2 vCPU (eg. GPR state). Only L2 VCPU state maybe be set by 372H_GUEST_VCPU_RUN(). 373 374All data in the GSB is big endian (as is standard in PAPR) 375 376The Guest state buffer has a header which gives the number of 377elements, followed by the GSB elements themselves. 378 379GSB header: 380 381+----------+----------+-------------------------------------------+ 382| Offset | Size | Purpose | 383| Bytes | Bytes | | 384+==========+==========+===========================================+ 385| 0 | 4 | Number of elements | 386+----------+----------+-------------------------------------------+ 387| 4 | | Guest state buffer elements | 388+----------+----------+-------------------------------------------+ 389 390GSB element: 391 392+----------+----------+-------------------------------------------+ 393| Offset | Size | Purpose | 394| Bytes | Bytes | | 395+==========+==========+===========================================+ 396| 0 | 2 | ID | 397+----------+----------+-------------------------------------------+ 398| 2 | 2 | Size of Value | 399+----------+----------+-------------------------------------------+ 400| 4 | As above | Value | 401+----------+----------+-------------------------------------------+ 402 403The ID in the GSB element specifies what is to be set. This includes 404archtected state like GPRs, VSRs, SPRs, plus also some meta data about 405the partition like the timebase offset and partition scoped page 406table information. 407 408+--------+-------+----+--------+----------------------------------+ 409| ID | Size | RW | Thread | Details | 410| | Bytes | | Guest | | 411| | | | Scope | | 412+========+=======+====+========+==================================+ 413| 0x0000 | | RW | TG | NOP element | 414+--------+-------+----+--------+----------------------------------+ 415| 0x0001 | 0x08 | R | G | Size of L0 vCPU state. See: | 416| | | | | H_GUEST_GET_STATE: | 417| | | | | flags = takeOwnershipOfVcpuState | 418+--------+-------+----+--------+----------------------------------+ 419| 0x0002 | 0x08 | R | G | Size Run vCPU out buffer | 420+--------+-------+----+--------+----------------------------------+ 421| 0x0003 | 0x04 | RW | G | Logical PVR | 422+--------+-------+----+--------+----------------------------------+ 423| 0x0004 | 0x08 | RW | G | TB Offset (L1 relative) | 424+--------+-------+----+--------+----------------------------------+ 425| 0x0005 | 0x18 | RW | G |Partition scoped page tbl info: | 426| | | | | | 427| | | | |- 0x00 Addr part scope table | 428| | | | |- 0x08 Num addr bits | 429| | | | |- 0x10 Size root dir | 430+--------+-------+----+--------+----------------------------------+ 431| 0x0006 | 0x10 | RW | G |Process Table Information: | 432| | | | | | 433| | | | |- 0x0 Addr proc scope table | 434| | | | |- 0x8 Table size. | 435+--------+-------+----+--------+----------------------------------+ 436| 0x0007-| | | | Reserved | 437| 0x0BFF | | | | | 438+--------+-------+----+--------+----------------------------------+ 439| 0x0C00 | 0x10 | RW | T |Run vCPU Input Buffer: | 440| | | | | | 441| | | | |- 0x0 Addr of buffer | 442| | | | |- 0x8 Buffer Size. | 443+--------+-------+----+--------+----------------------------------+ 444| 0x0C01 | 0x10 | RW | T |Run vCPU Output Buffer: | 445| | | | | | 446| | | | |- 0x0 Addr of buffer | 447| | | | |- 0x8 Buffer Size. | 448+--------+-------+----+--------+----------------------------------+ 449| 0x0C02 | 0x08 | RW | T | vCPU VPA Address | 450+--------+-------+----+--------+----------------------------------+ 451| 0x0C03-| | | | Reserved | 452| 0x0FFF | | | | | 453+--------+-------+----+--------+----------------------------------+ 454| 0x1000-| 0x08 | RW | T | GPR 0-31 | 455| 0x101F | | | | | 456+--------+-------+----+--------+----------------------------------+ 457| 0x1020 | 0x08 | T | T | HDEC expiry TB | 458+--------+-------+----+--------+----------------------------------+ 459| 0x1021 | 0x08 | RW | T | NIA | 460+--------+-------+----+--------+----------------------------------+ 461| 0x1022 | 0x08 | RW | T | MSR | 462+--------+-------+----+--------+----------------------------------+ 463| 0x1023 | 0x08 | RW | T | LR | 464+--------+-------+----+--------+----------------------------------+ 465| 0x1024 | 0x08 | RW | T | XER | 466+--------+-------+----+--------+----------------------------------+ 467| 0x1025 | 0x08 | RW | T | CTR | 468+--------+-------+----+--------+----------------------------------+ 469| 0x1026 | 0x08 | RW | T | CFAR | 470+--------+-------+----+--------+----------------------------------+ 471| 0x1027 | 0x08 | RW | T | SRR0 | 472+--------+-------+----+--------+----------------------------------+ 473| 0x1028 | 0x08 | RW | T | SRR1 | 474+--------+-------+----+--------+----------------------------------+ 475| 0x1029 | 0x08 | RW | T | DAR | 476+--------+-------+----+--------+----------------------------------+ 477| 0x102A | 0x08 | RW | T | DEC expiry TB | 478+--------+-------+----+--------+----------------------------------+ 479| 0x102B | 0x08 | RW | T | VTB | 480+--------+-------+----+--------+----------------------------------+ 481| 0x102C | 0x08 | RW | T | LPCR | 482+--------+-------+----+--------+----------------------------------+ 483| 0x102D | 0x08 | RW | T | HFSCR | 484+--------+-------+----+--------+----------------------------------+ 485| 0x102E | 0x08 | RW | T | FSCR | 486+--------+-------+----+--------+----------------------------------+ 487| 0x102F | 0x08 | RW | T | FPSCR | 488+--------+-------+----+--------+----------------------------------+ 489| 0x1030 | 0x08 | RW | T | DAWR0 | 490+--------+-------+----+--------+----------------------------------+ 491| 0x1031 | 0x08 | RW | T | DAWR1 | 492+--------+-------+----+--------+----------------------------------+ 493| 0x1032 | 0x08 | RW | T | CIABR | 494+--------+-------+----+--------+----------------------------------+ 495| 0x1033 | 0x08 | RW | T | PURR | 496+--------+-------+----+--------+----------------------------------+ 497| 0x1034 | 0x08 | RW | T | SPURR | 498+--------+-------+----+--------+----------------------------------+ 499| 0x1035 | 0x08 | RW | T | IC | 500+--------+-------+----+--------+----------------------------------+ 501| 0x1036-| 0x08 | RW | T | SPRG 0-3 | 502| 0x1039 | | | | | 503+--------+-------+----+--------+----------------------------------+ 504| 0x103A | 0x08 | W | T | PPR | 505+--------+-------+----+--------+----------------------------------+ 506| 0x103B | 0x08 | RW | T | MMCR 0-3 | 507| 0x103E | | | | | 508+--------+-------+----+--------+----------------------------------+ 509| 0x103F | 0x08 | RW | T | MMCRA | 510+--------+-------+----+--------+----------------------------------+ 511| 0x1040 | 0x08 | RW | T | SIER | 512+--------+-------+----+--------+----------------------------------+ 513| 0x1041 | 0x08 | RW | T | SIER 2 | 514+--------+-------+----+--------+----------------------------------+ 515| 0x1042 | 0x08 | RW | T | SIER 3 | 516+--------+-------+----+--------+----------------------------------+ 517| 0x1043 | 0x08 | RW | T | BESCR | 518+--------+-------+----+--------+----------------------------------+ 519| 0x1044 | 0x08 | RW | T | EBBHR | 520+--------+-------+----+--------+----------------------------------+ 521| 0x1045 | 0x08 | RW | T | EBBRR | 522+--------+-------+----+--------+----------------------------------+ 523| 0x1046 | 0x08 | RW | T | AMR | 524+--------+-------+----+--------+----------------------------------+ 525| 0x1047 | 0x08 | RW | T | IAMR | 526+--------+-------+----+--------+----------------------------------+ 527| 0x1048 | 0x08 | RW | T | AMOR | 528+--------+-------+----+--------+----------------------------------+ 529| 0x1049 | 0x08 | RW | T | UAMOR | 530+--------+-------+----+--------+----------------------------------+ 531| 0x104A | 0x08 | RW | T | SDAR | 532+--------+-------+----+--------+----------------------------------+ 533| 0x104B | 0x08 | RW | T | SIAR | 534+--------+-------+----+--------+----------------------------------+ 535| 0x104C | 0x08 | RW | T | DSCR | 536+--------+-------+----+--------+----------------------------------+ 537| 0x104D | 0x08 | RW | T | TAR | 538+--------+-------+----+--------+----------------------------------+ 539| 0x104E | 0x08 | RW | T | DEXCR | 540+--------+-------+----+--------+----------------------------------+ 541| 0x104F | 0x08 | RW | T | HDEXCR | 542+--------+-------+----+--------+----------------------------------+ 543| 0x1050 | 0x08 | RW | T | HASHKEYR | 544+--------+-------+----+--------+----------------------------------+ 545| 0x1051 | 0x08 | RW | T | HASHPKEYR | 546+--------+-------+----+--------+----------------------------------+ 547| 0x1052 | 0x08 | RW | T | CTRL | 548+--------+-------+----+--------+----------------------------------+ 549| 0x1053 | 0x08 | RW | T | DPDES | 550+--------+-------+----+--------+----------------------------------+ 551| 0x1054-| | | | Reserved | 552| 0x1FFF | | | | | 553+--------+-------+----+--------+----------------------------------+ 554| 0x2000 | 0x04 | RW | T | CR | 555+--------+-------+----+--------+----------------------------------+ 556| 0x2001 | 0x04 | RW | T | PIDR | 557+--------+-------+----+--------+----------------------------------+ 558| 0x2002 | 0x04 | RW | T | DSISR | 559+--------+-------+----+--------+----------------------------------+ 560| 0x2003 | 0x04 | RW | T | VSCR | 561+--------+-------+----+--------+----------------------------------+ 562| 0x2004 | 0x04 | RW | T | VRSAVE | 563+--------+-------+----+--------+----------------------------------+ 564| 0x2005 | 0x04 | RW | T | DAWRX0 | 565+--------+-------+----+--------+----------------------------------+ 566| 0x2006 | 0x04 | RW | T | DAWRX1 | 567+--------+-------+----+--------+----------------------------------+ 568| 0x2007-| 0x04 | RW | T | PMC 1-6 | 569| 0x200c | | | | | 570+--------+-------+----+--------+----------------------------------+ 571| 0x200D | 0x04 | RW | T | WORT | 572+--------+-------+----+--------+----------------------------------+ 573| 0x200E | 0x04 | RW | T | PSPB | 574+--------+-------+----+--------+----------------------------------+ 575| 0x200F-| | | | Reserved | 576| 0x2FFF | | | | | 577+--------+-------+----+--------+----------------------------------+ 578| 0x3000-| 0x10 | RW | T | VSR 0-63 | 579| 0x303F | | | | | 580+--------+-------+----+--------+----------------------------------+ 581| 0x3040-| | | | Reserved | 582| 0xEFFF | | | | | 583+--------+-------+----+--------+----------------------------------+ 584| 0xF000 | 0x08 | R | T | HDAR | 585+--------+-------+----+--------+----------------------------------+ 586| 0xF001 | 0x04 | R | T | HDSISR | 587+--------+-------+----+--------+----------------------------------+ 588| 0xF002 | 0x04 | R | T | HEIR | 589+--------+-------+----+--------+----------------------------------+ 590| 0xF003 | 0x08 | R | T | ASDR | 591+--------+-------+----+--------+----------------------------------+ 592 593 594Miscellaneous info 595================== 596 597State not in ptregs/hvregs 598-------------------------- 599 600In the v1 API, some state is not in the ptregs/hvstate. This includes 601the vector register and some SPRs. For the L1 to set this state for 602the L2, the L1 loads up these hardware registers before the 603h_enter_nested() call and the L0 ensures they end up as the L2 state 604(by not touching them). 605 606The v2 API removes this and explicitly sets this state via the GSB. 607 608L1 Implementation details: Caching state 609---------------------------------------- 610 611In the v1 API, all state is sent from the L1 to the L0 and vice versa 612on every h_enter_nested() hcall. If the L0 is not currently running 613any L2s, the L0 has no state information about them. The only 614exception to this is the location of the partition table, registered 615via h_set_partition_table(). 616 617The v2 API changes this so that the L0 retains the L2 state even when 618it's vCPUs are no longer running. This means that the L1 only needs to 619communicate with the L0 about L2 state when it needs to modify the L2 620state, or when it's value is out of date. This provides an opportunity 621for performance optimisation. 622 623When a vCPU exits from a H_GUEST_RUN_VCPU() call, the L1 internally 624marks all L2 state as invalid. This means that if the L1 wants to know 625the L2 state (say via a kvm_get_one_reg() call), it needs call 626H_GUEST_GET_STATE() to get that state. Once it's read, it's marked as 627valid in L1 until the L2 is run again. 628 629Also, when an L1 modifies L2 vcpu state, it doesn't need to write it 630to the L0 until that L2 vcpu runs again. Hence when the L1 updates 631state (say via a kvm_set_one_reg() call), it writes to an internal L1 632copy and only flushes this copy to the L0 when the L2 runs again via 633the H_GUEST_VCPU_RUN() input buffer. 634 635This lazy updating of state by the L1 avoids unnecessary 636H_GUEST_{G|S}ET_STATE() calls. 637