1.. SPDX-License-Identifier: GPL-2.0 2 3ACPI PRM CXL Address Translation 4================================ 5 6Document 7-------- 8 9CXL Revision 3.2, Version 1.0 10 11License 12------- 13 14SPDX-License Identifier: CC-BY-4.0 15 16Creator/Contributors 17-------------------- 18 19- Robert Richter, AMD et al. 20 21Summary of the Change 22--------------------- 23 24The CXL Fixed Memory Window Structures (CFMWS) describe zero or more Host 25Physical Address (HPA) windows associated with one or more CXL Host Bridges. 26Each HPA range of a CXL Host Bridge is represented by a CFMWS entry. An HPA 27range may include addresses currently assigned to CXL.mem devices, or an OS may 28assign ranges from an address window to a device. 29 30Host-managed Device Memory is Device-attached memory that is mapped to system 31coherent address space and accessible to the Host using standard write-back 32semantics. The managed address range is configured in the CXL HDM Decoder 33registers of the device. An HDM Decoder in a device is responsible for 34converting HPA into DPA by stripping off specific address bits. 35 36CXL devices and CXL bridges use the same HPA space. It is common across all 37components that belong to the same host domain. The view of the address region 38must be consistent on the CXL.mem path between the Host and the Device. 39 40This is described in the *CXL 3.2 specification* (Table 1-1, 3.3.1, 418.2.4.20, 9.13.1, 9.18.1.3). [#cxl-spec-3.2]_ 42 43Depending on the interconnect architecture of the platform, components attached 44to a host may not share the same host physical address space. Those platforms 45need address translation to convert an HPA between the host and the attached 46component, such as a CXL device. The translation mechanism is host-specific and 47implementation dependent. 48 49For example, x86 AMD platforms use a Data Fabric that manages access to physical 50memory. Devices have their own memory space and can be configured to use 51'Normalized addresses' different from System Physical Addresses (SPA). Address 52translation is then needed. For details, see 53:doc:`x86 AMD Address Translation </admin-guide/RAS/address-translation>`. 54 55Those AMD platforms provide PRM [#prm-spec]_ handlers in firmware to perform 56various types of address translation, including for CXL endpoints. AMD Zen5 57systems implement the ACPI PRM CXL Address Translation firmware call. The ACPI 58PRM handler has a specific GUID to uniquely identify platforms with support for 59Normalized addressing. This is documented in the *ACPI v6.5 Porting Guide* 60(Address Translation - CXL DPA to System Physical Address). [#amd-ppr-58088]_ 61 62When in Normalized address mode, HDM decoder address ranges must be configured 63and handled differently. Hardware addresses used in the HDM decoder 64configurations of an endpoint are not SPA and need to be translated from the 65address range of the endpoint to that of the CXL host bridge. This is especially 66important for finding an endpoint's associated CXL Host Bridge and HPA window 67described in the CFMWS. Additionally, the interleave decoding is done by the 68Data Fabric and the endpoint does not perform decoding when converting HPA to 69DPA. Instead, interleaving is switched off for the endpoint (1-way). Finally, 70address translation might also be needed to inspect the endpoint's hardware 71addresses, such as during profiling, tracing, or error handling. 72 73For example, with Normalized addressing the HDM decoders could look as follows:: 74 75 ------------------------------- 76 | Root Decoder (CFMWS) | 77 | SPA Range: 0x850000000 | 78 | Size: 0x8000000000 (512 GB) | 79 | Interleave Ways: 1 | 80 ------------------------------- 81 | 82 v 83 ------------------------------- 84 | Host Bridge Decoder (HDM) | 85 | SPA Range: 0x850000000 | 86 | Size: 0x8000000000 (512 GB) | 87 | Interleave Ways: 4 | 88 | Targets: endpoint5,8,11,13 | 89 | Granularity: 256 | 90 ------------------------------- 91 | 92 -----------------------------+------------------------------ 93 | | | | 94 v v v v 95 ------------------- ------------------- ------------------- ------------------- 96 | endpoint5 | | endpoint8 | | endpoint11 | | endpoint13 | 97 | decoder5.0 | | decoder8.0 | | decoder11.0 | | decoder13.0 | 98 | PCIe: | | PCIe: | | PCIe: | | PCIe: | 99 | 0000:e2:00.0 | | 0000:e3:00.0 | | 0000:e4:00.0 | | 0000:e1:00.0 | 100 | DPA: | | DPA: | | DPA: | | DPA: | 101 | Start: 0x0 | | Start: 0x0 | | Start: 0x0 | | Start: 0x0 | 102 | Size: | | Size: | | Size: | | Size: | 103 | 0x2000000000 | | 0x2000000000 | | 0x2000000000 | | 0x2000000000 | 104 | (128 GB) | | (128 GB) | | (128 GB) | | (128 GB) | 105 | Interleaving: | | Interleaving: | | Interleaving: | | Interleaving: | 106 | Ways: 1 | | Ways: 1 | | Ways: 1 | | Ways: 1 | 107 | Gran: 256 | | Gran: 256 | | Gran: 256 | | Gran: 256 | 108 ------------------- ------------------- ------------------- ------------------- 109 | | | | 110 v v v v 111 DPA DPA DPA DPA 112 113This shows the representation in sysfs: 114 115.. code-block:: none 116 117 /sys/bus/cxl/devices/endpoint5/decoder5.0/interleave_granularity:256 118 /sys/bus/cxl/devices/endpoint5/decoder5.0/interleave_ways:1 119 /sys/bus/cxl/devices/endpoint5/decoder5.0/size:0x2000000000 120 /sys/bus/cxl/devices/endpoint5/decoder5.0/start:0x0 121 /sys/bus/cxl/devices/endpoint8/decoder8.0/interleave_granularity:256 122 /sys/bus/cxl/devices/endpoint8/decoder8.0/interleave_ways:1 123 /sys/bus/cxl/devices/endpoint8/decoder8.0/size:0x2000000000 124 /sys/bus/cxl/devices/endpoint8/decoder8.0/start:0x0 125 /sys/bus/cxl/devices/endpoint11/decoder11.0/interleave_granularity:256 126 /sys/bus/cxl/devices/endpoint11/decoder11.0/interleave_ways:1 127 /sys/bus/cxl/devices/endpoint11/decoder11.0/size:0x2000000000 128 /sys/bus/cxl/devices/endpoint11/decoder11.0/start:0x0 129 /sys/bus/cxl/devices/endpoint13/decoder13.0/interleave_granularity:256 130 /sys/bus/cxl/devices/endpoint13/decoder13.0/interleave_ways:1 131 /sys/bus/cxl/devices/endpoint13/decoder13.0/size:0x2000000000 132 /sys/bus/cxl/devices/endpoint13/decoder13.0/start:0x0 133 134Note that the endpoint interleaving configurations use direct mapping (1-way). 135 136With PRM calls, the kernel can determine the following mappings: 137 138.. code-block:: none 139 140 cxl decoder5.0: address mapping found for 0000:e2:00.0 (hpa -> spa): 141 0x0+0x2000000000 -> 0x850000000+0x8000000000 ways:4 granularity:256 142 cxl decoder8.0: address mapping found for 0000:e3:00.0 (hpa -> spa): 143 0x0+0x2000000000 -> 0x850000000+0x8000000000 ways:4 granularity:256 144 cxl decoder11.0: address mapping found for 0000:e4:00.0 (hpa -> spa): 145 0x0+0x2000000000 -> 0x850000000+0x8000000000 ways:4 granularity:256 146 cxl decoder13.0: address mapping found for 0000:e1:00.0 (hpa -> spa): 147 0x0+0x2000000000 -> 0x850000000+0x8000000000 ways:4 granularity:256 148 149The corresponding CXL host bridge (HDM) decoders and root decoder (CFMWS) match 150the calculated endpoint mappings shown: 151 152.. code-block:: none 153 154 /sys/bus/cxl/devices/port1/decoder1.0/interleave_granularity:256 155 /sys/bus/cxl/devices/port1/decoder1.0/interleave_ways:4 156 /sys/bus/cxl/devices/port1/decoder1.0/size:0x8000000000 157 /sys/bus/cxl/devices/port1/decoder1.0/start:0x850000000 158 /sys/bus/cxl/devices/port1/decoder1.0/target_list:0,1,2,3 159 /sys/bus/cxl/devices/port1/decoder1.0/target_type:expander 160 /sys/bus/cxl/devices/root0/decoder0.0/interleave_granularity:256 161 /sys/bus/cxl/devices/root0/decoder0.0/interleave_ways:1 162 /sys/bus/cxl/devices/root0/decoder0.0/size:0x8000000000 163 /sys/bus/cxl/devices/root0/decoder0.0/start:0x850000000 164 /sys/bus/cxl/devices/root0/decoder0.0/target_list:7 165 166The following changes to the specification are needed: 167 168* Allow a CXL device to be in an HPA space other than the host's address space. 169 170* Allow the platform to use implementation-specific address translation when 171 crossing memory domains on the CXL.mem path between the host and the device. 172 173* Define a PRM handler method for converting device addresses to SPAs. 174 175* Specify that the platform shall provide the PRM handler method to the 176 Operating System to detect Normalized addressing and for determining Endpoint 177 SPA ranges and interleaving configurations. 178 179* Add reference to: 180 181 | Platform Runtime Mechanism Specification, Version 1.1 – November 2020 182 | https://uefi.org/sites/default/files/resources/PRM_Platform_Runtime_Mechanism_1_1_release_candidate.pdf 183 184Benefits of the Change 185---------------------- 186 187Without the change, the Operating System may be unable to determine the memory 188region and Root Decoder for an Endpoint and its corresponding HDM decoder. 189Region creation would fail. Platforms with a different interconnect architecture 190would fail to set up and use CXL. 191 192References 193---------- 194 195.. [#cxl-spec-3.2] Compute Express Link Specification, Revision 3.2, Version 1.0, 196 https://www.computeexpresslink.org/ 197 198.. [#amd-ppr-58088] AMD Family 1Ah Models 00h–0Fh and Models 10h–1Fh, 199 ACPI v6.5 Porting Guide, Publication # 58088, 200 https://www.amd.com/en/search/documentation/hub.html 201 202.. [#prm-spec] Platform Runtime Mechanism, Version: 1.1, 203 https://uefi.org/sites/default/files/resources/PRM_Platform_Runtime_Mechanism_1_1_release_candidate.pdf 204 205Detailed Description of the Change 206---------------------------------- 207 208The following describes the necessary changes to the *CXL 3.2 specification* 209[#cxl-spec-3.2]_: 210 211Add the following reference to the table: 212 213Table 1-2. Reference Documents 214 215+----------------------------+-------------------+---------------------------+ 216| Document | Chapter Reference | Document No./Location | 217+============================+===================+===========================+ 218| Platform Runtime Mechanism | Chapter 8, 9 | https://www.uefi.org/acpi | 219| Version: 1.1 | | | 220+----------------------------+-------------------+---------------------------+ 221 222Add the following paragraphs to the end of the section: 223 224**8.2.4.20 CXL HDM Decoder Capability Structure** 225 226"A device may use an HPA space that is not common to other components of the 227host domain. The platform is responsible for address translation when crossing 228HPA spaces. The Operating System must determine the interleaving configuration 229and perform address translation to the HPA ranges of the HDM decoders as needed. 230The translation mechanism is host-specific and implementation dependent. 231 232The platform indicates support of independent HPA spaces and the need for 233address translation by providing a Platform Runtime Mechanism (PRM) handler. The 234OS shall use that handler to perform the necessary translations from the DPA 235space to the HPA space. The handler is defined in Section 9.18.4 *PRM Handler 236for CXL DPA to System Physical Address Translation*." 237 238Add the following section and sub-section including tables: 239 240**9.18.4 PRM Handler for CXL DPA to System Physical Address Translation** 241 242"A platform may be configured to use 'Normalized addresses'. Host physical 243address (HPA) spaces are component-specific and differ from system physical 244addresses (SPAs). The endpoint has its own physical address space. All requests 245presented to the device already use Device Physical Addresses (DPAs). The CXL 246endpoint decoders have interleaving disabled (1-way interleaving) and the device 247does not perform HPA decoding to determine a DPA. 248 249The platform provides a PRM handler for CXL DPA to System Physical Address 250Translation. The PRM handler translates a Device Physical Address (DPA) to a 251System Physical Address (SPA) for a specified CXL endpoint. In the address space 252of the host, SPA and HPA are equivalent, and the OS shall use this handler to 253determine the HPA that corresponds to a device address, for example when 254configuring HDM decoders on platforms with Normalized addressing. The GUID and 255the parameter buffer format of the handler are specified in section 9.18.4.1. If 256the OS identifies the PRM handler, the platform supports Normalized addressing 257and the OS must perform DPA address translation as needed." 258 259**9.18.4.1 PRM Handler Invocation** 260 261"The OS calls the PRM handler for CXL DPA to System Physical Address Translation 262using the direct invocation mechanism. Details of calling a PRM handler are 263described in the Platform Runtime Mechanism (PRM) specification. 264 265The PRM handler is identified by the following GUID: 266 267 EE41B397-25D4-452C-AD54-48C6E3480B94 268 269The caller allocates and prepares a Parameter Buffer, then passes the PRM 270handler GUID and a pointer to the Parameter Buffer to invoke the handler. The 271Parameter Buffer is described in Table 9-32." 272 273**Table 9-32. PRM Parameter Buffer used for CXL DPA to System Physical Address Translation** 274 275+-------------+-----------+------------------------------------------------------------------------+ 276| Byte Offset | Length in | Description | 277| | Bytes | | 278+=============+===========+========================================================================+ 279| 00h | 8 | **CXL Device Physical Address (DPA)**: CXL DPA (e.g., from | 280| | | CXL Component Event Log) | 281+-------------+-----------+------------------------------------------------------------------------+ 282| 08h | 4 | **CXL Endpoint SBDF**: | 283| | | | 284| | | - Byte 3 - PCIe Segment | 285| | | - Byte 2 - Bus Number | 286| | | - Byte 1: | 287| | | - Device Number Bits[7:3] | 288| | | - Function Number Bits[2:0] | 289| | | - Byte 0 - RESERVED (MBZ) | 290| | | | 291+-------------+-----------+------------------------------------------------------------------------+ 292| 0Ch | 8 | **Output Buffer**: Virtual Address Pointer to the buffer, | 293| | | as defined in Table 9-33. | 294+-------------+-----------+------------------------------------------------------------------------+ 295 296**Table 9-33. PRM Output Buffer used for CXL DPA to System Physical Address Translation** 297 298+-------------+-----------+------------------------------------------------------------------------+ 299| Byte Offset | Length in | Description | 300| | Bytes | | 301+=============+===========+========================================================================+ 302| 00h | 8 | **System Physical Address (SPA)**: The SPA converted | 303| | | from the CXL DPA. | 304+-------------+-----------+------------------------------------------------------------------------+ 305