1.. SPDX-License-Identifier: GPL-2.0 2 3============ 4x86 Topology 5============ 6 7This documents and clarifies the main aspects of x86 topology modelling and 8representation in the kernel. Update/change when doing changes to the 9respective code. 10 11The architecture-agnostic topology definitions are in 12Documentation/admin-guide/cputopology.rst. This file holds x86-specific 13differences/specialities which must not necessarily apply to the generic 14definitions. Thus, the way to read up on Linux topology on x86 is to start 15with the generic one and look at this one in parallel for the x86 specifics. 16 17Needless to say, code should use the generic functions - this file is *only* 18here to *document* the inner workings of x86 topology. 19 20Started by Thomas Gleixner <tglx@linutronix.de> and Borislav Petkov <bp@alien8.de>. 21 22The main aim of the topology facilities is to present adequate interfaces to 23code which needs to know/query/use the structure of the running system wrt 24threads, cores, packages, etc. 25 26The kernel does not care about the concept of physical sockets because a 27socket has no relevance to software. It's an electromechanical component. In 28the past a socket always contained a single package (see below), but with the 29advent of Multi Chip Modules (MCM) a socket can hold more than one package. So 30there might be still references to sockets in the code, but they are of 31historical nature and should be cleaned up. 32 33The topology of a system is described in the units of: 34 35 - packages 36 - cores 37 - threads 38 39Package 40======= 41Packages contain a number of cores plus shared resources, e.g. DRAM 42controller, shared caches etc. 43 44Modern systems may also use the term 'Die' for package. 45 46AMD nomenclature for package is 'Node'. 47 48Package-related topology information in the kernel: 49 50 - topology_num_threads_per_package() 51 52 The number of threads in a package. 53 54 - topology_num_cores_per_package() 55 56 The number of cores in a package. 57 58 - topology_max_dies_per_package() 59 60 The maximum number of dies in a package. 61 62 - cpuinfo_x86.topo.die_id: 63 64 The physical ID of the die. 65 66 - cpuinfo_x86.topo.pkg_id: 67 68 The physical ID of the package. This information is retrieved via CPUID 69 and deduced from the APIC IDs of the cores in the package. 70 71 Modern systems use this value for the socket. There may be multiple 72 packages within a socket. This value may differ from topo.die_id. 73 74 - cpuinfo_x86.topo.logical_pkg_id: 75 76 The logical ID of the package. As we do not trust BIOSes to enumerate the 77 packages in a consistent way, we introduced the concept of logical package 78 ID so we can sanely calculate the number of maximum possible packages in 79 the system and have the packages enumerated linearly. 80 81 - topology_max_packages(): 82 83 The maximum possible number of packages in the system. Helpful for per 84 package facilities to preallocate per package information. 85 86 - cpuinfo_x86.topo.llc_id: 87 88 - On Intel, the first APIC ID of the list of CPUs sharing the Last Level 89 Cache 90 91 - On AMD, the Node ID or Core Complex ID containing the Last Level 92 Cache. In general, it is a number identifying an LLC uniquely on the 93 system. 94 95Cores 96===== 97A core consists of 1 or more threads. It does not matter whether the threads 98are SMT- or CMT-type threads. 99 100AMDs nomenclature for a CMT core is "Compute Unit". The kernel always uses 101"core". 102 103Threads 104======= 105A thread is a single scheduling unit. It's the equivalent to a logical Linux 106CPU. 107 108AMDs nomenclature for CMT threads is "Compute Unit Core". The kernel always 109uses "thread". 110 111Thread-related topology information in the kernel: 112 113 - topology_core_cpumask(): 114 115 The cpumask contains all online threads in the package to which a thread 116 belongs. 117 118 The number of online threads is also printed in /proc/cpuinfo "siblings." 119 120 - topology_sibling_cpumask(): 121 122 The cpumask contains all online threads in the core to which a thread 123 belongs. 124 125 - topology_logical_package_id(): 126 127 The logical package ID to which a thread belongs. 128 129 - topology_physical_package_id(): 130 131 The physical package ID to which a thread belongs. 132 133 - topology_core_id(); 134 135 The ID of the core to which a thread belongs. It is also printed in /proc/cpuinfo 136 "core_id." 137 138 - topology_logical_core_id(); 139 140 The logical core ID to which a thread belongs. 141 142 143 144System topology enumeration 145=========================== 146 147The topology on x86 systems can be discovered using a combination of vendor 148specific CPUID leaves which enumerate the processor topology and the cache 149hierarchy. 150 151The CPUID leaves in their preferred order of parsing for each x86 vendor is as 152follows: 153 1541) AMD 155 156 1) CPUID leaf 0x80000026 [Extended CPU Topology] (Core::X86::Cpuid::ExCpuTopology) 157 158 The extended CPUID leaf 0x80000026 is the extension of the CPUID leaf 0xB 159 and provides the topology information of Core, Complex, CCD (Die), and 160 Socket in each level. 161 162 Support for the leaf is discovered by checking if the maximum extended 163 CPUID level is >= 0x80000026 and then checking if `LogProcAtThisLevel` 164 in `EBX[15:0]` at a particular level (starting from 0) is non-zero. 165 166 The `LevelType` in `ECX[15:8]` at the level provides the topology domain 167 the level describes - Core, Complex, CCD(Die), or the Socket. 168 169 The kernel uses the `CoreMaskWidth` from `EAX[4:0]` to discover the 170 number of bits that need to be right-shifted from `ExtendedLocalApicId` 171 in `EDX[31:0]` in order to get a unique Topology ID for the topology 172 level. CPUs with the same Topology ID share the resources at that level. 173 174 CPUID leaf 0x80000026 also provides more information regarding the power 175 and efficiency rankings, and about the core type on AMD processors with 176 heterogeneous characteristics. 177 178 If CPUID leaf 0x80000026 is supported, further parsing is not required. 179 180 2) CPUID leaf 0x0000000B [Extended Topology Enumeration] (Core::X86::Cpuid::ExtTopEnum) 181 182 The extended CPUID leaf 0x0000000B is the predecessor on the extended 183 CPUID leaf 0x80000026 and only describes the core, and the socket domains 184 of the processor topology. 185 186 The support for the leaf is discovered by checking if the maximum supported 187 CPUID level is >= 0xB and then if `EBX[31:0]` at a particular level 188 (starting from 0) is non-zero. 189 190 The `LevelType` in `ECX[15:8]` at the level provides the topology domain 191 that the level describes - Thread, or Processor (Socket). 192 193 The kernel uses the `CoreMaskWidth` from `EAX[4:0]` to discover the 194 number of bits that need to be right-shifted from the `ExtendedLocalApicId` 195 in `EDX[31:0]` to get a unique Topology ID for that topology level. CPUs 196 sharing the Topology ID share the resources at that level. 197 198 If CPUID leaf 0xB is supported, further parsing is not required. 199 200 201 3) CPUID leaf 0x80000008 ECX [Size Identifiers] (Core::X86::Cpuid::SizeId) 202 203 If neither the CPUID leaf 0x80000026 nor 0xB is supported, the number of 204 CPUs on the package is detected using the Size Identifier leaf 205 0x80000008 ECX. 206 207 The support for the leaf is discovered by checking if the supported 208 extended CPUID level is >= 0x80000008. 209 210 The shifts from the APIC ID for the Socket ID is calculated from the 211 `ApicIdSize` field in `ECX[15:12]` if it is non-zero. 212 213 If `ApicIdSize` is reported to be zero, the shift is calculated as the 214 order of the `number of threads` calculated from `NC` field in 215 `ECX[7:0]` which describes the `number of threads - 1` on the package. 216 217 Unless Extended APIC ID is supported, the APIC ID used to find the 218 Socket ID is from the `LocalApicId` field of CPUID leaf 0x00000001 219 `EBX[31:24]`. 220 221 The topology parsing continues to detect if Extended APIC ID is 222 supported or not. 223 224 225 4) CPUID leaf 0x8000001E [Extended APIC ID, Core Identifiers, Node Identifiers] 226 (Core::X86::Cpuid::{ExtApicId,CoreId,NodeId}) 227 228 The support for Extended APIC ID can be detected by checking for the 229 presence of `TopologyExtensions` in `ECX[22]` of CPUID leaf 0x80000001 230 [Feature Identifiers] (Core::X86::Cpuid::FeatureExtIdEcx). 231 232 If Topology Extensions is supported, the APIC ID from `ExtendedApicId` 233 from CPUID leaf 0x8000001E `EAX[31:0]` should be preferred over that from 234 `LocalApicId` field of CPUID leaf 0x00000001 `EBX[31:24]` for topology 235 enumeration. 236 237 On processors of Family 0x17 and above that do not support CPUID leaf 238 0x80000026 or CPUID leaf 0xB, the shifts from the APIC ID for the Core 239 ID is calculated using the order of `number of threads per core` 240 calculated using the `ThreadsPerCore` field in `EBX[15:8]` which 241 describes `number of threads per core - 1`. 242 243 On Processors of Family 0x15, the Core ID from `EBX[7:0]` is used as the 244 `cu_id` (Compute Unit ID) to detect CPUs that share the compute units. 245 246 247 All AMD processors that support the `TopologyExtensions` feature store the 248 `NodeId` from the `ECX[7:0]` of CPUID leaf 0x8000001E 249 (Core::X86::Cpuid::NodeId) as the per-CPU `node_id`. On older processors, 250 the `node_id` was discovered using MSR_FAM10H_NODE_ID MSR (MSR 251 0x0xc001_100c). The presence of the NODE_ID MSR was detected by checking 252 `ECX[19]` of CPUID leaf 0x80000001 [Feature Identifiers] 253 (Core::X86::Cpuid::FeatureExtIdEcx). 254 255 2562) Intel 257 258 On Intel platforms, the CPUID leaves that enumerate the processor 259 topology are as follows: 260 261 1) CPUID leaf 0x1F (V2 Extended Topology Enumeration Leaf) 262 263 The CPUID leaf 0x1F is the extension of the CPUID leaf 0xB and provides 264 the topology information of Core, Module, Tile, Die, DieGrp, and Socket 265 in each level. 266 267 The support for the leaf is discovered by checking if the supported 268 CPUID level is >= 0x1F and then `EBX[31:0]` at a particular level 269 (starting from 0) is non-zero. 270 271 The `Domain Type` in `ECX[15:8]` of the sub-leaf provides the topology 272 domain that the level describes - Core, Module, Tile, Die, DieGrp, and 273 Socket. 274 275 The kernel uses the value from `EAX[4:0]` to discover the number of 276 bits that need to be right shifted from the `x2APIC ID` in `EDX[31:0]` 277 to get a unique Topology ID for the topology level. CPUs with the same 278 Topology ID share the resources at that level. 279 280 If CPUID leaf 0x1F is supported, further parsing is not required. 281 282 283 2) CPUID leaf 0x0000000B (Extended Topology Enumeration Leaf) 284 285 The extended CPUID leaf 0x0000000B is the predecessor of the V2 Extended 286 Topology Enumeration Leaf 0x1F and only describes the core, and the 287 socket domains of the processor topology. 288 289 The support for the leaf is iscovered by checking if the supported CPUID 290 level is >= 0xB and then checking if `EBX[31:0]` at a particular level 291 (starting from 0) is non-zero. 292 293 CPUID leaf 0x0000000B shares the same layout as CPUID leaf 0x1F and 294 should be enumerated in a similar manner. 295 296 If CPUID leaf 0xB is supported, further parsing is not required. 297 298 299 3) CPUID leaf 0x00000004 (Deterministic Cache Parameters Leaf) 300 301 On Intel processors that support neither CPUID leaf 0x1F, nor CPUID leaf 302 0xB, the shifts for the SMT domains is calculated using the number of 303 CPUs sharing the L1 cache. 304 305 Processors that feature Hyper-Threading is detected using `EDX[28]` of 306 CPUID leaf 0x1 (Basic CPUID Information). 307 308 The order of `Maximum number of addressable IDs for logical processors 309 sharing this cache` from `EAX[25:14]` of level-0 of CPUID 0x4 provides 310 the shifts from the APIC ID required to compute the Core ID. 311 312 The APIC ID and Package information is computed using the data from 313 CPUID leaf 0x1. 314 315 316 4) CPUID leaf 0x00000001 (Basic CPUID Information) 317 318 The mask and shifts to derive the Physical Package (socket) ID is 319 computed using the `Maximum number of addressable IDs for logical 320 processors in this physical package` from `EBX[23:16]` of CPUID leaf 321 0x1. 322 323 The APIC ID on the legacy platforms is derived from the `Initial APIC 324 ID` field from `EBX[31:24]` of CPUID leaf 0x1. 325 326 3273) Centaur and Zhaoxin 328 329 Similar to Intel, Centaur and Zhaoxin use a combination of CPUID leaf 330 0x00000004 (Deterministic Cache Parameters Leaf) and CPUID leaf 0x00000001 331 (Basic CPUID Information) to derive the topology information. 332 333 334 335System topology examples 336======================== 337 338.. note:: 339 The alternative Linux CPU enumeration depends on how the BIOS enumerates the 340 threads. Many BIOSes enumerate all threads 0 first and then all threads 1. 341 That has the "advantage" that the logical Linux CPU numbers of threads 0 stay 342 the same whether threads are enabled or not. That's merely an implementation 343 detail and has no practical impact. 344 3451) Single Package, Single Core:: 346 347 [package 0] -> [core 0] -> [thread 0] -> Linux CPU 0 348 3492) Single Package, Dual Core 350 351 a) One thread per core:: 352 353 [package 0] -> [core 0] -> [thread 0] -> Linux CPU 0 354 -> [core 1] -> [thread 0] -> Linux CPU 1 355 356 b) Two threads per core:: 357 358 [package 0] -> [core 0] -> [thread 0] -> Linux CPU 0 359 -> [thread 1] -> Linux CPU 1 360 -> [core 1] -> [thread 0] -> Linux CPU 2 361 -> [thread 1] -> Linux CPU 3 362 363 Alternative enumeration:: 364 365 [package 0] -> [core 0] -> [thread 0] -> Linux CPU 0 366 -> [thread 1] -> Linux CPU 2 367 -> [core 1] -> [thread 0] -> Linux CPU 1 368 -> [thread 1] -> Linux CPU 3 369 370 AMD nomenclature for CMT systems:: 371 372 [node 0] -> [Compute Unit 0] -> [Compute Unit Core 0] -> Linux CPU 0 373 -> [Compute Unit Core 1] -> Linux CPU 1 374 -> [Compute Unit 1] -> [Compute Unit Core 0] -> Linux CPU 2 375 -> [Compute Unit Core 1] -> Linux CPU 3 376 3774) Dual Package, Dual Core 378 379 a) One thread per core:: 380 381 [package 0] -> [core 0] -> [thread 0] -> Linux CPU 0 382 -> [core 1] -> [thread 0] -> Linux CPU 1 383 384 [package 1] -> [core 0] -> [thread 0] -> Linux CPU 2 385 -> [core 1] -> [thread 0] -> Linux CPU 3 386 387 b) Two threads per core:: 388 389 [package 0] -> [core 0] -> [thread 0] -> Linux CPU 0 390 -> [thread 1] -> Linux CPU 1 391 -> [core 1] -> [thread 0] -> Linux CPU 2 392 -> [thread 1] -> Linux CPU 3 393 394 [package 1] -> [core 0] -> [thread 0] -> Linux CPU 4 395 -> [thread 1] -> Linux CPU 5 396 -> [core 1] -> [thread 0] -> Linux CPU 6 397 -> [thread 1] -> Linux CPU 7 398 399 Alternative enumeration:: 400 401 [package 0] -> [core 0] -> [thread 0] -> Linux CPU 0 402 -> [thread 1] -> Linux CPU 4 403 -> [core 1] -> [thread 0] -> Linux CPU 1 404 -> [thread 1] -> Linux CPU 5 405 406 [package 1] -> [core 0] -> [thread 0] -> Linux CPU 2 407 -> [thread 1] -> Linux CPU 6 408 -> [core 1] -> [thread 0] -> Linux CPU 3 409 -> [thread 1] -> Linux CPU 7 410 411 AMD nomenclature for CMT systems:: 412 413 [node 0] -> [Compute Unit 0] -> [Compute Unit Core 0] -> Linux CPU 0 414 -> [Compute Unit Core 1] -> Linux CPU 1 415 -> [Compute Unit 1] -> [Compute Unit Core 0] -> Linux CPU 2 416 -> [Compute Unit Core 1] -> Linux CPU 3 417 418 [node 1] -> [Compute Unit 0] -> [Compute Unit Core 0] -> Linux CPU 4 419 -> [Compute Unit Core 1] -> Linux CPU 5 420 -> [Compute Unit 1] -> [Compute Unit Core 0] -> Linux CPU 6 421 -> [Compute Unit Core 1] -> Linux CPU 7 422