1.. SPDX-License-Identifier: GPL-2.0 2 3============ 4x86 Topology 5============ 6 7This documents and clarifies the main aspects of x86 topology modelling and 8representation in the kernel. Update/change when doing changes to the 9respective code. 10 11The architecture-agnostic topology definitions are in 12Documentation/admin-guide/cputopology.rst. This file holds x86-specific 13differences/specialities which must not necessarily apply to the generic 14definitions. Thus, the way to read up on Linux topology on x86 is to start 15with the generic one and look at this one in parallel for the x86 specifics. 16 17Needless to say, code should use the generic functions - this file is *only* 18here to *document* the inner workings of x86 topology. 19 20Started by Thomas Gleixner <tglx@linutronix.de> and Borislav Petkov <bp@alien8.de>. 21 22The main aim of the topology facilities is to present adequate interfaces to 23code which needs to know/query/use the structure of the running system wrt 24threads, cores, packages, etc. 25 26The kernel does not care about the concept of physical sockets because a 27socket has no relevance to software. It's an electromechanical component. In 28the past a socket always contained a single package (see below), but with the 29advent of Multi Chip Modules (MCM) a socket can hold more than one package. So 30there might be still references to sockets in the code, but they are of 31historical nature and should be cleaned up. 32 33The topology of a system is described in the units of: 34 35 - packages 36 - cores 37 - threads 38 39Package 40======= 41Packages contain a number of cores plus shared resources, e.g. DRAM 42controller, shared caches etc. 43 44Modern systems may also use the term 'Die' for package. 45 46AMD nomenclature for package is 'Node'. 47 48Package-related topology information in the kernel: 49 50 - cpuinfo_x86.x86_max_cores: 51 52 The number of cores in a package. This information is retrieved via CPUID. 53 54 - cpuinfo_x86.x86_max_dies: 55 56 The number of dies in a package. This information is retrieved via CPUID. 57 58 - cpuinfo_x86.topo.die_id: 59 60 The physical ID of the die. This information is retrieved via CPUID. 61 62 - cpuinfo_x86.topo.pkg_id: 63 64 The physical ID of the package. This information is retrieved via CPUID 65 and deduced from the APIC IDs of the cores in the package. 66 67 Modern systems use this value for the socket. There may be multiple 68 packages within a socket. This value may differ from topo.die_id. 69 70 - cpuinfo_x86.topo.logical_pkg_id: 71 72 The logical ID of the package. As we do not trust BIOSes to enumerate the 73 packages in a consistent way, we introduced the concept of logical package 74 ID so we can sanely calculate the number of maximum possible packages in 75 the system and have the packages enumerated linearly. 76 77 - topology_max_packages(): 78 79 The maximum possible number of packages in the system. Helpful for per 80 package facilities to preallocate per package information. 81 82 - cpuinfo_x86.topo.llc_id: 83 84 - On Intel, the first APIC ID of the list of CPUs sharing the Last Level 85 Cache 86 87 - On AMD, the Node ID or Core Complex ID containing the Last Level 88 Cache. In general, it is a number identifying an LLC uniquely on the 89 system. 90 91Cores 92===== 93A core consists of 1 or more threads. It does not matter whether the threads 94are SMT- or CMT-type threads. 95 96AMDs nomenclature for a CMT core is "Compute Unit". The kernel always uses 97"core". 98 99Core-related topology information in the kernel: 100 101 - smp_num_siblings: 102 103 The number of threads in a core. The number of threads in a package can be 104 calculated by:: 105 106 threads_per_package = cpuinfo_x86.x86_max_cores * smp_num_siblings 107 108 109Threads 110======= 111A thread is a single scheduling unit. It's the equivalent to a logical Linux 112CPU. 113 114AMDs nomenclature for CMT threads is "Compute Unit Core". The kernel always 115uses "thread". 116 117Thread-related topology information in the kernel: 118 119 - topology_core_cpumask(): 120 121 The cpumask contains all online threads in the package to which a thread 122 belongs. 123 124 The number of online threads is also printed in /proc/cpuinfo "siblings." 125 126 - topology_sibling_cpumask(): 127 128 The cpumask contains all online threads in the core to which a thread 129 belongs. 130 131 - topology_logical_package_id(): 132 133 The logical package ID to which a thread belongs. 134 135 - topology_physical_package_id(): 136 137 The physical package ID to which a thread belongs. 138 139 - topology_core_id(); 140 141 The ID of the core to which a thread belongs. It is also printed in /proc/cpuinfo 142 "core_id." 143 144 145 146System topology examples 147======================== 148 149.. note:: 150 The alternative Linux CPU enumeration depends on how the BIOS enumerates the 151 threads. Many BIOSes enumerate all threads 0 first and then all threads 1. 152 That has the "advantage" that the logical Linux CPU numbers of threads 0 stay 153 the same whether threads are enabled or not. That's merely an implementation 154 detail and has no practical impact. 155 1561) Single Package, Single Core:: 157 158 [package 0] -> [core 0] -> [thread 0] -> Linux CPU 0 159 1602) Single Package, Dual Core 161 162 a) One thread per core:: 163 164 [package 0] -> [core 0] -> [thread 0] -> Linux CPU 0 165 -> [core 1] -> [thread 0] -> Linux CPU 1 166 167 b) Two threads per core:: 168 169 [package 0] -> [core 0] -> [thread 0] -> Linux CPU 0 170 -> [thread 1] -> Linux CPU 1 171 -> [core 1] -> [thread 0] -> Linux CPU 2 172 -> [thread 1] -> Linux CPU 3 173 174 Alternative enumeration:: 175 176 [package 0] -> [core 0] -> [thread 0] -> Linux CPU 0 177 -> [thread 1] -> Linux CPU 2 178 -> [core 1] -> [thread 0] -> Linux CPU 1 179 -> [thread 1] -> Linux CPU 3 180 181 AMD nomenclature for CMT systems:: 182 183 [node 0] -> [Compute Unit 0] -> [Compute Unit Core 0] -> Linux CPU 0 184 -> [Compute Unit Core 1] -> Linux CPU 1 185 -> [Compute Unit 1] -> [Compute Unit Core 0] -> Linux CPU 2 186 -> [Compute Unit Core 1] -> Linux CPU 3 187 1884) Dual Package, Dual Core 189 190 a) One thread per core:: 191 192 [package 0] -> [core 0] -> [thread 0] -> Linux CPU 0 193 -> [core 1] -> [thread 0] -> Linux CPU 1 194 195 [package 1] -> [core 0] -> [thread 0] -> Linux CPU 2 196 -> [core 1] -> [thread 0] -> Linux CPU 3 197 198 b) Two threads per core:: 199 200 [package 0] -> [core 0] -> [thread 0] -> Linux CPU 0 201 -> [thread 1] -> Linux CPU 1 202 -> [core 1] -> [thread 0] -> Linux CPU 2 203 -> [thread 1] -> Linux CPU 3 204 205 [package 1] -> [core 0] -> [thread 0] -> Linux CPU 4 206 -> [thread 1] -> Linux CPU 5 207 -> [core 1] -> [thread 0] -> Linux CPU 6 208 -> [thread 1] -> Linux CPU 7 209 210 Alternative enumeration:: 211 212 [package 0] -> [core 0] -> [thread 0] -> Linux CPU 0 213 -> [thread 1] -> Linux CPU 4 214 -> [core 1] -> [thread 0] -> Linux CPU 1 215 -> [thread 1] -> Linux CPU 5 216 217 [package 1] -> [core 0] -> [thread 0] -> Linux CPU 2 218 -> [thread 1] -> Linux CPU 6 219 -> [core 1] -> [thread 0] -> Linux CPU 3 220 -> [thread 1] -> Linux CPU 7 221 222 AMD nomenclature for CMT systems:: 223 224 [node 0] -> [Compute Unit 0] -> [Compute Unit Core 0] -> Linux CPU 0 225 -> [Compute Unit Core 1] -> Linux CPU 1 226 -> [Compute Unit 1] -> [Compute Unit Core 0] -> Linux CPU 2 227 -> [Compute Unit Core 1] -> Linux CPU 3 228 229 [node 1] -> [Compute Unit 0] -> [Compute Unit Core 0] -> Linux CPU 4 230 -> [Compute Unit Core 1] -> Linux CPU 5 231 -> [Compute Unit 1] -> [Compute Unit Core 0] -> Linux CPU 6 232 -> [Compute Unit Core 1] -> Linux CPU 7 233