xref: /linux/Documentation/arch/x86/topology.rst (revision a65879b4584f98e6c1b80380f55ca8cfca82cb47)
1.. SPDX-License-Identifier: GPL-2.0
2
3============
4x86 Topology
5============
6
7This documents and clarifies the main aspects of x86 topology modelling and
8representation in the kernel. Update/change when doing changes to the
9respective code.
10
11The architecture-agnostic topology definitions are in
12Documentation/admin-guide/cputopology.rst. This file holds x86-specific
13differences/specialities which must not necessarily apply to the generic
14definitions. Thus, the way to read up on Linux topology on x86 is to start
15with the generic one and look at this one in parallel for the x86 specifics.
16
17Needless to say, code should use the generic functions - this file is *only*
18here to *document* the inner workings of x86 topology.
19
20Started by Thomas Gleixner <tglx@linutronix.de> and Borislav Petkov <bp@alien8.de>.
21
22The main aim of the topology facilities is to present adequate interfaces to
23code which needs to know/query/use the structure of the running system wrt
24threads, cores, packages, etc.
25
26The kernel does not care about the concept of physical sockets because a
27socket has no relevance to software. It's an electromechanical component. In
28the past a socket always contained a single package (see below), but with the
29advent of Multi Chip Modules (MCM) a socket can hold more than one package. So
30there might be still references to sockets in the code, but they are of
31historical nature and should be cleaned up.
32
33The topology of a system is described in the units of:
34
35    - packages
36    - cores
37    - threads
38
39Package
40=======
41Packages contain a number of cores plus shared resources, e.g. DRAM
42controller, shared caches etc.
43
44Modern systems may also use the term 'Die' for package.
45
46AMD nomenclature for package is 'Node'.
47
48Package-related topology information in the kernel:
49
50  - topology_num_threads_per_package()
51
52    The number of threads in a package.
53
54  - topology_num_cores_per_package()
55
56    The number of cores in a package.
57
58  - topology_max_dies_per_package()
59
60    The maximum number of dies in a package.
61
62  - cpuinfo_x86.topo.die_id:
63
64    The physical ID of the die.
65
66  - cpuinfo_x86.topo.pkg_id:
67
68    The physical ID of the package. This information is retrieved via CPUID
69    and deduced from the APIC IDs of the cores in the package.
70
71    Modern systems use this value for the socket. There may be multiple
72    packages within a socket. This value may differ from topo.die_id.
73
74  - cpuinfo_x86.topo.logical_pkg_id:
75
76    The logical ID of the package. As we do not trust BIOSes to enumerate the
77    packages in a consistent way, we introduced the concept of logical package
78    ID so we can sanely calculate the number of maximum possible packages in
79    the system and have the packages enumerated linearly.
80
81  - topology_max_packages():
82
83    The maximum possible number of packages in the system. Helpful for per
84    package facilities to preallocate per package information.
85
86  - cpuinfo_x86.topo.llc_id:
87
88      - On Intel, the first APIC ID of the list of CPUs sharing the Last Level
89        Cache
90
91      - On AMD, the Node ID or Core Complex ID containing the Last Level
92        Cache. In general, it is a number identifying an LLC uniquely on the
93        system.
94
95Cores
96=====
97A core consists of 1 or more threads. It does not matter whether the threads
98are SMT- or CMT-type threads.
99
100AMDs nomenclature for a CMT core is "Compute Unit". The kernel always uses
101"core".
102
103Threads
104=======
105A thread is a single scheduling unit. It's the equivalent to a logical Linux
106CPU.
107
108AMDs nomenclature for CMT threads is "Compute Unit Core". The kernel always
109uses "thread".
110
111Thread-related topology information in the kernel:
112
113  - topology_core_cpumask():
114
115    The cpumask contains all online threads in the package to which a thread
116    belongs.
117
118    The number of online threads is also printed in /proc/cpuinfo "siblings."
119
120  - topology_sibling_cpumask():
121
122    The cpumask contains all online threads in the core to which a thread
123    belongs.
124
125  - topology_logical_package_id():
126
127    The logical package ID to which a thread belongs.
128
129  - topology_physical_package_id():
130
131    The physical package ID to which a thread belongs.
132
133  - topology_core_id();
134
135    The ID of the core to which a thread belongs. It is also printed in /proc/cpuinfo
136    "core_id."
137
138  - topology_logical_core_id();
139
140    The logical core ID to which a thread belongs.
141
142
143
144System topology enumeration
145===========================
146
147The topology on x86 systems can be discovered using a combination of vendor
148specific CPUID leaves which enumerate the processor topology and the cache
149hierarchy.
150
151The CPUID leaves in their preferred order of parsing for each x86 vendor is as
152follows:
153
1541) AMD
155
156   1) CPUID leaf 0x80000026 [Extended CPU Topology] (Core::X86::Cpuid::ExCpuTopology)
157
158      The extended CPUID leaf 0x80000026 is the extension of the CPUID leaf 0xB
159      and provides the topology information of Core, Complex, CCD (Die), and
160      Socket in each level.
161
162      Support for the leaf is discovered by checking if the maximum extended
163      CPUID level is >= 0x80000026 and then checking if `LogProcAtThisLevel`
164      in `EBX[15:0]` at a particular level (starting from 0) is non-zero.
165
166      The `LevelType` in `ECX[15:8]` at the level provides the topology domain
167      the level describes - Core, Complex, CCD(Die), or the Socket.
168
169      The kernel uses the `CoreMaskWidth` from `EAX[4:0]` to discover the
170      number of bits that need to be right-shifted from `ExtendedLocalApicId`
171      in `EDX[31:0]` in order to get a unique Topology ID for the topology
172      level. CPUs with the same Topology ID share the resources at that level.
173
174      CPUID leaf 0x80000026 also provides more information regarding the power
175      and efficiency rankings, and about the core type on AMD processors with
176      heterogeneous characteristics.
177
178      If CPUID leaf 0x80000026 is supported, further parsing is not required.
179
180   2) CPUID leaf 0x0000000B [Extended Topology Enumeration] (Core::X86::Cpuid::ExtTopEnum)
181
182      The extended CPUID leaf 0x0000000B is the predecessor on the extended
183      CPUID leaf 0x80000026 and only describes the core, and the socket domains
184      of the processor topology.
185
186      The support for the leaf is discovered by checking if the maximum supported
187      CPUID level is >= 0xB and then if `EBX[31:0]` at a particular level
188      (starting from 0) is non-zero.
189
190      The `LevelType` in `ECX[15:8]` at the level provides the topology domain
191      that the level describes - Thread, or Processor (Socket).
192
193      The kernel uses the `CoreMaskWidth` from `EAX[4:0]` to discover the
194      number of bits that need to be right-shifted from the `ExtendedLocalApicId`
195      in `EDX[31:0]` to get a unique Topology ID for that topology level. CPUs
196      sharing the Topology ID share the resources at that level.
197
198      If CPUID leaf 0xB is supported, further parsing is not required.
199
200
201   3) CPUID leaf 0x80000008 ECX [Size Identifiers] (Core::X86::Cpuid::SizeId)
202
203      If neither the CPUID leaf 0x80000026 nor 0xB is supported, the number of
204      CPUs on the package is detected using the Size Identifier leaf
205      0x80000008 ECX.
206
207      The support for the leaf is discovered by checking if the supported
208      extended CPUID level is >= 0x80000008.
209
210      The shifts from the APIC ID for the Socket ID is calculated from the
211      `ApicIdSize` field in `ECX[15:12]` if it is non-zero.
212
213      If `ApicIdSize` is reported to be zero, the shift is calculated as the
214      order of the `number of threads` calculated from `NC` field in
215      `ECX[7:0]` which describes the `number of threads - 1` on the package.
216
217      Unless Extended APIC ID is supported, the APIC ID used to find the
218      Socket ID is from the `LocalApicId` field of CPUID leaf 0x00000001
219      `EBX[31:24]`.
220
221      The topology parsing continues to detect if Extended APIC ID is
222      supported or not.
223
224
225   4) CPUID leaf 0x8000001E [Extended APIC ID, Core Identifiers, Node Identifiers]
226      (Core::X86::Cpuid::{ExtApicId,CoreId,NodeId})
227
228      The support for Extended APIC ID can be detected by checking for the
229      presence of `TopologyExtensions` in `ECX[22]` of CPUID leaf 0x80000001
230      [Feature Identifiers] (Core::X86::Cpuid::FeatureExtIdEcx).
231
232      If Topology Extensions is supported, the APIC ID from `ExtendedApicId`
233      from CPUID leaf 0x8000001E `EAX[31:0]` should be preferred over that from
234      `LocalApicId` field of CPUID leaf 0x00000001 `EBX[31:24]` for topology
235      enumeration.
236
237      On processors of Family 0x17 and above that do not support CPUID leaf
238      0x80000026 or CPUID leaf 0xB, the shifts from the APIC ID for the Core
239      ID is calculated using the order of `number of threads per core`
240      calculated using the `ThreadsPerCore` field in `EBX[15:8]` which
241      describes `number of threads per core - 1`.
242
243      On Processors of Family 0x15, the Core ID from `EBX[7:0]` is used as the
244      `cu_id` (Compute Unit ID) to detect CPUs that share the compute units.
245
246
247   All AMD processors that support the `TopologyExtensions` feature store the
248   `NodeId` from the `ECX[7:0]` of CPUID leaf 0x8000001E
249   (Core::X86::Cpuid::NodeId) as the per-CPU `node_id`. On older processors,
250   the `node_id` was discovered using MSR_FAM10H_NODE_ID MSR (MSR
251   0x0xc001_100c). The presence of the NODE_ID MSR was detected by checking
252   `ECX[19]` of CPUID leaf 0x80000001 [Feature Identifiers]
253   (Core::X86::Cpuid::FeatureExtIdEcx).
254
255
2562) Intel
257
258   On Intel platforms, the CPUID leaves that enumerate the processor
259   topology are as follows:
260
261   1) CPUID leaf 0x1F (V2 Extended Topology Enumeration Leaf)
262
263      The CPUID leaf 0x1F is the extension of the CPUID leaf 0xB and provides
264      the topology information of Core, Module, Tile, Die, DieGrp, and Socket
265      in each level.
266
267      The support for the leaf is discovered by checking if the supported
268      CPUID level is >= 0x1F and then `EBX[31:0]` at a particular level
269      (starting from 0) is non-zero.
270
271      The `Domain Type` in `ECX[15:8]` of the sub-leaf provides the topology
272      domain that the level describes - Core, Module, Tile, Die, DieGrp, and
273      Socket.
274
275      The kernel uses the value from `EAX[4:0]` to discover the number of
276      bits that need to be right shifted from the `x2APIC ID` in `EDX[31:0]`
277      to get a unique Topology ID for the topology level. CPUs with the same
278      Topology ID share the resources at that level.
279
280      If CPUID leaf 0x1F is supported, further parsing is not required.
281
282
283   2) CPUID leaf 0x0000000B (Extended Topology Enumeration Leaf)
284
285      The extended CPUID leaf 0x0000000B is the predecessor of the V2 Extended
286      Topology Enumeration Leaf 0x1F and only describes the core, and the
287      socket domains of the processor topology.
288
289      The support for the leaf is iscovered by checking if the supported CPUID
290      level is >= 0xB and then checking if `EBX[31:0]` at a particular level
291      (starting from 0) is non-zero.
292
293      CPUID leaf 0x0000000B shares the same layout as CPUID leaf 0x1F and
294      should be enumerated in a similar manner.
295
296      If CPUID leaf 0xB is supported, further parsing is not required.
297
298
299   3) CPUID leaf 0x00000004 (Deterministic Cache Parameters Leaf)
300
301      On Intel processors that support neither CPUID leaf 0x1F, nor CPUID leaf
302      0xB, the shifts for the SMT domains is calculated using the number of
303      CPUs sharing the L1 cache.
304
305      Processors that feature Hyper-Threading is detected using `EDX[28]` of
306      CPUID leaf 0x1 (Basic CPUID Information).
307
308      The order of `Maximum number of addressable IDs for logical processors
309      sharing this cache` from `EAX[25:14]` of level-0 of CPUID 0x4 provides
310      the shifts from the APIC ID required to compute the Core ID.
311
312      The APIC ID and Package information is computed using the data from
313      CPUID leaf 0x1.
314
315
316   4) CPUID leaf 0x00000001 (Basic CPUID Information)
317
318      The mask and shifts to derive the Physical Package (socket) ID is
319      computed using the `Maximum number of addressable IDs for logical
320      processors in this physical package` from `EBX[23:16]` of CPUID leaf
321      0x1.
322
323     The APIC ID on the legacy platforms is derived from the `Initial APIC
324     ID` field from `EBX[31:24]` of CPUID leaf 0x1.
325
326
3273) Centaur and Zhaoxin
328
329   Similar to Intel, Centaur and Zhaoxin use a combination of CPUID leaf
330   0x00000004 (Deterministic Cache Parameters Leaf) and CPUID leaf 0x00000001
331   (Basic CPUID Information) to derive the topology information.
332
333
334
335System topology examples
336========================
337
338.. note::
339  The alternative Linux CPU enumeration depends on how the BIOS enumerates the
340  threads. Many BIOSes enumerate all threads 0 first and then all threads 1.
341  That has the "advantage" that the logical Linux CPU numbers of threads 0 stay
342  the same whether threads are enabled or not. That's merely an implementation
343  detail and has no practical impact.
344
3451) Single Package, Single Core::
346
347   [package 0] -> [core 0] -> [thread 0] -> Linux CPU 0
348
3492) Single Package, Dual Core
350
351   a) One thread per core::
352
353	[package 0] -> [core 0] -> [thread 0] -> Linux CPU 0
354		    -> [core 1] -> [thread 0] -> Linux CPU 1
355
356   b) Two threads per core::
357
358	[package 0] -> [core 0] -> [thread 0] -> Linux CPU 0
359				-> [thread 1] -> Linux CPU 1
360		    -> [core 1] -> [thread 0] -> Linux CPU 2
361				-> [thread 1] -> Linux CPU 3
362
363      Alternative enumeration::
364
365	[package 0] -> [core 0] -> [thread 0] -> Linux CPU 0
366				-> [thread 1] -> Linux CPU 2
367		    -> [core 1] -> [thread 0] -> Linux CPU 1
368				-> [thread 1] -> Linux CPU 3
369
370      AMD nomenclature for CMT systems::
371
372	[node 0] -> [Compute Unit 0] -> [Compute Unit Core 0] -> Linux CPU 0
373				     -> [Compute Unit Core 1] -> Linux CPU 1
374		 -> [Compute Unit 1] -> [Compute Unit Core 0] -> Linux CPU 2
375				     -> [Compute Unit Core 1] -> Linux CPU 3
376
3774) Dual Package, Dual Core
378
379   a) One thread per core::
380
381	[package 0] -> [core 0] -> [thread 0] -> Linux CPU 0
382		    -> [core 1] -> [thread 0] -> Linux CPU 1
383
384	[package 1] -> [core 0] -> [thread 0] -> Linux CPU 2
385		    -> [core 1] -> [thread 0] -> Linux CPU 3
386
387   b) Two threads per core::
388
389	[package 0] -> [core 0] -> [thread 0] -> Linux CPU 0
390				-> [thread 1] -> Linux CPU 1
391		    -> [core 1] -> [thread 0] -> Linux CPU 2
392				-> [thread 1] -> Linux CPU 3
393
394	[package 1] -> [core 0] -> [thread 0] -> Linux CPU 4
395				-> [thread 1] -> Linux CPU 5
396		    -> [core 1] -> [thread 0] -> Linux CPU 6
397				-> [thread 1] -> Linux CPU 7
398
399      Alternative enumeration::
400
401	[package 0] -> [core 0] -> [thread 0] -> Linux CPU 0
402				-> [thread 1] -> Linux CPU 4
403		    -> [core 1] -> [thread 0] -> Linux CPU 1
404				-> [thread 1] -> Linux CPU 5
405
406	[package 1] -> [core 0] -> [thread 0] -> Linux CPU 2
407				-> [thread 1] -> Linux CPU 6
408		    -> [core 1] -> [thread 0] -> Linux CPU 3
409				-> [thread 1] -> Linux CPU 7
410
411      AMD nomenclature for CMT systems::
412
413	[node 0] -> [Compute Unit 0] -> [Compute Unit Core 0] -> Linux CPU 0
414				     -> [Compute Unit Core 1] -> Linux CPU 1
415		 -> [Compute Unit 1] -> [Compute Unit Core 0] -> Linux CPU 2
416				     -> [Compute Unit Core 1] -> Linux CPU 3
417
418	[node 1] -> [Compute Unit 0] -> [Compute Unit Core 0] -> Linux CPU 4
419				     -> [Compute Unit Core 1] -> Linux CPU 5
420		 -> [Compute Unit 1] -> [Compute Unit Core 0] -> Linux CPU 6
421				     -> [Compute Unit Core 1] -> Linux CPU 7
422