1.. SPDX-License-Identifier: GPL-2.0 2.. include:: <isonum.txt> 3 4===================================================== 5User Interface for Resource Control feature (resctrl) 6===================================================== 7 8:Copyright: |copy| 2016 Intel Corporation 9:Authors: - Fenghua Yu <fenghua.yu@intel.com> 10 - Tony Luck <tony.luck@intel.com> 11 - Vikas Shivappa <vikas.shivappa@intel.com> 12 13 14Intel refers to this feature as Intel Resource Director Technology(Intel(R) RDT). 15AMD refers to this feature as AMD Platform Quality of Service(AMD QoS). 16 17This feature is enabled by the CONFIG_X86_CPU_RESCTRL and the x86 /proc/cpuinfo 18flag bits: 19 20=============================================================== ================================ 21RDT (Resource Director Technology) Allocation "rdt_a" 22CAT (Cache Allocation Technology) "cat_l3", "cat_l2" 23CDP (Code and Data Prioritization) "cdp_l3", "cdp_l2" 24CQM (Cache QoS Monitoring) "cqm_llc", "cqm_occup_llc" 25MBM (Memory Bandwidth Monitoring) "cqm_mbm_total", "cqm_mbm_local" 26MBA (Memory Bandwidth Allocation) "mba" 27SMBA (Slow Memory Bandwidth Allocation) "" 28BMEC (Bandwidth Monitoring Event Configuration) "" 29ABMC (Assignable Bandwidth Monitoring Counters) "" 30SDCIAE (Smart Data Cache Injection Allocation Enforcement) "" 31=============================================================== ================================ 32 33Historically, new features were made visible by default in /proc/cpuinfo. This 34resulted in the feature flags becoming hard to parse by humans. Adding a new 35flag to /proc/cpuinfo should be avoided if user space can obtain information 36about the feature from resctrl's info directory. 37 38To use the feature mount the file system:: 39 40 # mount -t resctrl resctrl [-o cdp[,cdpl2][,mba_MBps][,debug]] /sys/fs/resctrl 41 42mount options are: 43 44"cdp": 45 Enable code/data prioritization in L3 cache allocations. 46"cdpl2": 47 Enable code/data prioritization in L2 cache allocations. 48"mba_MBps": 49 Enable the MBA Software Controller(mba_sc) to specify MBA 50 bandwidth in MiBps 51"debug": 52 Make debug files accessible. Available debug files are annotated with 53 "Available only with debug option". 54 55L2 and L3 CDP are controlled separately. 56 57RDT features are orthogonal. A particular system may support only 58monitoring, only control, or both monitoring and control. Cache 59pseudo-locking is a unique way of using cache control to "pin" or 60"lock" data in the cache. Details can be found in 61"Cache Pseudo-Locking". 62 63 64The mount succeeds if either of allocation or monitoring is present, but 65only those files and directories supported by the system will be created. 66For more details on the behavior of the interface during monitoring 67and allocation, see the "Resource alloc and monitor groups" section. 68 69Info directory 70============== 71 72The 'info' directory contains information about the enabled 73resources. Each resource has its own subdirectory. The subdirectory 74names reflect the resource names. 75 76Most of the files in the resource's subdirectory are read-only, and 77describe properties of the resource. Resources that support global 78configuration options also include writable files that can be used 79to modify those settings. 80 81Each subdirectory contains the following files with respect to 82allocation: 83 84Cache resource(L3/L2) subdirectory contains the following files 85related to allocation: 86 87"num_closids": 88 The number of CLOSIDs which are valid for this 89 resource. The kernel uses the smallest number of 90 CLOSIDs of all enabled resources as limit. 91"cbm_mask": 92 The bitmask which is valid for this resource. 93 This mask is equivalent to 100%. 94"min_cbm_bits": 95 The minimum number of consecutive bits which 96 must be set when writing a mask. 97 98"shareable_bits": 99 Bitmask of shareable resource with other executing entities 100 (e.g. I/O). Applies to all instances of this resource. User 101 can use this when setting up exclusive cache partitions. 102 Note that some platforms support devices that have their 103 own settings for cache use which can over-ride these bits. 104 105 When "io_alloc" is enabled, a portion of each cache instance can 106 be configured for shared use between hardware and software. 107 "bit_usage" should be used to see which portions of each cache 108 instance is configured for hardware use via "io_alloc" feature 109 because every cache instance can have its "io_alloc" bitmask 110 configured independently via "io_alloc_cbm". 111 112"bit_usage": 113 Annotated capacity bitmasks showing how all 114 instances of the resource are used. The legend is: 115 116 "0": 117 Corresponding region is unused. When the system's 118 resources have been allocated and a "0" is found 119 in "bit_usage" it is a sign that resources are 120 wasted. 121 122 "H": 123 Corresponding region is used by hardware only 124 but available for software use. If a resource 125 has bits set in "shareable_bits" or "io_alloc_cbm" 126 but not all of these bits appear in the resource 127 groups' schemata then the bits appearing in 128 "shareable_bits" or "io_alloc_cbm" but no 129 resource group will be marked as "H". 130 "X": 131 Corresponding region is available for sharing and 132 used by hardware and software. These are the bits 133 that appear in "shareable_bits" or "io_alloc_cbm" 134 as well as a resource group's allocation. 135 "S": 136 Corresponding region is used by software 137 and available for sharing. 138 "E": 139 Corresponding region is used exclusively by 140 one resource group. No sharing allowed. 141 "P": 142 Corresponding region is pseudo-locked. No 143 sharing allowed. 144"sparse_masks": 145 Indicates if non-contiguous 1s value in CBM is supported. 146 147 "0": 148 Only contiguous 1s value in CBM is supported. 149 "1": 150 Non-contiguous 1s value in CBM is supported. 151 152"io_alloc": 153 "io_alloc" enables system software to configure the portion of 154 the cache allocated for I/O traffic. File may only exist if the 155 system supports this feature on some of its cache resources. 156 157 "disabled": 158 Resource supports "io_alloc" but the feature is disabled. 159 Portions of cache used for allocation of I/O traffic cannot 160 be configured. 161 "enabled": 162 Portions of cache used for allocation of I/O traffic 163 can be configured using "io_alloc_cbm". 164 "not supported": 165 Support not available for this resource. 166 167 The feature can be modified by writing to the interface, for example: 168 169 To enable:: 170 171 # echo 1 > /sys/fs/resctrl/info/L3/io_alloc 172 173 To disable:: 174 175 # echo 0 > /sys/fs/resctrl/info/L3/io_alloc 176 177 The underlying implementation may reduce resources available to 178 general (CPU) cache allocation. See architecture specific notes 179 below. Depending on usage requirements the feature can be enabled 180 or disabled. 181 182 On AMD systems, io_alloc feature is supported by the L3 Smart 183 Data Cache Injection Allocation Enforcement (SDCIAE). The CLOSID for 184 io_alloc is the highest CLOSID supported by the resource. When 185 io_alloc is enabled, the highest CLOSID is dedicated to io_alloc and 186 no longer available for general (CPU) cache allocation. When CDP is 187 enabled, io_alloc routes I/O traffic using the highest CLOSID allocated 188 for the instruction cache (CDP_CODE), making this CLOSID no longer 189 available for general (CPU) cache allocation for both the CDP_CODE 190 and CDP_DATA resources. 191 192"io_alloc_cbm": 193 Capacity bitmasks that describe the portions of cache instances to 194 which I/O traffic from supported I/O devices are routed when "io_alloc" 195 is enabled. 196 197 CBMs are displayed in the following format: 198 199 <cache_id0>=<cbm>;<cache_id1>=<cbm>;... 200 201 Example:: 202 203 # cat /sys/fs/resctrl/info/L3/io_alloc_cbm 204 0=ffff;1=ffff 205 206 CBMs can be configured by writing to the interface. 207 208 Example:: 209 210 # echo 1=ff > /sys/fs/resctrl/info/L3/io_alloc_cbm 211 # cat /sys/fs/resctrl/info/L3/io_alloc_cbm 212 0=ffff;1=00ff 213 214 # echo "0=ff;1=f" > /sys/fs/resctrl/info/L3/io_alloc_cbm 215 # cat /sys/fs/resctrl/info/L3/io_alloc_cbm 216 0=00ff;1=000f 217 218 An ID of "*" configures all domains with the provided CBM. 219 220 Example on a system that does not require a minimum number of consecutive bits in the mask:: 221 222 # echo "*=0" > /sys/fs/resctrl/info/L3/io_alloc_cbm 223 # cat /sys/fs/resctrl/info/L3/io_alloc_cbm 224 0=0;1=0 225 226 When CDP is enabled "io_alloc_cbm" associated with the CDP_DATA and CDP_CODE 227 resources may reflect the same values. For example, values read from and 228 written to /sys/fs/resctrl/info/L3DATA/io_alloc_cbm may be reflected by 229 /sys/fs/resctrl/info/L3CODE/io_alloc_cbm and vice versa. 230 231Memory bandwidth(MB) subdirectory contains the following files 232with respect to allocation: 233 234"min_bandwidth": 235 The minimum memory bandwidth percentage which 236 user can request. 237 238"bandwidth_gran": 239 The approximate granularity in which the memory bandwidth 240 percentage is allocated. The allocated bandwidth percentage is 241 rounded up or down to the closest control step available on the 242 hardware. The available hardware steps are no larger than this 243 value. 244 245"delay_linear": 246 Indicates if the delay scale is linear or 247 non-linear. This field is purely informational 248 only. 249 250"thread_throttle_mode": 251 Indicator on Intel systems of how tasks running on threads 252 of a physical core are throttled in cases where they 253 request different memory bandwidth percentages: 254 255 "max": 256 the smallest percentage is applied 257 to all threads 258 "per-thread": 259 bandwidth percentages are directly applied to 260 the threads running on the core 261 262If L3 monitoring is available there will be an "L3_MON" directory 263with the following files: 264 265"num_rmids": 266 The number of RMIDs supported by hardware for 267 L3 monitoring events. 268 269"mon_features": 270 Lists the monitoring events if 271 monitoring is enabled for the resource. 272 Example:: 273 274 # cat /sys/fs/resctrl/info/L3_MON/mon_features 275 llc_occupancy 276 mbm_total_bytes 277 mbm_local_bytes 278 279 If the system supports Bandwidth Monitoring Event 280 Configuration (BMEC), then the bandwidth events will 281 be configurable. The output will be:: 282 283 # cat /sys/fs/resctrl/info/L3_MON/mon_features 284 llc_occupancy 285 mbm_total_bytes 286 mbm_total_bytes_config 287 mbm_local_bytes 288 mbm_local_bytes_config 289 290"mbm_total_bytes_config", "mbm_local_bytes_config": 291 Read/write files containing the configuration for the mbm_total_bytes 292 and mbm_local_bytes events, respectively, when the Bandwidth 293 Monitoring Event Configuration (BMEC) feature is supported. 294 The event configuration settings are domain specific and affect 295 all the CPUs in the domain. When either event configuration is 296 changed, the bandwidth counters for all RMIDs of both events 297 (mbm_total_bytes as well as mbm_local_bytes) are cleared for that 298 domain. The next read for every RMID will report "Unavailable" 299 and subsequent reads will report the valid value. 300 301 Following are the types of events supported: 302 303 ==== ======================================================== 304 Bits Description 305 ==== ======================================================== 306 6 Dirty Victims from the QOS domain to all types of memory 307 5 Reads to slow memory in the non-local NUMA domain 308 4 Reads to slow memory in the local NUMA domain 309 3 Non-temporal writes to non-local NUMA domain 310 2 Non-temporal writes to local NUMA domain 311 1 Reads to memory in the non-local NUMA domain 312 0 Reads to memory in the local NUMA domain 313 ==== ======================================================== 314 315 By default, the mbm_total_bytes configuration is set to 0x7f to count 316 all the event types and the mbm_local_bytes configuration is set to 317 0x15 to count all the local memory events. 318 319 Examples: 320 321 * To view the current configuration:: 322 :: 323 324 # cat /sys/fs/resctrl/info/L3_MON/mbm_total_bytes_config 325 0=0x7f;1=0x7f;2=0x7f;3=0x7f 326 327 # cat /sys/fs/resctrl/info/L3_MON/mbm_local_bytes_config 328 0=0x15;1=0x15;3=0x15;4=0x15 329 330 * To change the mbm_total_bytes to count only reads on domain 0, 331 the bits 0, 1, 4 and 5 needs to be set, which is 110011b in binary 332 (in hexadecimal 0x33): 333 :: 334 335 # echo "0=0x33" > /sys/fs/resctrl/info/L3_MON/mbm_total_bytes_config 336 337 # cat /sys/fs/resctrl/info/L3_MON/mbm_total_bytes_config 338 0=0x33;1=0x7f;2=0x7f;3=0x7f 339 340 * To change the mbm_local_bytes to count all the slow memory reads on 341 domain 0 and 1, the bits 4 and 5 needs to be set, which is 110000b 342 in binary (in hexadecimal 0x30): 343 :: 344 345 # echo "0=0x30;1=0x30" > /sys/fs/resctrl/info/L3_MON/mbm_local_bytes_config 346 347 # cat /sys/fs/resctrl/info/L3_MON/mbm_local_bytes_config 348 0=0x30;1=0x30;3=0x15;4=0x15 349 350"mbm_assign_mode": 351 The supported counter assignment modes. The enclosed brackets indicate which mode 352 is enabled. The MBM events associated with counters may reset when "mbm_assign_mode" 353 is changed. 354 :: 355 356 # cat /sys/fs/resctrl/info/L3_MON/mbm_assign_mode 357 [mbm_event] 358 default 359 360 "mbm_event": 361 362 mbm_event mode allows users to assign a hardware counter to an RMID, event 363 pair and monitor the bandwidth usage as long as it is assigned. The hardware 364 continues to track the assigned counter until it is explicitly unassigned by 365 the user. Each event within a resctrl group can be assigned independently. 366 367 In this mode, a monitoring event can only accumulate data while it is backed 368 by a hardware counter. Use "mbm_L3_assignments" found in each CTRL_MON and MON 369 group to specify which of the events should have a counter assigned. The number 370 of counters available is described in the "num_mbm_cntrs" file. Changing the 371 mode may cause all counters on the resource to reset. 372 373 Moving to mbm_event counter assignment mode requires users to assign the counters 374 to the events. Otherwise, the MBM event counters will return 'Unassigned' when read. 375 376 The mode is beneficial for AMD platforms that support more CTRL_MON 377 and MON groups than available hardware counters. By default, this 378 feature is enabled on AMD platforms with the ABMC (Assignable Bandwidth 379 Monitoring Counters) capability, ensuring counters remain assigned even 380 when the corresponding RMID is not actively used by any processor. 381 382 "default": 383 384 In default mode, resctrl assumes there is a hardware counter for each 385 event within every CTRL_MON and MON group. On AMD platforms, it is 386 recommended to use the mbm_event mode, if supported, to prevent reset of MBM 387 events between reads resulting from hardware re-allocating counters. This can 388 result in misleading values or display "Unavailable" if no counter is assigned 389 to the event. 390 391 * To enable "mbm_event" counter assignment mode: 392 :: 393 394 # echo "mbm_event" > /sys/fs/resctrl/info/L3_MON/mbm_assign_mode 395 396 * To enable "default" monitoring mode: 397 :: 398 399 # echo "default" > /sys/fs/resctrl/info/L3_MON/mbm_assign_mode 400 401"num_mbm_cntrs": 402 The maximum number of counters (total of available and assigned counters) in 403 each domain when the system supports mbm_event mode. 404 405 For example, on a system with maximum of 32 memory bandwidth monitoring 406 counters in each of its L3 domains: 407 :: 408 409 # cat /sys/fs/resctrl/info/L3_MON/num_mbm_cntrs 410 0=32;1=32 411 412"available_mbm_cntrs": 413 The number of counters available for assignment in each domain when mbm_event 414 mode is enabled on the system. 415 416 For example, on a system with 30 available [hardware] assignable counters 417 in each of its L3 domains: 418 :: 419 420 # cat /sys/fs/resctrl/info/L3_MON/available_mbm_cntrs 421 0=30;1=30 422 423"event_configs": 424 Directory that exists when "mbm_event" counter assignment mode is supported. 425 Contains a sub-directory for each MBM event that can be assigned to a counter. 426 427 Two MBM events are supported by default: mbm_local_bytes and mbm_total_bytes. 428 Each MBM event's sub-directory contains a file named "event_filter" that is 429 used to view and (if writable) modify which memory transactions the MBM event 430 is configured with. The file is accessible only when "mbm_event" counter 431 assignment mode is enabled. 432 433 List of memory transaction types supported: 434 435 ========================== ======================================================== 436 Name Description 437 ========================== ======================================================== 438 dirty_victim_writes_all Dirty Victims from the QOS domain to all types of memory 439 remote_reads_slow_memory Reads to slow memory in the non-local NUMA domain 440 local_reads_slow_memory Reads to slow memory in the local NUMA domain 441 remote_non_temporal_writes Non-temporal writes to non-local NUMA domain 442 local_non_temporal_writes Non-temporal writes to local NUMA domain 443 remote_reads Reads to memory in the non-local NUMA domain 444 local_reads Reads to memory in the local NUMA domain 445 ========================== ======================================================== 446 447 For example:: 448 449 # cat /sys/fs/resctrl/info/L3_MON/event_configs/mbm_total_bytes/event_filter 450 local_reads,remote_reads,local_non_temporal_writes,remote_non_temporal_writes, 451 local_reads_slow_memory,remote_reads_slow_memory,dirty_victim_writes_all 452 453 # cat /sys/fs/resctrl/info/L3_MON/event_configs/mbm_local_bytes/event_filter 454 local_reads,local_non_temporal_writes,local_reads_slow_memory 455 456 The memory transactions the MBM event is configured with can be changed 457 if "event_filter" is writable. 458 459 For example:: 460 461 # echo "local_reads, local_non_temporal_writes" > 462 /sys/fs/resctrl/info/L3_MON/event_configs/mbm_total_bytes/event_filter 463 464 # cat /sys/fs/resctrl/info/L3_MON/event_configs/mbm_total_bytes/event_filter 465 local_reads,local_non_temporal_writes 466 467"mbm_assign_on_mkdir": 468 Exists when "mbm_event" counter assignment mode is supported. Accessible 469 only when "mbm_event" counter assignment mode is enabled. 470 471 Determines if a counter will automatically be assigned to an RMID, MBM event 472 pair when its associated monitor group is created via mkdir. Enabled by default 473 on boot, also when switched from "default" mode to "mbm_event" counter assignment 474 mode. Users can disable this capability by writing to the interface. 475 476 "0": 477 Auto assignment is disabled. 478 "1": 479 Auto assignment is enabled. 480 481 Automatic counter assignment is done with best effort. If auto 482 assignment is enabled but there are not enough available counters then 483 monitor group creation could succeed while one or more events belonging 484 to the group may not have a counter assigned in all domains. Consult 485 mbm_L3_assignments for counter assignment states of the new groups. 486 487 Example:: 488 489 # echo 0 > /sys/fs/resctrl/info/L3_MON/mbm_assign_on_mkdir 490 # cat /sys/fs/resctrl/info/L3_MON/mbm_assign_on_mkdir 491 0 492 493"max_threshold_occupancy": 494 Read/write file provides the largest value (in 495 bytes) at which a previously used LLC_occupancy 496 counter can be considered for reuse. 497 498If telemetry monitoring is available there will be a "PERF_PKG_MON" directory 499with the following files: 500 501"num_rmids": 502 The number of RMIDs for telemetry monitoring events. 503 504 On Intel resctrl will not enable telemetry events if the number of 505 RMIDs that can be tracked concurrently is lower than the total number 506 of RMIDs supported. Telemetry events can be force-enabled with the 507 "rdt=" kernel parameter, but this may reduce the number of 508 monitoring groups that can be created. 509 510"mon_features": 511 Lists the telemetry monitoring events that are enabled on this system. 512 513The upper bound for how many "CTRL_MON" + "MON" can be created 514is the smaller of the L3_MON and PERF_PKG_MON "num_rmids" values. 515 516Finally, in the top level of the "info" directory there is a file 517named "last_cmd_status". This is reset with every "command" issued 518via the file system (making new directories or writing to any of the 519control files). If the command was successful, it will read as "ok". 520If the command failed, it will provide more information that can be 521conveyed in the error returns from file operations. E.g. 522:: 523 524 # echo L3:0=f7 > schemata 525 bash: echo: write error: Invalid argument 526 # cat info/last_cmd_status 527 mask f7 has non-consecutive 1-bits 528 529Resource alloc and monitor groups 530================================= 531 532Resource groups are represented as directories in the resctrl file 533system. The default group is the root directory which, immediately 534after mounting, owns all the tasks and cpus in the system and can make 535full use of all resources. 536 537On a system with RDT control features additional directories can be 538created in the root directory that specify different amounts of each 539resource (see "schemata" below). The root and these additional top level 540directories are referred to as "CTRL_MON" groups below. 541 542On a system with RDT monitoring the root directory and other top level 543directories contain a directory named "mon_groups" in which additional 544directories can be created to monitor subsets of tasks in the CTRL_MON 545group that is their ancestor. These are called "MON" groups in the rest 546of this document. 547 548Removing a directory will move all tasks and cpus owned by the group it 549represents to the parent. Removing one of the created CTRL_MON groups 550will automatically remove all MON groups below it. 551 552Moving MON group directories to a new parent CTRL_MON group is supported 553for the purpose of changing the resource allocations of a MON group 554without impacting its monitoring data or assigned tasks. This operation 555is not allowed for MON groups which monitor CPUs. No other move 556operation is currently allowed other than simply renaming a CTRL_MON or 557MON group. 558 559All groups contain the following files: 560 561"tasks": 562 Reading this file shows the list of all tasks that belong to 563 this group. Writing a task id to the file will add a task to the 564 group. Multiple tasks can be added by separating the task ids 565 with commas. Tasks will be assigned sequentially. Multiple 566 failures are not supported. A single failure encountered while 567 attempting to assign a task will cause the operation to abort and 568 already added tasks before the failure will remain in the group. 569 Failures will be logged to /sys/fs/resctrl/info/last_cmd_status. 570 571 If the group is a CTRL_MON group the task is removed from 572 whichever previous CTRL_MON group owned the task and also from 573 any MON group that owned the task. If the group is a MON group, 574 then the task must already belong to the CTRL_MON parent of this 575 group. The task is removed from any previous MON group. 576 577 When writing to this file, a task id of 0 is interpreted as the 578 task id of the currently running task. On reading the file, a task 579 id of 0 will never be shown and there is no representation of the 580 idle tasks. Instead, a CPU's idle task is always considered as a 581 member of the group owning the CPU. 582 583"cpus": 584 Reading this file shows a bitmask of the logical CPUs owned by 585 this group. Writing a mask to this file will add and remove 586 CPUs to/from this group. As with the tasks file a hierarchy is 587 maintained where MON groups may only include CPUs owned by the 588 parent CTRL_MON group. 589 When the resource group is in pseudo-locked mode this file will 590 only be readable, reflecting the CPUs associated with the 591 pseudo-locked region. 592 593 594"cpus_list": 595 Just like "cpus", only using ranges of CPUs instead of bitmasks. 596 597 598When control is enabled all CTRL_MON groups will also contain: 599 600"schemata": 601 A list of all the resources available to this group. 602 Each resource has its own line and format - see below for details. 603 604"size": 605 Mirrors the display of the "schemata" file to display the size in 606 bytes of each allocation instead of the bits representing the 607 allocation. 608 609"mode": 610 The "mode" of the resource group dictates the sharing of its 611 allocations. A "shareable" resource group allows sharing of its 612 allocations while an "exclusive" resource group does not. A 613 cache pseudo-locked region is created by first writing 614 "pseudo-locksetup" to the "mode" file before writing the cache 615 pseudo-locked region's schemata to the resource group's "schemata" 616 file. On successful pseudo-locked region creation the mode will 617 automatically change to "pseudo-locked". 618 619"ctrl_hw_id": 620 Available only with debug option. The identifier used by hardware 621 for the control group. On x86 this is the CLOSID. 622 623When monitoring is enabled all MON groups will also contain: 624 625"mon_data": 626 This contains directories for each monitor domain. 627 628 If L3 monitoring is enabled, there will be a "mon_L3_XX" directory for 629 each instance of an L3 cache. Each directory contains files for the enabled 630 L3 events (e.g. "llc_occupancy", "mbm_total_bytes", and "mbm_local_bytes"). 631 632 If telemetry monitoring is enabled, there will be a "mon_PERF_PKG_YY" 633 directory for each physical processor package. Each directory contains 634 files for the enabled telemetry events (e.g. "core_energy". "activity", 635 "uops_retired", etc.) 636 637 The info/`*`/mon_features files provide the full list of enabled 638 event/file names. 639 640 "core energy" reports a floating point number for the energy (in Joules) 641 consumed by cores (registers, arithmetic units, TLB and L1/L2 caches) 642 during execution of instructions summed across all logical CPUs on a 643 package for the current monitoring group. 644 645 "activity" also reports a floating point value (in nanofarads). This provides 646 an estimate of work done independent of the frequency that the CPUs used 647 for execution. 648 649 Note that "core energy" and "activity" only measure energy/activity in the 650 "core" of the CPU (arithmetic units, TLB, L1 and L2 caches, etc.). They 651 do not include L3 cache, memory, I/O devices etc. 652 653 All other events report decimal integer values. 654 655 In a MON group these files provide a read out of the current value of 656 the event for all tasks in the group. In CTRL_MON groups these files 657 provide the sum for all tasks in the CTRL_MON group and all tasks in 658 MON groups. Please see example section for more details on usage. 659 660 On systems with Sub-NUMA Cluster (SNC) enabled there are extra 661 directories for each node (located within the "mon_L3_XX" directory 662 for the L3 cache they occupy). These are named "mon_sub_L3_YY" 663 where "YY" is the node number. 664 665 When the 'mbm_event' counter assignment mode is enabled, reading 666 an MBM event of a MON group returns 'Unassigned' if no hardware 667 counter is assigned to it. For CTRL_MON groups, 'Unassigned' is 668 returned if the MBM event does not have an assigned counter in the 669 CTRL_MON group nor in any of its associated MON groups. 670 671"mon_hw_id": 672 Available only with debug option. The identifier used by hardware 673 for the monitor group. On x86 this is the RMID. 674 675When monitoring is enabled all MON groups may also contain: 676 677"mbm_L3_assignments": 678 Exists when "mbm_event" counter assignment mode is supported and lists the 679 counter assignment states of the group. 680 681 The assignment list is displayed in the following format: 682 683 <Event>:<Domain ID>=<Assignment state>;<Domain ID>=<Assignment state> 684 685 Event: A valid MBM event in the 686 /sys/fs/resctrl/info/L3_MON/event_configs directory. 687 688 Domain ID: A valid domain ID. When writing, '*' applies the changes 689 to all the domains. 690 691 Assignment states: 692 693 _ : No counter assigned. 694 695 e : Counter assigned exclusively. 696 697 Example: 698 699 To display the counter assignment states for the default group. 700 :: 701 702 # cd /sys/fs/resctrl 703 # cat /sys/fs/resctrl/mbm_L3_assignments 704 mbm_total_bytes:0=e;1=e 705 mbm_local_bytes:0=e;1=e 706 707 Assignments can be modified by writing to the interface. 708 709 Examples: 710 711 To unassign the counter associated with the mbm_total_bytes event on domain 0: 712 :: 713 714 # echo "mbm_total_bytes:0=_" > /sys/fs/resctrl/mbm_L3_assignments 715 # cat /sys/fs/resctrl/mbm_L3_assignments 716 mbm_total_bytes:0=_;1=e 717 mbm_local_bytes:0=e;1=e 718 719 To unassign the counter associated with the mbm_total_bytes event on all the domains: 720 :: 721 722 # echo "mbm_total_bytes:*=_" > /sys/fs/resctrl/mbm_L3_assignments 723 # cat /sys/fs/resctrl/mbm_L3_assignments 724 mbm_total_bytes:0=_;1=_ 725 mbm_local_bytes:0=e;1=e 726 727 To assign a counter associated with the mbm_total_bytes event on all domains in 728 exclusive mode: 729 :: 730 731 # echo "mbm_total_bytes:*=e" > /sys/fs/resctrl/mbm_L3_assignments 732 # cat /sys/fs/resctrl/mbm_L3_assignments 733 mbm_total_bytes:0=e;1=e 734 mbm_local_bytes:0=e;1=e 735 736When the "mba_MBps" mount option is used all CTRL_MON groups will also contain: 737 738"mba_MBps_event": 739 Reading this file shows which memory bandwidth event is used 740 as input to the software feedback loop that keeps memory bandwidth 741 below the value specified in the schemata file. Writing the 742 name of one of the supported memory bandwidth events found in 743 /sys/fs/resctrl/info/L3_MON/mon_features changes the input 744 event. 745 746Resource allocation rules 747------------------------- 748 749When a task is running the following rules define which resources are 750available to it: 751 7521) If the task is a member of a non-default group, then the schemata 753 for that group is used. 754 7552) Else if the task belongs to the default group, but is running on a 756 CPU that is assigned to some specific group, then the schemata for the 757 CPU's group is used. 758 7593) Otherwise the schemata for the default group is used. 760 761Resource monitoring rules 762------------------------- 7631) If a task is a member of a MON group, or non-default CTRL_MON group 764 then RDT events for the task will be reported in that group. 765 7662) If a task is a member of the default CTRL_MON group, but is running 767 on a CPU that is assigned to some specific group, then the RDT events 768 for the task will be reported in that group. 769 7703) Otherwise RDT events for the task will be reported in the root level 771 "mon_data" group. 772 773 774Notes on cache occupancy monitoring and control 775=============================================== 776When moving a task from one group to another you should remember that 777this only affects *new* cache allocations by the task. E.g. you may have 778a task in a monitor group showing 3 MB of cache occupancy. If you move 779to a new group and immediately check the occupancy of the old and new 780groups you will likely see that the old group is still showing 3 MB and 781the new group zero. When the task accesses locations still in cache from 782before the move, the h/w does not update any counters. On a busy system 783you will likely see the occupancy in the old group go down as cache lines 784are evicted and re-used while the occupancy in the new group rises as 785the task accesses memory and loads into the cache are counted based on 786membership in the new group. 787 788The same applies to cache allocation control. Moving a task to a group 789with a smaller cache partition will not evict any cache lines. The 790process may continue to use them from the old partition. 791 792Hardware uses CLOSid(Class of service ID) and an RMID(Resource monitoring ID) 793to identify a control group and a monitoring group respectively. Each of 794the resource groups are mapped to these IDs based on the kind of group. The 795number of CLOSid and RMID are limited by the hardware and hence the creation of 796a "CTRL_MON" directory may fail if we run out of either CLOSID or RMID 797and creation of "MON" group may fail if we run out of RMIDs. 798 799max_threshold_occupancy - generic concepts 800------------------------------------------ 801 802Note that an RMID once freed may not be immediately available for use as 803the RMID is still tagged the cache lines of the previous user of RMID. 804Hence such RMIDs are placed on limbo list and checked back if the cache 805occupancy has gone down. If there is a time when system has a lot of 806limbo RMIDs but which are not ready to be used, user may see an -EBUSY 807during mkdir. 808 809max_threshold_occupancy is a user configurable value to determine the 810occupancy at which an RMID can be freed. 811 812The mon_llc_occupancy_limbo tracepoint gives the precise occupancy in bytes 813for a subset of RMID that are not immediately available for allocation. 814This can't be relied on to produce output every second, it may be necessary 815to attempt to create an empty monitor group to force an update. Output may 816only be produced if creation of a control or monitor group fails. 817 818Schemata files - general concepts 819--------------------------------- 820Each line in the file describes one resource. The line starts with 821the name of the resource, followed by specific values to be applied 822in each of the instances of that resource on the system. 823 824Cache IDs 825--------- 826On current generation systems there is one L3 cache per socket and L2 827caches are generally just shared by the hyperthreads on a core, but this 828isn't an architectural requirement. We could have multiple separate L3 829caches on a socket, multiple cores could share an L2 cache. So instead 830of using "socket" or "core" to define the set of logical cpus sharing 831a resource we use a "Cache ID". At a given cache level this will be a 832unique number across the whole system (but it isn't guaranteed to be a 833contiguous sequence, there may be gaps). To find the ID for each logical 834CPU look in /sys/devices/system/cpu/cpu*/cache/index*/id 835 836Cache Bit Masks (CBM) 837--------------------- 838For cache resources we describe the portion of the cache that is available 839for allocation using a bitmask. The maximum value of the mask is defined 840by each cpu model (and may be different for different cache levels). It 841is found using CPUID, but is also provided in the "info" directory of 842the resctrl file system in "info/{resource}/cbm_mask". Some Intel hardware 843requires that these masks have all the '1' bits in a contiguous block. So 8440x3, 0x6 and 0xC are legal 4-bit masks with two bits set, but 0x5, 0x9 845and 0xA are not. Check /sys/fs/resctrl/info/{resource}/sparse_masks 846if non-contiguous 1s value is supported. On a system with a 20-bit mask 847each bit represents 5% of the capacity of the cache. You could partition 848the cache into four equal parts with masks: 0x1f, 0x3e0, 0x7c00, 0xf8000. 849 850Notes on Sub-NUMA Cluster mode 851============================== 852When SNC mode is enabled, Linux may load balance tasks between Sub-NUMA 853nodes much more readily than between regular NUMA nodes since the CPUs 854on Sub-NUMA nodes share the same L3 cache and the system may report 855the NUMA distance between Sub-NUMA nodes with a lower value than used 856for regular NUMA nodes. 857 858The top-level monitoring files in each "mon_L3_XX" directory provide 859the sum of data across all SNC nodes sharing an L3 cache instance. 860Users who bind tasks to the CPUs of a specific Sub-NUMA node can read 861the "llc_occupancy", "mbm_total_bytes", and "mbm_local_bytes" in the 862"mon_sub_L3_YY" directories to get node local data. 863 864Memory bandwidth allocation is still performed at the L3 cache 865level. I.e. throttling controls are applied to all SNC nodes. 866 867L3 cache allocation bitmaps also apply to all SNC nodes. But note that 868the amount of L3 cache represented by each bit is divided by the number 869of SNC nodes per L3 cache. E.g. with a 100MB cache on a system with 10-bit 870allocation masks each bit normally represents 10MB. With SNC mode enabled 871with two SNC nodes per L3 cache, each bit only represents 5MB. 872 873Memory bandwidth Allocation and monitoring 874========================================== 875 876For Memory bandwidth resource, by default the user controls the resource 877by indicating the percentage of total memory bandwidth. 878 879The minimum bandwidth percentage value for each cpu model is predefined 880and can be looked up through "info/MB/min_bandwidth". The bandwidth 881granularity that is allocated is also dependent on the cpu model and can 882be looked up at "info/MB/bandwidth_gran". The available bandwidth 883control steps are, approximately, min_bw + N * bw_gran. The steps may 884appear irregular due to rounding to an exact percentage: bw_gran is the 885maximum interval between the percentage values corresponding to any two 886adjacent steps in the hardware. 887 888The bandwidth throttling is a core specific mechanism on some of Intel 889SKUs. Using a high bandwidth and a low bandwidth setting on two threads 890sharing a core may result in both threads being throttled to use the 891low bandwidth (see "thread_throttle_mode"). 892 893The fact that Memory bandwidth allocation(MBA) may be a core 894specific mechanism where as memory bandwidth monitoring(MBM) is done at 895the package level may lead to confusion when users try to apply control 896via the MBA and then monitor the bandwidth to see if the controls are 897effective. Below are such scenarios: 898 8991. User may *not* see increase in actual bandwidth when percentage 900 values are increased: 901 902This can occur when aggregate L2 external bandwidth is more than L3 903external bandwidth. Consider an SKL SKU with 24 cores on a package and 904where L2 external is 10GBps (hence aggregate L2 external bandwidth is 905240GBps) and L3 external bandwidth is 100GBps. Now a workload with '20 906threads, having 50% bandwidth, each consuming 5GBps' consumes the max L3 907bandwidth of 100GBps although the percentage value specified is only 50% 908<< 100%. Hence increasing the bandwidth percentage will not yield any 909more bandwidth. This is because although the L2 external bandwidth still 910has capacity, the L3 external bandwidth is fully used. Also note that 911this would be dependent on number of cores the benchmark is run on. 912 9132. Same bandwidth percentage may mean different actual bandwidth 914 depending on # of threads: 915 916For the same SKU in #1, a 'single thread, with 10% bandwidth' and '4 917thread, with 10% bandwidth' can consume up to 10GBps and 40GBps although 918they have same percentage bandwidth of 10%. This is simply because as 919threads start using more cores in an rdtgroup, the actual bandwidth may 920increase or vary although user specified bandwidth percentage is same. 921 922In order to mitigate this and make the interface more user friendly, 923resctrl added support for specifying the bandwidth in MiBps as well. The 924kernel underneath would use a software feedback mechanism or a "Software 925Controller(mba_sc)" which reads the actual bandwidth using MBM counters 926and adjust the memory bandwidth percentages to ensure:: 927 928 "actual bandwidth < user specified bandwidth". 929 930By default, the schemata would take the bandwidth percentage values 931where as user can switch to the "MBA software controller" mode using 932a mount option 'mba_MBps'. The schemata format is specified in the below 933sections. 934 935L3 schemata file details (code and data prioritization disabled) 936---------------------------------------------------------------- 937With CDP disabled the L3 schemata format is:: 938 939 L3:<cache_id0>=<cbm>;<cache_id1>=<cbm>;... 940 941L3 schemata file details (CDP enabled via mount option to resctrl) 942------------------------------------------------------------------ 943When CDP is enabled L3 control is split into two separate resources 944so you can specify independent masks for code and data like this:: 945 946 L3DATA:<cache_id0>=<cbm>;<cache_id1>=<cbm>;... 947 L3CODE:<cache_id0>=<cbm>;<cache_id1>=<cbm>;... 948 949L2 schemata file details 950------------------------ 951CDP is supported at L2 using the 'cdpl2' mount option. The schemata 952format is either:: 953 954 L2:<cache_id0>=<cbm>;<cache_id1>=<cbm>;... 955 956or 957 958 L2DATA:<cache_id0>=<cbm>;<cache_id1>=<cbm>;... 959 L2CODE:<cache_id0>=<cbm>;<cache_id1>=<cbm>;... 960 961 962Memory bandwidth Allocation (default mode) 963------------------------------------------ 964 965Memory b/w domain is L3 cache. 966:: 967 968 MB:<cache_id0>=bandwidth0;<cache_id1>=bandwidth1;... 969 970Memory bandwidth Allocation specified in MiBps 971---------------------------------------------- 972 973Memory bandwidth domain is L3 cache. 974:: 975 976 MB:<cache_id0>=bw_MiBps0;<cache_id1>=bw_MiBps1;... 977 978Slow Memory Bandwidth Allocation (SMBA) 979--------------------------------------- 980AMD hardware supports Slow Memory Bandwidth Allocation (SMBA). 981CXL.memory is the only supported "slow" memory device. With the 982support of SMBA, the hardware enables bandwidth allocation on 983the slow memory devices. If there are multiple such devices in 984the system, the throttling logic groups all the slow sources 985together and applies the limit on them as a whole. 986 987The presence of SMBA (with CXL.memory) is independent of slow memory 988devices presence. If there are no such devices on the system, then 989configuring SMBA will have no impact on the performance of the system. 990 991The bandwidth domain for slow memory is L3 cache. Its schemata file 992is formatted as: 993:: 994 995 SMBA:<cache_id0>=bandwidth0;<cache_id1>=bandwidth1;... 996 997Reading/writing the schemata file 998--------------------------------- 999Reading the schemata file will show the state of all resources 1000on all domains. When writing you only need to specify those values 1001which you wish to change. E.g. 1002:: 1003 1004 # cat schemata 1005 L3DATA:0=fffff;1=fffff;2=fffff;3=fffff 1006 L3CODE:0=fffff;1=fffff;2=fffff;3=fffff 1007 # echo "L3DATA:2=3c0;" > schemata 1008 # cat schemata 1009 L3DATA:0=fffff;1=fffff;2=3c0;3=fffff 1010 L3CODE:0=fffff;1=fffff;2=fffff;3=fffff 1011 1012Reading/writing the schemata file (on AMD systems) 1013-------------------------------------------------- 1014Reading the schemata file will show the current bandwidth limit on all 1015domains. The allocated resources are in multiples of one eighth GB/s. 1016When writing to the file, you need to specify what cache id you wish to 1017configure the bandwidth limit. 1018 1019For example, to allocate 2GB/s limit on the first cache id: 1020 1021:: 1022 1023 # cat schemata 1024 MB:0=2048;1=2048;2=2048;3=2048 1025 L3:0=ffff;1=ffff;2=ffff;3=ffff 1026 1027 # echo "MB:1=16" > schemata 1028 # cat schemata 1029 MB:0=2048;1= 16;2=2048;3=2048 1030 L3:0=ffff;1=ffff;2=ffff;3=ffff 1031 1032Reading/writing the schemata file (on AMD systems) with SMBA feature 1033-------------------------------------------------------------------- 1034Reading and writing the schemata file is the same as without SMBA in 1035above section. 1036 1037For example, to allocate 8GB/s limit on the first cache id: 1038 1039:: 1040 1041 # cat schemata 1042 SMBA:0=2048;1=2048;2=2048;3=2048 1043 MB:0=2048;1=2048;2=2048;3=2048 1044 L3:0=ffff;1=ffff;2=ffff;3=ffff 1045 1046 # echo "SMBA:1=64" > schemata 1047 # cat schemata 1048 SMBA:0=2048;1= 64;2=2048;3=2048 1049 MB:0=2048;1=2048;2=2048;3=2048 1050 L3:0=ffff;1=ffff;2=ffff;3=ffff 1051 1052Cache Pseudo-Locking 1053==================== 1054CAT enables a user to specify the amount of cache space that an 1055application can fill. Cache pseudo-locking builds on the fact that a 1056CPU can still read and write data pre-allocated outside its current 1057allocated area on a cache hit. With cache pseudo-locking, data can be 1058preloaded into a reserved portion of cache that no application can 1059fill, and from that point on will only serve cache hits. The cache 1060pseudo-locked memory is made accessible to user space where an 1061application can map it into its virtual address space and thus have 1062a region of memory with reduced average read latency. 1063 1064The creation of a cache pseudo-locked region is triggered by a request 1065from the user to do so that is accompanied by a schemata of the region 1066to be pseudo-locked. The cache pseudo-locked region is created as follows: 1067 1068- Create a CAT allocation CLOSNEW with a CBM matching the schemata 1069 from the user of the cache region that will contain the pseudo-locked 1070 memory. This region must not overlap with any current CAT allocation/CLOS 1071 on the system and no future overlap with this cache region is allowed 1072 while the pseudo-locked region exists. 1073- Create a contiguous region of memory of the same size as the cache 1074 region. 1075- Flush the cache, disable hardware prefetchers, disable preemption. 1076- Make CLOSNEW the active CLOS and touch the allocated memory to load 1077 it into the cache. 1078- Set the previous CLOS as active. 1079- At this point the closid CLOSNEW can be released - the cache 1080 pseudo-locked region is protected as long as its CBM does not appear in 1081 any CAT allocation. Even though the cache pseudo-locked region will from 1082 this point on not appear in any CBM of any CLOS an application running with 1083 any CLOS will be able to access the memory in the pseudo-locked region since 1084 the region continues to serve cache hits. 1085- The contiguous region of memory loaded into the cache is exposed to 1086 user-space as a character device. 1087 1088Cache pseudo-locking increases the probability that data will remain 1089in the cache via carefully configuring the CAT feature and controlling 1090application behavior. There is no guarantee that data is placed in 1091cache. Instructions like INVD, WBINVD, CLFLUSH, etc. can still evict 1092“locked” data from cache. Power management C-states may shrink or 1093power off cache. Deeper C-states will automatically be restricted on 1094pseudo-locked region creation. 1095 1096It is required that an application using a pseudo-locked region runs 1097with affinity to the cores (or a subset of the cores) associated 1098with the cache on which the pseudo-locked region resides. A sanity check 1099within the code will not allow an application to map pseudo-locked memory 1100unless it runs with affinity to cores associated with the cache on which the 1101pseudo-locked region resides. The sanity check is only done during the 1102initial mmap() handling, there is no enforcement afterwards and the 1103application self needs to ensure it remains affine to the correct cores. 1104 1105Pseudo-locking is accomplished in two stages: 1106 11071) During the first stage the system administrator allocates a portion 1108 of cache that should be dedicated to pseudo-locking. At this time an 1109 equivalent portion of memory is allocated, loaded into allocated 1110 cache portion, and exposed as a character device. 11112) During the second stage a user-space application maps (mmap()) the 1112 pseudo-locked memory into its address space. 1113 1114Cache Pseudo-Locking Interface 1115------------------------------ 1116A pseudo-locked region is created using the resctrl interface as follows: 1117 11181) Create a new resource group by creating a new directory in /sys/fs/resctrl. 11192) Change the new resource group's mode to "pseudo-locksetup" by writing 1120 "pseudo-locksetup" to the "mode" file. 11213) Write the schemata of the pseudo-locked region to the "schemata" file. All 1122 bits within the schemata should be "unused" according to the "bit_usage" 1123 file. 1124 1125On successful pseudo-locked region creation the "mode" file will contain 1126"pseudo-locked" and a new character device with the same name as the resource 1127group will exist in /dev/pseudo_lock. This character device can be mmap()'ed 1128by user space in order to obtain access to the pseudo-locked memory region. 1129 1130An example of cache pseudo-locked region creation and usage can be found below. 1131 1132Cache Pseudo-Locking Debugging Interface 1133---------------------------------------- 1134The pseudo-locking debugging interface is enabled by default (if 1135CONFIG_DEBUG_FS is enabled) and can be found in /sys/kernel/debug/resctrl. 1136 1137There is no explicit way for the kernel to test if a provided memory 1138location is present in the cache. The pseudo-locking debugging interface uses 1139the tracing infrastructure to provide two ways to measure cache residency of 1140the pseudo-locked region: 1141 11421) Memory access latency using the pseudo_lock_mem_latency tracepoint. Data 1143 from these measurements are best visualized using a hist trigger (see 1144 example below). In this test the pseudo-locked region is traversed at 1145 a stride of 32 bytes while hardware prefetchers and preemption 1146 are disabled. This also provides a substitute visualization of cache 1147 hits and misses. 11482) Cache hit and miss measurements using model specific precision counters if 1149 available. Depending on the levels of cache on the system the pseudo_lock_l2 1150 and pseudo_lock_l3 tracepoints are available. 1151 1152When a pseudo-locked region is created a new debugfs directory is created for 1153it in debugfs as /sys/kernel/debug/resctrl/<newdir>. A single 1154write-only file, pseudo_lock_measure, is present in this directory. The 1155measurement of the pseudo-locked region depends on the number written to this 1156debugfs file: 1157 11581: 1159 writing "1" to the pseudo_lock_measure file will trigger the latency 1160 measurement captured in the pseudo_lock_mem_latency tracepoint. See 1161 example below. 11622: 1163 writing "2" to the pseudo_lock_measure file will trigger the L2 cache 1164 residency (cache hits and misses) measurement captured in the 1165 pseudo_lock_l2 tracepoint. See example below. 11663: 1167 writing "3" to the pseudo_lock_measure file will trigger the L3 cache 1168 residency (cache hits and misses) measurement captured in the 1169 pseudo_lock_l3 tracepoint. 1170 1171All measurements are recorded with the tracing infrastructure. This requires 1172the relevant tracepoints to be enabled before the measurement is triggered. 1173 1174Example of latency debugging interface 1175~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1176In this example a pseudo-locked region named "newlock" was created. Here is 1177how we can measure the latency in cycles of reading from this region and 1178visualize this data with a histogram that is available if CONFIG_HIST_TRIGGERS 1179is set:: 1180 1181 # :> /sys/kernel/tracing/trace 1182 # echo 'hist:keys=latency' > /sys/kernel/tracing/events/resctrl/pseudo_lock_mem_latency/trigger 1183 # echo 1 > /sys/kernel/tracing/events/resctrl/pseudo_lock_mem_latency/enable 1184 # echo 1 > /sys/kernel/debug/resctrl/newlock/pseudo_lock_measure 1185 # echo 0 > /sys/kernel/tracing/events/resctrl/pseudo_lock_mem_latency/enable 1186 # cat /sys/kernel/tracing/events/resctrl/pseudo_lock_mem_latency/hist 1187 1188 # event histogram 1189 # 1190 # trigger info: hist:keys=latency:vals=hitcount:sort=hitcount:size=2048 [active] 1191 # 1192 1193 { latency: 456 } hitcount: 1 1194 { latency: 50 } hitcount: 83 1195 { latency: 36 } hitcount: 96 1196 { latency: 44 } hitcount: 174 1197 { latency: 48 } hitcount: 195 1198 { latency: 46 } hitcount: 262 1199 { latency: 42 } hitcount: 693 1200 { latency: 40 } hitcount: 3204 1201 { latency: 38 } hitcount: 3484 1202 1203 Totals: 1204 Hits: 8192 1205 Entries: 9 1206 Dropped: 0 1207 1208Example of cache hits/misses debugging 1209~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1210In this example a pseudo-locked region named "newlock" was created on the L2 1211cache of a platform. Here is how we can obtain details of the cache hits 1212and misses using the platform's precision counters. 1213:: 1214 1215 # :> /sys/kernel/tracing/trace 1216 # echo 1 > /sys/kernel/tracing/events/resctrl/pseudo_lock_l2/enable 1217 # echo 2 > /sys/kernel/debug/resctrl/newlock/pseudo_lock_measure 1218 # echo 0 > /sys/kernel/tracing/events/resctrl/pseudo_lock_l2/enable 1219 # cat /sys/kernel/tracing/trace 1220 1221 # tracer: nop 1222 # 1223 # _-----=> irqs-off 1224 # / _----=> need-resched 1225 # | / _---=> hardirq/softirq 1226 # || / _--=> preempt-depth 1227 # ||| / delay 1228 # TASK-PID CPU# |||| TIMESTAMP FUNCTION 1229 # | | | |||| | | 1230 pseudo_lock_mea-1672 [002] .... 3132.860500: pseudo_lock_l2: hits=4097 miss=0 1231 1232 1233Examples for RDT allocation usage 1234~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1235 12361) Example 1 1237 1238On a two socket machine (one L3 cache per socket) with just four bits 1239for cache bit masks, minimum b/w of 10% with a memory bandwidth 1240granularity of 10%. 1241:: 1242 1243 # mount -t resctrl resctrl /sys/fs/resctrl 1244 # cd /sys/fs/resctrl 1245 # mkdir p0 p1 1246 # echo "L3:0=3;1=c\nMB:0=50;1=50" > /sys/fs/resctrl/p0/schemata 1247 # echo "L3:0=3;1=3\nMB:0=50;1=50" > /sys/fs/resctrl/p1/schemata 1248 1249The default resource group is unmodified, so we have access to all parts 1250of all caches (its schemata file reads "L3:0=f;1=f"). 1251 1252Tasks that are under the control of group "p0" may only allocate from the 1253"lower" 50% on cache ID 0, and the "upper" 50% of cache ID 1. 1254Tasks in group "p1" use the "lower" 50% of cache on both sockets. 1255 1256Similarly, tasks that are under the control of group "p0" may use a 1257maximum memory b/w of 50% on socket0 and 50% on socket 1. 1258Tasks in group "p1" may also use 50% memory b/w on both sockets. 1259Note that unlike cache masks, memory b/w cannot specify whether these 1260allocations can overlap or not. The allocations specifies the maximum 1261b/w that the group may be able to use and the system admin can configure 1262the b/w accordingly. 1263 1264If resctrl is using the software controller (mba_sc) then user can enter the 1265max b/w in MB rather than the percentage values. 1266:: 1267 1268 # echo "L3:0=3;1=c\nMB:0=1024;1=500" > /sys/fs/resctrl/p0/schemata 1269 # echo "L3:0=3;1=3\nMB:0=1024;1=500" > /sys/fs/resctrl/p1/schemata 1270 1271In the above example the tasks in "p1" and "p0" on socket 0 would use a max b/w 1272of 1024MB where as on socket 1 they would use 500MB. 1273 12742) Example 2 1275 1276Again two sockets, but this time with a more realistic 20-bit mask. 1277 1278Two real time tasks pid=1234 running on processor 0 and pid=5678 running on 1279processor 1 on socket 0 on a 2-socket and dual core machine. To avoid noisy 1280neighbors, each of the two real-time tasks exclusively occupies one quarter 1281of L3 cache on socket 0. 1282:: 1283 1284 # mount -t resctrl resctrl /sys/fs/resctrl 1285 # cd /sys/fs/resctrl 1286 1287First we reset the schemata for the default group so that the "upper" 128850% of the L3 cache on socket 0 and 50% of memory b/w cannot be used by 1289ordinary tasks:: 1290 1291 # echo "L3:0=3ff;1=fffff\nMB:0=50;1=100" > schemata 1292 1293Next we make a resource group for our first real time task and give 1294it access to the "top" 25% of the cache on socket 0. 1295:: 1296 1297 # mkdir p0 1298 # echo "L3:0=f8000;1=fffff" > p0/schemata 1299 1300Finally we move our first real time task into this resource group. We 1301also use taskset(1) to ensure the task always runs on a dedicated CPU 1302on socket 0. Most uses of resource groups will also constrain which 1303processors tasks run on. 1304:: 1305 1306 # echo 1234 > p0/tasks 1307 # taskset -cp 1 1234 1308 1309Ditto for the second real time task (with the remaining 25% of cache):: 1310 1311 # mkdir p1 1312 # echo "L3:0=7c00;1=fffff" > p1/schemata 1313 # echo 5678 > p1/tasks 1314 # taskset -cp 2 5678 1315 1316For the same 2 socket system with memory b/w resource and CAT L3 the 1317schemata would look like(Assume min_bandwidth 10 and bandwidth_gran is 131810): 1319 1320For our first real time task this would request 20% memory b/w on socket 0. 1321:: 1322 1323 # echo -e "L3:0=f8000;1=fffff\nMB:0=20;1=100" > p0/schemata 1324 1325For our second real time task this would request an other 20% memory b/w 1326on socket 0. 1327:: 1328 1329 # echo -e "L3:0=f8000;1=fffff\nMB:0=20;1=100" > p0/schemata 1330 13313) Example 3 1332 1333A single socket system which has real-time tasks running on core 4-7 and 1334non real-time workload assigned to core 0-3. The real-time tasks share text 1335and data, so a per task association is not required and due to interaction 1336with the kernel it's desired that the kernel on these cores shares L3 with 1337the tasks. 1338:: 1339 1340 # mount -t resctrl resctrl /sys/fs/resctrl 1341 # cd /sys/fs/resctrl 1342 1343First we reset the schemata for the default group so that the "upper" 134450% of the L3 cache on socket 0, and 50% of memory bandwidth on socket 0 1345cannot be used by ordinary tasks:: 1346 1347 # echo "L3:0=3ff\nMB:0=50" > schemata 1348 1349Next we make a resource group for our real time cores and give it access 1350to the "top" 50% of the cache on socket 0 and 50% of memory bandwidth on 1351socket 0. 1352:: 1353 1354 # mkdir p0 1355 # echo "L3:0=ffc00\nMB:0=50" > p0/schemata 1356 1357Finally we move core 4-7 over to the new group and make sure that the 1358kernel and the tasks running there get 50% of the cache. They should 1359also get 50% of memory bandwidth assuming that the cores 4-7 are SMT 1360siblings and only the real time threads are scheduled on the cores 4-7. 1361:: 1362 1363 # echo F0 > p0/cpus 1364 13654) Example 4 1366 1367The resource groups in previous examples were all in the default "shareable" 1368mode allowing sharing of their cache allocations. If one resource group 1369configures a cache allocation then nothing prevents another resource group 1370to overlap with that allocation. 1371 1372In this example a new exclusive resource group will be created on a L2 CAT 1373system with two L2 cache instances that can be configured with an 8-bit 1374capacity bitmask. The new exclusive resource group will be configured to use 137525% of each cache instance. 1376:: 1377 1378 # mount -t resctrl resctrl /sys/fs/resctrl/ 1379 # cd /sys/fs/resctrl 1380 1381First, we observe that the default group is configured to allocate to all L2 1382cache:: 1383 1384 # cat schemata 1385 L2:0=ff;1=ff 1386 1387We could attempt to create the new resource group at this point, but it will 1388fail because of the overlap with the schemata of the default group:: 1389 1390 # mkdir p0 1391 # echo 'L2:0=0x3;1=0x3' > p0/schemata 1392 # cat p0/mode 1393 shareable 1394 # echo exclusive > p0/mode 1395 -sh: echo: write error: Invalid argument 1396 # cat info/last_cmd_status 1397 schemata overlaps 1398 1399To ensure that there is no overlap with another resource group the default 1400resource group's schemata has to change, making it possible for the new 1401resource group to become exclusive. 1402:: 1403 1404 # echo 'L2:0=0xfc;1=0xfc' > schemata 1405 # echo exclusive > p0/mode 1406 # grep . p0/* 1407 p0/cpus:0 1408 p0/mode:exclusive 1409 p0/schemata:L2:0=03;1=03 1410 p0/size:L2:0=262144;1=262144 1411 1412A new resource group will on creation not overlap with an exclusive resource 1413group:: 1414 1415 # mkdir p1 1416 # grep . p1/* 1417 p1/cpus:0 1418 p1/mode:shareable 1419 p1/schemata:L2:0=fc;1=fc 1420 p1/size:L2:0=786432;1=786432 1421 1422The bit_usage will reflect how the cache is used:: 1423 1424 # cat info/L2/bit_usage 1425 0=SSSSSSEE;1=SSSSSSEE 1426 1427A resource group cannot be forced to overlap with an exclusive resource group:: 1428 1429 # echo 'L2:0=0x1;1=0x1' > p1/schemata 1430 -sh: echo: write error: Invalid argument 1431 # cat info/last_cmd_status 1432 overlaps with exclusive group 1433 1434Example of Cache Pseudo-Locking 1435~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1436Lock portion of L2 cache from cache id 1 using CBM 0x3. Pseudo-locked 1437region is exposed at /dev/pseudo_lock/newlock that can be provided to 1438application for argument to mmap(). 1439:: 1440 1441 # mount -t resctrl resctrl /sys/fs/resctrl/ 1442 # cd /sys/fs/resctrl 1443 1444Ensure that there are bits available that can be pseudo-locked, since only 1445unused bits can be pseudo-locked the bits to be pseudo-locked needs to be 1446removed from the default resource group's schemata:: 1447 1448 # cat info/L2/bit_usage 1449 0=SSSSSSSS;1=SSSSSSSS 1450 # echo 'L2:1=0xfc' > schemata 1451 # cat info/L2/bit_usage 1452 0=SSSSSSSS;1=SSSSSS00 1453 1454Create a new resource group that will be associated with the pseudo-locked 1455region, indicate that it will be used for a pseudo-locked region, and 1456configure the requested pseudo-locked region capacity bitmask:: 1457 1458 # mkdir newlock 1459 # echo pseudo-locksetup > newlock/mode 1460 # echo 'L2:1=0x3' > newlock/schemata 1461 1462On success the resource group's mode will change to pseudo-locked, the 1463bit_usage will reflect the pseudo-locked region, and the character device 1464exposing the pseudo-locked region will exist:: 1465 1466 # cat newlock/mode 1467 pseudo-locked 1468 # cat info/L2/bit_usage 1469 0=SSSSSSSS;1=SSSSSSPP 1470 # ls -l /dev/pseudo_lock/newlock 1471 crw------- 1 root root 243, 0 Apr 3 05:01 /dev/pseudo_lock/newlock 1472 1473:: 1474 1475 /* 1476 * Example code to access one page of pseudo-locked cache region 1477 * from user space. 1478 */ 1479 #define _GNU_SOURCE 1480 #include <fcntl.h> 1481 #include <sched.h> 1482 #include <stdio.h> 1483 #include <stdlib.h> 1484 #include <unistd.h> 1485 #include <sys/mman.h> 1486 1487 /* 1488 * It is required that the application runs with affinity to only 1489 * cores associated with the pseudo-locked region. Here the cpu 1490 * is hardcoded for convenience of example. 1491 */ 1492 static int cpuid = 2; 1493 1494 int main(int argc, char *argv[]) 1495 { 1496 cpu_set_t cpuset; 1497 long page_size; 1498 void *mapping; 1499 int dev_fd; 1500 int ret; 1501 1502 page_size = sysconf(_SC_PAGESIZE); 1503 1504 CPU_ZERO(&cpuset); 1505 CPU_SET(cpuid, &cpuset); 1506 ret = sched_setaffinity(0, sizeof(cpuset), &cpuset); 1507 if (ret < 0) { 1508 perror("sched_setaffinity"); 1509 exit(EXIT_FAILURE); 1510 } 1511 1512 dev_fd = open("/dev/pseudo_lock/newlock", O_RDWR); 1513 if (dev_fd < 0) { 1514 perror("open"); 1515 exit(EXIT_FAILURE); 1516 } 1517 1518 mapping = mmap(0, page_size, PROT_READ | PROT_WRITE, MAP_SHARED, 1519 dev_fd, 0); 1520 if (mapping == MAP_FAILED) { 1521 perror("mmap"); 1522 close(dev_fd); 1523 exit(EXIT_FAILURE); 1524 } 1525 1526 /* Application interacts with pseudo-locked memory @mapping */ 1527 1528 ret = munmap(mapping, page_size); 1529 if (ret < 0) { 1530 perror("munmap"); 1531 close(dev_fd); 1532 exit(EXIT_FAILURE); 1533 } 1534 1535 close(dev_fd); 1536 exit(EXIT_SUCCESS); 1537 } 1538 1539Locking between applications 1540---------------------------- 1541 1542Certain operations on the resctrl filesystem, composed of read/writes 1543to/from multiple files, must be atomic. 1544 1545As an example, the allocation of an exclusive reservation of L3 cache 1546involves: 1547 1548 1. Read the cbmmasks from each directory or the per-resource "bit_usage" 1549 2. Find a contiguous set of bits in the global CBM bitmask that is clear 1550 in any of the directory cbmmasks 1551 3. Create a new directory 1552 4. Set the bits found in step 2 to the new directory "schemata" file 1553 1554If two applications attempt to allocate space concurrently then they can 1555end up allocating the same bits so the reservations are shared instead of 1556exclusive. 1557 1558To coordinate atomic operations on the resctrlfs and to avoid the problem 1559above, the following locking procedure is recommended: 1560 1561Locking is based on flock, which is available in libc and also as a shell 1562script command 1563 1564Write lock: 1565 1566 A) Take flock(LOCK_EX) on /sys/fs/resctrl 1567 B) Read/write the directory structure. 1568 C) funlock 1569 1570Read lock: 1571 1572 A) Take flock(LOCK_SH) on /sys/fs/resctrl 1573 B) If success read the directory structure. 1574 C) funlock 1575 1576Example with bash:: 1577 1578 # Atomically read directory structure 1579 $ flock -s /sys/fs/resctrl/ find /sys/fs/resctrl 1580 1581 # Read directory contents and create new subdirectory 1582 1583 $ cat create-dir.sh 1584 find /sys/fs/resctrl/ > output.txt 1585 mask = function-of(output.txt) 1586 mkdir /sys/fs/resctrl/newres/ 1587 echo mask > /sys/fs/resctrl/newres/schemata 1588 1589 $ flock /sys/fs/resctrl/ ./create-dir.sh 1590 1591Example with C:: 1592 1593 /* 1594 * Example code do take advisory locks 1595 * before accessing resctrl filesystem 1596 */ 1597 #include <sys/file.h> 1598 #include <stdlib.h> 1599 1600 void resctrl_take_shared_lock(int fd) 1601 { 1602 int ret; 1603 1604 /* take shared lock on resctrl filesystem */ 1605 ret = flock(fd, LOCK_SH); 1606 if (ret) { 1607 perror("flock"); 1608 exit(-1); 1609 } 1610 } 1611 1612 void resctrl_take_exclusive_lock(int fd) 1613 { 1614 int ret; 1615 1616 /* release lock on resctrl filesystem */ 1617 ret = flock(fd, LOCK_EX); 1618 if (ret) { 1619 perror("flock"); 1620 exit(-1); 1621 } 1622 } 1623 1624 void resctrl_release_lock(int fd) 1625 { 1626 int ret; 1627 1628 /* take shared lock on resctrl filesystem */ 1629 ret = flock(fd, LOCK_UN); 1630 if (ret) { 1631 perror("flock"); 1632 exit(-1); 1633 } 1634 } 1635 1636 void main(void) 1637 { 1638 int fd, ret; 1639 1640 fd = open("/sys/fs/resctrl", O_DIRECTORY); 1641 if (fd == -1) { 1642 perror("open"); 1643 exit(-1); 1644 } 1645 resctrl_take_shared_lock(fd); 1646 /* code to read directory contents */ 1647 resctrl_release_lock(fd); 1648 1649 resctrl_take_exclusive_lock(fd); 1650 /* code to read and write directory contents */ 1651 resctrl_release_lock(fd); 1652 } 1653 1654Examples for RDT Monitoring along with allocation usage 1655======================================================= 1656Reading monitored data 1657---------------------- 1658Reading an event file (for ex: mon_data/mon_L3_00/llc_occupancy) would 1659show the current snapshot of LLC occupancy of the corresponding MON 1660group or CTRL_MON group. 1661 1662 1663Example 1 (Monitor CTRL_MON group and subset of tasks in CTRL_MON group) 1664------------------------------------------------------------------------ 1665On a two socket machine (one L3 cache per socket) with just four bits 1666for cache bit masks:: 1667 1668 # mount -t resctrl resctrl /sys/fs/resctrl 1669 # cd /sys/fs/resctrl 1670 # mkdir p0 p1 1671 # echo "L3:0=3;1=c" > /sys/fs/resctrl/p0/schemata 1672 # echo "L3:0=3;1=3" > /sys/fs/resctrl/p1/schemata 1673 # echo 5678 > p1/tasks 1674 # echo 5679 > p1/tasks 1675 1676The default resource group is unmodified, so we have access to all parts 1677of all caches (its schemata file reads "L3:0=f;1=f"). 1678 1679Tasks that are under the control of group "p0" may only allocate from the 1680"lower" 50% on cache ID 0, and the "upper" 50% of cache ID 1. 1681Tasks in group "p1" use the "lower" 50% of cache on both sockets. 1682 1683Create monitor groups and assign a subset of tasks to each monitor group. 1684:: 1685 1686 # cd /sys/fs/resctrl/p1/mon_groups 1687 # mkdir m11 m12 1688 # echo 5678 > m11/tasks 1689 # echo 5679 > m12/tasks 1690 1691fetch data (data shown in bytes) 1692:: 1693 1694 # cat m11/mon_data/mon_L3_00/llc_occupancy 1695 16234000 1696 # cat m11/mon_data/mon_L3_01/llc_occupancy 1697 14789000 1698 # cat m12/mon_data/mon_L3_00/llc_occupancy 1699 16789000 1700 1701The parent ctrl_mon group shows the aggregated data. 1702:: 1703 1704 # cat /sys/fs/resctrl/p1/mon_data/mon_l3_00/llc_occupancy 1705 31234000 1706 1707Example 2 (Monitor a task from its creation) 1708-------------------------------------------- 1709On a two socket machine (one L3 cache per socket):: 1710 1711 # mount -t resctrl resctrl /sys/fs/resctrl 1712 # cd /sys/fs/resctrl 1713 # mkdir p0 p1 1714 1715An RMID is allocated to the group once its created and hence the <cmd> 1716below is monitored from its creation. 1717:: 1718 1719 # echo $$ > /sys/fs/resctrl/p1/tasks 1720 # <cmd> 1721 1722Fetch the data:: 1723 1724 # cat /sys/fs/resctrl/p1/mon_data/mon_l3_00/llc_occupancy 1725 31789000 1726 1727Example 3 (Monitor without CAT support or before creating CAT groups) 1728--------------------------------------------------------------------- 1729 1730Assume a system like HSW has only CQM and no CAT support. In this case 1731the resctrl will still mount but cannot create CTRL_MON directories. 1732But user can create different MON groups within the root group thereby 1733able to monitor all tasks including kernel threads. 1734 1735This can also be used to profile jobs cache size footprint before being 1736able to allocate them to different allocation groups. 1737:: 1738 1739 # mount -t resctrl resctrl /sys/fs/resctrl 1740 # cd /sys/fs/resctrl 1741 # mkdir mon_groups/m01 1742 # mkdir mon_groups/m02 1743 1744 # echo 3478 > /sys/fs/resctrl/mon_groups/m01/tasks 1745 # echo 2467 > /sys/fs/resctrl/mon_groups/m02/tasks 1746 1747Monitor the groups separately and also get per domain data. From the 1748below its apparent that the tasks are mostly doing work on 1749domain(socket) 0. 1750:: 1751 1752 # cat /sys/fs/resctrl/mon_groups/m01/mon_L3_00/llc_occupancy 1753 31234000 1754 # cat /sys/fs/resctrl/mon_groups/m01/mon_L3_01/llc_occupancy 1755 34555 1756 # cat /sys/fs/resctrl/mon_groups/m02/mon_L3_00/llc_occupancy 1757 31234000 1758 # cat /sys/fs/resctrl/mon_groups/m02/mon_L3_01/llc_occupancy 1759 32789 1760 1761 1762Example 4 (Monitor real time tasks) 1763----------------------------------- 1764 1765A single socket system which has real time tasks running on cores 4-7 1766and non real time tasks on other cpus. We want to monitor the cache 1767occupancy of the real time threads on these cores. 1768:: 1769 1770 # mount -t resctrl resctrl /sys/fs/resctrl 1771 # cd /sys/fs/resctrl 1772 # mkdir p1 1773 1774Move the cpus 4-7 over to p1:: 1775 1776 # echo f0 > p1/cpus 1777 1778View the llc occupancy snapshot:: 1779 1780 # cat /sys/fs/resctrl/p1/mon_data/mon_L3_00/llc_occupancy 1781 11234000 1782 1783 1784Examples on working with mbm_assign_mode 1785======================================== 1786 1787a. Check if MBM counter assignment mode is supported. 1788:: 1789 1790 # mount -t resctrl resctrl /sys/fs/resctrl/ 1791 1792 # cat /sys/fs/resctrl/info/L3_MON/mbm_assign_mode 1793 [mbm_event] 1794 default 1795 1796The "mbm_event" mode is detected and enabled. 1797 1798b. Check how many assignable counters are supported. 1799:: 1800 1801 # cat /sys/fs/resctrl/info/L3_MON/num_mbm_cntrs 1802 0=32;1=32 1803 1804c. Check how many assignable counters are available for assignment in each domain. 1805:: 1806 1807 # cat /sys/fs/resctrl/info/L3_MON/available_mbm_cntrs 1808 0=30;1=30 1809 1810d. To list the default group's assign states. 1811:: 1812 1813 # cat /sys/fs/resctrl/mbm_L3_assignments 1814 mbm_total_bytes:0=e;1=e 1815 mbm_local_bytes:0=e;1=e 1816 1817e. To unassign the counter associated with the mbm_total_bytes event on domain 0. 1818:: 1819 1820 # echo "mbm_total_bytes:0=_" > /sys/fs/resctrl/mbm_L3_assignments 1821 # cat /sys/fs/resctrl/mbm_L3_assignments 1822 mbm_total_bytes:0=_;1=e 1823 mbm_local_bytes:0=e;1=e 1824 1825f. To unassign the counter associated with the mbm_total_bytes event on all domains. 1826:: 1827 1828 # echo "mbm_total_bytes:*=_" > /sys/fs/resctrl/mbm_L3_assignments 1829 # cat /sys/fs/resctrl/mbm_L3_assignment 1830 mbm_total_bytes:0=_;1=_ 1831 mbm_local_bytes:0=e;1=e 1832 1833g. To assign a counter associated with the mbm_total_bytes event on all domains in 1834exclusive mode. 1835:: 1836 1837 # echo "mbm_total_bytes:*=e" > /sys/fs/resctrl/mbm_L3_assignments 1838 # cat /sys/fs/resctrl/mbm_L3_assignments 1839 mbm_total_bytes:0=e;1=e 1840 mbm_local_bytes:0=e;1=e 1841 1842h. Read the events mbm_total_bytes and mbm_local_bytes of the default group. There is 1843no change in reading the events with the assignment. 1844:: 1845 1846 # cat /sys/fs/resctrl/mon_data/mon_L3_00/mbm_total_bytes 1847 779247936 1848 # cat /sys/fs/resctrl/mon_data/mon_L3_01/mbm_total_bytes 1849 562324232 1850 # cat /sys/fs/resctrl/mon_data/mon_L3_00/mbm_local_bytes 1851 212122123 1852 # cat /sys/fs/resctrl/mon_data/mon_L3_01/mbm_local_bytes 1853 121212144 1854 1855i. Check the event configurations. 1856:: 1857 1858 # cat /sys/fs/resctrl/info/L3_MON/event_configs/mbm_total_bytes/event_filter 1859 local_reads,remote_reads,local_non_temporal_writes,remote_non_temporal_writes, 1860 local_reads_slow_memory,remote_reads_slow_memory,dirty_victim_writes_all 1861 1862 # cat /sys/fs/resctrl/info/L3_MON/event_configs/mbm_local_bytes/event_filter 1863 local_reads,local_non_temporal_writes,local_reads_slow_memory 1864 1865j. Change the event configuration for mbm_local_bytes. 1866:: 1867 1868 # echo "local_reads, local_non_temporal_writes, local_reads_slow_memory, remote_reads" > 1869 /sys/fs/resctrl/info/L3_MON/event_configs/mbm_local_bytes/event_filter 1870 1871 # cat /sys/fs/resctrl/info/L3_MON/event_configs/mbm_local_bytes/event_filter 1872 local_reads,local_non_temporal_writes,local_reads_slow_memory,remote_reads 1873 1874k. Now read the local events again. The first read may come back with "Unavailable" 1875status. The subsequent read of mbm_local_bytes will display the current value. 1876:: 1877 1878 # cat /sys/fs/resctrl/mon_data/mon_L3_00/mbm_local_bytes 1879 Unavailable 1880 # cat /sys/fs/resctrl/mon_data/mon_L3_00/mbm_local_bytes 1881 2252323 1882 # cat /sys/fs/resctrl/mon_data/mon_L3_01/mbm_local_bytes 1883 Unavailable 1884 # cat /sys/fs/resctrl/mon_data/mon_L3_01/mbm_local_bytes 1885 1566565 1886 1887l. Users have the option to go back to 'default' mbm_assign_mode if required. This can be 1888done using the following command. Note that switching the mbm_assign_mode may reset all 1889the MBM counters (and thus all MBM events) of all the resctrl groups. 1890:: 1891 1892 # echo "default" > /sys/fs/resctrl/info/L3_MON/mbm_assign_mode 1893 # cat /sys/fs/resctrl/info/L3_MON/mbm_assign_mode 1894 mbm_event 1895 [default] 1896 1897m. Unmount the resctrl filesystem. 1898:: 1899 1900 # umount /sys/fs/resctrl/ 1901 1902Intel RDT Errata 1903================ 1904 1905Intel MBM Counters May Report System Memory Bandwidth Incorrectly 1906----------------------------------------------------------------- 1907 1908Errata SKX99 for Skylake server and BDF102 for Broadwell server. 1909 1910Problem: Intel Memory Bandwidth Monitoring (MBM) counters track metrics 1911according to the assigned Resource Monitor ID (RMID) for that logical 1912core. The IA32_QM_CTR register (MSR 0xC8E), used to report these 1913metrics, may report incorrect system bandwidth for certain RMID values. 1914 1915Implication: Due to the errata, system memory bandwidth may not match 1916what is reported. 1917 1918Workaround: MBM total and local readings are corrected according to the 1919following correction factor table: 1920 1921+---------------+---------------+---------------+-----------------+ 1922|core count |rmid count |rmid threshold |correction factor| 1923+---------------+---------------+---------------+-----------------+ 1924|1 |8 |0 |1.000000 | 1925+---------------+---------------+---------------+-----------------+ 1926|2 |16 |0 |1.000000 | 1927+---------------+---------------+---------------+-----------------+ 1928|3 |24 |15 |0.969650 | 1929+---------------+---------------+---------------+-----------------+ 1930|4 |32 |0 |1.000000 | 1931+---------------+---------------+---------------+-----------------+ 1932|6 |48 |31 |0.969650 | 1933+---------------+---------------+---------------+-----------------+ 1934|7 |56 |47 |1.142857 | 1935+---------------+---------------+---------------+-----------------+ 1936|8 |64 |0 |1.000000 | 1937+---------------+---------------+---------------+-----------------+ 1938|9 |72 |63 |1.185115 | 1939+---------------+---------------+---------------+-----------------+ 1940|10 |80 |63 |1.066553 | 1941+---------------+---------------+---------------+-----------------+ 1942|11 |88 |79 |1.454545 | 1943+---------------+---------------+---------------+-----------------+ 1944|12 |96 |0 |1.000000 | 1945+---------------+---------------+---------------+-----------------+ 1946|13 |104 |95 |1.230769 | 1947+---------------+---------------+---------------+-----------------+ 1948|14 |112 |95 |1.142857 | 1949+---------------+---------------+---------------+-----------------+ 1950|15 |120 |95 |1.066667 | 1951+---------------+---------------+---------------+-----------------+ 1952|16 |128 |0 |1.000000 | 1953+---------------+---------------+---------------+-----------------+ 1954|17 |136 |127 |1.254863 | 1955+---------------+---------------+---------------+-----------------+ 1956|18 |144 |127 |1.185255 | 1957+---------------+---------------+---------------+-----------------+ 1958|19 |152 |0 |1.000000 | 1959+---------------+---------------+---------------+-----------------+ 1960|20 |160 |127 |1.066667 | 1961+---------------+---------------+---------------+-----------------+ 1962|21 |168 |0 |1.000000 | 1963+---------------+---------------+---------------+-----------------+ 1964|22 |176 |159 |1.454334 | 1965+---------------+---------------+---------------+-----------------+ 1966|23 |184 |0 |1.000000 | 1967+---------------+---------------+---------------+-----------------+ 1968|24 |192 |127 |0.969744 | 1969+---------------+---------------+---------------+-----------------+ 1970|25 |200 |191 |1.280246 | 1971+---------------+---------------+---------------+-----------------+ 1972|26 |208 |191 |1.230921 | 1973+---------------+---------------+---------------+-----------------+ 1974|27 |216 |0 |1.000000 | 1975+---------------+---------------+---------------+-----------------+ 1976|28 |224 |191 |1.143118 | 1977+---------------+---------------+---------------+-----------------+ 1978 1979If rmid > rmid threshold, MBM total and local values should be multiplied 1980by the correction factor. 1981 1982See: 1983 19841. Erratum SKX99 in Intel Xeon Processor Scalable Family Specification Update: 1985http://web.archive.org/web/20200716124958/https://www.intel.com/content/www/us/en/processors/xeon/scalable/xeon-scalable-spec-update.html 1986 19872. Erratum BDF102 in Intel Xeon E5-2600 v4 Processor Product Family Specification Update: 1988http://web.archive.org/web/20191125200531/https://www.intel.com/content/dam/www/public/us/en/documents/specification-updates/xeon-e5-v4-spec-update.pdf 1989 19903. The errata in Intel Resource Director Technology (Intel RDT) on 2nd Generation Intel Xeon Scalable Processors Reference Manual: 1991https://software.intel.com/content/www/us/en/develop/articles/intel-resource-director-technology-rdt-reference-manual.html 1992 1993for further information. 1994