1.. SPDX-License-Identifier: GPL-2.0 2.. include:: <isonum.txt> 3 4=========================================== 5User Interface for Resource Control feature 6=========================================== 7 8:Copyright: |copy| 2016 Intel Corporation 9:Authors: - Fenghua Yu <fenghua.yu@intel.com> 10 - Tony Luck <tony.luck@intel.com> 11 - Vikas Shivappa <vikas.shivappa@intel.com> 12 13 14Intel refers to this feature as Intel Resource Director Technology(Intel(R) RDT). 15AMD refers to this feature as AMD Platform Quality of Service(AMD QoS). 16 17This feature is enabled by the CONFIG_X86_CPU_RESCTRL and the x86 /proc/cpuinfo 18flag bits: 19 20=============================================== ================================ 21RDT (Resource Director Technology) Allocation "rdt_a" 22CAT (Cache Allocation Technology) "cat_l3", "cat_l2" 23CDP (Code and Data Prioritization) "cdp_l3", "cdp_l2" 24CQM (Cache QoS Monitoring) "cqm_llc", "cqm_occup_llc" 25MBM (Memory Bandwidth Monitoring) "cqm_mbm_total", "cqm_mbm_local" 26MBA (Memory Bandwidth Allocation) "mba" 27SMBA (Slow Memory Bandwidth Allocation) "" 28BMEC (Bandwidth Monitoring Event Configuration) "" 29=============================================== ================================ 30 31Historically, new features were made visible by default in /proc/cpuinfo. This 32resulted in the feature flags becoming hard to parse by humans. Adding a new 33flag to /proc/cpuinfo should be avoided if user space can obtain information 34about the feature from resctrl's info directory. 35 36To use the feature mount the file system:: 37 38 # mount -t resctrl resctrl [-o cdp[,cdpl2][,mba_MBps][,debug]] /sys/fs/resctrl 39 40mount options are: 41 42"cdp": 43 Enable code/data prioritization in L3 cache allocations. 44"cdpl2": 45 Enable code/data prioritization in L2 cache allocations. 46"mba_MBps": 47 Enable the MBA Software Controller(mba_sc) to specify MBA 48 bandwidth in MBps 49"debug": 50 Make debug files accessible. Available debug files are annotated with 51 "Available only with debug option". 52 53L2 and L3 CDP are controlled separately. 54 55RDT features are orthogonal. A particular system may support only 56monitoring, only control, or both monitoring and control. Cache 57pseudo-locking is a unique way of using cache control to "pin" or 58"lock" data in the cache. Details can be found in 59"Cache Pseudo-Locking". 60 61 62The mount succeeds if either of allocation or monitoring is present, but 63only those files and directories supported by the system will be created. 64For more details on the behavior of the interface during monitoring 65and allocation, see the "Resource alloc and monitor groups" section. 66 67Info directory 68============== 69 70The 'info' directory contains information about the enabled 71resources. Each resource has its own subdirectory. The subdirectory 72names reflect the resource names. 73 74Each subdirectory contains the following files with respect to 75allocation: 76 77Cache resource(L3/L2) subdirectory contains the following files 78related to allocation: 79 80"num_closids": 81 The number of CLOSIDs which are valid for this 82 resource. The kernel uses the smallest number of 83 CLOSIDs of all enabled resources as limit. 84"cbm_mask": 85 The bitmask which is valid for this resource. 86 This mask is equivalent to 100%. 87"min_cbm_bits": 88 The minimum number of consecutive bits which 89 must be set when writing a mask. 90 91"shareable_bits": 92 Bitmask of shareable resource with other executing 93 entities (e.g. I/O). User can use this when 94 setting up exclusive cache partitions. Note that 95 some platforms support devices that have their 96 own settings for cache use which can over-ride 97 these bits. 98"bit_usage": 99 Annotated capacity bitmasks showing how all 100 instances of the resource are used. The legend is: 101 102 "0": 103 Corresponding region is unused. When the system's 104 resources have been allocated and a "0" is found 105 in "bit_usage" it is a sign that resources are 106 wasted. 107 108 "H": 109 Corresponding region is used by hardware only 110 but available for software use. If a resource 111 has bits set in "shareable_bits" but not all 112 of these bits appear in the resource groups' 113 schematas then the bits appearing in 114 "shareable_bits" but no resource group will 115 be marked as "H". 116 "X": 117 Corresponding region is available for sharing and 118 used by hardware and software. These are the 119 bits that appear in "shareable_bits" as 120 well as a resource group's allocation. 121 "S": 122 Corresponding region is used by software 123 and available for sharing. 124 "E": 125 Corresponding region is used exclusively by 126 one resource group. No sharing allowed. 127 "P": 128 Corresponding region is pseudo-locked. No 129 sharing allowed. 130"sparse_masks": 131 Indicates if non-contiguous 1s value in CBM is supported. 132 133 "0": 134 Only contiguous 1s value in CBM is supported. 135 "1": 136 Non-contiguous 1s value in CBM is supported. 137 138Memory bandwidth(MB) subdirectory contains the following files 139with respect to allocation: 140 141"min_bandwidth": 142 The minimum memory bandwidth percentage which 143 user can request. 144 145"bandwidth_gran": 146 The granularity in which the memory bandwidth 147 percentage is allocated. The allocated 148 b/w percentage is rounded off to the next 149 control step available on the hardware. The 150 available bandwidth control steps are: 151 min_bandwidth + N * bandwidth_gran. 152 153"delay_linear": 154 Indicates if the delay scale is linear or 155 non-linear. This field is purely informational 156 only. 157 158"thread_throttle_mode": 159 Indicator on Intel systems of how tasks running on threads 160 of a physical core are throttled in cases where they 161 request different memory bandwidth percentages: 162 163 "max": 164 the smallest percentage is applied 165 to all threads 166 "per-thread": 167 bandwidth percentages are directly applied to 168 the threads running on the core 169 170If RDT monitoring is available there will be an "L3_MON" directory 171with the following files: 172 173"num_rmids": 174 The number of RMIDs available. This is the 175 upper bound for how many "CTRL_MON" + "MON" 176 groups can be created. 177 178"mon_features": 179 Lists the monitoring events if 180 monitoring is enabled for the resource. 181 Example:: 182 183 # cat /sys/fs/resctrl/info/L3_MON/mon_features 184 llc_occupancy 185 mbm_total_bytes 186 mbm_local_bytes 187 188 If the system supports Bandwidth Monitoring Event 189 Configuration (BMEC), then the bandwidth events will 190 be configurable. The output will be:: 191 192 # cat /sys/fs/resctrl/info/L3_MON/mon_features 193 llc_occupancy 194 mbm_total_bytes 195 mbm_total_bytes_config 196 mbm_local_bytes 197 mbm_local_bytes_config 198 199"mbm_total_bytes_config", "mbm_local_bytes_config": 200 Read/write files containing the configuration for the mbm_total_bytes 201 and mbm_local_bytes events, respectively, when the Bandwidth 202 Monitoring Event Configuration (BMEC) feature is supported. 203 The event configuration settings are domain specific and affect 204 all the CPUs in the domain. When either event configuration is 205 changed, the bandwidth counters for all RMIDs of both events 206 (mbm_total_bytes as well as mbm_local_bytes) are cleared for that 207 domain. The next read for every RMID will report "Unavailable" 208 and subsequent reads will report the valid value. 209 210 Following are the types of events supported: 211 212 ==== ======================================================== 213 Bits Description 214 ==== ======================================================== 215 6 Dirty Victims from the QOS domain to all types of memory 216 5 Reads to slow memory in the non-local NUMA domain 217 4 Reads to slow memory in the local NUMA domain 218 3 Non-temporal writes to non-local NUMA domain 219 2 Non-temporal writes to local NUMA domain 220 1 Reads to memory in the non-local NUMA domain 221 0 Reads to memory in the local NUMA domain 222 ==== ======================================================== 223 224 By default, the mbm_total_bytes configuration is set to 0x7f to count 225 all the event types and the mbm_local_bytes configuration is set to 226 0x15 to count all the local memory events. 227 228 Examples: 229 230 * To view the current configuration:: 231 :: 232 233 # cat /sys/fs/resctrl/info/L3_MON/mbm_total_bytes_config 234 0=0x7f;1=0x7f;2=0x7f;3=0x7f 235 236 # cat /sys/fs/resctrl/info/L3_MON/mbm_local_bytes_config 237 0=0x15;1=0x15;3=0x15;4=0x15 238 239 * To change the mbm_total_bytes to count only reads on domain 0, 240 the bits 0, 1, 4 and 5 needs to be set, which is 110011b in binary 241 (in hexadecimal 0x33): 242 :: 243 244 # echo "0=0x33" > /sys/fs/resctrl/info/L3_MON/mbm_total_bytes_config 245 246 # cat /sys/fs/resctrl/info/L3_MON/mbm_total_bytes_config 247 0=0x33;1=0x7f;2=0x7f;3=0x7f 248 249 * To change the mbm_local_bytes to count all the slow memory reads on 250 domain 0 and 1, the bits 4 and 5 needs to be set, which is 110000b 251 in binary (in hexadecimal 0x30): 252 :: 253 254 # echo "0=0x30;1=0x30" > /sys/fs/resctrl/info/L3_MON/mbm_local_bytes_config 255 256 # cat /sys/fs/resctrl/info/L3_MON/mbm_local_bytes_config 257 0=0x30;1=0x30;3=0x15;4=0x15 258 259"max_threshold_occupancy": 260 Read/write file provides the largest value (in 261 bytes) at which a previously used LLC_occupancy 262 counter can be considered for re-use. 263 264Finally, in the top level of the "info" directory there is a file 265named "last_cmd_status". This is reset with every "command" issued 266via the file system (making new directories or writing to any of the 267control files). If the command was successful, it will read as "ok". 268If the command failed, it will provide more information that can be 269conveyed in the error returns from file operations. E.g. 270:: 271 272 # echo L3:0=f7 > schemata 273 bash: echo: write error: Invalid argument 274 # cat info/last_cmd_status 275 mask f7 has non-consecutive 1-bits 276 277Resource alloc and monitor groups 278================================= 279 280Resource groups are represented as directories in the resctrl file 281system. The default group is the root directory which, immediately 282after mounting, owns all the tasks and cpus in the system and can make 283full use of all resources. 284 285On a system with RDT control features additional directories can be 286created in the root directory that specify different amounts of each 287resource (see "schemata" below). The root and these additional top level 288directories are referred to as "CTRL_MON" groups below. 289 290On a system with RDT monitoring the root directory and other top level 291directories contain a directory named "mon_groups" in which additional 292directories can be created to monitor subsets of tasks in the CTRL_MON 293group that is their ancestor. These are called "MON" groups in the rest 294of this document. 295 296Removing a directory will move all tasks and cpus owned by the group it 297represents to the parent. Removing one of the created CTRL_MON groups 298will automatically remove all MON groups below it. 299 300Moving MON group directories to a new parent CTRL_MON group is supported 301for the purpose of changing the resource allocations of a MON group 302without impacting its monitoring data or assigned tasks. This operation 303is not allowed for MON groups which monitor CPUs. No other move 304operation is currently allowed other than simply renaming a CTRL_MON or 305MON group. 306 307All groups contain the following files: 308 309"tasks": 310 Reading this file shows the list of all tasks that belong to 311 this group. Writing a task id to the file will add a task to the 312 group. Multiple tasks can be added by separating the task ids 313 with commas. Tasks will be assigned sequentially. Multiple 314 failures are not supported. A single failure encountered while 315 attempting to assign a task will cause the operation to abort and 316 already added tasks before the failure will remain in the group. 317 Failures will be logged to /sys/fs/resctrl/info/last_cmd_status. 318 319 If the group is a CTRL_MON group the task is removed from 320 whichever previous CTRL_MON group owned the task and also from 321 any MON group that owned the task. If the group is a MON group, 322 then the task must already belong to the CTRL_MON parent of this 323 group. The task is removed from any previous MON group. 324 325 326"cpus": 327 Reading this file shows a bitmask of the logical CPUs owned by 328 this group. Writing a mask to this file will add and remove 329 CPUs to/from this group. As with the tasks file a hierarchy is 330 maintained where MON groups may only include CPUs owned by the 331 parent CTRL_MON group. 332 When the resource group is in pseudo-locked mode this file will 333 only be readable, reflecting the CPUs associated with the 334 pseudo-locked region. 335 336 337"cpus_list": 338 Just like "cpus", only using ranges of CPUs instead of bitmasks. 339 340 341When control is enabled all CTRL_MON groups will also contain: 342 343"schemata": 344 A list of all the resources available to this group. 345 Each resource has its own line and format - see below for details. 346 347"size": 348 Mirrors the display of the "schemata" file to display the size in 349 bytes of each allocation instead of the bits representing the 350 allocation. 351 352"mode": 353 The "mode" of the resource group dictates the sharing of its 354 allocations. A "shareable" resource group allows sharing of its 355 allocations while an "exclusive" resource group does not. A 356 cache pseudo-locked region is created by first writing 357 "pseudo-locksetup" to the "mode" file before writing the cache 358 pseudo-locked region's schemata to the resource group's "schemata" 359 file. On successful pseudo-locked region creation the mode will 360 automatically change to "pseudo-locked". 361 362"ctrl_hw_id": 363 Available only with debug option. The identifier used by hardware 364 for the control group. On x86 this is the CLOSID. 365 366When monitoring is enabled all MON groups will also contain: 367 368"mon_data": 369 This contains a set of files organized by L3 domain and by 370 RDT event. E.g. on a system with two L3 domains there will 371 be subdirectories "mon_L3_00" and "mon_L3_01". Each of these 372 directories have one file per event (e.g. "llc_occupancy", 373 "mbm_total_bytes", and "mbm_local_bytes"). In a MON group these 374 files provide a read out of the current value of the event for 375 all tasks in the group. In CTRL_MON groups these files provide 376 the sum for all tasks in the CTRL_MON group and all tasks in 377 MON groups. Please see example section for more details on usage. 378 379"mon_hw_id": 380 Available only with debug option. The identifier used by hardware 381 for the monitor group. On x86 this is the RMID. 382 383Resource allocation rules 384------------------------- 385 386When a task is running the following rules define which resources are 387available to it: 388 3891) If the task is a member of a non-default group, then the schemata 390 for that group is used. 391 3922) Else if the task belongs to the default group, but is running on a 393 CPU that is assigned to some specific group, then the schemata for the 394 CPU's group is used. 395 3963) Otherwise the schemata for the default group is used. 397 398Resource monitoring rules 399------------------------- 4001) If a task is a member of a MON group, or non-default CTRL_MON group 401 then RDT events for the task will be reported in that group. 402 4032) If a task is a member of the default CTRL_MON group, but is running 404 on a CPU that is assigned to some specific group, then the RDT events 405 for the task will be reported in that group. 406 4073) Otherwise RDT events for the task will be reported in the root level 408 "mon_data" group. 409 410 411Notes on cache occupancy monitoring and control 412=============================================== 413When moving a task from one group to another you should remember that 414this only affects *new* cache allocations by the task. E.g. you may have 415a task in a monitor group showing 3 MB of cache occupancy. If you move 416to a new group and immediately check the occupancy of the old and new 417groups you will likely see that the old group is still showing 3 MB and 418the new group zero. When the task accesses locations still in cache from 419before the move, the h/w does not update any counters. On a busy system 420you will likely see the occupancy in the old group go down as cache lines 421are evicted and re-used while the occupancy in the new group rises as 422the task accesses memory and loads into the cache are counted based on 423membership in the new group. 424 425The same applies to cache allocation control. Moving a task to a group 426with a smaller cache partition will not evict any cache lines. The 427process may continue to use them from the old partition. 428 429Hardware uses CLOSid(Class of service ID) and an RMID(Resource monitoring ID) 430to identify a control group and a monitoring group respectively. Each of 431the resource groups are mapped to these IDs based on the kind of group. The 432number of CLOSid and RMID are limited by the hardware and hence the creation of 433a "CTRL_MON" directory may fail if we run out of either CLOSID or RMID 434and creation of "MON" group may fail if we run out of RMIDs. 435 436max_threshold_occupancy - generic concepts 437------------------------------------------ 438 439Note that an RMID once freed may not be immediately available for use as 440the RMID is still tagged the cache lines of the previous user of RMID. 441Hence such RMIDs are placed on limbo list and checked back if the cache 442occupancy has gone down. If there is a time when system has a lot of 443limbo RMIDs but which are not ready to be used, user may see an -EBUSY 444during mkdir. 445 446max_threshold_occupancy is a user configurable value to determine the 447occupancy at which an RMID can be freed. 448 449Schemata files - general concepts 450--------------------------------- 451Each line in the file describes one resource. The line starts with 452the name of the resource, followed by specific values to be applied 453in each of the instances of that resource on the system. 454 455Cache IDs 456--------- 457On current generation systems there is one L3 cache per socket and L2 458caches are generally just shared by the hyperthreads on a core, but this 459isn't an architectural requirement. We could have multiple separate L3 460caches on a socket, multiple cores could share an L2 cache. So instead 461of using "socket" or "core" to define the set of logical cpus sharing 462a resource we use a "Cache ID". At a given cache level this will be a 463unique number across the whole system (but it isn't guaranteed to be a 464contiguous sequence, there may be gaps). To find the ID for each logical 465CPU look in /sys/devices/system/cpu/cpu*/cache/index*/id 466 467Cache Bit Masks (CBM) 468--------------------- 469For cache resources we describe the portion of the cache that is available 470for allocation using a bitmask. The maximum value of the mask is defined 471by each cpu model (and may be different for different cache levels). It 472is found using CPUID, but is also provided in the "info" directory of 473the resctrl file system in "info/{resource}/cbm_mask". Some Intel hardware 474requires that these masks have all the '1' bits in a contiguous block. So 4750x3, 0x6 and 0xC are legal 4-bit masks with two bits set, but 0x5, 0x9 476and 0xA are not. Check /sys/fs/resctrl/info/{resource}/sparse_masks 477if non-contiguous 1s value is supported. On a system with a 20-bit mask 478each bit represents 5% of the capacity of the cache. You could partition 479the cache into four equal parts with masks: 0x1f, 0x3e0, 0x7c00, 0xf8000. 480 481Memory bandwidth Allocation and monitoring 482========================================== 483 484For Memory bandwidth resource, by default the user controls the resource 485by indicating the percentage of total memory bandwidth. 486 487The minimum bandwidth percentage value for each cpu model is predefined 488and can be looked up through "info/MB/min_bandwidth". The bandwidth 489granularity that is allocated is also dependent on the cpu model and can 490be looked up at "info/MB/bandwidth_gran". The available bandwidth 491control steps are: min_bw + N * bw_gran. Intermediate values are rounded 492to the next control step available on the hardware. 493 494The bandwidth throttling is a core specific mechanism on some of Intel 495SKUs. Using a high bandwidth and a low bandwidth setting on two threads 496sharing a core may result in both threads being throttled to use the 497low bandwidth (see "thread_throttle_mode"). 498 499The fact that Memory bandwidth allocation(MBA) may be a core 500specific mechanism where as memory bandwidth monitoring(MBM) is done at 501the package level may lead to confusion when users try to apply control 502via the MBA and then monitor the bandwidth to see if the controls are 503effective. Below are such scenarios: 504 5051. User may *not* see increase in actual bandwidth when percentage 506 values are increased: 507 508This can occur when aggregate L2 external bandwidth is more than L3 509external bandwidth. Consider an SKL SKU with 24 cores on a package and 510where L2 external is 10GBps (hence aggregate L2 external bandwidth is 511240GBps) and L3 external bandwidth is 100GBps. Now a workload with '20 512threads, having 50% bandwidth, each consuming 5GBps' consumes the max L3 513bandwidth of 100GBps although the percentage value specified is only 50% 514<< 100%. Hence increasing the bandwidth percentage will not yield any 515more bandwidth. This is because although the L2 external bandwidth still 516has capacity, the L3 external bandwidth is fully used. Also note that 517this would be dependent on number of cores the benchmark is run on. 518 5192. Same bandwidth percentage may mean different actual bandwidth 520 depending on # of threads: 521 522For the same SKU in #1, a 'single thread, with 10% bandwidth' and '4 523thread, with 10% bandwidth' can consume upto 10GBps and 40GBps although 524they have same percentage bandwidth of 10%. This is simply because as 525threads start using more cores in an rdtgroup, the actual bandwidth may 526increase or vary although user specified bandwidth percentage is same. 527 528In order to mitigate this and make the interface more user friendly, 529resctrl added support for specifying the bandwidth in MBps as well. The 530kernel underneath would use a software feedback mechanism or a "Software 531Controller(mba_sc)" which reads the actual bandwidth using MBM counters 532and adjust the memory bandwidth percentages to ensure:: 533 534 "actual bandwidth < user specified bandwidth". 535 536By default, the schemata would take the bandwidth percentage values 537where as user can switch to the "MBA software controller" mode using 538a mount option 'mba_MBps'. The schemata format is specified in the below 539sections. 540 541L3 schemata file details (code and data prioritization disabled) 542---------------------------------------------------------------- 543With CDP disabled the L3 schemata format is:: 544 545 L3:<cache_id0>=<cbm>;<cache_id1>=<cbm>;... 546 547L3 schemata file details (CDP enabled via mount option to resctrl) 548------------------------------------------------------------------ 549When CDP is enabled L3 control is split into two separate resources 550so you can specify independent masks for code and data like this:: 551 552 L3DATA:<cache_id0>=<cbm>;<cache_id1>=<cbm>;... 553 L3CODE:<cache_id0>=<cbm>;<cache_id1>=<cbm>;... 554 555L2 schemata file details 556------------------------ 557CDP is supported at L2 using the 'cdpl2' mount option. The schemata 558format is either:: 559 560 L2:<cache_id0>=<cbm>;<cache_id1>=<cbm>;... 561 562or 563 564 L2DATA:<cache_id0>=<cbm>;<cache_id1>=<cbm>;... 565 L2CODE:<cache_id0>=<cbm>;<cache_id1>=<cbm>;... 566 567 568Memory bandwidth Allocation (default mode) 569------------------------------------------ 570 571Memory b/w domain is L3 cache. 572:: 573 574 MB:<cache_id0>=bandwidth0;<cache_id1>=bandwidth1;... 575 576Memory bandwidth Allocation specified in MBps 577--------------------------------------------- 578 579Memory bandwidth domain is L3 cache. 580:: 581 582 MB:<cache_id0>=bw_MBps0;<cache_id1>=bw_MBps1;... 583 584Slow Memory Bandwidth Allocation (SMBA) 585--------------------------------------- 586AMD hardware supports Slow Memory Bandwidth Allocation (SMBA). 587CXL.memory is the only supported "slow" memory device. With the 588support of SMBA, the hardware enables bandwidth allocation on 589the slow memory devices. If there are multiple such devices in 590the system, the throttling logic groups all the slow sources 591together and applies the limit on them as a whole. 592 593The presence of SMBA (with CXL.memory) is independent of slow memory 594devices presence. If there are no such devices on the system, then 595configuring SMBA will have no impact on the performance of the system. 596 597The bandwidth domain for slow memory is L3 cache. Its schemata file 598is formatted as: 599:: 600 601 SMBA:<cache_id0>=bandwidth0;<cache_id1>=bandwidth1;... 602 603Reading/writing the schemata file 604--------------------------------- 605Reading the schemata file will show the state of all resources 606on all domains. When writing you only need to specify those values 607which you wish to change. E.g. 608:: 609 610 # cat schemata 611 L3DATA:0=fffff;1=fffff;2=fffff;3=fffff 612 L3CODE:0=fffff;1=fffff;2=fffff;3=fffff 613 # echo "L3DATA:2=3c0;" > schemata 614 # cat schemata 615 L3DATA:0=fffff;1=fffff;2=3c0;3=fffff 616 L3CODE:0=fffff;1=fffff;2=fffff;3=fffff 617 618Reading/writing the schemata file (on AMD systems) 619-------------------------------------------------- 620Reading the schemata file will show the current bandwidth limit on all 621domains. The allocated resources are in multiples of one eighth GB/s. 622When writing to the file, you need to specify what cache id you wish to 623configure the bandwidth limit. 624 625For example, to allocate 2GB/s limit on the first cache id: 626 627:: 628 629 # cat schemata 630 MB:0=2048;1=2048;2=2048;3=2048 631 L3:0=ffff;1=ffff;2=ffff;3=ffff 632 633 # echo "MB:1=16" > schemata 634 # cat schemata 635 MB:0=2048;1= 16;2=2048;3=2048 636 L3:0=ffff;1=ffff;2=ffff;3=ffff 637 638Reading/writing the schemata file (on AMD systems) with SMBA feature 639-------------------------------------------------------------------- 640Reading and writing the schemata file is the same as without SMBA in 641above section. 642 643For example, to allocate 8GB/s limit on the first cache id: 644 645:: 646 647 # cat schemata 648 SMBA:0=2048;1=2048;2=2048;3=2048 649 MB:0=2048;1=2048;2=2048;3=2048 650 L3:0=ffff;1=ffff;2=ffff;3=ffff 651 652 # echo "SMBA:1=64" > schemata 653 # cat schemata 654 SMBA:0=2048;1= 64;2=2048;3=2048 655 MB:0=2048;1=2048;2=2048;3=2048 656 L3:0=ffff;1=ffff;2=ffff;3=ffff 657 658Cache Pseudo-Locking 659==================== 660CAT enables a user to specify the amount of cache space that an 661application can fill. Cache pseudo-locking builds on the fact that a 662CPU can still read and write data pre-allocated outside its current 663allocated area on a cache hit. With cache pseudo-locking, data can be 664preloaded into a reserved portion of cache that no application can 665fill, and from that point on will only serve cache hits. The cache 666pseudo-locked memory is made accessible to user space where an 667application can map it into its virtual address space and thus have 668a region of memory with reduced average read latency. 669 670The creation of a cache pseudo-locked region is triggered by a request 671from the user to do so that is accompanied by a schemata of the region 672to be pseudo-locked. The cache pseudo-locked region is created as follows: 673 674- Create a CAT allocation CLOSNEW with a CBM matching the schemata 675 from the user of the cache region that will contain the pseudo-locked 676 memory. This region must not overlap with any current CAT allocation/CLOS 677 on the system and no future overlap with this cache region is allowed 678 while the pseudo-locked region exists. 679- Create a contiguous region of memory of the same size as the cache 680 region. 681- Flush the cache, disable hardware prefetchers, disable preemption. 682- Make CLOSNEW the active CLOS and touch the allocated memory to load 683 it into the cache. 684- Set the previous CLOS as active. 685- At this point the closid CLOSNEW can be released - the cache 686 pseudo-locked region is protected as long as its CBM does not appear in 687 any CAT allocation. Even though the cache pseudo-locked region will from 688 this point on not appear in any CBM of any CLOS an application running with 689 any CLOS will be able to access the memory in the pseudo-locked region since 690 the region continues to serve cache hits. 691- The contiguous region of memory loaded into the cache is exposed to 692 user-space as a character device. 693 694Cache pseudo-locking increases the probability that data will remain 695in the cache via carefully configuring the CAT feature and controlling 696application behavior. There is no guarantee that data is placed in 697cache. Instructions like INVD, WBINVD, CLFLUSH, etc. can still evict 698“locked” data from cache. Power management C-states may shrink or 699power off cache. Deeper C-states will automatically be restricted on 700pseudo-locked region creation. 701 702It is required that an application using a pseudo-locked region runs 703with affinity to the cores (or a subset of the cores) associated 704with the cache on which the pseudo-locked region resides. A sanity check 705within the code will not allow an application to map pseudo-locked memory 706unless it runs with affinity to cores associated with the cache on which the 707pseudo-locked region resides. The sanity check is only done during the 708initial mmap() handling, there is no enforcement afterwards and the 709application self needs to ensure it remains affine to the correct cores. 710 711Pseudo-locking is accomplished in two stages: 712 7131) During the first stage the system administrator allocates a portion 714 of cache that should be dedicated to pseudo-locking. At this time an 715 equivalent portion of memory is allocated, loaded into allocated 716 cache portion, and exposed as a character device. 7172) During the second stage a user-space application maps (mmap()) the 718 pseudo-locked memory into its address space. 719 720Cache Pseudo-Locking Interface 721------------------------------ 722A pseudo-locked region is created using the resctrl interface as follows: 723 7241) Create a new resource group by creating a new directory in /sys/fs/resctrl. 7252) Change the new resource group's mode to "pseudo-locksetup" by writing 726 "pseudo-locksetup" to the "mode" file. 7273) Write the schemata of the pseudo-locked region to the "schemata" file. All 728 bits within the schemata should be "unused" according to the "bit_usage" 729 file. 730 731On successful pseudo-locked region creation the "mode" file will contain 732"pseudo-locked" and a new character device with the same name as the resource 733group will exist in /dev/pseudo_lock. This character device can be mmap()'ed 734by user space in order to obtain access to the pseudo-locked memory region. 735 736An example of cache pseudo-locked region creation and usage can be found below. 737 738Cache Pseudo-Locking Debugging Interface 739---------------------------------------- 740The pseudo-locking debugging interface is enabled by default (if 741CONFIG_DEBUG_FS is enabled) and can be found in /sys/kernel/debug/resctrl. 742 743There is no explicit way for the kernel to test if a provided memory 744location is present in the cache. The pseudo-locking debugging interface uses 745the tracing infrastructure to provide two ways to measure cache residency of 746the pseudo-locked region: 747 7481) Memory access latency using the pseudo_lock_mem_latency tracepoint. Data 749 from these measurements are best visualized using a hist trigger (see 750 example below). In this test the pseudo-locked region is traversed at 751 a stride of 32 bytes while hardware prefetchers and preemption 752 are disabled. This also provides a substitute visualization of cache 753 hits and misses. 7542) Cache hit and miss measurements using model specific precision counters if 755 available. Depending on the levels of cache on the system the pseudo_lock_l2 756 and pseudo_lock_l3 tracepoints are available. 757 758When a pseudo-locked region is created a new debugfs directory is created for 759it in debugfs as /sys/kernel/debug/resctrl/<newdir>. A single 760write-only file, pseudo_lock_measure, is present in this directory. The 761measurement of the pseudo-locked region depends on the number written to this 762debugfs file: 763 7641: 765 writing "1" to the pseudo_lock_measure file will trigger the latency 766 measurement captured in the pseudo_lock_mem_latency tracepoint. See 767 example below. 7682: 769 writing "2" to the pseudo_lock_measure file will trigger the L2 cache 770 residency (cache hits and misses) measurement captured in the 771 pseudo_lock_l2 tracepoint. See example below. 7723: 773 writing "3" to the pseudo_lock_measure file will trigger the L3 cache 774 residency (cache hits and misses) measurement captured in the 775 pseudo_lock_l3 tracepoint. 776 777All measurements are recorded with the tracing infrastructure. This requires 778the relevant tracepoints to be enabled before the measurement is triggered. 779 780Example of latency debugging interface 781~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 782In this example a pseudo-locked region named "newlock" was created. Here is 783how we can measure the latency in cycles of reading from this region and 784visualize this data with a histogram that is available if CONFIG_HIST_TRIGGERS 785is set:: 786 787 # :> /sys/kernel/tracing/trace 788 # echo 'hist:keys=latency' > /sys/kernel/tracing/events/resctrl/pseudo_lock_mem_latency/trigger 789 # echo 1 > /sys/kernel/tracing/events/resctrl/pseudo_lock_mem_latency/enable 790 # echo 1 > /sys/kernel/debug/resctrl/newlock/pseudo_lock_measure 791 # echo 0 > /sys/kernel/tracing/events/resctrl/pseudo_lock_mem_latency/enable 792 # cat /sys/kernel/tracing/events/resctrl/pseudo_lock_mem_latency/hist 793 794 # event histogram 795 # 796 # trigger info: hist:keys=latency:vals=hitcount:sort=hitcount:size=2048 [active] 797 # 798 799 { latency: 456 } hitcount: 1 800 { latency: 50 } hitcount: 83 801 { latency: 36 } hitcount: 96 802 { latency: 44 } hitcount: 174 803 { latency: 48 } hitcount: 195 804 { latency: 46 } hitcount: 262 805 { latency: 42 } hitcount: 693 806 { latency: 40 } hitcount: 3204 807 { latency: 38 } hitcount: 3484 808 809 Totals: 810 Hits: 8192 811 Entries: 9 812 Dropped: 0 813 814Example of cache hits/misses debugging 815~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 816In this example a pseudo-locked region named "newlock" was created on the L2 817cache of a platform. Here is how we can obtain details of the cache hits 818and misses using the platform's precision counters. 819:: 820 821 # :> /sys/kernel/tracing/trace 822 # echo 1 > /sys/kernel/tracing/events/resctrl/pseudo_lock_l2/enable 823 # echo 2 > /sys/kernel/debug/resctrl/newlock/pseudo_lock_measure 824 # echo 0 > /sys/kernel/tracing/events/resctrl/pseudo_lock_l2/enable 825 # cat /sys/kernel/tracing/trace 826 827 # tracer: nop 828 # 829 # _-----=> irqs-off 830 # / _----=> need-resched 831 # | / _---=> hardirq/softirq 832 # || / _--=> preempt-depth 833 # ||| / delay 834 # TASK-PID CPU# |||| TIMESTAMP FUNCTION 835 # | | | |||| | | 836 pseudo_lock_mea-1672 [002] .... 3132.860500: pseudo_lock_l2: hits=4097 miss=0 837 838 839Examples for RDT allocation usage 840~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 841 8421) Example 1 843 844On a two socket machine (one L3 cache per socket) with just four bits 845for cache bit masks, minimum b/w of 10% with a memory bandwidth 846granularity of 10%. 847:: 848 849 # mount -t resctrl resctrl /sys/fs/resctrl 850 # cd /sys/fs/resctrl 851 # mkdir p0 p1 852 # echo "L3:0=3;1=c\nMB:0=50;1=50" > /sys/fs/resctrl/p0/schemata 853 # echo "L3:0=3;1=3\nMB:0=50;1=50" > /sys/fs/resctrl/p1/schemata 854 855The default resource group is unmodified, so we have access to all parts 856of all caches (its schemata file reads "L3:0=f;1=f"). 857 858Tasks that are under the control of group "p0" may only allocate from the 859"lower" 50% on cache ID 0, and the "upper" 50% of cache ID 1. 860Tasks in group "p1" use the "lower" 50% of cache on both sockets. 861 862Similarly, tasks that are under the control of group "p0" may use a 863maximum memory b/w of 50% on socket0 and 50% on socket 1. 864Tasks in group "p1" may also use 50% memory b/w on both sockets. 865Note that unlike cache masks, memory b/w cannot specify whether these 866allocations can overlap or not. The allocations specifies the maximum 867b/w that the group may be able to use and the system admin can configure 868the b/w accordingly. 869 870If resctrl is using the software controller (mba_sc) then user can enter the 871max b/w in MB rather than the percentage values. 872:: 873 874 # echo "L3:0=3;1=c\nMB:0=1024;1=500" > /sys/fs/resctrl/p0/schemata 875 # echo "L3:0=3;1=3\nMB:0=1024;1=500" > /sys/fs/resctrl/p1/schemata 876 877In the above example the tasks in "p1" and "p0" on socket 0 would use a max b/w 878of 1024MB where as on socket 1 they would use 500MB. 879 8802) Example 2 881 882Again two sockets, but this time with a more realistic 20-bit mask. 883 884Two real time tasks pid=1234 running on processor 0 and pid=5678 running on 885processor 1 on socket 0 on a 2-socket and dual core machine. To avoid noisy 886neighbors, each of the two real-time tasks exclusively occupies one quarter 887of L3 cache on socket 0. 888:: 889 890 # mount -t resctrl resctrl /sys/fs/resctrl 891 # cd /sys/fs/resctrl 892 893First we reset the schemata for the default group so that the "upper" 89450% of the L3 cache on socket 0 and 50% of memory b/w cannot be used by 895ordinary tasks:: 896 897 # echo "L3:0=3ff;1=fffff\nMB:0=50;1=100" > schemata 898 899Next we make a resource group for our first real time task and give 900it access to the "top" 25% of the cache on socket 0. 901:: 902 903 # mkdir p0 904 # echo "L3:0=f8000;1=fffff" > p0/schemata 905 906Finally we move our first real time task into this resource group. We 907also use taskset(1) to ensure the task always runs on a dedicated CPU 908on socket 0. Most uses of resource groups will also constrain which 909processors tasks run on. 910:: 911 912 # echo 1234 > p0/tasks 913 # taskset -cp 1 1234 914 915Ditto for the second real time task (with the remaining 25% of cache):: 916 917 # mkdir p1 918 # echo "L3:0=7c00;1=fffff" > p1/schemata 919 # echo 5678 > p1/tasks 920 # taskset -cp 2 5678 921 922For the same 2 socket system with memory b/w resource and CAT L3 the 923schemata would look like(Assume min_bandwidth 10 and bandwidth_gran is 92410): 925 926For our first real time task this would request 20% memory b/w on socket 0. 927:: 928 929 # echo -e "L3:0=f8000;1=fffff\nMB:0=20;1=100" > p0/schemata 930 931For our second real time task this would request an other 20% memory b/w 932on socket 0. 933:: 934 935 # echo -e "L3:0=f8000;1=fffff\nMB:0=20;1=100" > p0/schemata 936 9373) Example 3 938 939A single socket system which has real-time tasks running on core 4-7 and 940non real-time workload assigned to core 0-3. The real-time tasks share text 941and data, so a per task association is not required and due to interaction 942with the kernel it's desired that the kernel on these cores shares L3 with 943the tasks. 944:: 945 946 # mount -t resctrl resctrl /sys/fs/resctrl 947 # cd /sys/fs/resctrl 948 949First we reset the schemata for the default group so that the "upper" 95050% of the L3 cache on socket 0, and 50% of memory bandwidth on socket 0 951cannot be used by ordinary tasks:: 952 953 # echo "L3:0=3ff\nMB:0=50" > schemata 954 955Next we make a resource group for our real time cores and give it access 956to the "top" 50% of the cache on socket 0 and 50% of memory bandwidth on 957socket 0. 958:: 959 960 # mkdir p0 961 # echo "L3:0=ffc00\nMB:0=50" > p0/schemata 962 963Finally we move core 4-7 over to the new group and make sure that the 964kernel and the tasks running there get 50% of the cache. They should 965also get 50% of memory bandwidth assuming that the cores 4-7 are SMT 966siblings and only the real time threads are scheduled on the cores 4-7. 967:: 968 969 # echo F0 > p0/cpus 970 9714) Example 4 972 973The resource groups in previous examples were all in the default "shareable" 974mode allowing sharing of their cache allocations. If one resource group 975configures a cache allocation then nothing prevents another resource group 976to overlap with that allocation. 977 978In this example a new exclusive resource group will be created on a L2 CAT 979system with two L2 cache instances that can be configured with an 8-bit 980capacity bitmask. The new exclusive resource group will be configured to use 98125% of each cache instance. 982:: 983 984 # mount -t resctrl resctrl /sys/fs/resctrl/ 985 # cd /sys/fs/resctrl 986 987First, we observe that the default group is configured to allocate to all L2 988cache:: 989 990 # cat schemata 991 L2:0=ff;1=ff 992 993We could attempt to create the new resource group at this point, but it will 994fail because of the overlap with the schemata of the default group:: 995 996 # mkdir p0 997 # echo 'L2:0=0x3;1=0x3' > p0/schemata 998 # cat p0/mode 999 shareable 1000 # echo exclusive > p0/mode 1001 -sh: echo: write error: Invalid argument 1002 # cat info/last_cmd_status 1003 schemata overlaps 1004 1005To ensure that there is no overlap with another resource group the default 1006resource group's schemata has to change, making it possible for the new 1007resource group to become exclusive. 1008:: 1009 1010 # echo 'L2:0=0xfc;1=0xfc' > schemata 1011 # echo exclusive > p0/mode 1012 # grep . p0/* 1013 p0/cpus:0 1014 p0/mode:exclusive 1015 p0/schemata:L2:0=03;1=03 1016 p0/size:L2:0=262144;1=262144 1017 1018A new resource group will on creation not overlap with an exclusive resource 1019group:: 1020 1021 # mkdir p1 1022 # grep . p1/* 1023 p1/cpus:0 1024 p1/mode:shareable 1025 p1/schemata:L2:0=fc;1=fc 1026 p1/size:L2:0=786432;1=786432 1027 1028The bit_usage will reflect how the cache is used:: 1029 1030 # cat info/L2/bit_usage 1031 0=SSSSSSEE;1=SSSSSSEE 1032 1033A resource group cannot be forced to overlap with an exclusive resource group:: 1034 1035 # echo 'L2:0=0x1;1=0x1' > p1/schemata 1036 -sh: echo: write error: Invalid argument 1037 # cat info/last_cmd_status 1038 overlaps with exclusive group 1039 1040Example of Cache Pseudo-Locking 1041~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1042Lock portion of L2 cache from cache id 1 using CBM 0x3. Pseudo-locked 1043region is exposed at /dev/pseudo_lock/newlock that can be provided to 1044application for argument to mmap(). 1045:: 1046 1047 # mount -t resctrl resctrl /sys/fs/resctrl/ 1048 # cd /sys/fs/resctrl 1049 1050Ensure that there are bits available that can be pseudo-locked, since only 1051unused bits can be pseudo-locked the bits to be pseudo-locked needs to be 1052removed from the default resource group's schemata:: 1053 1054 # cat info/L2/bit_usage 1055 0=SSSSSSSS;1=SSSSSSSS 1056 # echo 'L2:1=0xfc' > schemata 1057 # cat info/L2/bit_usage 1058 0=SSSSSSSS;1=SSSSSS00 1059 1060Create a new resource group that will be associated with the pseudo-locked 1061region, indicate that it will be used for a pseudo-locked region, and 1062configure the requested pseudo-locked region capacity bitmask:: 1063 1064 # mkdir newlock 1065 # echo pseudo-locksetup > newlock/mode 1066 # echo 'L2:1=0x3' > newlock/schemata 1067 1068On success the resource group's mode will change to pseudo-locked, the 1069bit_usage will reflect the pseudo-locked region, and the character device 1070exposing the pseudo-locked region will exist:: 1071 1072 # cat newlock/mode 1073 pseudo-locked 1074 # cat info/L2/bit_usage 1075 0=SSSSSSSS;1=SSSSSSPP 1076 # ls -l /dev/pseudo_lock/newlock 1077 crw------- 1 root root 243, 0 Apr 3 05:01 /dev/pseudo_lock/newlock 1078 1079:: 1080 1081 /* 1082 * Example code to access one page of pseudo-locked cache region 1083 * from user space. 1084 */ 1085 #define _GNU_SOURCE 1086 #include <fcntl.h> 1087 #include <sched.h> 1088 #include <stdio.h> 1089 #include <stdlib.h> 1090 #include <unistd.h> 1091 #include <sys/mman.h> 1092 1093 /* 1094 * It is required that the application runs with affinity to only 1095 * cores associated with the pseudo-locked region. Here the cpu 1096 * is hardcoded for convenience of example. 1097 */ 1098 static int cpuid = 2; 1099 1100 int main(int argc, char *argv[]) 1101 { 1102 cpu_set_t cpuset; 1103 long page_size; 1104 void *mapping; 1105 int dev_fd; 1106 int ret; 1107 1108 page_size = sysconf(_SC_PAGESIZE); 1109 1110 CPU_ZERO(&cpuset); 1111 CPU_SET(cpuid, &cpuset); 1112 ret = sched_setaffinity(0, sizeof(cpuset), &cpuset); 1113 if (ret < 0) { 1114 perror("sched_setaffinity"); 1115 exit(EXIT_FAILURE); 1116 } 1117 1118 dev_fd = open("/dev/pseudo_lock/newlock", O_RDWR); 1119 if (dev_fd < 0) { 1120 perror("open"); 1121 exit(EXIT_FAILURE); 1122 } 1123 1124 mapping = mmap(0, page_size, PROT_READ | PROT_WRITE, MAP_SHARED, 1125 dev_fd, 0); 1126 if (mapping == MAP_FAILED) { 1127 perror("mmap"); 1128 close(dev_fd); 1129 exit(EXIT_FAILURE); 1130 } 1131 1132 /* Application interacts with pseudo-locked memory @mapping */ 1133 1134 ret = munmap(mapping, page_size); 1135 if (ret < 0) { 1136 perror("munmap"); 1137 close(dev_fd); 1138 exit(EXIT_FAILURE); 1139 } 1140 1141 close(dev_fd); 1142 exit(EXIT_SUCCESS); 1143 } 1144 1145Locking between applications 1146---------------------------- 1147 1148Certain operations on the resctrl filesystem, composed of read/writes 1149to/from multiple files, must be atomic. 1150 1151As an example, the allocation of an exclusive reservation of L3 cache 1152involves: 1153 1154 1. Read the cbmmasks from each directory or the per-resource "bit_usage" 1155 2. Find a contiguous set of bits in the global CBM bitmask that is clear 1156 in any of the directory cbmmasks 1157 3. Create a new directory 1158 4. Set the bits found in step 2 to the new directory "schemata" file 1159 1160If two applications attempt to allocate space concurrently then they can 1161end up allocating the same bits so the reservations are shared instead of 1162exclusive. 1163 1164To coordinate atomic operations on the resctrlfs and to avoid the problem 1165above, the following locking procedure is recommended: 1166 1167Locking is based on flock, which is available in libc and also as a shell 1168script command 1169 1170Write lock: 1171 1172 A) Take flock(LOCK_EX) on /sys/fs/resctrl 1173 B) Read/write the directory structure. 1174 C) funlock 1175 1176Read lock: 1177 1178 A) Take flock(LOCK_SH) on /sys/fs/resctrl 1179 B) If success read the directory structure. 1180 C) funlock 1181 1182Example with bash:: 1183 1184 # Atomically read directory structure 1185 $ flock -s /sys/fs/resctrl/ find /sys/fs/resctrl 1186 1187 # Read directory contents and create new subdirectory 1188 1189 $ cat create-dir.sh 1190 find /sys/fs/resctrl/ > output.txt 1191 mask = function-of(output.txt) 1192 mkdir /sys/fs/resctrl/newres/ 1193 echo mask > /sys/fs/resctrl/newres/schemata 1194 1195 $ flock /sys/fs/resctrl/ ./create-dir.sh 1196 1197Example with C:: 1198 1199 /* 1200 * Example code do take advisory locks 1201 * before accessing resctrl filesystem 1202 */ 1203 #include <sys/file.h> 1204 #include <stdlib.h> 1205 1206 void resctrl_take_shared_lock(int fd) 1207 { 1208 int ret; 1209 1210 /* take shared lock on resctrl filesystem */ 1211 ret = flock(fd, LOCK_SH); 1212 if (ret) { 1213 perror("flock"); 1214 exit(-1); 1215 } 1216 } 1217 1218 void resctrl_take_exclusive_lock(int fd) 1219 { 1220 int ret; 1221 1222 /* release lock on resctrl filesystem */ 1223 ret = flock(fd, LOCK_EX); 1224 if (ret) { 1225 perror("flock"); 1226 exit(-1); 1227 } 1228 } 1229 1230 void resctrl_release_lock(int fd) 1231 { 1232 int ret; 1233 1234 /* take shared lock on resctrl filesystem */ 1235 ret = flock(fd, LOCK_UN); 1236 if (ret) { 1237 perror("flock"); 1238 exit(-1); 1239 } 1240 } 1241 1242 void main(void) 1243 { 1244 int fd, ret; 1245 1246 fd = open("/sys/fs/resctrl", O_DIRECTORY); 1247 if (fd == -1) { 1248 perror("open"); 1249 exit(-1); 1250 } 1251 resctrl_take_shared_lock(fd); 1252 /* code to read directory contents */ 1253 resctrl_release_lock(fd); 1254 1255 resctrl_take_exclusive_lock(fd); 1256 /* code to read and write directory contents */ 1257 resctrl_release_lock(fd); 1258 } 1259 1260Examples for RDT Monitoring along with allocation usage 1261======================================================= 1262Reading monitored data 1263---------------------- 1264Reading an event file (for ex: mon_data/mon_L3_00/llc_occupancy) would 1265show the current snapshot of LLC occupancy of the corresponding MON 1266group or CTRL_MON group. 1267 1268 1269Example 1 (Monitor CTRL_MON group and subset of tasks in CTRL_MON group) 1270------------------------------------------------------------------------ 1271On a two socket machine (one L3 cache per socket) with just four bits 1272for cache bit masks:: 1273 1274 # mount -t resctrl resctrl /sys/fs/resctrl 1275 # cd /sys/fs/resctrl 1276 # mkdir p0 p1 1277 # echo "L3:0=3;1=c" > /sys/fs/resctrl/p0/schemata 1278 # echo "L3:0=3;1=3" > /sys/fs/resctrl/p1/schemata 1279 # echo 5678 > p1/tasks 1280 # echo 5679 > p1/tasks 1281 1282The default resource group is unmodified, so we have access to all parts 1283of all caches (its schemata file reads "L3:0=f;1=f"). 1284 1285Tasks that are under the control of group "p0" may only allocate from the 1286"lower" 50% on cache ID 0, and the "upper" 50% of cache ID 1. 1287Tasks in group "p1" use the "lower" 50% of cache on both sockets. 1288 1289Create monitor groups and assign a subset of tasks to each monitor group. 1290:: 1291 1292 # cd /sys/fs/resctrl/p1/mon_groups 1293 # mkdir m11 m12 1294 # echo 5678 > m11/tasks 1295 # echo 5679 > m12/tasks 1296 1297fetch data (data shown in bytes) 1298:: 1299 1300 # cat m11/mon_data/mon_L3_00/llc_occupancy 1301 16234000 1302 # cat m11/mon_data/mon_L3_01/llc_occupancy 1303 14789000 1304 # cat m12/mon_data/mon_L3_00/llc_occupancy 1305 16789000 1306 1307The parent ctrl_mon group shows the aggregated data. 1308:: 1309 1310 # cat /sys/fs/resctrl/p1/mon_data/mon_l3_00/llc_occupancy 1311 31234000 1312 1313Example 2 (Monitor a task from its creation) 1314-------------------------------------------- 1315On a two socket machine (one L3 cache per socket):: 1316 1317 # mount -t resctrl resctrl /sys/fs/resctrl 1318 # cd /sys/fs/resctrl 1319 # mkdir p0 p1 1320 1321An RMID is allocated to the group once its created and hence the <cmd> 1322below is monitored from its creation. 1323:: 1324 1325 # echo $$ > /sys/fs/resctrl/p1/tasks 1326 # <cmd> 1327 1328Fetch the data:: 1329 1330 # cat /sys/fs/resctrl/p1/mon_data/mon_l3_00/llc_occupancy 1331 31789000 1332 1333Example 3 (Monitor without CAT support or before creating CAT groups) 1334--------------------------------------------------------------------- 1335 1336Assume a system like HSW has only CQM and no CAT support. In this case 1337the resctrl will still mount but cannot create CTRL_MON directories. 1338But user can create different MON groups within the root group thereby 1339able to monitor all tasks including kernel threads. 1340 1341This can also be used to profile jobs cache size footprint before being 1342able to allocate them to different allocation groups. 1343:: 1344 1345 # mount -t resctrl resctrl /sys/fs/resctrl 1346 # cd /sys/fs/resctrl 1347 # mkdir mon_groups/m01 1348 # mkdir mon_groups/m02 1349 1350 # echo 3478 > /sys/fs/resctrl/mon_groups/m01/tasks 1351 # echo 2467 > /sys/fs/resctrl/mon_groups/m02/tasks 1352 1353Monitor the groups separately and also get per domain data. From the 1354below its apparent that the tasks are mostly doing work on 1355domain(socket) 0. 1356:: 1357 1358 # cat /sys/fs/resctrl/mon_groups/m01/mon_L3_00/llc_occupancy 1359 31234000 1360 # cat /sys/fs/resctrl/mon_groups/m01/mon_L3_01/llc_occupancy 1361 34555 1362 # cat /sys/fs/resctrl/mon_groups/m02/mon_L3_00/llc_occupancy 1363 31234000 1364 # cat /sys/fs/resctrl/mon_groups/m02/mon_L3_01/llc_occupancy 1365 32789 1366 1367 1368Example 4 (Monitor real time tasks) 1369----------------------------------- 1370 1371A single socket system which has real time tasks running on cores 4-7 1372and non real time tasks on other cpus. We want to monitor the cache 1373occupancy of the real time threads on these cores. 1374:: 1375 1376 # mount -t resctrl resctrl /sys/fs/resctrl 1377 # cd /sys/fs/resctrl 1378 # mkdir p1 1379 1380Move the cpus 4-7 over to p1:: 1381 1382 # echo f0 > p1/cpus 1383 1384View the llc occupancy snapshot:: 1385 1386 # cat /sys/fs/resctrl/p1/mon_data/mon_L3_00/llc_occupancy 1387 11234000 1388 1389Intel RDT Errata 1390================ 1391 1392Intel MBM Counters May Report System Memory Bandwidth Incorrectly 1393----------------------------------------------------------------- 1394 1395Errata SKX99 for Skylake server and BDF102 for Broadwell server. 1396 1397Problem: Intel Memory Bandwidth Monitoring (MBM) counters track metrics 1398according to the assigned Resource Monitor ID (RMID) for that logical 1399core. The IA32_QM_CTR register (MSR 0xC8E), used to report these 1400metrics, may report incorrect system bandwidth for certain RMID values. 1401 1402Implication: Due to the errata, system memory bandwidth may not match 1403what is reported. 1404 1405Workaround: MBM total and local readings are corrected according to the 1406following correction factor table: 1407 1408+---------------+---------------+---------------+-----------------+ 1409|core count |rmid count |rmid threshold |correction factor| 1410+---------------+---------------+---------------+-----------------+ 1411|1 |8 |0 |1.000000 | 1412+---------------+---------------+---------------+-----------------+ 1413|2 |16 |0 |1.000000 | 1414+---------------+---------------+---------------+-----------------+ 1415|3 |24 |15 |0.969650 | 1416+---------------+---------------+---------------+-----------------+ 1417|4 |32 |0 |1.000000 | 1418+---------------+---------------+---------------+-----------------+ 1419|6 |48 |31 |0.969650 | 1420+---------------+---------------+---------------+-----------------+ 1421|7 |56 |47 |1.142857 | 1422+---------------+---------------+---------------+-----------------+ 1423|8 |64 |0 |1.000000 | 1424+---------------+---------------+---------------+-----------------+ 1425|9 |72 |63 |1.185115 | 1426+---------------+---------------+---------------+-----------------+ 1427|10 |80 |63 |1.066553 | 1428+---------------+---------------+---------------+-----------------+ 1429|11 |88 |79 |1.454545 | 1430+---------------+---------------+---------------+-----------------+ 1431|12 |96 |0 |1.000000 | 1432+---------------+---------------+---------------+-----------------+ 1433|13 |104 |95 |1.230769 | 1434+---------------+---------------+---------------+-----------------+ 1435|14 |112 |95 |1.142857 | 1436+---------------+---------------+---------------+-----------------+ 1437|15 |120 |95 |1.066667 | 1438+---------------+---------------+---------------+-----------------+ 1439|16 |128 |0 |1.000000 | 1440+---------------+---------------+---------------+-----------------+ 1441|17 |136 |127 |1.254863 | 1442+---------------+---------------+---------------+-----------------+ 1443|18 |144 |127 |1.185255 | 1444+---------------+---------------+---------------+-----------------+ 1445|19 |152 |0 |1.000000 | 1446+---------------+---------------+---------------+-----------------+ 1447|20 |160 |127 |1.066667 | 1448+---------------+---------------+---------------+-----------------+ 1449|21 |168 |0 |1.000000 | 1450+---------------+---------------+---------------+-----------------+ 1451|22 |176 |159 |1.454334 | 1452+---------------+---------------+---------------+-----------------+ 1453|23 |184 |0 |1.000000 | 1454+---------------+---------------+---------------+-----------------+ 1455|24 |192 |127 |0.969744 | 1456+---------------+---------------+---------------+-----------------+ 1457|25 |200 |191 |1.280246 | 1458+---------------+---------------+---------------+-----------------+ 1459|26 |208 |191 |1.230921 | 1460+---------------+---------------+---------------+-----------------+ 1461|27 |216 |0 |1.000000 | 1462+---------------+---------------+---------------+-----------------+ 1463|28 |224 |191 |1.143118 | 1464+---------------+---------------+---------------+-----------------+ 1465 1466If rmid > rmid threshold, MBM total and local values should be multiplied 1467by the correction factor. 1468 1469See: 1470 14711. Erratum SKX99 in Intel Xeon Processor Scalable Family Specification Update: 1472http://web.archive.org/web/20200716124958/https://www.intel.com/content/www/us/en/processors/xeon/scalable/xeon-scalable-spec-update.html 1473 14742. Erratum BDF102 in Intel Xeon E5-2600 v4 Processor Product Family Specification Update: 1475http://web.archive.org/web/20191125200531/https://www.intel.com/content/dam/www/public/us/en/documents/specification-updates/xeon-e5-v4-spec-update.pdf 1476 14773. The errata in Intel Resource Director Technology (Intel RDT) on 2nd Generation Intel Xeon Scalable Processors Reference Manual: 1478https://software.intel.com/content/www/us/en/develop/articles/intel-resource-director-technology-rdt-reference-manual.html 1479 1480for further information. 1481