Lines Matching +full:resource +full:- +full:files
1 .. SPDX-License-Identifier: GPL-2.0
5 User Interface for Resource Control feature
9 :Authors: - Fenghua Yu <fenghua.yu@intel.com>
10 - Tony Luck <tony.luck@intel.com>
11 - Vikas Shivappa <vikas.shivappa@intel.com>
14 Intel refers to this feature as Intel Resource Director Technology(Intel(R) RDT).
21 RDT (Resource Director Technology) Allocation "rdt_a"
38 # mount -t resctrl resctrl [-o cdp[,cdpl2][,mba_MBps][,debug]] /sys/fs/resctrl
50 Make debug files accessible. Available debug files are annotated with
57 pseudo-locking is a unique way of using cache control to "pin" or
59 "Cache Pseudo-Locking".
63 only those files and directories supported by the system will be created.
65 and allocation, see the "Resource alloc and monitor groups" section.
71 resources. Each resource has its own subdirectory. The subdirectory
72 names reflect the resource names.
74 Each subdirectory contains the following files with respect to
77 Cache resource(L3/L2) subdirectory contains the following files
82 resource. The kernel uses the smallest number of
85 The bitmask which is valid for this resource.
92 Bitmask of shareable resource with other executing
96 own settings for cache use which can over-ride
100 instances of the resource are used. The legend is:
110 but available for software use. If a resource
112 of these bits appear in the resource groups'
114 "shareable_bits" but no resource group will
120 well as a resource group's allocation.
126 one resource group. No sharing allowed.
128 Corresponding region is pseudo-locked. No
131 Indicates if non-contiguous 1s value in CBM is supported.
136 Non-contiguous 1s value in CBM is supported.
138 Memory bandwidth(MB) subdirectory contains the following files
155 non-linear. This field is purely informational
166 "per-thread":
171 with the following files:
180 monitoring is enabled for the resource.
200 Read/write files containing the configuration for the mbm_total_bytes
216 5 Reads to slow memory in the non-local NUMA domain
218 3 Non-temporal writes to non-local NUMA domain
219 2 Non-temporal writes to local NUMA domain
220 1 Reads to memory in the non-local NUMA domain
262 counter can be considered for re-use.
267 control files). If the command was successful, it will read as "ok".
275 mask f7 has non-consecutive 1-bits
277 Resource alloc and monitor groups
280 Resource groups are represented as directories in the resctrl file
287 resource (see "schemata" below). The root and these additional top level
301 for the purpose of changing the resource allocations of a MON group
307 All groups contain the following files:
332 When the resource group is in pseudo-locked mode this file will
334 pseudo-locked region.
345 Each resource has its own line and format - see below for details.
353 The "mode" of the resource group dictates the sharing of its
354 allocations. A "shareable" resource group allows sharing of its
355 allocations while an "exclusive" resource group does not. A
356 cache pseudo-locked region is created by first writing
357 "pseudo-locksetup" to the "mode" file before writing the cache
358 pseudo-locked region's schemata to the resource group's "schemata"
359 file. On successful pseudo-locked region creation the mode will
360 automatically change to "pseudo-locked".
369 This contains a set of files organized by L3 domain and by
374 files provide a read out of the current value of the event for
375 all tasks in the group. In CTRL_MON groups these files provide
378 On systems with Sub-NUMA Cluster (SNC) enabled there are extra
387 Resource allocation rules
388 -------------------------
393 1) If the task is a member of a non-default group, then the schemata
402 Resource monitoring rules
403 -------------------------
404 1) If a task is a member of a MON group, or non-default CTRL_MON group
425 are evicted and re-used while the occupancy in the new group rises as
433 Hardware uses CLOSid(Class of service ID) and an RMID(Resource monitoring ID)
435 the resource groups are mapped to these IDs based on the kind of group. The
440 max_threshold_occupancy - generic concepts
441 ------------------------------------------
447 limbo RMIDs but which are not ready to be used, user may see an -EBUSY
459 Schemata files - general concepts
460 ---------------------------------
461 Each line in the file describes one resource. The line starts with
462 the name of the resource, followed by specific values to be applied
463 in each of the instances of that resource on the system.
466 ---------
472 a resource we use a "Cache ID". At a given cache level this will be a
478 ---------------------
483 the resctrl file system in "info/{resource}/cbm_mask". Some Intel hardware
485 0x3, 0x6 and 0xC are legal 4-bit masks with two bits set, but 0x5, 0x9
486 and 0xA are not. Check /sys/fs/resctrl/info/{resource}/sparse_masks
487 if non-contiguous 1s value is supported. On a system with a 20-bit mask
491 Notes on Sub-NUMA Cluster mode
493 When SNC mode is enabled, Linux may load balance tasks between Sub-NUMA
495 on Sub-NUMA nodes share the same L3 cache and the system may report
496 the NUMA distance between Sub-NUMA nodes with a lower value than used
499 The top-level monitoring files in each "mon_L3_XX" directory provide
501 Users who bind tasks to the CPUs of a specific Sub-NUMA node can read
510 of SNC nodes per L3 cache. E.g. with a 100MB cache on a system with 10-bit
517 For Memory bandwidth resource, by default the user controls the resource
575 ----------------------------------------------------------------
581 ------------------------------------------------------------------
589 ------------------------
602 ------------------------------------------
610 ----------------------------------------------
618 ---------------------------------------
637 ---------------------------------
652 --------------------------------------------------
672 --------------------------------------------------------------------
691 Cache Pseudo-Locking
694 application can fill. Cache pseudo-locking builds on the fact that a
695 CPU can still read and write data pre-allocated outside its current
696 allocated area on a cache hit. With cache pseudo-locking, data can be
699 pseudo-locked memory is made accessible to user space where an
703 The creation of a cache pseudo-locked region is triggered by a request
705 to be pseudo-locked. The cache pseudo-locked region is created as follows:
707 - Create a CAT allocation CLOSNEW with a CBM matching the schemata
708 from the user of the cache region that will contain the pseudo-locked
711 while the pseudo-locked region exists.
712 - Create a contiguous region of memory of the same size as the cache
714 - Flush the cache, disable hardware prefetchers, disable preemption.
715 - Make CLOSNEW the active CLOS and touch the allocated memory to load
717 - Set the previous CLOS as active.
718 - At this point the closid CLOSNEW can be released - the cache
719 pseudo-locked region is protected as long as its CBM does not appear in
720 any CAT allocation. Even though the cache pseudo-locked region will from
722 any CLOS will be able to access the memory in the pseudo-locked region since
724 - The contiguous region of memory loaded into the cache is exposed to
725 user-space as a character device.
727 Cache pseudo-locking increases the probability that data will remain
731 “locked” data from cache. Power management C-states may shrink or
732 power off cache. Deeper C-states will automatically be restricted on
733 pseudo-locked region creation.
735 It is required that an application using a pseudo-locked region runs
737 with the cache on which the pseudo-locked region resides. A sanity check
738 within the code will not allow an application to map pseudo-locked memory
740 pseudo-locked region resides. The sanity check is only done during the
744 Pseudo-locking is accomplished in two stages:
747 of cache that should be dedicated to pseudo-locking. At this time an
750 2) During the second stage a user-space application maps (mmap()) the
751 pseudo-locked memory into its address space.
753 Cache Pseudo-Locking Interface
754 ------------------------------
755 A pseudo-locked region is created using the resctrl interface as follows:
757 1) Create a new resource group by creating a new directory in /sys/fs/resctrl.
758 2) Change the new resource group's mode to "pseudo-locksetup" by writing
759 "pseudo-locksetup" to the "mode" file.
760 3) Write the schemata of the pseudo-locked region to the "schemata" file. All
764 On successful pseudo-locked region creation the "mode" file will contain
765 "pseudo-locked" and a new character device with the same name as the resource
767 by user space in order to obtain access to the pseudo-locked memory region.
769 An example of cache pseudo-locked region creation and usage can be found below.
771 Cache Pseudo-Locking Debugging Interface
772 ----------------------------------------
773 The pseudo-locking debugging interface is enabled by default (if
777 location is present in the cache. The pseudo-locking debugging interface uses
779 the pseudo-locked region:
783 example below). In this test the pseudo-locked region is traversed at
791 When a pseudo-locked region is created a new debugfs directory is created for
793 write-only file, pseudo_lock_measure, is present in this directory. The
794 measurement of the pseudo-locked region depends on the number written to this
815 In this example a pseudo-locked region named "newlock" was created. Here is
849 In this example a pseudo-locked region named "newlock" was created on the L2
862 # _-----=> irqs-off
863 # / _----=> need-resched
864 # | / _---=> hardirq/softirq
865 # || / _--=> preempt-depth
867 # TASK-PID CPU# |||| TIMESTAMP FUNCTION
869 pseudo_lock_mea-1672 [002] .... 3132.860500: pseudo_lock_l2: hits=4097 miss=0
882 # mount -t resctrl resctrl /sys/fs/resctrl
888 The default resource group is unmodified, so we have access to all parts
915 Again two sockets, but this time with a more realistic 20-bit mask.
918 processor 1 on socket 0 on a 2-socket and dual core machine. To avoid noisy
919 neighbors, each of the two real-time tasks exclusively occupies one quarter
923 # mount -t resctrl resctrl /sys/fs/resctrl
932 Next we make a resource group for our first real time task and give
939 Finally we move our first real time task into this resource group. We
941 on socket 0. Most uses of resource groups will also constrain which
946 # taskset -cp 1 1234
953 # taskset -cp 2 5678
955 For the same 2 socket system with memory b/w resource and CAT L3 the
962 # echo -e "L3:0=f8000;1=fffff\nMB:0=20;1=100" > p0/schemata
968 # echo -e "L3:0=f8000;1=fffff\nMB:0=20;1=100" > p0/schemata
972 A single socket system which has real-time tasks running on core 4-7 and
973 non real-time workload assigned to core 0-3. The real-time tasks share text
979 # mount -t resctrl resctrl /sys/fs/resctrl
988 Next we make a resource group for our real time cores and give it access
996 Finally we move core 4-7 over to the new group and make sure that the
998 also get 50% of memory bandwidth assuming that the cores 4-7 are SMT
999 siblings and only the real time threads are scheduled on the cores 4-7.
1006 The resource groups in previous examples were all in the default "shareable"
1007 mode allowing sharing of their cache allocations. If one resource group
1008 configures a cache allocation then nothing prevents another resource group
1011 In this example a new exclusive resource group will be created on a L2 CAT
1012 system with two L2 cache instances that can be configured with an 8-bit
1013 capacity bitmask. The new exclusive resource group will be configured to use
1017 # mount -t resctrl resctrl /sys/fs/resctrl/
1026 We could attempt to create the new resource group at this point, but it will
1034 -sh: echo: write error: Invalid argument
1038 To ensure that there is no overlap with another resource group the default
1039 resource group's schemata has to change, making it possible for the new
1040 resource group to become exclusive.
1051 A new resource group will on creation not overlap with an exclusive resource
1066 A resource group cannot be forced to overlap with an exclusive resource group::
1069 -sh: echo: write error: Invalid argument
1073 Example of Cache Pseudo-Locking
1075 Lock portion of L2 cache from cache id 1 using CBM 0x3. Pseudo-locked
1080 # mount -t resctrl resctrl /sys/fs/resctrl/
1083 Ensure that there are bits available that can be pseudo-locked, since only
1084 unused bits can be pseudo-locked the bits to be pseudo-locked needs to be
1085 removed from the default resource group's schemata::
1093 Create a new resource group that will be associated with the pseudo-locked
1094 region, indicate that it will be used for a pseudo-locked region, and
1095 configure the requested pseudo-locked region capacity bitmask::
1098 # echo pseudo-locksetup > newlock/mode
1101 On success the resource group's mode will change to pseudo-locked, the
1102 bit_usage will reflect the pseudo-locked region, and the character device
1103 exposing the pseudo-locked region will exist::
1106 pseudo-locked
1109 # ls -l /dev/pseudo_lock/newlock
1110 crw------- 1 root root 243, 0 Apr 3 05:01 /dev/pseudo_lock/newlock
1115 * Example code to access one page of pseudo-locked cache region
1128 * cores associated with the pseudo-locked region. Here the cpu
1165 /* Application interacts with pseudo-locked memory @mapping */
1179 ----------------------------
1182 to/from multiple files, must be atomic.
1187 1. Read the cbmmasks from each directory or the per-resource "bit_usage"
1218 $ flock -s /sys/fs/resctrl/ find /sys/fs/resctrl
1222 $ cat create-dir.sh
1224 mask = function-of(output.txt)
1228 $ flock /sys/fs/resctrl/ ./create-dir.sh
1247 exit(-1);
1259 exit(-1);
1271 exit(-1);
1280 if (fd == -1) {
1282 exit(-1);
1296 ----------------------
1303 ------------------------------------------------------------------------
1307 # mount -t resctrl resctrl /sys/fs/resctrl
1315 The default resource group is unmodified, so we have access to all parts
1347 --------------------------------------------
1350 # mount -t resctrl resctrl /sys/fs/resctrl
1367 ---------------------------------------------------------------------
1378 # mount -t resctrl resctrl /sys/fs/resctrl
1402 -----------------------------------
1404 A single socket system which has real time tasks running on cores 4-7
1409 # mount -t resctrl resctrl /sys/fs/resctrl
1413 Move the cpus 4-7 over to p1::
1426 -----------------------------------------------------------------
1431 according to the assigned Resource Monitor ID (RMID) for that logical
1441 +---------------+---------------+---------------+-----------------+
1443 +---------------+---------------+---------------+-----------------+
1445 +---------------+---------------+---------------+-----------------+
1447 +---------------+---------------+---------------+-----------------+
1449 +---------------+---------------+---------------+-----------------+
1451 +---------------+---------------+---------------+-----------------+
1453 +---------------+---------------+---------------+-----------------+
1455 +---------------+---------------+---------------+-----------------+
1457 +---------------+---------------+---------------+-----------------+
1459 +---------------+---------------+---------------+-----------------+
1461 +---------------+---------------+---------------+-----------------+
1463 +---------------+---------------+---------------+-----------------+
1465 +---------------+---------------+---------------+-----------------+
1467 +---------------+---------------+---------------+-----------------+
1469 +---------------+---------------+---------------+-----------------+
1471 +---------------+---------------+---------------+-----------------+
1473 +---------------+---------------+---------------+-----------------+
1475 +---------------+---------------+---------------+-----------------+
1477 +---------------+---------------+---------------+-----------------+
1479 +---------------+---------------+---------------+-----------------+
1481 +---------------+---------------+---------------+-----------------+
1483 +---------------+---------------+---------------+-----------------+
1485 +---------------+---------------+---------------+-----------------+
1487 +---------------+---------------+---------------+-----------------+
1489 +---------------+---------------+---------------+-----------------+
1491 +---------------+---------------+---------------+-----------------+
1493 +---------------+---------------+---------------+-----------------+
1495 +---------------+---------------+---------------+-----------------+
1497 +---------------+---------------+---------------+-----------------+
1505 …958/https://www.intel.com/content/www/us/en/processors/xeon/scalable/xeon-scalable-spec-update.html
1507 2. Erratum BDF102 in Intel Xeon E5-2600 v4 Processor Product Family Specification Update:
1508 …w.intel.com/content/dam/www/public/us/en/documents/specification-updates/xeon-e5-v4-spec-update.pdf
1510 3. The errata in Intel Resource Director Technology (Intel RDT) on 2nd Generation Intel Xeon Scalab…
1511 …are.intel.com/content/www/us/en/develop/articles/intel-resource-director-technology-rdt-reference-…