1.. _cgroup-v2: 2 3================ 4Control Group v2 5================ 6 7:Date: October, 2015 8:Author: Tejun Heo <tj@kernel.org> 9 10This is the authoritative documentation on the design, interface and 11conventions of cgroup v2. It describes all userland-visible aspects 12of cgroup including core and specific controller behaviors. All 13future changes must be reflected in this document. Documentation for 14v1 is available under :ref:`Documentation/admin-guide/cgroup-v1/index.rst <cgroup-v1>`. 15 16.. CONTENTS 17 18 [Whenever any new section is added to this document, please also add 19 an entry here.] 20 21 1. Introduction 22 1-1. Terminology 23 1-2. What is cgroup? 24 2. Basic Operations 25 2-1. Mounting 26 2-2. Organizing Processes and Threads 27 2-2-1. Processes 28 2-2-2. Threads 29 2-3. [Un]populated Notification 30 2-4. Controlling Controllers 31 2-4-1. Availability 32 2-4-2. Enabling and Disabling 33 2-4-3. Top-down Constraint 34 2-4-4. No Internal Process Constraint 35 2-5. Delegation 36 2-5-1. Model of Delegation 37 2-5-2. Delegation Containment 38 2-6. Guidelines 39 2-6-1. Organize Once and Control 40 2-6-2. Avoid Name Collisions 41 3. Resource Distribution Models 42 3-1. Weights 43 3-2. Limits 44 3-3. Protections 45 3-4. Allocations 46 4. Interface Files 47 4-1. Format 48 4-2. Conventions 49 4-3. Core Interface Files 50 5. Controllers 51 5-1. CPU 52 5-1-1. CPU Interface Files 53 5-2. Memory 54 5-2-1. Memory Interface Files 55 5-2-2. Usage Guidelines 56 5-2-3. Reclaim Protection 57 5-2-4. Memory Ownership 58 5-3. IO 59 5-3-1. IO Interface Files 60 5-3-2. Writeback 61 5-3-3. IO Latency 62 5-3-3-1. How IO Latency Throttling Works 63 5-3-3-2. IO Latency Interface Files 64 5-3-4. IO Priority 65 5-4. PID 66 5-4-1. PID Interface Files 67 5-5. Cpuset 68 5.5-1. Cpuset Interface Files 69 5-6. Device controller 70 5-7. RDMA 71 5-7-1. RDMA Interface Files 72 5-8. DMEM 73 5-8-1. DMEM Interface Files 74 5-9. HugeTLB 75 5.9-1. HugeTLB Interface Files 76 5-10. Misc 77 5.10-1 Misc Interface Files 78 5.10-2 Migration and Ownership 79 5-11. Others 80 5-11-1. perf_event 81 5-N. Non-normative information 82 5-N-1. CPU controller root cgroup process behaviour 83 5-N-2. IO controller root cgroup process behaviour 84 6. Namespace 85 6-1. Basics 86 6-2. The Root and Views 87 6-3. Migration and setns(2) 88 6-4. Interaction with Other Namespaces 89 P. Information on Kernel Programming 90 P-1. Filesystem Support for Writeback 91 D. Deprecated v1 Core Features 92 R. Issues with v1 and Rationales for v2 93 R-1. Multiple Hierarchies 94 R-2. Thread Granularity 95 R-3. Competition Between Inner Nodes and Threads 96 R-4. Other Interface Issues 97 R-5. Controller Issues and Remedies 98 R-5-1. Memory 99 100 101Introduction 102============ 103 104Terminology 105----------- 106 107"cgroup" stands for "control group" and is never capitalized. The 108singular form is used to designate the whole feature and also as a 109qualifier as in "cgroup controllers". When explicitly referring to 110multiple individual control groups, the plural form "cgroups" is used. 111 112 113What is cgroup? 114--------------- 115 116cgroup is a mechanism to organize processes hierarchically and 117distribute system resources along the hierarchy in a controlled and 118configurable manner. 119 120cgroup is largely composed of two parts - the core and controllers. 121cgroup core is primarily responsible for hierarchically organizing 122processes. A cgroup controller is usually responsible for 123distributing a specific type of system resource along the hierarchy 124although there are utility controllers which serve purposes other than 125resource distribution. 126 127cgroups form a tree structure and every process in the system belongs 128to one and only one cgroup. All threads of a process belong to the 129same cgroup. On creation, all processes are put in the cgroup that 130the parent process belongs to at the time. A process can be migrated 131to another cgroup. Migration of a process doesn't affect already 132existing descendant processes. 133 134Following certain structural constraints, controllers may be enabled or 135disabled selectively on a cgroup. All controller behaviors are 136hierarchical - if a controller is enabled on a cgroup, it affects all 137processes which belong to the cgroups consisting the inclusive 138sub-hierarchy of the cgroup. When a controller is enabled on a nested 139cgroup, it always restricts the resource distribution further. The 140restrictions set closer to the root in the hierarchy can not be 141overridden from further away. 142 143 144Basic Operations 145================ 146 147Mounting 148-------- 149 150Unlike v1, cgroup v2 has only single hierarchy. The cgroup v2 151hierarchy can be mounted with the following mount command:: 152 153 # mount -t cgroup2 none $MOUNT_POINT 154 155cgroup2 filesystem has the magic number 0x63677270 ("cgrp"). All 156controllers which support v2 and are not bound to a v1 hierarchy are 157automatically bound to the v2 hierarchy and show up at the root. 158Controllers which are not in active use in the v2 hierarchy can be 159bound to other hierarchies. This allows mixing v2 hierarchy with the 160legacy v1 multiple hierarchies in a fully backward compatible way. 161 162A controller can be moved across hierarchies only after the controller 163is no longer referenced in its current hierarchy. Because per-cgroup 164controller states are destroyed asynchronously and controllers may 165have lingering references, a controller may not show up immediately on 166the v2 hierarchy after the final umount of the previous hierarchy. 167Similarly, a controller should be fully disabled to be moved out of 168the unified hierarchy and it may take some time for the disabled 169controller to become available for other hierarchies; furthermore, due 170to inter-controller dependencies, other controllers may need to be 171disabled too. 172 173While useful for development and manual configurations, moving 174controllers dynamically between the v2 and other hierarchies is 175strongly discouraged for production use. It is recommended to decide 176the hierarchies and controller associations before starting using the 177controllers after system boot. 178 179During transition to v2, system management software might still 180automount the v1 cgroup filesystem and so hijack all controllers 181during boot, before manual intervention is possible. To make testing 182and experimenting easier, the kernel parameter cgroup_no_v1= allows 183disabling controllers in v1 and make them always available in v2. 184 185cgroup v2 currently supports the following mount options. 186 187 nsdelegate 188 Consider cgroup namespaces as delegation boundaries. This 189 option is system wide and can only be set on mount or modified 190 through remount from the init namespace. The mount option is 191 ignored on non-init namespace mounts. Please refer to the 192 Delegation section for details. 193 194 favordynmods 195 Reduce the latencies of dynamic cgroup modifications such as 196 task migrations and controller on/offs at the cost of making 197 hot path operations such as forks and exits more expensive. 198 The static usage pattern of creating a cgroup, enabling 199 controllers, and then seeding it with CLONE_INTO_CGROUP is 200 not affected by this option. 201 202 memory_localevents 203 Only populate memory.events with data for the current cgroup, 204 and not any subtrees. This is legacy behaviour, the default 205 behaviour without this option is to include subtree counts. 206 This option is system wide and can only be set on mount or 207 modified through remount from the init namespace. The mount 208 option is ignored on non-init namespace mounts. 209 210 memory_recursiveprot 211 Recursively apply memory.min and memory.low protection to 212 entire subtrees, without requiring explicit downward 213 propagation into leaf cgroups. This allows protecting entire 214 subtrees from one another, while retaining free competition 215 within those subtrees. This should have been the default 216 behavior but is a mount-option to avoid regressing setups 217 relying on the original semantics (e.g. specifying bogusly 218 high 'bypass' protection values at higher tree levels). 219 220 memory_hugetlb_accounting 221 Count HugeTLB memory usage towards the cgroup's overall 222 memory usage for the memory controller (for the purpose of 223 statistics reporting and memory protection). This is a new 224 behavior that could regress existing setups, so it must be 225 explicitly opted in with this mount option. 226 227 A few caveats to keep in mind: 228 229 * There is no HugeTLB pool management involved in the memory 230 controller. The pre-allocated pool does not belong to anyone. 231 Specifically, when a new HugeTLB folio is allocated to 232 the pool, it is not accounted for from the perspective of the 233 memory controller. It is only charged to a cgroup when it is 234 actually used (for e.g at page fault time). Host memory 235 overcommit management has to consider this when configuring 236 hard limits. In general, HugeTLB pool management should be 237 done via other mechanisms (such as the HugeTLB controller). 238 * Failure to charge a HugeTLB folio to the memory controller 239 results in SIGBUS. This could happen even if the HugeTLB pool 240 still has pages available (but the cgroup limit is hit and 241 reclaim attempt fails). 242 * Charging HugeTLB memory towards the memory controller affects 243 memory protection and reclaim dynamics. Any userspace tuning 244 (of low, min limits for e.g) needs to take this into account. 245 * HugeTLB pages utilized while this option is not selected 246 will not be tracked by the memory controller (even if cgroup 247 v2 is remounted later on). 248 249 pids_localevents 250 The option restores v1-like behavior of pids.events:max, that is only 251 local (inside cgroup proper) fork failures are counted. Without this 252 option pids.events.max represents any pids.max enforcemnt across 253 cgroup's subtree. 254 255 256 257Organizing Processes and Threads 258-------------------------------- 259 260Processes 261~~~~~~~~~ 262 263Initially, only the root cgroup exists to which all processes belong. 264A child cgroup can be created by creating a sub-directory:: 265 266 # mkdir $CGROUP_NAME 267 268A given cgroup may have multiple child cgroups forming a tree 269structure. Each cgroup has a read-writable interface file 270"cgroup.procs". When read, it lists the PIDs of all processes which 271belong to the cgroup one-per-line. The PIDs are not ordered and the 272same PID may show up more than once if the process got moved to 273another cgroup and then back or the PID got recycled while reading. 274 275A process can be migrated into a cgroup by writing its PID to the 276target cgroup's "cgroup.procs" file. Only one process can be migrated 277on a single write(2) call. If a process is composed of multiple 278threads, writing the PID of any thread migrates all threads of the 279process. 280 281When a process forks a child process, the new process is born into the 282cgroup that the forking process belongs to at the time of the 283operation. After exit, a process stays associated with the cgroup 284that it belonged to at the time of exit until it's reaped; however, a 285zombie process does not appear in "cgroup.procs" and thus can't be 286moved to another cgroup. 287 288A cgroup which doesn't have any children or live processes can be 289destroyed by removing the directory. Note that a cgroup which doesn't 290have any children and is associated only with zombie processes is 291considered empty and can be removed:: 292 293 # rmdir $CGROUP_NAME 294 295"/proc/$PID/cgroup" lists a process's cgroup membership. If legacy 296cgroup is in use in the system, this file may contain multiple lines, 297one for each hierarchy. The entry for cgroup v2 is always in the 298format "0::$PATH":: 299 300 # cat /proc/842/cgroup 301 ... 302 0::/test-cgroup/test-cgroup-nested 303 304If the process becomes a zombie and the cgroup it was associated with 305is removed subsequently, " (deleted)" is appended to the path:: 306 307 # cat /proc/842/cgroup 308 ... 309 0::/test-cgroup/test-cgroup-nested (deleted) 310 311 312Threads 313~~~~~~~ 314 315cgroup v2 supports thread granularity for a subset of controllers to 316support use cases requiring hierarchical resource distribution across 317the threads of a group of processes. By default, all threads of a 318process belong to the same cgroup, which also serves as the resource 319domain to host resource consumptions which are not specific to a 320process or thread. The thread mode allows threads to be spread across 321a subtree while still maintaining the common resource domain for them. 322 323Controllers which support thread mode are called threaded controllers. 324The ones which don't are called domain controllers. 325 326Marking a cgroup threaded makes it join the resource domain of its 327parent as a threaded cgroup. The parent may be another threaded 328cgroup whose resource domain is further up in the hierarchy. The root 329of a threaded subtree, that is, the nearest ancestor which is not 330threaded, is called threaded domain or thread root interchangeably and 331serves as the resource domain for the entire subtree. 332 333Inside a threaded subtree, threads of a process can be put in 334different cgroups and are not subject to the no internal process 335constraint - threaded controllers can be enabled on non-leaf cgroups 336whether they have threads in them or not. 337 338As the threaded domain cgroup hosts all the domain resource 339consumptions of the subtree, it is considered to have internal 340resource consumptions whether there are processes in it or not and 341can't have populated child cgroups which aren't threaded. Because the 342root cgroup is not subject to no internal process constraint, it can 343serve both as a threaded domain and a parent to domain cgroups. 344 345The current operation mode or type of the cgroup is shown in the 346"cgroup.type" file which indicates whether the cgroup is a normal 347domain, a domain which is serving as the domain of a threaded subtree, 348or a threaded cgroup. 349 350On creation, a cgroup is always a domain cgroup and can be made 351threaded by writing "threaded" to the "cgroup.type" file. The 352operation is single direction:: 353 354 # echo threaded > cgroup.type 355 356Once threaded, the cgroup can't be made a domain again. To enable the 357thread mode, the following conditions must be met. 358 359- As the cgroup will join the parent's resource domain. The parent 360 must either be a valid (threaded) domain or a threaded cgroup. 361 362- When the parent is an unthreaded domain, it must not have any domain 363 controllers enabled or populated domain children. The root is 364 exempt from this requirement. 365 366Topology-wise, a cgroup can be in an invalid state. Please consider 367the following topology:: 368 369 A (threaded domain) - B (threaded) - C (domain, just created) 370 371C is created as a domain but isn't connected to a parent which can 372host child domains. C can't be used until it is turned into a 373threaded cgroup. "cgroup.type" file will report "domain (invalid)" in 374these cases. Operations which fail due to invalid topology use 375EOPNOTSUPP as the errno. 376 377A domain cgroup is turned into a threaded domain when one of its child 378cgroup becomes threaded or threaded controllers are enabled in the 379"cgroup.subtree_control" file while there are processes in the cgroup. 380A threaded domain reverts to a normal domain when the conditions 381clear. 382 383When read, "cgroup.threads" contains the list of the thread IDs of all 384threads in the cgroup. Except that the operations are per-thread 385instead of per-process, "cgroup.threads" has the same format and 386behaves the same way as "cgroup.procs". While "cgroup.threads" can be 387written to in any cgroup, as it can only move threads inside the same 388threaded domain, its operations are confined inside each threaded 389subtree. 390 391The threaded domain cgroup serves as the resource domain for the whole 392subtree, and, while the threads can be scattered across the subtree, 393all the processes are considered to be in the threaded domain cgroup. 394"cgroup.procs" in a threaded domain cgroup contains the PIDs of all 395processes in the subtree and is not readable in the subtree proper. 396However, "cgroup.procs" can be written to from anywhere in the subtree 397to migrate all threads of the matching process to the cgroup. 398 399Only threaded controllers can be enabled in a threaded subtree. When 400a threaded controller is enabled inside a threaded subtree, it only 401accounts for and controls resource consumptions associated with the 402threads in the cgroup and its descendants. All consumptions which 403aren't tied to a specific thread belong to the threaded domain cgroup. 404 405Because a threaded subtree is exempt from no internal process 406constraint, a threaded controller must be able to handle competition 407between threads in a non-leaf cgroup and its child cgroups. Each 408threaded controller defines how such competitions are handled. 409 410Currently, the following controllers are threaded and can be enabled 411in a threaded cgroup:: 412 413- cpu 414- cpuset 415- perf_event 416- pids 417 418[Un]populated Notification 419-------------------------- 420 421Each non-root cgroup has a "cgroup.events" file which contains 422"populated" field indicating whether the cgroup's sub-hierarchy has 423live processes in it. Its value is 0 if there is no live process in 424the cgroup and its descendants; otherwise, 1. poll and [id]notify 425events are triggered when the value changes. This can be used, for 426example, to start a clean-up operation after all processes of a given 427sub-hierarchy have exited. The populated state updates and 428notifications are recursive. Consider the following sub-hierarchy 429where the numbers in the parentheses represent the numbers of processes 430in each cgroup:: 431 432 A(4) - B(0) - C(1) 433 \ D(0) 434 435A, B and C's "populated" fields would be 1 while D's 0. After the one 436process in C exits, B and C's "populated" fields would flip to "0" and 437file modified events will be generated on the "cgroup.events" files of 438both cgroups. 439 440 441Controlling Controllers 442----------------------- 443 444Availability 445~~~~~~~~~~~~ 446 447A controller is available in a cgroup when it is supported by the kernel (i.e., 448compiled in, not disabled and not attached to a v1 hierarchy) and listed in the 449"cgroup.controllers" file. Availability means the controller's interface files 450are exposed in the cgroup’s directory, allowing the distribution of the target 451resource to be observed or controlled within that cgroup. 452 453Enabling and Disabling 454~~~~~~~~~~~~~~~~~~~~~~ 455 456Each cgroup has a "cgroup.controllers" file which lists all 457controllers available for the cgroup to enable:: 458 459 # cat cgroup.controllers 460 cpu io memory 461 462No controller is enabled by default. Controllers can be enabled and 463disabled by writing to the "cgroup.subtree_control" file:: 464 465 # echo "+cpu +memory -io" > cgroup.subtree_control 466 467Only controllers which are listed in "cgroup.controllers" can be 468enabled. When multiple operations are specified as above, either they 469all succeed or fail. If multiple operations on the same controller 470are specified, the last one is effective. 471 472Enabling a controller in a cgroup indicates that the distribution of 473the target resource across its immediate children will be controlled. 474Consider the following sub-hierarchy. The enabled controllers are 475listed in parentheses:: 476 477 A(cpu,memory) - B(memory) - C() 478 \ D() 479 480As A has "cpu" and "memory" enabled, A will control the distribution 481of CPU cycles and memory to its children, in this case, B. As B has 482"memory" enabled but not "CPU", C and D will compete freely on CPU 483cycles but their division of memory available to B will be controlled. 484 485As a controller regulates the distribution of the target resource to 486the cgroup's children, enabling it creates the controller's interface 487files in the child cgroups. In the above example, enabling "cpu" on B 488would create the "cpu." prefixed controller interface files in C and 489D. Likewise, disabling "memory" from B would remove the "memory." 490prefixed controller interface files from C and D. This means that the 491controller interface files - anything which doesn't start with 492"cgroup." are owned by the parent rather than the cgroup itself. 493 494 495Top-down Constraint 496~~~~~~~~~~~~~~~~~~~ 497 498Resources are distributed top-down and a cgroup can further distribute 499a resource only if the resource has been distributed to it from the 500parent. This means that all non-root "cgroup.subtree_control" files 501can only contain controllers which are enabled in the parent's 502"cgroup.subtree_control" file. A controller can be enabled only if 503the parent has the controller enabled and a controller can't be 504disabled if one or more children have it enabled. 505 506 507No Internal Process Constraint 508~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 509 510Non-root cgroups can distribute domain resources to their children 511only when they don't have any processes of their own. In other words, 512only domain cgroups which don't contain any processes can have domain 513controllers enabled in their "cgroup.subtree_control" files. 514 515This guarantees that, when a domain controller is looking at the part 516of the hierarchy which has it enabled, processes are always only on 517the leaves. This rules out situations where child cgroups compete 518against internal processes of the parent. 519 520The root cgroup is exempt from this restriction. Root contains 521processes and anonymous resource consumption which can't be associated 522with any other cgroups and requires special treatment from most 523controllers. How resource consumption in the root cgroup is governed 524is up to each controller (for more information on this topic please 525refer to the Non-normative information section in the Controllers 526chapter). 527 528Note that the restriction doesn't get in the way if there is no 529enabled controller in the cgroup's "cgroup.subtree_control". This is 530important as otherwise it wouldn't be possible to create children of a 531populated cgroup. To control resource distribution of a cgroup, the 532cgroup must create children and transfer all its processes to the 533children before enabling controllers in its "cgroup.subtree_control" 534file. 535 536 537Delegation 538---------- 539 540Model of Delegation 541~~~~~~~~~~~~~~~~~~~ 542 543A cgroup can be delegated in two ways. First, to a less privileged 544user by granting write access of the directory and its "cgroup.procs", 545"cgroup.threads" and "cgroup.subtree_control" files to the user. 546Second, if the "nsdelegate" mount option is set, automatically to a 547cgroup namespace on namespace creation. 548 549Because the resource control interface files in a given directory 550control the distribution of the parent's resources, the delegatee 551shouldn't be allowed to write to them. For the first method, this is 552achieved by not granting access to these files. For the second, files 553outside the namespace should be hidden from the delegatee by the means 554of at least mount namespacing, and the kernel rejects writes to all 555files on a namespace root from inside the cgroup namespace, except for 556those files listed in "/sys/kernel/cgroup/delegate" (including 557"cgroup.procs", "cgroup.threads", "cgroup.subtree_control", etc.). 558 559The end results are equivalent for both delegation types. Once 560delegated, the user can build sub-hierarchy under the directory, 561organize processes inside it as it sees fit and further distribute the 562resources it received from the parent. The limits and other settings 563of all resource controllers are hierarchical and regardless of what 564happens in the delegated sub-hierarchy, nothing can escape the 565resource restrictions imposed by the parent. 566 567Currently, cgroup doesn't impose any restrictions on the number of 568cgroups in or nesting depth of a delegated sub-hierarchy; however, 569this may be limited explicitly in the future. 570 571 572Delegation Containment 573~~~~~~~~~~~~~~~~~~~~~~ 574 575A delegated sub-hierarchy is contained in the sense that processes 576can't be moved into or out of the sub-hierarchy by the delegatee. 577 578For delegations to a less privileged user, this is achieved by 579requiring the following conditions for a process with a non-root euid 580to migrate a target process into a cgroup by writing its PID to the 581"cgroup.procs" file. 582 583- The writer must have write access to the "cgroup.procs" file. 584 585- The writer must have write access to the "cgroup.procs" file of the 586 common ancestor of the source and destination cgroups. 587 588The above two constraints ensure that while a delegatee may migrate 589processes around freely in the delegated sub-hierarchy it can't pull 590in from or push out to outside the sub-hierarchy. 591 592For an example, let's assume cgroups C0 and C1 have been delegated to 593user U0 who created C00, C01 under C0 and C10 under C1 as follows and 594all processes under C0 and C1 belong to U0:: 595 596 ~~~~~~~~~~~~~ - C0 - C00 597 ~ cgroup ~ \ C01 598 ~ hierarchy ~ 599 ~~~~~~~~~~~~~ - C1 - C10 600 601Let's also say U0 wants to write the PID of a process which is 602currently in C10 into "C00/cgroup.procs". U0 has write access to the 603file; however, the common ancestor of the source cgroup C10 and the 604destination cgroup C00 is above the points of delegation and U0 would 605not have write access to its "cgroup.procs" files and thus the write 606will be denied with -EACCES. 607 608For delegations to namespaces, containment is achieved by requiring 609that both the source and destination cgroups are reachable from the 610namespace of the process which is attempting the migration. If either 611is not reachable, the migration is rejected with -ENOENT. 612 613 614Guidelines 615---------- 616 617Organize Once and Control 618~~~~~~~~~~~~~~~~~~~~~~~~~ 619 620Migrating a process across cgroups is a relatively expensive operation 621and stateful resources such as memory are not moved together with the 622process. This is an explicit design decision as there often exist 623inherent trade-offs between migration and various hot paths in terms 624of synchronization cost. 625 626As such, migrating processes across cgroups frequently as a means to 627apply different resource restrictions is discouraged. A workload 628should be assigned to a cgroup according to the system's logical and 629resource structure once on start-up. Dynamic adjustments to resource 630distribution can be made by changing controller configuration through 631the interface files. 632 633 634Avoid Name Collisions 635~~~~~~~~~~~~~~~~~~~~~ 636 637Interface files for a cgroup and its children cgroups occupy the same 638directory and it is possible to create children cgroups which collide 639with interface files. 640 641All cgroup core interface files are prefixed with "cgroup." and each 642controller's interface files are prefixed with the controller name and 643a dot. A controller's name is composed of lower case alphabets and 644'_'s but never begins with an '_' so it can be used as the prefix 645character for collision avoidance. Also, interface file names won't 646start or end with terms which are often used in categorizing workloads 647such as job, service, slice, unit or workload. 648 649cgroup doesn't do anything to prevent name collisions and it's the 650user's responsibility to avoid them. 651 652 653Resource Distribution Models 654============================ 655 656cgroup controllers implement several resource distribution schemes 657depending on the resource type and expected use cases. This section 658describes major schemes in use along with their expected behaviors. 659 660 661Weights 662------- 663 664A parent's resource is distributed by adding up the weights of all 665active children and giving each the fraction matching the ratio of its 666weight against the sum. As only children which can make use of the 667resource at the moment participate in the distribution, this is 668work-conserving. Due to the dynamic nature, this model is usually 669used for stateless resources. 670 671All weights are in the range [1, 10000] with the default at 100. This 672allows symmetric multiplicative biases in both directions at fine 673enough granularity while staying in the intuitive range. 674 675As long as the weight is in range, all configuration combinations are 676valid and there is no reason to reject configuration changes or 677process migrations. 678 679"cpu.weight" proportionally distributes CPU cycles to active children 680and is an example of this type. 681 682 683.. _cgroupv2-limits-distributor: 684 685Limits 686------ 687 688A child can only consume up to the configured amount of the resource. 689Limits can be over-committed - the sum of the limits of children can 690exceed the amount of resource available to the parent. 691 692Limits are in the range [0, max] and defaults to "max", which is noop. 693 694As limits can be over-committed, all configuration combinations are 695valid and there is no reason to reject configuration changes or 696process migrations. 697 698"io.max" limits the maximum BPS and/or IOPS that a cgroup can consume 699on an IO device and is an example of this type. 700 701.. _cgroupv2-protections-distributor: 702 703Protections 704----------- 705 706A cgroup is protected up to the configured amount of the resource 707as long as the usages of all its ancestors are under their 708protected levels. Protections can be hard guarantees or best effort 709soft boundaries. Protections can also be over-committed in which case 710only up to the amount available to the parent is protected among 711children. 712 713Protections are in the range [0, max] and defaults to 0, which is 714noop. 715 716As protections can be over-committed, all configuration combinations 717are valid and there is no reason to reject configuration changes or 718process migrations. 719 720"memory.low" implements best-effort memory protection and is an 721example of this type. 722 723 724Allocations 725----------- 726 727A cgroup is exclusively allocated a certain amount of a finite 728resource. Allocations can't be over-committed - the sum of the 729allocations of children can not exceed the amount of resource 730available to the parent. 731 732Allocations are in the range [0, max] and defaults to 0, which is no 733resource. 734 735As allocations can't be over-committed, some configuration 736combinations are invalid and should be rejected. Also, if the 737resource is mandatory for execution of processes, process migrations 738may be rejected. 739 740 741Interface Files 742=============== 743 744Format 745------ 746 747All interface files should be in one of the following formats whenever 748possible:: 749 750 New-line separated values 751 (when only one value can be written at once) 752 753 VAL0\n 754 VAL1\n 755 ... 756 757 Space separated values 758 (when read-only or multiple values can be written at once) 759 760 VAL0 VAL1 ...\n 761 762 Flat keyed 763 764 KEY0 VAL0\n 765 KEY1 VAL1\n 766 ... 767 768 Nested keyed 769 770 KEY0 SUB_KEY0=VAL00 SUB_KEY1=VAL01... 771 KEY1 SUB_KEY0=VAL10 SUB_KEY1=VAL11... 772 ... 773 774For a writable file, the format for writing should generally match 775reading; however, controllers may allow omitting later fields or 776implement restricted shortcuts for most common use cases. 777 778For both flat and nested keyed files, only the values for a single key 779can be written at a time. For nested keyed files, the sub key pairs 780may be specified in any order and not all pairs have to be specified. 781 782 783Conventions 784----------- 785 786- Settings for a single feature should be contained in a single file. 787 788- The root cgroup should be exempt from resource control and thus 789 shouldn't have resource control interface files. 790 791- The default time unit is microseconds. If a different unit is ever 792 used, an explicit unit suffix must be present. 793 794- A parts-per quantity should use a percentage decimal with at least 795 two digit fractional part - e.g. 13.40. 796 797- If a controller implements weight based resource distribution, its 798 interface file should be named "weight" and have the range [1, 799 10000] with 100 as the default. The values are chosen to allow 800 enough and symmetric bias in both directions while keeping it 801 intuitive (the default is 100%). 802 803- If a controller implements an absolute resource guarantee and/or 804 limit, the interface files should be named "min" and "max" 805 respectively. If a controller implements best effort resource 806 guarantee and/or limit, the interface files should be named "low" 807 and "high" respectively. 808 809 In the above four control files, the special token "max" should be 810 used to represent upward infinity for both reading and writing. 811 812- If a setting has a configurable default value and keyed specific 813 overrides, the default entry should be keyed with "default" and 814 appear as the first entry in the file. 815 816 The default value can be updated by writing either "default $VAL" or 817 "$VAL". 818 819 When writing to update a specific override, "default" can be used as 820 the value to indicate removal of the override. Override entries 821 with "default" as the value must not appear when read. 822 823 For example, a setting which is keyed by major:minor device numbers 824 with integer values may look like the following:: 825 826 # cat cgroup-example-interface-file 827 default 150 828 8:0 300 829 830 The default value can be updated by:: 831 832 # echo 125 > cgroup-example-interface-file 833 834 or:: 835 836 # echo "default 125" > cgroup-example-interface-file 837 838 An override can be set by:: 839 840 # echo "8:16 170" > cgroup-example-interface-file 841 842 and cleared by:: 843 844 # echo "8:0 default" > cgroup-example-interface-file 845 # cat cgroup-example-interface-file 846 default 125 847 8:16 170 848 849- For events which are not very high frequency, an interface file 850 "events" should be created which lists event key value pairs. 851 Whenever a notifiable event happens, file modified event should be 852 generated on the file. 853 854 855Core Interface Files 856-------------------- 857 858All cgroup core files are prefixed with "cgroup." 859 860 cgroup.type 861 A read-write single value file which exists on non-root 862 cgroups. 863 864 When read, it indicates the current type of the cgroup, which 865 can be one of the following values. 866 867 - "domain" : A normal valid domain cgroup. 868 869 - "domain threaded" : A threaded domain cgroup which is 870 serving as the root of a threaded subtree. 871 872 - "domain invalid" : A cgroup which is in an invalid state. 873 It can't be populated or have controllers enabled. It may 874 be allowed to become a threaded cgroup. 875 876 - "threaded" : A threaded cgroup which is a member of a 877 threaded subtree. 878 879 A cgroup can be turned into a threaded cgroup by writing 880 "threaded" to this file. 881 882 cgroup.procs 883 A read-write new-line separated values file which exists on 884 all cgroups. 885 886 When read, it lists the PIDs of all processes which belong to 887 the cgroup one-per-line. The PIDs are not ordered and the 888 same PID may show up more than once if the process got moved 889 to another cgroup and then back or the PID got recycled while 890 reading. 891 892 A PID can be written to migrate the process associated with 893 the PID to the cgroup. The writer should match all of the 894 following conditions. 895 896 - It must have write access to the "cgroup.procs" file. 897 898 - It must have write access to the "cgroup.procs" file of the 899 common ancestor of the source and destination cgroups. 900 901 When delegating a sub-hierarchy, write access to this file 902 should be granted along with the containing directory. 903 904 In a threaded cgroup, reading this file fails with EOPNOTSUPP 905 as all the processes belong to the thread root. Writing is 906 supported and moves every thread of the process to the cgroup. 907 908 cgroup.threads 909 A read-write new-line separated values file which exists on 910 all cgroups. 911 912 When read, it lists the TIDs of all threads which belong to 913 the cgroup one-per-line. The TIDs are not ordered and the 914 same TID may show up more than once if the thread got moved to 915 another cgroup and then back or the TID got recycled while 916 reading. 917 918 A TID can be written to migrate the thread associated with the 919 TID to the cgroup. The writer should match all of the 920 following conditions. 921 922 - It must have write access to the "cgroup.threads" file. 923 924 - The cgroup that the thread is currently in must be in the 925 same resource domain as the destination cgroup. 926 927 - It must have write access to the "cgroup.procs" file of the 928 common ancestor of the source and destination cgroups. 929 930 When delegating a sub-hierarchy, write access to this file 931 should be granted along with the containing directory. 932 933 cgroup.controllers 934 A read-only space separated values file which exists on all 935 cgroups. 936 937 It shows space separated list of all controllers available to 938 the cgroup. The controllers are not ordered. 939 940 cgroup.subtree_control 941 A read-write space separated values file which exists on all 942 cgroups. Starts out empty. 943 944 When read, it shows space separated list of the controllers 945 which are enabled to control resource distribution from the 946 cgroup to its children. 947 948 Space separated list of controllers prefixed with '+' or '-' 949 can be written to enable or disable controllers. A controller 950 name prefixed with '+' enables the controller and '-' 951 disables. If a controller appears more than once on the list, 952 the last one is effective. When multiple enable and disable 953 operations are specified, either all succeed or all fail. 954 955 cgroup.events 956 A read-only flat-keyed file which exists on non-root cgroups. 957 The following entries are defined. Unless specified 958 otherwise, a value change in this file generates a file 959 modified event. 960 961 populated 962 1 if the cgroup or its descendants contains any live 963 processes; otherwise, 0. 964 frozen 965 1 if the cgroup is frozen; otherwise, 0. 966 967 cgroup.max.descendants 968 A read-write single value files. The default is "max". 969 970 Maximum allowed number of descent cgroups. 971 If the actual number of descendants is equal or larger, 972 an attempt to create a new cgroup in the hierarchy will fail. 973 974 cgroup.max.depth 975 A read-write single value files. The default is "max". 976 977 Maximum allowed descent depth below the current cgroup. 978 If the actual descent depth is equal or larger, 979 an attempt to create a new child cgroup will fail. 980 981 cgroup.stat 982 A read-only flat-keyed file with the following entries: 983 984 nr_descendants 985 Total number of visible descendant cgroups. 986 987 nr_dying_descendants 988 Total number of dying descendant cgroups. A cgroup becomes 989 dying after being deleted by a user. The cgroup will remain 990 in dying state for some time undefined time (which can depend 991 on system load) before being completely destroyed. 992 993 A process can't enter a dying cgroup under any circumstances, 994 a dying cgroup can't revive. 995 996 A dying cgroup can consume system resources not exceeding 997 limits, which were active at the moment of cgroup deletion. 998 999 nr_subsys_<cgroup_subsys> 1000 Total number of live cgroup subsystems (e.g memory 1001 cgroup) at and beneath the current cgroup. 1002 1003 nr_dying_subsys_<cgroup_subsys> 1004 Total number of dying cgroup subsystems (e.g. memory 1005 cgroup) at and beneath the current cgroup. 1006 1007 cgroup.stat.local 1008 A read-only flat-keyed file which exists in non-root cgroups. 1009 The following entry is defined: 1010 1011 frozen_usec 1012 Cumulative time that this cgroup has spent between freezing and 1013 thawing, regardless of whether by self or ancestor groups. 1014 NB: (not) reaching "frozen" state is not accounted here. 1015 1016 Using the following ASCII representation of a cgroup's freezer 1017 state, :: 1018 1019 1 _____ 1020 frozen 0 __/ \__ 1021 ab cd 1022 1023 the duration being measured is the span between a and c. 1024 1025 cgroup.freeze 1026 A read-write single value file which exists on non-root cgroups. 1027 Allowed values are "0" and "1". The default is "0". 1028 1029 Writing "1" to the file causes freezing of the cgroup and all 1030 descendant cgroups. This means that all belonging processes will 1031 be stopped and will not run until the cgroup will be explicitly 1032 unfrozen. Freezing of the cgroup may take some time; when this action 1033 is completed, the "frozen" value in the cgroup.events control file 1034 will be updated to "1" and the corresponding notification will be 1035 issued. 1036 1037 A cgroup can be frozen either by its own settings, or by settings 1038 of any ancestor cgroups. If any of ancestor cgroups is frozen, the 1039 cgroup will remain frozen. 1040 1041 Processes in the frozen cgroup can be killed by a fatal signal. 1042 They also can enter and leave a frozen cgroup: either by an explicit 1043 move by a user, or if freezing of the cgroup races with fork(). 1044 If a process is moved to a frozen cgroup, it stops. If a process is 1045 moved out of a frozen cgroup, it becomes running. 1046 1047 Frozen status of a cgroup doesn't affect any cgroup tree operations: 1048 it's possible to delete a frozen (and empty) cgroup, as well as 1049 create new sub-cgroups. 1050 1051 cgroup.kill 1052 A write-only single value file which exists in non-root cgroups. 1053 The only allowed value is "1". 1054 1055 Writing "1" to the file causes the cgroup and all descendant cgroups to 1056 be killed. This means that all processes located in the affected cgroup 1057 tree will be killed via SIGKILL. 1058 1059 Killing a cgroup tree will deal with concurrent forks appropriately and 1060 is protected against migrations. 1061 1062 In a threaded cgroup, writing this file fails with EOPNOTSUPP as 1063 killing cgroups is a process directed operation, i.e. it affects 1064 the whole thread-group. 1065 1066 cgroup.pressure 1067 A read-write single value file that allowed values are "0" and "1". 1068 The default is "1". 1069 1070 Writing "0" to the file will disable the cgroup PSI accounting. 1071 Writing "1" to the file will re-enable the cgroup PSI accounting. 1072 1073 This control attribute is not hierarchical, so disable or enable PSI 1074 accounting in a cgroup does not affect PSI accounting in descendants 1075 and doesn't need pass enablement via ancestors from root. 1076 1077 The reason this control attribute exists is that PSI accounts stalls for 1078 each cgroup separately and aggregates it at each level of the hierarchy. 1079 This may cause non-negligible overhead for some workloads when under 1080 deep level of the hierarchy, in which case this control attribute can 1081 be used to disable PSI accounting in the non-leaf cgroups. 1082 1083 irq.pressure 1084 A read-write nested-keyed file. 1085 1086 Shows pressure stall information for IRQ/SOFTIRQ. See 1087 :ref:`Documentation/accounting/psi.rst <psi>` for details. 1088 1089Controllers 1090=========== 1091 1092.. _cgroup-v2-cpu: 1093 1094CPU 1095--- 1096 1097The "cpu" controllers regulates distribution of CPU cycles. This 1098controller implements weight and absolute bandwidth limit models for 1099normal scheduling policy and absolute bandwidth allocation model for 1100realtime scheduling policy. 1101 1102In all the above models, cycles distribution is defined only on a temporal 1103base and it does not account for the frequency at which tasks are executed. 1104The (optional) utilization clamping support allows to hint the schedutil 1105cpufreq governor about the minimum desired frequency which should always be 1106provided by a CPU, as well as the maximum desired frequency, which should not 1107be exceeded by a CPU. 1108 1109WARNING: cgroup2 cpu controller doesn't yet support the (bandwidth) control of 1110realtime processes. For a kernel built with the CONFIG_RT_GROUP_SCHED option 1111enabled for group scheduling of realtime processes, the cpu controller can only 1112be enabled when all RT processes are in the root cgroup. Be aware that system 1113management software may already have placed RT processes into non-root cgroups 1114during the system boot process, and these processes may need to be moved to the 1115root cgroup before the cpu controller can be enabled with a 1116CONFIG_RT_GROUP_SCHED enabled kernel. 1117 1118With CONFIG_RT_GROUP_SCHED disabled, this limitation does not apply and some of 1119the interface files either affect realtime processes or account for them. See 1120the following section for details. Only the cpu controller is affected by 1121CONFIG_RT_GROUP_SCHED. Other controllers can be used for the resource control of 1122realtime processes irrespective of CONFIG_RT_GROUP_SCHED. 1123 1124 1125CPU Interface Files 1126~~~~~~~~~~~~~~~~~~~ 1127 1128The interaction of a process with the cpu controller depends on its scheduling 1129policy and the underlying scheduler. From the point of view of the cpu controller, 1130processes can be categorized as follows: 1131 1132* Processes under the fair-class scheduler 1133* Processes under a BPF scheduler with the corresponding ``cgroup_set_*`` callback 1134* Everything else: ``SCHED_{FIFO,RR,DEADLINE}`` and processes under a BPF scheduler 1135 without the corresponding ``cgroup_set_*`` callback 1136 1137For details on when a process is under the fair-class scheduler or a BPF scheduler, 1138check out :ref:`Documentation/scheduler/sched-ext.rst <sched-ext>`. 1139 1140For each of the following interface files, the above categories 1141will be referred to. All time durations are in microseconds. 1142 1143 cpu.stat 1144 A read-only flat-keyed file. 1145 This file exists whether the controller is enabled or not. 1146 1147 It always reports the following three stats, which account for all the 1148 processes in the cgroup (including those in descendant cgroups): 1149 1150 - usage_usec 1151 - user_usec 1152 - system_usec 1153 1154 and the following five when the controller is enabled, which account for 1155 only the processes under the fair-class scheduler: 1156 1157 - nr_periods 1158 - nr_throttled 1159 - throttled_usec 1160 - nr_bursts 1161 - burst_usec 1162 1163 Note that the above five CFS bandwidth stats are non-hierarchical; 1164 they only account for throttling caused by this cgroup's own bandwidth 1165 limit, not including throttling inherited from ancestor cgroups. 1166 1167 cpu.stat.local 1168 A read-only flat-keyed file. 1169 This file exists whether the controller is enabled or not. 1170 1171 It reports the following stat when the controller is enabled: 1172 1173 - throttled_usec 1174 1175 Unlike the ``throttled_usec`` reported by ``cpu.stat`` which 1176 accounts for throttling caused by this cgroup's own CFS 1177 bandwidth limit, ``cpu.stat.local`` reports the actual 1178 throttling time incurred by this cgroup's own runqueues, 1179 which may include throttling inherited from ancestor 1180 cgroup bandwidth limits. 1181 1182 When the controller is not enabled, this stat is not reported. 1183 1184 cpu.weight 1185 A read-write single value file which exists on non-root 1186 cgroups. The default is "100". 1187 1188 For non idle groups (cpu.idle = 0), the weight is in the 1189 range [1, 10000]. 1190 1191 If the cgroup has been configured to be SCHED_IDLE (cpu.idle = 1), 1192 then the weight will show as a 0. 1193 1194 This file affects only processes under the fair-class scheduler and a BPF 1195 scheduler with the ``cgroup_set_weight`` callback depending on what the 1196 callback actually does. 1197 1198 cpu.weight.nice 1199 A read-write single value file which exists on non-root 1200 cgroups. The default is "0". 1201 1202 The nice value is in the range [-20, 19]. 1203 1204 This interface file is an alternative interface for 1205 "cpu.weight" and allows reading and setting weight using the 1206 same values used by nice(2). Because the range is smaller and 1207 granularity is coarser for the nice values, the read value is 1208 the closest approximation of the current weight. 1209 1210 This file affects only processes under the fair-class scheduler and a BPF 1211 scheduler with the ``cgroup_set_weight`` callback depending on what the 1212 callback actually does. 1213 1214 cpu.max 1215 A read-write two value file which exists on non-root cgroups. 1216 The default is "max 100000". 1217 1218 The maximum bandwidth limit. It's in the following format:: 1219 1220 $MAX $PERIOD 1221 1222 which indicates that the group may consume up to $MAX in each 1223 $PERIOD duration. "max" for $MAX indicates no limit. If only 1224 one number is written, $MAX is updated. 1225 1226 This file affects only processes under the fair-class scheduler and a BPF 1227 scheduler with the ``cgroup_set_bandwidth`` callback depending on what 1228 the callback actually does. 1229 1230 cpu.max.burst 1231 A read-write single value file which exists on non-root 1232 cgroups. The default is "0". 1233 1234 The burst in the range [0, $MAX]. 1235 1236 This file affects only processes under the fair-class scheduler and a BPF 1237 scheduler with the ``cgroup_set_bandwidth`` callback depending on what 1238 the callback actually does. 1239 1240 cpu.pressure 1241 A read-write nested-keyed file. 1242 1243 Shows pressure stall information for CPU. See 1244 :ref:`Documentation/accounting/psi.rst <psi>` for details. 1245 1246 This file accounts for all the processes in the cgroup. 1247 1248 cpu.uclamp.min 1249 A read-write single value file which exists on non-root cgroups. 1250 The default is "0", i.e. no utilization boosting. 1251 1252 The requested minimum utilization (protection) as a percentage 1253 rational number, e.g. 12.34 for 12.34%. 1254 1255 This interface allows reading and setting minimum utilization clamp 1256 values similar to the sched_setattr(2). This minimum utilization 1257 value is used to clamp the task specific minimum utilization clamp, 1258 including those of realtime processes. 1259 1260 The requested minimum utilization (protection) is always capped by 1261 the current value for the maximum utilization (limit), i.e. 1262 `cpu.uclamp.max`. 1263 1264 This file affects all the processes in the cgroup. 1265 1266 cpu.uclamp.max 1267 A read-write single value file which exists on non-root cgroups. 1268 The default is "max". i.e. no utilization capping 1269 1270 The requested maximum utilization (limit) as a percentage rational 1271 number, e.g. 98.76 for 98.76%. 1272 1273 This interface allows reading and setting maximum utilization clamp 1274 values similar to the sched_setattr(2). This maximum utilization 1275 value is used to clamp the task specific maximum utilization clamp, 1276 including those of realtime processes. 1277 1278 This file affects all the processes in the cgroup. 1279 1280 cpu.idle 1281 A read-write single value file which exists on non-root cgroups. 1282 The default is 0. 1283 1284 This is the cgroup analog of the per-task SCHED_IDLE sched policy. 1285 Setting this value to a 1 will make the scheduling policy of the 1286 cgroup SCHED_IDLE. The threads inside the cgroup will retain their 1287 own relative priorities, but the cgroup itself will be treated as 1288 very low priority relative to its peers. 1289 1290 This file affects only processes under the fair-class scheduler and a BPF 1291 scheduler with the ``cgroup_set_idle`` callback depending on what the 1292 callback actually does. 1293 1294Memory 1295------ 1296 1297The "memory" controller regulates distribution of memory. Memory is 1298stateful and implements both limit and protection models. Due to the 1299intertwining between memory usage and reclaim pressure and the 1300stateful nature of memory, the distribution model is relatively 1301complex. 1302 1303While not completely water-tight, all major memory usages by a given 1304cgroup are tracked so that the total memory consumption can be 1305accounted and controlled to a reasonable extent. Currently, the 1306following types of memory usages are tracked. 1307 1308- Userland memory - page cache and anonymous memory. 1309 1310- Kernel data structures such as dentries and inodes. 1311 1312- TCP socket buffers. 1313 1314The above list may expand in the future for better coverage. 1315 1316 1317Memory Interface Files 1318~~~~~~~~~~~~~~~~~~~~~~ 1319 1320All memory amounts are in bytes. If a value which is not aligned to 1321PAGE_SIZE is written, the value may be rounded up to the closest 1322PAGE_SIZE multiple when read back. 1323 1324For the limit files described below, an empty or all-whitespace 1325write is accepted and sets the limit to 0. To disable a limit, 1326write "max"; to set it to zero explicitly, write "0". 1327 1328 memory.current 1329 A read-only single value file which exists on non-root 1330 cgroups. 1331 1332 The total amount of memory currently being used by the cgroup 1333 and its descendants. 1334 1335 memory.min 1336 A read-write single value file which exists on non-root 1337 cgroups. The default is "0". 1338 1339 Hard memory protection. If the memory usage of a cgroup 1340 is within its effective min boundary, the cgroup's memory 1341 won't be reclaimed under any conditions. If there is no 1342 unprotected reclaimable memory available, OOM killer 1343 is invoked. Above the effective min boundary (or 1344 effective low boundary if it is higher), pages are reclaimed 1345 proportionally to the overage, reducing reclaim pressure for 1346 smaller overages. 1347 1348 Effective min boundary is limited by memory.min values of 1349 ancestor cgroups. If there is memory.min overcommitment 1350 (child cgroup or cgroups are requiring more protected memory 1351 than parent will allow), then each child cgroup will get 1352 the part of parent's protection proportional to its 1353 actual memory usage below memory.min. 1354 1355 Putting more memory than generally available under this 1356 protection is discouraged and may lead to constant OOMs. 1357 1358 memory.low 1359 A read-write single value file which exists on non-root 1360 cgroups. The default is "0". 1361 1362 Best-effort memory protection. If the memory usage of a 1363 cgroup is within its effective low boundary, the cgroup's 1364 memory won't be reclaimed unless there is no reclaimable 1365 memory available in unprotected cgroups. 1366 Above the effective low boundary (or 1367 effective min boundary if it is higher), pages are reclaimed 1368 proportionally to the overage, reducing reclaim pressure for 1369 smaller overages. 1370 1371 Effective low boundary is limited by memory.low values of 1372 ancestor cgroups. If there is memory.low overcommitment 1373 (child cgroup or cgroups are requiring more protected memory 1374 than parent will allow), then each child cgroup will get 1375 the part of parent's protection proportional to its 1376 actual memory usage below memory.low. 1377 1378 Putting more memory than generally available under this 1379 protection is discouraged. 1380 1381 memory.high 1382 A read-write single value file which exists on non-root 1383 cgroups. The default is "max". 1384 1385 Memory usage throttle limit. If a cgroup's usage goes 1386 over the high boundary, the processes of the cgroup are 1387 throttled and put under heavy reclaim pressure. 1388 1389 Going over the high limit never invokes the OOM killer and 1390 under extreme conditions the limit may be breached. The high 1391 limit should be used in scenarios where an external process 1392 monitors the limited cgroup to alleviate heavy reclaim 1393 pressure. 1394 1395 If memory.high is opened with O_NONBLOCK then the synchronous 1396 reclaim is bypassed. This is useful for admin processes that 1397 need to dynamically adjust the job's memory limits without 1398 expending their own CPU resources on memory reclamation. The 1399 job will trigger the reclaim and/or get throttled on its 1400 next charge request. 1401 1402 Please note that with O_NONBLOCK, there is a chance that the 1403 target memory cgroup may take indefinite amount of time to 1404 reduce usage below the limit due to delayed charge request or 1405 busy-hitting its memory to slow down reclaim. 1406 1407 memory.max 1408 A read-write single value file which exists on non-root 1409 cgroups. The default is "max". 1410 1411 Memory usage hard limit. This is the main mechanism to limit 1412 memory usage of a cgroup. If a cgroup's memory usage reaches 1413 this limit and can't be reduced, the OOM killer is invoked in 1414 the cgroup. Under certain circumstances, the usage may go 1415 over the limit temporarily. 1416 1417 In default configuration regular 0-order allocations always 1418 succeed unless OOM killer chooses current task as a victim. 1419 1420 Some kinds of allocations don't invoke the OOM killer. 1421 Caller could retry them differently, return into userspace 1422 as -ENOMEM or silently ignore in cases like disk readahead. 1423 1424 If memory.max is opened with O_NONBLOCK, then the synchronous 1425 reclaim and oom-kill are bypassed. This is useful for admin 1426 processes that need to dynamically adjust the job's memory limits 1427 without expending their own CPU resources on memory reclamation. 1428 The job will trigger the reclaim and/or oom-kill on its next 1429 charge request. 1430 1431 Please note that with O_NONBLOCK, there is a chance that the 1432 target memory cgroup may take indefinite amount of time to 1433 reduce usage below the limit due to delayed charge request or 1434 busy-hitting its memory to slow down reclaim. 1435 1436 memory.reclaim 1437 A write-only nested-keyed file which exists for all cgroups. 1438 1439 This is a simple interface to trigger memory reclaim in the 1440 target cgroup. 1441 1442 Example:: 1443 1444 echo "1G" > memory.reclaim 1445 1446 Please note that the kernel can over or under reclaim from 1447 the target cgroup. If less bytes are reclaimed than the 1448 specified amount, -EAGAIN is returned. 1449 1450 Please note that the proactive reclaim (triggered by this 1451 interface) is not meant to indicate memory pressure on the 1452 memory cgroup. Therefore socket memory balancing triggered by 1453 the memory reclaim normally is not exercised in this case. 1454 This means that the networking layer will not adapt based on 1455 reclaim induced by memory.reclaim. 1456 1457The following nested keys are defined. 1458 1459 ========== ================================ 1460 swappiness Swappiness value to reclaim with 1461 ========== ================================ 1462 1463 Specifying a swappiness value instructs the kernel to perform 1464 the reclaim with that swappiness value. Note that this has the 1465 same semantics as vm.swappiness applied to memcg reclaim with 1466 all the existing limitations and potential future extensions. 1467 1468 The valid range for swappiness is [0-200, max], setting 1469 swappiness=max exclusively reclaims anonymous memory. 1470 1471 memory.peak 1472 A read-write single value file which exists on non-root cgroups. 1473 1474 The max memory usage recorded for the cgroup and its descendants since 1475 either the creation of the cgroup or the most recent reset for that FD. 1476 1477 A write of any non-empty string to this file resets it to the 1478 current memory usage for subsequent reads through the same 1479 file descriptor. 1480 1481 memory.oom.group 1482 A read-write single value file which exists on non-root 1483 cgroups. The default value is "0". 1484 1485 Determines whether the cgroup should be treated as 1486 an indivisible workload by the OOM killer. If set, 1487 all tasks belonging to the cgroup or to its descendants 1488 (if the memory cgroup is not a leaf cgroup) are killed 1489 together or not at all. This can be used to avoid 1490 partial kills to guarantee workload integrity. 1491 1492 Tasks with the OOM protection (oom_score_adj set to -1000) 1493 are treated as an exception and are never killed. 1494 1495 If the OOM killer is invoked in a cgroup, it's not going 1496 to kill any tasks outside of this cgroup, regardless 1497 memory.oom.group values of ancestor cgroups. 1498 1499 memory.events 1500 A read-only flat-keyed file which exists on non-root cgroups. 1501 The following entries are defined. Unless specified 1502 otherwise, a value change in this file generates a file 1503 modified event. 1504 1505 Note that all fields in this file are hierarchical and the 1506 file modified event can be generated due to an event down the 1507 hierarchy. For the local events at the cgroup level see 1508 memory.events.local. 1509 1510 low 1511 The number of times the cgroup is reclaimed due to 1512 high memory pressure even though its usage is under 1513 the low boundary. This usually indicates that the low 1514 boundary is over-committed. 1515 1516 high 1517 The number of times processes of the cgroup are 1518 throttled and routed to perform direct memory reclaim 1519 because the high memory boundary was exceeded. For a 1520 cgroup whose memory usage is capped by the high limit 1521 rather than global memory pressure, this event's 1522 occurrences are expected. 1523 1524 max 1525 The number of times the cgroup's memory usage was 1526 about to go over the max boundary. If direct reclaim 1527 fails to bring it down, the cgroup goes to OOM state. 1528 1529 oom 1530 The number of time the cgroup's memory usage was 1531 reached the limit and allocation was about to fail. 1532 1533 This event is not raised if the OOM killer is not 1534 considered as an option, e.g. for failed high-order 1535 allocations or if caller asked to not retry attempts. 1536 1537 oom_kill 1538 The number of processes belonging to this cgroup 1539 killed by any kind of OOM killer. 1540 1541 oom_group_kill 1542 The number of times a group OOM has occurred. 1543 1544 sock_throttled 1545 The number of times network sockets associated with 1546 this cgroup are throttled. 1547 1548 memory.events.local 1549 Similar to memory.events but the fields in the file are local 1550 to the cgroup i.e. not hierarchical. The file modified event 1551 generated on this file reflects only the local events. 1552 1553 memory.stat 1554 A read-only flat-keyed file which exists on non-root cgroups. 1555 1556 This breaks down the cgroup's memory footprint into different 1557 types of memory, type-specific details, and other information 1558 on the state and past events of the memory management system. 1559 1560 All memory amounts are in bytes. 1561 1562 The entries are ordered to be human readable, and new entries 1563 can show up in the middle. Don't rely on items remaining in a 1564 fixed position; use the keys to look up specific values! 1565 1566 If the entry has no per-node counter (or not show in the 1567 memory.numa_stat). We use 'npn' (non-per-node) as the tag 1568 to indicate that it will not show in the memory.numa_stat. 1569 1570 anon 1571 Amount of memory used in anonymous mappings such as 1572 brk(), sbrk(), and mmap(MAP_ANONYMOUS). Note that 1573 some kernel configurations might account complete larger 1574 allocations (e.g., THP) if only some, but not all the 1575 memory of such an allocation is mapped anymore. 1576 1577 file 1578 Amount of memory used to cache filesystem data, 1579 including tmpfs and shared memory. 1580 1581 kernel (npn) 1582 Amount of total kernel memory, including 1583 (kernel_stack, pagetables, percpu, vmalloc, slab) in 1584 addition to other kernel memory use cases. 1585 1586 kernel_stack 1587 Amount of memory allocated to kernel stacks. 1588 1589 pagetables 1590 Amount of memory allocated for page tables. 1591 1592 sec_pagetables 1593 Amount of memory allocated for secondary page tables, 1594 this currently includes KVM mmu allocations on x86 1595 and arm64 and IOMMU page tables. 1596 1597 percpu (npn) 1598 Amount of memory used for storing per-cpu kernel 1599 data structures. 1600 1601 sock (npn) 1602 Amount of memory used in network transmission buffers 1603 1604 vmalloc 1605 Amount of memory used for vmap backed memory. 1606 1607 shmem 1608 Amount of cached filesystem data that is swap-backed, 1609 such as tmpfs, shm segments, shared anonymous mmap()s 1610 1611 zswap 1612 Amount of memory consumed by the zswap compression backend. 1613 1614 zswapped 1615 Amount of application memory swapped out to zswap. 1616 1617 file_mapped 1618 Amount of cached filesystem data mapped with mmap(). Note 1619 that some kernel configurations might account complete 1620 larger allocations (e.g., THP) if only some, but not 1621 not all the memory of such an allocation is mapped. 1622 1623 file_dirty 1624 Amount of cached filesystem data that was modified but 1625 not yet written back to disk 1626 1627 file_writeback 1628 Amount of cached filesystem data that was modified and 1629 is currently being written back to disk 1630 1631 swapcached 1632 Amount of swap cached in memory. The swapcache is accounted 1633 against both memory and swap usage. 1634 1635 anon_thp 1636 Amount of memory used in anonymous mappings backed by 1637 transparent hugepages 1638 1639 file_thp 1640 Amount of cached filesystem data backed by transparent 1641 hugepages 1642 1643 shmem_thp 1644 Amount of shm, tmpfs, shared anonymous mmap()s backed by 1645 transparent hugepages 1646 1647 inactive_anon, active_anon, inactive_file, active_file, unevictable 1648 Amount of memory, swap-backed and filesystem-backed, 1649 on the internal memory management lists used by the 1650 page reclaim algorithm. 1651 1652 As these represent internal list state (eg. shmem pages are on anon 1653 memory management lists), inactive_foo + active_foo may not be equal to 1654 the value for the foo counter, since the foo counter is type-based, not 1655 list-based. 1656 1657 slab_reclaimable 1658 Part of "slab" that might be reclaimed, such as 1659 dentries and inodes. 1660 1661 slab_unreclaimable 1662 Part of "slab" that cannot be reclaimed on memory 1663 pressure. 1664 1665 slab (npn) 1666 Amount of memory used for storing in-kernel data 1667 structures. 1668 1669 workingset_refault_anon 1670 Number of refaults of previously evicted anonymous pages. 1671 1672 workingset_refault_file 1673 Number of refaults of previously evicted file pages. 1674 1675 workingset_activate_anon 1676 Number of refaulted anonymous pages that were immediately 1677 activated. 1678 1679 workingset_activate_file 1680 Number of refaulted file pages that were immediately activated. 1681 1682 workingset_restore_anon 1683 Number of restored anonymous pages which have been detected as 1684 an active workingset before they got reclaimed. 1685 1686 workingset_restore_file 1687 Number of restored file pages which have been detected as an 1688 active workingset before they got reclaimed. 1689 1690 workingset_nodereclaim 1691 Number of times a shadow node has been reclaimed 1692 1693 pswpin (npn) 1694 Number of pages swapped into memory 1695 1696 pswpout (npn) 1697 Number of pages swapped out of memory 1698 1699 pgscan (npn) 1700 Amount of scanned pages (in an inactive LRU list) 1701 1702 pgsteal (npn) 1703 Amount of reclaimed pages 1704 1705 pgscan_kswapd (npn) 1706 Amount of scanned pages by kswapd (in an inactive LRU list) 1707 1708 pgscan_direct (npn) 1709 Amount of scanned pages directly (in an inactive LRU list) 1710 1711 pgscan_khugepaged (npn) 1712 Amount of scanned pages by khugepaged (in an inactive LRU list) 1713 1714 pgscan_proactive (npn) 1715 Amount of scanned pages proactively (in an inactive LRU list) 1716 1717 pgsteal_kswapd (npn) 1718 Amount of reclaimed pages by kswapd 1719 1720 pgsteal_direct (npn) 1721 Amount of reclaimed pages directly 1722 1723 pgsteal_khugepaged (npn) 1724 Amount of reclaimed pages by khugepaged 1725 1726 pgsteal_proactive (npn) 1727 Amount of reclaimed pages proactively 1728 1729 pgfault (npn) 1730 Total number of page faults incurred 1731 1732 pgmajfault (npn) 1733 Number of major page faults incurred 1734 1735 pgrefill (npn) 1736 Amount of scanned pages (in an active LRU list) 1737 1738 pgactivate (npn) 1739 Amount of pages moved to the active LRU list 1740 1741 pgdeactivate (npn) 1742 Amount of pages moved to the inactive LRU list 1743 1744 pglazyfree (npn) 1745 Amount of pages postponed to be freed under memory pressure 1746 1747 pglazyfreed (npn) 1748 Amount of reclaimed lazyfree pages 1749 1750 swpin_zero 1751 Number of pages swapped into memory and filled with zero, where I/O 1752 was optimized out because the page content was detected to be zero 1753 during swapout. 1754 1755 swpout_zero 1756 Number of zero-filled pages swapped out with I/O skipped due to the 1757 content being detected as zero. 1758 1759 zswpin 1760 Number of pages moved in to memory from zswap. 1761 1762 zswpout 1763 Number of pages moved out of memory to zswap. 1764 1765 zswpwb 1766 Number of pages written from zswap to swap. 1767 1768 zswap_incomp 1769 Amount of memory used by incompressible pages currently stored in zswap 1770 without compression. These pages could not be compressed to 1771 a size smaller than PAGE_SIZE, so they are stored as-is. 1772 1773 thp_fault_alloc (npn) 1774 Number of transparent hugepages which were allocated to satisfy 1775 a page fault. This counter is not present when CONFIG_TRANSPARENT_HUGEPAGE 1776 is not set. 1777 1778 thp_collapse_alloc (npn) 1779 Number of transparent hugepages which were allocated to allow 1780 collapsing an existing range of pages. This counter is not 1781 present when CONFIG_TRANSPARENT_HUGEPAGE is not set. 1782 1783 thp_swpout (npn) 1784 Number of transparent hugepages which are swapout in one piece 1785 without splitting. 1786 1787 thp_swpout_fallback (npn) 1788 Number of transparent hugepages which were split before swapout. 1789 Usually because failed to allocate some continuous swap space 1790 for the huge page. 1791 1792 numa_pages_migrated (npn) 1793 Number of pages migrated by NUMA balancing. 1794 1795 numa_pte_updates (npn) 1796 Number of pages whose page table entries are modified by 1797 NUMA balancing to produce NUMA hinting faults on access. 1798 1799 numa_hint_faults (npn) 1800 Number of NUMA hinting faults. 1801 1802 pgdemote_kswapd 1803 Number of pages demoted by kswapd. 1804 1805 pgdemote_direct 1806 Number of pages demoted directly. 1807 1808 pgdemote_khugepaged 1809 Number of pages demoted by khugepaged. 1810 1811 pgdemote_proactive 1812 Number of pages demoted by proactively. 1813 1814 hugetlb 1815 Amount of memory used by hugetlb pages. This metric only shows 1816 up if hugetlb usage is accounted for in memory.current (i.e. 1817 cgroup is mounted with the memory_hugetlb_accounting option). 1818 1819 memory.numa_stat 1820 A read-only nested-keyed file which exists on non-root cgroups. 1821 1822 This breaks down the cgroup's memory footprint into different 1823 types of memory, type-specific details, and other information 1824 per node on the state of the memory management system. 1825 1826 This is useful for providing visibility into the NUMA locality 1827 information within an memcg since the pages are allowed to be 1828 allocated from any physical node. One of the use case is evaluating 1829 application performance by combining this information with the 1830 application's CPU allocation. 1831 1832 All memory amounts are in bytes. 1833 1834 The output format of memory.numa_stat is:: 1835 1836 type N0=<bytes in node 0> N1=<bytes in node 1> ... 1837 1838 The entries are ordered to be human readable, and new entries 1839 can show up in the middle. Don't rely on items remaining in a 1840 fixed position; use the keys to look up specific values! 1841 1842 The entries can refer to the memory.stat. 1843 1844 memory.swap.current 1845 A read-only single value file which exists on non-root 1846 cgroups. 1847 1848 The total amount of swap currently being used by the cgroup 1849 and its descendants. 1850 1851 memory.swap.high 1852 A read-write single value file which exists on non-root 1853 cgroups. The default is "max". 1854 1855 Swap usage throttle limit. If a cgroup's swap usage exceeds 1856 this limit, all its further allocations will be throttled to 1857 allow userspace to implement custom out-of-memory procedures. 1858 1859 This limit marks a point of no return for the cgroup. It is NOT 1860 designed to manage the amount of swapping a workload does 1861 during regular operation. Compare to memory.swap.max, which 1862 prohibits swapping past a set amount, but lets the cgroup 1863 continue unimpeded as long as other memory can be reclaimed. 1864 1865 Healthy workloads are not expected to reach this limit. 1866 1867 memory.swap.peak 1868 A read-write single value file which exists on non-root cgroups. 1869 1870 The max swap usage recorded for the cgroup and its descendants since 1871 the creation of the cgroup or the most recent reset for that FD. 1872 1873 A write of any non-empty string to this file resets it to the 1874 current memory usage for subsequent reads through the same 1875 file descriptor. 1876 1877 memory.swap.max 1878 A read-write single value file which exists on non-root 1879 cgroups. The default is "max". 1880 1881 Swap usage hard limit. If a cgroup's swap usage reaches this 1882 limit, anonymous memory of the cgroup will not be swapped out. 1883 1884 memory.swap.events 1885 A read-only flat-keyed file which exists on non-root cgroups. 1886 The following entries are defined. Unless specified 1887 otherwise, a value change in this file generates a file 1888 modified event. 1889 1890 high 1891 The number of times the cgroup's swap usage was over 1892 the high threshold. 1893 1894 max 1895 The number of times the cgroup's swap usage was about 1896 to go over the max boundary and swap allocation 1897 failed. 1898 1899 fail 1900 The number of times swap allocation failed either 1901 because of running out of swap system-wide or max 1902 limit. 1903 1904 When reduced under the current usage, the existing swap 1905 entries are reclaimed gradually and the swap usage may stay 1906 higher than the limit for an extended period of time. This 1907 reduces the impact on the workload and memory management. 1908 1909 memory.zswap.current 1910 A read-only single value file which exists on non-root 1911 cgroups. 1912 1913 The total amount of memory consumed by the zswap compression 1914 backend. 1915 1916 memory.zswap.max 1917 A read-write single value file which exists on non-root 1918 cgroups. The default is "max". 1919 1920 Zswap usage hard limit. If a cgroup's zswap pool reaches this 1921 limit, it will refuse to take any more stores before existing 1922 entries fault back in or are written out to disk. 1923 1924 memory.zswap.writeback 1925 A read-write single value file. The default value is "1". 1926 Note that this setting is hierarchical, i.e. the writeback would be 1927 implicitly disabled for child cgroups if the upper hierarchy 1928 does so. 1929 1930 When this is set to 0, all swapping attempts to swapping devices 1931 are disabled. This included both zswap writebacks, and swapping due 1932 to zswap store failures. If the zswap store failures are recurring 1933 (for e.g if the pages are incompressible), users can observe 1934 reclaim inefficiency after disabling writeback (because the same 1935 pages might be rejected again and again). 1936 1937 Note that this is subtly different from setting memory.swap.max to 1938 0, as it still allows for pages to be written to the zswap pool. 1939 This setting has no effect if zswap is disabled, and swapping 1940 is allowed unless memory.swap.max is set to 0. 1941 1942 memory.pressure 1943 A read-write nested-keyed file. 1944 1945 Shows pressure stall information for memory. See 1946 :ref:`Documentation/accounting/psi.rst <psi>` for details. 1947 1948 1949Usage Guidelines 1950~~~~~~~~~~~~~~~~ 1951 1952"memory.high" is the main mechanism to control memory usage. 1953Over-committing on high limit (sum of high limits > available memory) 1954and letting global memory pressure to distribute memory according to 1955usage is a viable strategy. 1956 1957Because breach of the high limit doesn't trigger the OOM killer but 1958throttles the offending cgroup, a management agent has ample 1959opportunities to monitor and take appropriate actions such as granting 1960more memory or terminating the workload. 1961 1962Determining whether a cgroup has enough memory is not trivial as 1963memory usage doesn't indicate whether the workload can benefit from 1964more memory. For example, a workload which writes data received from 1965network to a file can use all available memory but can also operate as 1966performant with a small amount of memory. A measure of memory 1967pressure - how much the workload is being impacted due to lack of 1968memory - is necessary to determine whether a workload needs more 1969memory; unfortunately, memory pressure monitoring mechanism isn't 1970implemented yet. 1971 1972Reclaim Protection 1973~~~~~~~~~~~~~~~~~~ 1974 1975The protection configured with "memory.low" or "memory.min" applies relatively 1976to the target of the reclaim (i.e. any of memory cgroup limits, proactive 1977memory.reclaim or global reclaim apparently located in the root cgroup). 1978The protection value configured for B applies unchanged to the reclaim 1979targeting A (i.e. caused by competition with the sibling E):: 1980 1981 root - ... - A - B - C 1982 \ ` D 1983 ` E 1984 1985When the reclaim targets ancestors of A, the effective protection of B is 1986capped by the protection value configured for A (and any other intermediate 1987ancestors between A and the target). 1988 1989To express indifference about relative sibling protection, it is suggested to 1990use memory_recursiveprot. Configuring all descendants of a parent with finite 1991protection to "max" works but it may unnecessarily skew memory.events:low 1992field. 1993 1994Memory Ownership 1995~~~~~~~~~~~~~~~~ 1996 1997A memory area is charged to the cgroup which instantiated it and stays 1998charged to the cgroup until the area is released. Migrating a process 1999to a different cgroup doesn't move the memory usages that it 2000instantiated while in the previous cgroup to the new cgroup. 2001 2002A memory area may be used by processes belonging to different cgroups. 2003To which cgroup the area will be charged is in-deterministic; however, 2004over time, the memory area is likely to end up in a cgroup which has 2005enough memory allowance to avoid high reclaim pressure. 2006 2007If a cgroup sweeps a considerable amount of memory which is expected 2008to be accessed repeatedly by other cgroups, it may make sense to use 2009POSIX_FADV_DONTNEED to relinquish the ownership of memory areas 2010belonging to the affected files to ensure correct memory ownership. 2011 2012 2013IO 2014-- 2015 2016The "io" controller regulates the distribution of IO resources. This 2017controller implements both weight based and absolute bandwidth or IOPS 2018limit distribution. Absolute BPS and IOPS limits are enforced by 2019blk-throttle and apply to all devices, while weight based proportional 2020distribution is provided by the iocost cost model controller 2021(CONFIG_BLK_CGROUP_IOCOST) and, when the BFQ I/O scheduler is in use 2022for a device, by BFQ's own cgroup support. Latency-based protection 2023(CONFIG_BLK_CGROUP_IOLATENCY) and I/O priority assignment 2024(CONFIG_BLK_CGROUP_IOPRIO) are also available. 2025 2026 2027IO Interface Files 2028~~~~~~~~~~~~~~~~~~ 2029 2030 io.stat 2031 A read-only nested-keyed file. 2032 2033 Lines are keyed by $MAJ:$MIN device numbers and not ordered. 2034 The following nested keys are defined. 2035 2036 ====== ===================== 2037 rbytes Bytes read 2038 wbytes Bytes written 2039 rios Number of read IOs 2040 wios Number of write IOs 2041 dbytes Bytes discarded 2042 dios Number of discard IOs 2043 ====== ===================== 2044 2045 An example read output follows:: 2046 2047 8:16 rbytes=1459200 wbytes=314773504 rios=192 wios=353 dbytes=0 dios=0 2048 8:0 rbytes=90430464 wbytes=299008000 rios=8950 wios=1252 dbytes=50331648 dios=3021 2049 2050 io.cost.qos 2051 A read-write nested-keyed file which exists only on the root 2052 cgroup. 2053 2054 This file configures the Quality of Service of the IO cost 2055 model based controller (CONFIG_BLK_CGROUP_IOCOST) which 2056 currently implements "io.weight" proportional control. Lines 2057 are keyed by $MAJ:$MIN device numbers and not ordered. The 2058 line for a given device is populated on the first write for 2059 the device on "io.cost.qos" or "io.cost.model". The following 2060 nested keys are defined. 2061 2062 ====== ===================================== 2063 enable Weight-based control enable 2064 ctrl "auto" or "user" 2065 rpct Read latency percentile [0, 100] 2066 rlat Read latency threshold 2067 wpct Write latency percentile [0, 100] 2068 wlat Write latency threshold 2069 min Minimum scaling percentage [1, 10000] 2070 max Maximum scaling percentage [1, 10000] 2071 ====== ===================================== 2072 2073 The controller is disabled by default and can be enabled by 2074 setting "enable" to 1. "rpct" and "wpct" parameters default 2075 to zero and the controller uses internal device saturation 2076 state to adjust the overall IO rate between "min" and "max". 2077 2078 When a better control quality is needed, latency QoS 2079 parameters can be configured. For example:: 2080 2081 8:16 enable=1 ctrl=auto rpct=95.00 rlat=75000 wpct=95.00 wlat=150000 min=50.00 max=150.0 2082 2083 shows that on sdb, the controller is enabled, will consider 2084 the device saturated if the 95th percentile of read completion 2085 latencies is above 75ms or write 150ms, and adjust the overall 2086 IO issue rate between 50% and 150% accordingly. 2087 2088 The lower the saturation point, the better the latency QoS at 2089 the cost of aggregate bandwidth. The narrower the allowed 2090 adjustment range between "min" and "max", the more conformant 2091 to the cost model the IO behavior. Note that the IO issue 2092 base rate may be far off from 100% and setting "min" and "max" 2093 blindly can lead to a significant loss of device capacity or 2094 control quality. "min" and "max" are useful for regulating 2095 devices which show wide temporary behavior changes - e.g. a 2096 ssd which accepts writes at the line speed for a while and 2097 then completely stalls for multiple seconds. 2098 2099 When "ctrl" is "auto", the parameters are controlled by the 2100 kernel and may change automatically. Setting "ctrl" to "user" 2101 or setting any of the percentile and latency parameters puts 2102 it into "user" mode and disables the automatic changes. The 2103 automatic mode can be restored by setting "ctrl" to "auto". 2104 2105 io.cost.model 2106 A read-write nested-keyed file which exists only on the root 2107 cgroup. 2108 2109 This file configures the cost model of the IO cost model based 2110 controller (CONFIG_BLK_CGROUP_IOCOST) which currently 2111 implements "io.weight" proportional control. Lines are keyed 2112 by $MAJ:$MIN device numbers and not ordered. The line for a 2113 given device is populated on the first write for the device on 2114 "io.cost.qos" or "io.cost.model". The following nested keys 2115 are defined. 2116 2117 ===== ================================ 2118 ctrl "auto" or "user" 2119 model The cost model in use - "linear" 2120 ===== ================================ 2121 2122 When "ctrl" is "auto", the kernel may change all parameters 2123 dynamically. When "ctrl" is set to "user" or any other 2124 parameters are written to, "ctrl" become "user" and the 2125 automatic changes are disabled. 2126 2127 When "model" is "linear", the following model parameters are 2128 defined. 2129 2130 ============= ======================================== 2131 [r|w]bps The maximum sequential IO throughput 2132 [r|w]seqiops The maximum 4k sequential IOs per second 2133 [r|w]randiops The maximum 4k random IOs per second 2134 flushiops The maximum flushes per second 2135 ============= ======================================== 2136 2137 From the above, the builtin linear model determines the base 2138 costs of a sequential and random IO and the cost coefficient 2139 for the IO size. While simple, this model can cover most 2140 common device classes acceptably. 2141 2142 "flushiops" determines the cost of a cache flush: a write with 2143 a preceding cache flush is charged one flush on top of its data 2144 cost, and a FUA write one more flush on devices without native 2145 FUA support. It is zero in the builtin profiles, so flushes 2146 stay free until it is configured. 2147 2148 The IO cost model isn't expected to be accurate in absolute 2149 sense and is scaled to the device behavior dynamically. 2150 2151 If needed, tools/cgroup/iocost_coef_gen.py can be used to 2152 generate device-specific coefficients. 2153 2154 io.weight 2155 A read-write flat-keyed file which exists on non-root cgroups. 2156 The default is "default 100". 2157 2158 The first line is the default weight applied to devices 2159 without specific override. The rest are overrides keyed by 2160 $MAJ:$MIN device numbers and not ordered. The weights are in 2161 the range [1, 10000] and specifies the relative amount IO time 2162 the cgroup can use in relation to its siblings. 2163 2164 The default weight can be updated by writing either "default 2165 $WEIGHT" or simply "$WEIGHT". Overrides can be set by writing 2166 "$MAJ:$MIN $WEIGHT" and unset by writing "$MAJ:$MIN default". 2167 2168 An example read output follows:: 2169 2170 default 100 2171 8:16 200 2172 8:0 50 2173 2174 io.max 2175 A read-write nested-keyed file which exists on non-root 2176 cgroups. 2177 2178 BPS and IOPS based IO limit. Lines are keyed by $MAJ:$MIN 2179 device numbers and not ordered. The following nested keys are 2180 defined. 2181 2182 ===== ================================== 2183 rbps Max read bytes per second 2184 wbps Max write bytes per second 2185 riops Max read IO operations per second 2186 wiops Max write IO operations per second 2187 ===== ================================== 2188 2189 When writing, any number of nested key-value pairs can be 2190 specified in any order. "max" can be specified as the value 2191 to remove a specific limit. If the same key is specified 2192 multiple times, the outcome is undefined. 2193 2194 BPS and IOPS are measured in each IO direction and IOs are 2195 delayed if limit is reached. Temporary bursts are allowed. 2196 2197 Setting read limit at 2M BPS and write at 120 IOPS for 8:16:: 2198 2199 echo "8:16 rbps=2097152 wiops=120" > io.max 2200 2201 Reading returns the following:: 2202 2203 8:16 rbps=2097152 wbps=max riops=max wiops=120 2204 2205 Write IOPS limit can be removed by writing the following:: 2206 2207 echo "8:16 wiops=max" > io.max 2208 2209 Reading now returns the following:: 2210 2211 8:16 rbps=2097152 wbps=max riops=max wiops=max 2212 2213 io.pressure 2214 A read-write nested-keyed file. 2215 2216 Shows pressure stall information for IO. See 2217 :ref:`Documentation/accounting/psi.rst <psi>` for details. 2218 2219 2220Writeback 2221~~~~~~~~~ 2222 2223Page cache is dirtied through buffered writes and shared mmaps and 2224written asynchronously to the backing filesystem by the writeback 2225mechanism. Writeback sits between the memory and IO domains and 2226regulates the proportion of dirty memory by balancing dirtying and 2227write IOs. 2228 2229The io controller, in conjunction with the memory controller, 2230implements control of page cache writeback IOs. The memory controller 2231defines the memory domain that dirty memory ratio is calculated and 2232maintained for and the io controller defines the io domain which 2233writes out dirty pages for the memory domain. Both system-wide and 2234per-cgroup dirty memory states are examined and the more restrictive 2235of the two is enforced. 2236 2237cgroup writeback requires explicit support from the underlying 2238filesystem. Currently, cgroup writeback is implemented on ext2, ext4, 2239btrfs, f2fs, and xfs. On other filesystems, all writeback IOs are 2240attributed to the root cgroup. 2241 2242There are inherent differences in memory and writeback management 2243which affects how cgroup ownership is tracked. Memory is tracked per 2244page while writeback per inode. For the purpose of writeback, an 2245inode is assigned to a cgroup and all IO requests to write dirty pages 2246from the inode are attributed to that cgroup. 2247 2248As cgroup ownership for memory is tracked per page, there can be pages 2249which are associated with different cgroups than the one the inode is 2250associated with. These are called foreign pages. The writeback 2251constantly keeps track of foreign pages and, if a particular foreign 2252cgroup becomes the majority over a certain period of time, switches 2253the ownership of the inode to that cgroup. 2254 2255While this model is enough for most use cases where a given inode is 2256mostly dirtied by a single cgroup even when the main writing cgroup 2257changes over time, use cases where multiple cgroups write to a single 2258inode simultaneously are not supported well. In such circumstances, a 2259significant portion of IOs are likely to be attributed incorrectly. 2260As memory controller assigns page ownership on the first use and 2261doesn't update it until the page is released, even if writeback 2262strictly follows page ownership, multiple cgroups dirtying overlapping 2263areas wouldn't work as expected. It's recommended to avoid such usage 2264patterns. 2265 2266The sysctl knobs which affect writeback behavior are applied to cgroup 2267writeback as follows. 2268 2269 vm.dirty_background_ratio, vm.dirty_ratio 2270 These ratios apply the same to cgroup writeback with the 2271 amount of available memory capped by limits imposed by the 2272 memory controller and system-wide clean memory. 2273 2274 vm.dirty_background_bytes, vm.dirty_bytes 2275 For cgroup writeback, this is calculated into ratio against 2276 total available memory and applied the same way as 2277 vm.dirty[_background]_ratio. 2278 2279 2280IO Latency 2281~~~~~~~~~~ 2282 2283This is a cgroup v2 controller for IO workload protection. You provide a group 2284with a latency target, and if the group misses its target the controller will 2285throttle any peers that have a lower latency target than the protected 2286workload. How a miss is detected depends on the device: on rotational devices 2287the average latency over the window must exceed the target, while on 2288non-rotational devices a miss is counted once enough of the IOs in the window 2289individually exceed the target. 2290 2291The limits are only applied at the peer level in the hierarchy. This means that 2292in the diagram below, only groups A, B, and C will influence each other, and 2293groups D and F will influence each other. Group G will influence nobody:: 2294 2295 [root] 2296 / | \ 2297 A B C 2298 / \ | 2299 D F G 2300 2301 2302So the ideal way to configure this is to set io.latency in groups A, B, and C. 2303Generally you do not want to set a value lower than the latency your device 2304supports. Experiment to find the value that works best for your workload. 2305Start at higher than the expected latency for your device and, with 2306blkcg_debug_stats enabled, observe io.stat for your workload group to get an 2307idea of the latency you see during normal operation. On rotational devices, 2308use the avg_lat value as a basis for your real setting, setting it 10-15% 2309higher. On non-rotational devices io.stat reports no average latency; set 2310the target based on your device and use the missed/total fields to verify it 2311is being met. 2312 2313How IO Latency Throttling Works 2314~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2315 2316io.latency is work conserving; so as long as everybody is meeting their latency 2317target the controller doesn't do anything. Once a group starts missing its 2318target it begins throttling any peer group that has a higher target than itself. 2319This throttling takes 2 forms: 2320 2321- Queue depth throttling. This is the number of outstanding IO's a group is 2322 allowed to have. We will clamp down relatively quickly, starting at no limit 2323 and going all the way down to 1 IO at a time. 2324 2325- Artificial delay induction. There are certain types of IO that cannot be 2326 throttled without possibly adversely affecting higher priority groups. This 2327 includes swapping and metadata IO. These types of IO are allowed to occur 2328 normally, however they are "charged" to the originating group. If the 2329 originating group is being throttled you will see the use_delay and delay_nsec 2330 fields in io.stat increase. The delay_nsec value is how many nanoseconds that 2331 are being added to any process that runs in this group. Because this number can 2332 grow quite large if there is a lot of swapping or metadata IO occurring we 2333 limit the individual delay events to 1 second at a time. 2334 2335Once the victimized group starts meeting its latency target again it will start 2336unthrottling any peer groups that were throttled previously. If the victimized 2337group simply stops doing IO the global counter will unthrottle appropriately. 2338 2339IO Latency Interface Files 2340~~~~~~~~~~~~~~~~~~~~~~~~~~ 2341 2342 io.latency 2343 This takes a similar format as the other controllers. 2344 2345 "MAJOR:MINOR target=<target time in microseconds>" 2346 2347 io.stat 2348 If the controller is enabled you will see extra stats in io.stat in 2349 addition to the normal ones. These debug stats are only emitted when 2350 the blkcg_debug_stats module parameter is enabled (it is disabled by 2351 default). 2352 2353 The reported latency fields depend on the device. Rotational devices 2354 report avg_lat and win; non-rotational devices report missed and total 2355 instead. missed and total are live counters for the current window and 2356 may change between reads. 2357 2358 depth 2359 This is the current queue depth for the group. 2360 2361 avg_lat 2362 (Rotational devices only.) This is an exponential moving 2363 average with a decay rate of 1/exp bound by the sampling 2364 interval. The decay rate interval can be calculated by 2365 multiplying the win value in io.stat by the corresponding number 2366 of samples based on the win value. 2367 2368 win 2369 (Rotational devices only.) The sampling window size in 2370 milliseconds. This is the minimum duration of time between 2371 evaluation events. Windows only elapse with IO activity. Idle 2372 periods extend the most recent window. 2373 2374 missed 2375 (Non-rotational devices only.) The number of IOs in the 2376 current window whose latency exceeded the target. A group is 2377 considered to be missing its target once missed reaches a 2378 certain ratio of total. 2379 2380 total 2381 (Non-rotational devices only.) The total number of IOs 2382 accounted in the current window. 2383 2384IO Priority 2385~~~~~~~~~~~ 2386 2387A single attribute controls the behavior of the I/O priority cgroup policy, 2388namely the io.prio.class attribute. The following values are accepted for 2389that attribute: 2390 2391 no-change 2392 Do not modify the I/O priority class. 2393 2394 promote-to-rt 2395 For requests that have a non-RT I/O priority class, change it into RT. 2396 Also change the priority level of these requests to 4. Do not modify 2397 the I/O priority of requests that have priority class RT. 2398 2399 restrict-to-be 2400 For requests that do not have an I/O priority class or that have I/O 2401 priority class RT, change it into BE. Also change the priority level 2402 of these requests to 0. Do not modify the I/O priority class of 2403 requests that have priority class IDLE. 2404 2405 idle 2406 Change the I/O priority class of all requests into IDLE, the lowest 2407 I/O priority class. 2408 2409 none-to-rt 2410 Deprecated. Just an alias for promote-to-rt. 2411 2412The following numerical values are associated with the I/O priority policies: 2413 2414+----------------+---+ 2415| no-change | 0 | 2416+----------------+---+ 2417| promote-to-rt | 1 | 2418+----------------+---+ 2419| restrict-to-be | 2 | 2420+----------------+---+ 2421| idle | 3 | 2422+----------------+---+ 2423 2424The numerical value that corresponds to each I/O priority class is as follows: 2425 2426+-------------------------------+---+ 2427| IOPRIO_CLASS_NONE | 0 | 2428+-------------------------------+---+ 2429| IOPRIO_CLASS_RT (real-time) | 1 | 2430+-------------------------------+---+ 2431| IOPRIO_CLASS_BE (best effort) | 2 | 2432+-------------------------------+---+ 2433| IOPRIO_CLASS_IDLE | 3 | 2434+-------------------------------+---+ 2435 2436The algorithm to set the I/O priority class for a request is as follows: 2437 2438- If I/O priority class policy is promote-to-rt, change the request I/O 2439 priority class to IOPRIO_CLASS_RT and change the request I/O priority 2440 level to 4. 2441- If I/O priority class policy is not promote-to-rt, translate the I/O priority 2442 class policy into a number, then change the request I/O priority class 2443 into the maximum of the I/O priority class policy number and the numerical 2444 I/O priority class. 2445 2446PID 2447--- 2448 2449The process number controller is used to allow a cgroup to stop any 2450new tasks from being fork()'d or clone()'d after a specified limit is 2451reached. 2452 2453The number of tasks in a cgroup can be exhausted in ways which other 2454controllers cannot prevent, thus warranting its own controller. For 2455example, a fork bomb is likely to exhaust the number of tasks before 2456hitting memory restrictions. 2457 2458Note that PIDs used in this controller refer to TIDs, process IDs as 2459used by the kernel. 2460 2461 2462PID Interface Files 2463~~~~~~~~~~~~~~~~~~~ 2464 2465 pids.max 2466 A read-write single value file which exists on non-root 2467 cgroups. The default is "max". 2468 2469 Hard limit of number of processes. 2470 2471 pids.current 2472 A read-only single value file which exists on non-root cgroups. 2473 2474 The number of processes currently in the cgroup and its 2475 descendants. 2476 2477 pids.peak 2478 A read-only single value file which exists on non-root cgroups. 2479 2480 The maximum value that the number of processes in the cgroup and its 2481 descendants has ever reached. 2482 2483 pids.events 2484 A read-only flat-keyed file which exists on non-root cgroups. Unless 2485 specified otherwise, a value change in this file generates a file 2486 modified event. The following entries are defined. 2487 2488 max 2489 The number of times the cgroup's total number of processes hit the pids.max 2490 limit (see also pids_localevents). 2491 2492 pids.events.local 2493 Similar to pids.events but the fields in the file are local 2494 to the cgroup i.e. not hierarchical. The file modified event 2495 generated on this file reflects only the local events. 2496 2497Organisational operations are not blocked by cgroup policies, so it is 2498possible to have pids.current > pids.max. This can be done by either 2499setting the limit to be smaller than pids.current, or attaching enough 2500processes to the cgroup such that pids.current is larger than 2501pids.max. However, it is not possible to violate a cgroup PID policy 2502through fork() or clone(). These will return -EAGAIN if the creation 2503of a new process would cause a cgroup policy to be violated. 2504 2505 2506Cpuset 2507------ 2508 2509The "cpuset" controller provides a mechanism for constraining 2510the CPU and memory node placement of tasks to only the resources 2511specified in the cpuset interface files in a task's current cgroup. 2512This is especially valuable on large NUMA systems where placing jobs 2513on properly sized subsets of the systems with careful processor and 2514memory placement to reduce cross-node memory access and contention 2515can improve overall system performance. 2516 2517The "cpuset" controller is hierarchical. That means the controller 2518cannot use CPUs or memory nodes not allowed in its parent. 2519 2520 2521Cpuset Interface Files 2522~~~~~~~~~~~~~~~~~~~~~~ 2523 2524 cpuset.cpus 2525 A read-write multiple values file which exists on non-root 2526 cpuset-enabled cgroups. 2527 2528 It lists the requested CPUs to be used by tasks within this 2529 cgroup. The actual list of CPUs to be granted, however, is 2530 subjected to constraints imposed by its parent and can differ 2531 from the requested CPUs. 2532 2533 The CPU numbers are comma-separated numbers or ranges. 2534 For example:: 2535 2536 # cat cpuset.cpus 2537 0-4,6,8-10 2538 2539 An empty value indicates that the cgroup is using the same 2540 setting as the nearest cgroup ancestor with a non-empty 2541 "cpuset.cpus" or all the available CPUs if none is found. 2542 2543 The value of "cpuset.cpus" stays constant until the next update 2544 and won't be affected by any CPU hotplug events. 2545 2546 cpuset.cpus.effective 2547 A read-only multiple values file which exists on all 2548 cpuset-enabled cgroups. 2549 2550 It lists the active CPUs available to tasks in this cgroup. 2551 2552 For a cgroup that is not a valid partition root, an empty 2553 "cpuset.cpus" makes "cpuset.cpus.effective" show all CPUs 2554 available from the parent cgroup. Otherwise, it is a subset 2555 of "cpuset.cpus" unless none of the requested CPUs can be 2556 granted. In that case, it is treated like an empty 2557 "cpuset.cpus". 2558 2559 For example:: 2560 2561 P (cpuset.cpus.effective=0-3) 2562 | 2563 +-- C0 (member): cpuset.cpus=1-2 2564 | cpuset.cpus.effective=1-2 2565 +-- C1 (invalid root): cpuset.cpus=4-5 2566 | cpuset.cpus.effective=0-3 2567 \-- C2 (member): cpuset.cpus=<empty> 2568 cpuset.cpus.effective=0-3 2569 2570 C0 gets CPUs 1-2 because they are available from P. 2571 None of C1's requested CPUs 4-5 are available from P, so C1 uses P's 2572 effective CPUs 0-3 instead. 2573 C2 has an empty "cpuset.cpus" and therefore also uses P's effective 2574 CPUs 0-3. 2575 2576 For a valid non-root partition root, "cpuset.cpus.effective" contains 2577 the active CPUs from "cpuset.cpus.exclusive.effective" except 2578 those granted to valid child partition roots. If 2579 "cpuset.cpus.exclusive" is set to a value different from 2580 "cpuset.cpus", the effective CPUs need not be a subset of 2581 "cpuset.cpus". 2582 2583 For example, if CPUs 2-5 are allocated to P for exclusive use, 2584 CPUs 2-4 are active, CPU 5 is offline, and none of CPUs 2-5 are 2585 assigned to a valid child partition root:: 2586 2587 top (root) 2588 | 2589 \-- P (valid partition root) 2590 cpuset.cpus=0-3 2591 cpuset.cpus.exclusive=2-5 2592 cpuset.cpus.exclusive.effective=2-5 2593 cpuset.cpus.effective=2-4 2594 2595 P has 2-5 in "cpuset.cpus.exclusive.effective", including the 2596 offline CPU 5. CPU 5 is excluded from "cpuset.cpus.effective", 2597 leaving 2-4. The result also shows that "cpuset.cpus.effective" 2598 need not be a subset of "cpuset.cpus", which is 0-3. 2599 2600 Its value will be affected by CPU hotplug events. 2601 2602 cpuset.mems 2603 A read-write multiple values file which exists on non-root 2604 cpuset-enabled cgroups. 2605 2606 It lists the requested memory nodes to be used by tasks within 2607 this cgroup. The actual list of memory nodes granted, however, 2608 is subjected to constraints imposed by its parent and can differ 2609 from the requested memory nodes. 2610 2611 The memory node numbers are comma-separated numbers or ranges. 2612 For example:: 2613 2614 # cat cpuset.mems 2615 0-1,3 2616 2617 An empty value indicates that the cgroup is using the same 2618 setting as the nearest cgroup ancestor with a non-empty 2619 "cpuset.mems" or all the available memory nodes if none 2620 is found. 2621 2622 The value of "cpuset.mems" stays constant until the next update 2623 and won't be affected by any memory nodes hotplug events. 2624 2625 Setting a non-empty value to "cpuset.mems" causes memory of 2626 tasks within the cgroup to be migrated to the designated nodes if 2627 they are currently using memory outside of the designated nodes. 2628 2629 There is a cost for this memory migration. The migration 2630 may not be complete and some memory pages may be left behind. 2631 So it is recommended that "cpuset.mems" should be set properly 2632 before spawning new tasks into the cpuset. Even if there is 2633 a need to change "cpuset.mems" with active tasks, it shouldn't 2634 be done frequently. 2635 2636 For a multithreaded process, the threadgroup leader is 2637 considered the owner of the group's memory. Memory policy 2638 rebinding and migration will only happen with respect to the 2639 threadgroup leader. To avoid unexpected results, non-leading 2640 threads shouldn't be put into another cgroup whose "cpuset.mems" 2641 doesn't fully overlap that of the threadgroup leader. 2642 2643 cpuset.mems.effective 2644 A read-only multiple values file which exists on all 2645 cpuset-enabled cgroups. 2646 2647 It lists the onlined memory nodes that are actually granted to 2648 this cgroup by its parent. These memory nodes are allowed to 2649 be used by tasks within the current cgroup. 2650 2651 If "cpuset.mems" is empty, it shows all the memory nodes from the 2652 parent cgroup that will be available to be used by this cgroup. 2653 Otherwise, it should be a subset of "cpuset.mems" unless none of 2654 the memory nodes listed in "cpuset.mems" can be granted. In this 2655 case, it will be treated just like an empty "cpuset.mems". 2656 2657 Its value will be affected by memory nodes hotplug events. 2658 2659 cpuset.cpus.exclusive 2660 A read-write multiple values file which exists on non-root 2661 cpuset-enabled cgroups. 2662 2663 It lists all the exclusive CPUs that are allowed to be used 2664 to create a new cpuset partition. Its value is not used 2665 unless the cgroup becomes a valid partition root. See the 2666 "cpuset.cpus.partition" section below for a description of what 2667 a cpuset partition is. 2668 2669 When the cgroup becomes a partition root, the actual exclusive 2670 CPUs that are allocated to that partition are listed in 2671 "cpuset.cpus.exclusive.effective" which may be different 2672 from "cpuset.cpus.exclusive". If "cpuset.cpus.exclusive" 2673 has previously been set, "cpuset.cpus.exclusive.effective" 2674 is always a subset of it. 2675 2676 Users can manually set it to a value that is different from 2677 "cpuset.cpus". One constraint in setting it is that the list of 2678 CPUs must be exclusive with respect to "cpuset.cpus.exclusive" 2679 and "cpuset.cpus.exclusive.effective" of its siblings. Another 2680 constraint is that it cannot be a superset of "cpuset.cpus" 2681 of its sibling in order to leave at least one CPU available to 2682 that sibling when the exclusive CPUs are taken away. 2683 2684 For a parent cgroup, any one of its exclusive CPUs can only 2685 be distributed to at most one of its child cgroups. Having an 2686 exclusive CPU appearing in two or more of its child cgroups is 2687 not allowed (the exclusivity rule). A value that violates the 2688 exclusivity rule will be rejected with a write error. 2689 2690 The root cgroup is a partition root and all its available CPUs 2691 are in its exclusive CPU set. 2692 2693 cpuset.cpus.exclusive.effective 2694 A read-only multiple values file which exists on all non-root 2695 cpuset-enabled cgroups. 2696 2697 This file shows the effective set of exclusive CPUs that 2698 can be used to create a partition root. The content 2699 of this file will always be a subset of its parent's 2700 "cpuset.cpus.exclusive.effective" if its parent is not the root 2701 cgroup. It will also be a subset of "cpuset.cpus.exclusive" 2702 if it is set. This file should only be non-empty if either 2703 "cpuset.cpus.exclusive" is set or when the current cpuset is 2704 a valid partition root. 2705 2706 cpuset.cpus.isolated 2707 A read-only and root cgroup only multiple values file. 2708 2709 This file shows the set of all isolated CPUs used in existing 2710 isolated partitions or isolated at boot time via the "domain" 2711 flag of "isolcpus". The latter remain listed even after any 2712 partition using them is released. 2713 2714 cpuset.cpus.partition 2715 A read-write single value file which exists on non-root 2716 cpuset-enabled cgroups. This flag is owned by the parent cgroup 2717 and is not delegatable. 2718 2719 It accepts only the following input values when written to. 2720 2721 ========== ===================================== 2722 "member" Non-root member of a partition 2723 "root" Partition root 2724 "isolated" Partition root without load balancing 2725 ========== ===================================== 2726 2727 A cpuset partition is a collection of cpuset-enabled cgroups with 2728 a partition root at the top of the hierarchy and its descendants 2729 except those that are separate partition roots themselves and 2730 their descendants. A partition has exclusive access to the 2731 set of exclusive CPUs allocated to it. Other cgroups outside 2732 of that partition cannot use any CPUs in that set. 2733 2734 There are two types of partitions - local and remote. A local 2735 partition is one whose parent cgroup is also a valid partition 2736 root. A remote partition is one whose parent cgroup is not a 2737 valid partition root itself. 2738 2739 Writing to "cpuset.cpus.exclusive" is optional for the creation 2740 of a local partition as its "cpuset.cpus.exclusive" file will 2741 assume an implicit value that is the same as "cpuset.cpus" if it 2742 is not set. Writing the proper "cpuset.cpus.exclusive" values 2743 down the cgroup hierarchy before the target partition root is 2744 mandatory for the creation of a remote partition. 2745 2746 Not all the CPUs requested in "cpuset.cpus.exclusive" can be 2747 used to form a new partition. Only those that were present 2748 in its parent's "cpuset.cpus.exclusive.effective" control 2749 file can be used. For partitions created without setting 2750 "cpuset.cpus.exclusive", exclusive CPUs specified in sibling's 2751 "cpuset.cpus.exclusive" or "cpuset.cpus.exclusive.effective" 2752 also cannot be used. 2753 2754 Currently, a remote partition cannot be created under a local 2755 partition. All the ancestors of a remote partition root except 2756 the root cgroup cannot be a partition root. 2757 2758 The root cgroup is always a partition root and its state cannot 2759 be changed. All other non-root cgroups start out as "member". 2760 Even though the "cpuset.cpus.exclusive*" and "cpuset.cpus" 2761 control files are not present in the root cgroup, they are 2762 implicitly the same as the "/sys/devices/system/cpu/possible" 2763 sysfs file. 2764 2765 When set to "root", the current cgroup is the root of a new 2766 partition or scheduling domain. The set of exclusive CPUs is 2767 determined by the value of its "cpuset.cpus.exclusive.effective". 2768 2769 When set to "isolated", the CPUs in that partition will be in 2770 an isolated state without any load balancing from the scheduler 2771 and excluded from the unbound workqueues. Tasks placed in such 2772 a partition with multiple CPUs should be carefully distributed 2773 and bound to each of the individual CPUs for optimal performance. 2774 2775 A partition root ("root" or "isolated") can be in one of the 2776 two possible states - valid or invalid. An invalid partition 2777 root is in a degraded state where some state information may 2778 be retained, but behaves more like a "member". 2779 2780 All possible state transitions among "member", "root" and 2781 "isolated" are allowed. 2782 2783 On read, the "cpuset.cpus.partition" file can show the following 2784 values. 2785 2786 ============================= ===================================== 2787 "member" Non-root member of a partition 2788 "root" Partition root 2789 "isolated" Partition root without load balancing 2790 "root invalid (<reason>)" Invalid partition root 2791 "isolated invalid (<reason>)" Invalid isolated partition root 2792 ============================= ===================================== 2793 2794 In the case of an invalid partition root, a descriptive string on 2795 why the partition is invalid is included within parentheses. 2796 2797 For a local partition root to be valid, the following conditions 2798 must be met. 2799 2800 1) The parent cgroup is a valid partition root. 2801 2) The "cpuset.cpus.exclusive.effective" file cannot be empty, 2802 though it may contain offline CPUs. 2803 3) The "cpuset.cpus.effective" cannot be empty unless there is 2804 no task associated with this partition. 2805 2806 For a remote partition root to be valid, all the above conditions 2807 except the first one must be met. 2808 2809 External events like hotplug or changes to "cpuset.cpus" or 2810 "cpuset.cpus.exclusive" can cause a valid partition root to 2811 become invalid and vice versa. Note that a task cannot be 2812 moved to a cgroup with empty "cpuset.cpus.effective". 2813 2814 A valid non-root parent partition may distribute out all its CPUs 2815 to its child local partitions when there is no task associated 2816 with it. 2817 2818 Care must be taken to change a valid partition root to "member" 2819 as all its child local partitions, if present, will become 2820 invalid causing disruption to tasks running in those child 2821 partitions. These inactivated partitions could be recovered if 2822 their parent is switched back to a partition root with a proper 2823 value in "cpuset.cpus" or "cpuset.cpus.exclusive". 2824 2825 Poll and inotify events are triggered whenever the state of 2826 "cpuset.cpus.partition" changes. That includes changes caused 2827 by write to "cpuset.cpus.partition", cpu hotplug or other 2828 changes that modify the validity status of the partition. 2829 This will allow user space agents to monitor unexpected changes 2830 to "cpuset.cpus.partition" without the need to do continuous 2831 polling. 2832 2833 A user can pre-configure certain CPUs to an isolated state 2834 with load balancing disabled at boot time with the "isolcpus" 2835 kernel boot command line option. If those CPUs are to be put 2836 into a partition, they have to be used in an isolated partition. 2837 2838 2839Device controller 2840----------------- 2841 2842Device controller manages access to device files. It includes both 2843creation of new device files (using mknod), and access to the 2844existing device files. 2845 2846Cgroup v2 device controller has no interface files and is implemented 2847on top of cgroup BPF. To control access to device files, a user may 2848create bpf programs of type BPF_PROG_TYPE_CGROUP_DEVICE and attach 2849them to cgroups with BPF_CGROUP_DEVICE flag. On an attempt to access a 2850device file, corresponding BPF programs will be executed, and depending 2851on the return value the attempt will succeed or fail with -EPERM. 2852 2853A BPF_PROG_TYPE_CGROUP_DEVICE program takes a pointer to the 2854bpf_cgroup_dev_ctx structure, which describes the device access attempt: 2855access type (mknod/read/write) and device (type, major and minor numbers). 2856If the program returns 0, the attempt fails with -EPERM, otherwise it 2857succeeds. 2858 2859An example of BPF_PROG_TYPE_CGROUP_DEVICE program may be found in 2860tools/testing/selftests/bpf/progs/dev_cgroup.c in the kernel source tree. 2861 2862 2863RDMA 2864---- 2865 2866The "rdma" controller regulates the distribution and accounting of 2867RDMA resources. 2868 2869RDMA devices from all network namespaces are listed. Each line starts with 2870the device name. If more than one device has the same name, ``index=N`` 2871follows the name, where ``N`` is the system-wide RDMA device index, unique 2872among registered devices. 2873 2874RDMA Interface Files 2875~~~~~~~~~~~~~~~~~~~~ 2876 2877 rdma.max 2878 A readwrite nested-keyed file that exists for all the cgroups 2879 except root that describes current configured resource limit 2880 for a RDMA/IB device. 2881 2882 Lines are keyed by device name and are not ordered. A write may 2883 include ``index=N`` after the device name. The index is optional 2884 when the name is globally unique. If multiple devices have that 2885 name, the index is required and a write without it fails with 2886 ``-ENOTUNIQ``. 2887 Each line contains space separated resource name and its configured 2888 limit that can be distributed. 2889 2890 The following nested keys are defined. 2891 2892 ========== ============================= 2893 hca_handle Maximum number of HCA Handles 2894 hca_object Maximum number of HCA Objects 2895 ========== ============================= 2896 2897 An example for mlx4 and ocrdma device follows:: 2898 2899 mlx4_0 hca_handle=2 hca_object=2000 2900 ocrdma1 hca_handle=3 hca_object=max 2901 2902 For devices with duplicate names, select the device by index:: 2903 2904 echo "rxe0 index=5 hca_handle=2" > rdma.max 2905 2906 rdma.current 2907 A read-only file that describes current resource usage. 2908 It exists for all the cgroup except root. 2909 2910 An example for mlx4 and ocrdma device follows:: 2911 2912 mlx4_0 hca_handle=1 hca_object=20 2913 ocrdma1 hca_handle=1 hca_object=23 2914 2915 rdma.peak 2916 A read-only nested-keyed file that exists for all the cgroups 2917 except root. It shows the historical high watermark of 2918 resource usage per device since the cgroup was created. 2919 2920 An example for mlx4 and ocrdma device follows:: 2921 2922 mlx4_0 hca_handle=1 hca_object=20 2923 ocrdma1 hca_handle=0 hca_object=23 2924 2925 rdma.events 2926 A read-only nested-keyed file which exists on non-root 2927 cgroups. The following nested keys are defined. 2928 2929 max 2930 The number of times a process in this cgroup or its 2931 descendants attempted an RDMA resource allocation that 2932 was rejected because a rdma.max limit in the subtree 2933 was reached. This is a hierarchical counter: the event 2934 is propagated upward to all ancestor cgroups. A value 2935 change in this file generates a file modified event. 2936 2937 alloc_fail 2938 The number of RDMA resource allocation attempts that 2939 originated in this cgroup or its descendants and failed 2940 due to a rdma.max limit being reached. This is a 2941 hierarchical counter propagated upward. 2942 2943 An example for mlx4 device follows:: 2944 2945 mlx4_0 hca_handle.max=5 hca_handle.alloc_fail=3 hca_object.max=0 hca_object.alloc_fail=0 2946 2947 rdma.events.local 2948 Similar to rdma.events but the fields in the file are local 2949 to the cgroup i.e. not hierarchical. The file modified event 2950 generated on this file reflects only the local events. 2951 2952 The following nested keys are defined. 2953 2954 max 2955 The number of times a process in this cgroup or its 2956 descendants attempted an RDMA resource allocation that 2957 was rejected because this cgroup's own rdma.max limit 2958 was reached. 2959 alloc_fail 2960 The number of RDMA resource allocation attempts 2961 originating from this cgroup that failed due to this 2962 cgroup's or an ancestor's rdma.max limit. 2963 2964 An example for mlx4 device follows:: 2965 2966 mlx4_0 hca_handle.max=5 hca_handle.alloc_fail=0 hca_object.max=0 hca_object.alloc_fail=0 2967 2968DMEM 2969---- 2970 2971The "dmem" controller regulates the distribution and accounting of 2972device memory regions. Because each memory region may have its own page size, 2973which does not have to be equal to the system page size, the units are always bytes. 2974 2975DMEM Interface Files 2976~~~~~~~~~~~~~~~~~~~~ 2977 2978 dmem.max, dmem.min, dmem.low 2979 A readwrite nested-keyed file that exists for all the cgroups 2980 except root that describes current configured resource limit 2981 for a region. 2982 2983 An example for xe follows:: 2984 2985 drm/0000:03:00.0/vram0 1073741824 2986 drm/0000:03:00.0/stolen max 2987 2988 The semantics are the same as for the memory cgroup controller, and are 2989 calculated in the same way. 2990 2991 dmem.peak 2992 A read-only nested-keyed file that exists on non-root cgroups. 2993 2994 The max device memory usage recorded for the cgroup and its 2995 descendants since the creation of the cgroup for each region. 2996 2997 dmem.capacity 2998 A read-only file that describes maximum region capacity. 2999 It only exists on the root cgroup. Not all memory can be 3000 allocated by cgroups, as the kernel reserves some for 3001 internal use. 3002 3003 An example for xe follows:: 3004 3005 drm/0000:03:00.0/vram0 8514437120 3006 drm/0000:03:00.0/stolen 67108864 3007 3008 dmem.current 3009 A read-only file that describes current resource usage. 3010 It exists for all the cgroup except root. 3011 3012 An example for xe follows:: 3013 3014 drm/0000:03:00.0/vram0 12550144 3015 drm/0000:03:00.0/stolen 8650752 3016 3017HugeTLB 3018------- 3019 3020The HugeTLB controller allows limiting the HugeTLB usage per control group and 3021enforces the controller limit during page fault. 3022 3023HugeTLB Interface Files 3024~~~~~~~~~~~~~~~~~~~~~~~ 3025 3026 hugetlb.<hugepagesize>.current 3027 Show current usage for "hugepagesize" hugetlb. It exists for all 3028 the cgroup except root. 3029 3030 hugetlb.<hugepagesize>.max 3031 Set/show the hard limit of "hugepagesize" hugetlb usage. 3032 The default value is "max". It exists for all the cgroup except root. 3033 3034 hugetlb.<hugepagesize>.events 3035 A read-only flat-keyed file which exists on non-root cgroups. 3036 3037 max 3038 The number of allocation failure due to HugeTLB limit 3039 3040 hugetlb.<hugepagesize>.events.local 3041 Similar to hugetlb.<hugepagesize>.events but the fields in the file 3042 are local to the cgroup i.e. not hierarchical. The file modified event 3043 generated on this file reflects only the local events. 3044 3045 hugetlb.<hugepagesize>.numa_stat 3046 Similar to memory.numa_stat, it shows the numa information of the 3047 hugetlb pages of <hugepagesize> in this cgroup. Only active in 3048 use hugetlb pages are included. The per-node values are in bytes. 3049 3050Misc 3051---- 3052 3053The Miscellaneous cgroup provides the resource limiting and tracking 3054mechanism for the scalar resources which cannot be abstracted like the other 3055cgroup resources. Controller is enabled by the CONFIG_CGROUP_MISC config 3056option. 3057 3058A resource can be added to the controller via enum misc_res_type{} in the 3059include/linux/misc_cgroup.h file and the corresponding name via misc_res_name[] 3060in the kernel/cgroup/misc.c file. Provider of the resource must set its 3061capacity prior to using the resource by calling misc_cg_set_capacity(). 3062 3063Once a capacity is set then the resource usage can be updated using charge and 3064uncharge APIs. All of the APIs to interact with misc controller are in 3065include/linux/misc_cgroup.h. 3066 3067Misc Interface Files 3068~~~~~~~~~~~~~~~~~~~~ 3069 3070Miscellaneous controller provides the following interface files. If two misc 3071resources (res_a and res_b) are registered then: 3072 3073 misc.capacity 3074 A read-only flat-keyed file shown only in the root cgroup. It shows 3075 miscellaneous scalar resources available on the platform along with 3076 their quantities:: 3077 3078 $ cat misc.capacity 3079 res_a 50 3080 res_b 10 3081 3082 misc.current 3083 A read-only flat-keyed file shown in the all cgroups. It shows 3084 the current usage of the resources in the cgroup and its children.:: 3085 3086 $ cat misc.current 3087 res_a 3 3088 res_b 0 3089 3090 misc.peak 3091 A read-only flat-keyed file shown in all cgroups. It shows the 3092 historical maximum usage of the resources in the cgroup and its 3093 children.:: 3094 3095 $ cat misc.peak 3096 res_a 10 3097 res_b 8 3098 3099 misc.max 3100 A read-write flat-keyed file shown in the non root cgroups. Allowed 3101 maximum usage of the resources in the cgroup and its children.:: 3102 3103 $ cat misc.max 3104 res_a max 3105 res_b 4 3106 3107 Limit can be set by:: 3108 3109 # echo res_a 1 > misc.max 3110 3111 Limit can be set to max by:: 3112 3113 # echo res_a max > misc.max 3114 3115 Limits can be set higher than the capacity value in the misc.capacity 3116 file. 3117 3118 misc.events 3119 A read-only flat-keyed file which exists on non-root cgroups. The 3120 following entries are defined. Unless specified otherwise, a value 3121 change in this file generates a file modified event. All fields in 3122 this file are hierarchical. 3123 3124 <res>.max 3125 The number of times the cgroup's resource usage was 3126 about to go over the max boundary. 3127 3128 misc.events.local 3129 Similar to misc.events but the fields in the file are local to the 3130 cgroup i.e. not hierarchical. The file modified event generated on 3131 this file reflects only the local events. 3132 3133Migration and Ownership 3134~~~~~~~~~~~~~~~~~~~~~~~ 3135 3136A miscellaneous scalar resource is charged to the cgroup in which it is used 3137first, and stays charged to that cgroup until that resource is freed. Migrating 3138a process to a different cgroup does not move the charge to the destination 3139cgroup where the process has moved. 3140 3141Others 3142------ 3143 3144perf_event 3145~~~~~~~~~~ 3146 3147perf_event controller, if not mounted on a legacy hierarchy, is 3148automatically enabled on the v2 hierarchy so that perf events can 3149always be filtered by cgroup v2 path. The controller can still be 3150moved to a legacy hierarchy after v2 hierarchy is populated. 3151 3152 3153Non-normative information 3154------------------------- 3155 3156This section contains information that isn't considered to be a part of 3157the stable kernel API and so is subject to change. 3158 3159 3160CPU controller root cgroup process behaviour 3161~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 3162 3163When distributing CPU cycles in the root cgroup each thread in this 3164cgroup is treated as if it was hosted in a separate child cgroup of the 3165root cgroup. This child cgroup weight is dependent on its thread nice 3166level. 3167 3168For details of this mapping see sched_prio_to_weight array in 3169kernel/sched/core.c file (values from this array should be scaled 3170appropriately so the neutral - nice 0 - value is 100 instead of 1024). 3171 3172 3173IO controller root cgroup process behaviour 3174~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 3175 3176Root cgroup processes are hosted in an implicit leaf child node. 3177When distributing IO resources this implicit child node is taken into 3178account as if it was a normal child cgroup of the root cgroup with a 3179weight value of 200. 3180 3181 3182Namespace 3183========= 3184 3185Basics 3186------ 3187 3188cgroup namespace provides a mechanism to virtualize the view of the 3189"/proc/$PID/cgroup" file and cgroup mounts. The CLONE_NEWCGROUP clone 3190flag can be used with clone(2) and unshare(2) to create a new cgroup 3191namespace. The process running inside the cgroup namespace will have 3192its "/proc/$PID/cgroup" output restricted to cgroupns root. The 3193cgroupns root is the cgroup of the process at the time of creation of 3194the cgroup namespace. 3195 3196Without cgroup namespace, the "/proc/$PID/cgroup" file shows the 3197complete path of the cgroup of a process. In a container setup where 3198a set of cgroups and namespaces are intended to isolate processes the 3199"/proc/$PID/cgroup" file may leak potential system level information 3200to the isolated processes. For example:: 3201 3202 # cat /proc/self/cgroup 3203 0::/batchjobs/container_id1 3204 3205The path '/batchjobs/container_id1' can be considered as system-data 3206and undesirable to expose to the isolated processes. cgroup namespace 3207can be used to restrict visibility of this path. For example, before 3208creating a cgroup namespace, one would see:: 3209 3210 # ls -l /proc/self/ns/cgroup 3211 lrwxrwxrwx 1 root root 0 2014-07-15 10:37 /proc/self/ns/cgroup -> cgroup:[4026531835] 3212 # cat /proc/self/cgroup 3213 0::/batchjobs/container_id1 3214 3215After unsharing a new namespace, the view changes:: 3216 3217 # ls -l /proc/self/ns/cgroup 3218 lrwxrwxrwx 1 root root 0 2014-07-15 10:35 /proc/self/ns/cgroup -> cgroup:[4026532183] 3219 # cat /proc/self/cgroup 3220 0::/ 3221 3222When some thread from a multi-threaded process unshares its cgroup 3223namespace, the new cgroupns gets applied to the entire process (all 3224the threads). This is natural for the v2 hierarchy; however, for the 3225legacy hierarchies, this may be unexpected. 3226 3227A cgroup namespace is alive as long as there are processes inside or 3228mounts pinning it. When the last usage goes away, the cgroup 3229namespace is destroyed. The cgroupns root and the actual cgroups 3230remain. 3231 3232 3233The Root and Views 3234------------------ 3235 3236The 'cgroupns root' for a cgroup namespace is the cgroup in which the 3237process calling unshare(2) is running. For example, if a process in 3238/batchjobs/container_id1 cgroup calls unshare, cgroup 3239/batchjobs/container_id1 becomes the cgroupns root. For the 3240init_cgroup_ns, this is the real root ('/') cgroup. 3241 3242The cgroupns root cgroup does not change even if the namespace creator 3243process later moves to a different cgroup:: 3244 3245 # ~/unshare -c # unshare cgroupns in some cgroup 3246 # cat /proc/self/cgroup 3247 0::/ 3248 # mkdir sub_cgrp_1 3249 # echo 0 > sub_cgrp_1/cgroup.procs 3250 # cat /proc/self/cgroup 3251 0::/sub_cgrp_1 3252 3253Each process gets its namespace-specific view of "/proc/$PID/cgroup" 3254 3255Processes running inside the cgroup namespace will be able to see 3256cgroup paths (in /proc/self/cgroup) only inside their root cgroup. 3257From within an unshared cgroupns:: 3258 3259 # sleep 100000 & 3260 [1] 7353 3261 # echo 7353 > sub_cgrp_1/cgroup.procs 3262 # cat /proc/7353/cgroup 3263 0::/sub_cgrp_1 3264 3265From the initial cgroup namespace, the real cgroup path will be 3266visible:: 3267 3268 $ cat /proc/7353/cgroup 3269 0::/batchjobs/container_id1/sub_cgrp_1 3270 3271From a sibling cgroup namespace (that is, a namespace rooted at a 3272different cgroup), the cgroup path relative to its own cgroup 3273namespace root will be shown. For instance, if PID 7353's cgroup 3274namespace root is at '/batchjobs/container_id2', then it will see:: 3275 3276 # cat /proc/7353/cgroup 3277 0::/../container_id2/sub_cgrp_1 3278 3279Note that the relative path always starts with '/' to indicate that 3280its relative to the cgroup namespace root of the caller. 3281 3282 3283Migration and setns(2) 3284---------------------- 3285 3286Processes inside a cgroup namespace can move into and out of the 3287namespace root if they have proper access to external cgroups. For 3288example, from inside a namespace with cgroupns root at 3289/batchjobs/container_id1, and assuming that the global hierarchy is 3290still accessible inside cgroupns:: 3291 3292 # cat /proc/7353/cgroup 3293 0::/sub_cgrp_1 3294 # echo 7353 > batchjobs/container_id2/cgroup.procs 3295 # cat /proc/7353/cgroup 3296 0::/../container_id2 3297 3298Note that this kind of setup is not encouraged. A task inside cgroup 3299namespace should only be exposed to its own cgroupns hierarchy. 3300 3301setns(2) to another cgroup namespace is allowed when: 3302 3303(a) the process has CAP_SYS_ADMIN against its current user namespace 3304(b) the process has CAP_SYS_ADMIN against the target cgroup 3305 namespace's userns 3306 3307No implicit cgroup changes happen with attaching to another cgroup 3308namespace. It is expected that the someone moves the attaching 3309process under the target cgroup namespace root. 3310 3311 3312Interaction with Other Namespaces 3313--------------------------------- 3314 3315Namespace specific cgroup hierarchy can be mounted by a process 3316running inside a non-init cgroup namespace:: 3317 3318 # mount -t cgroup2 none $MOUNT_POINT 3319 3320This will mount the unified cgroup hierarchy with cgroupns root as the 3321filesystem root. The process needs CAP_SYS_ADMIN against its user and 3322mount namespaces. 3323 3324The virtualization of /proc/self/cgroup file combined with restricting 3325the view of cgroup hierarchy by namespace-private cgroupfs mount 3326provides a properly isolated cgroup view inside the container. 3327 3328 3329Information on Kernel Programming 3330================================= 3331 3332This section contains kernel programming information in the areas 3333where interacting with cgroup is necessary. cgroup core and 3334controllers are not covered. 3335 3336 3337Filesystem Support for Writeback 3338-------------------------------- 3339 3340A filesystem can support cgroup writeback by updating 3341address_space_operations->writepages() to annotate bio's using the 3342following two functions. 3343 3344 wbc_init_bio(@wbc, @bio) 3345 Should be called for each bio carrying writeback data and 3346 associates the bio with the inode's owner cgroup and the 3347 corresponding request queue. This must be called after 3348 a queue (device) has been associated with the bio and 3349 before submission. 3350 3351 wbc_account_cgroup_owner(@wbc, @folio, @bytes) 3352 Should be called for each data segment being written out. 3353 While this function doesn't care exactly when it's called 3354 during the writeback session, it's the easiest and most 3355 natural to call it as data segments are added to a bio. 3356 3357With writeback bio's annotated, cgroup support can be enabled per 3358super_block by setting SB_I_CGROUPWB in ->s_iflags. This allows for 3359selective disabling of cgroup writeback support which is helpful when 3360certain filesystem features, e.g. journaled data mode, are 3361incompatible. 3362 3363wbc_init_bio() binds the specified bio to its cgroup. Depending on 3364the configuration, the bio may be executed at a lower priority and if 3365the writeback session is holding shared resources, e.g. a journal 3366entry, may lead to priority inversion. There is no one easy solution 3367for the problem. Filesystems can try to work around specific problem 3368cases by skipping wbc_init_bio() and using bio_associate_blkg() 3369directly. 3370 3371 3372Deprecated v1 Core Features 3373=========================== 3374 3375- Multiple hierarchies including named ones are not supported. 3376 3377- All v1 mount options are not supported. 3378 3379- The "tasks" file is removed and "cgroup.procs" is not sorted. 3380 3381- "cgroup.clone_children" is removed. 3382 3383- /proc/cgroups is meaningless for v2. Use "cgroup.controllers" or 3384 "cgroup.stat" files at the root instead. 3385 3386 3387Issues with v1 and Rationales for v2 3388==================================== 3389 3390Multiple Hierarchies 3391-------------------- 3392 3393cgroup v1 allowed an arbitrary number of hierarchies and each 3394hierarchy could host any number of controllers. While this seemed to 3395provide a high level of flexibility, it wasn't useful in practice. 3396 3397For example, as there is only one instance of each controller, utility 3398type controllers such as freezer which can be useful in all 3399hierarchies could only be used in one. The issue is exacerbated by 3400the fact that controllers couldn't be moved to another hierarchy once 3401hierarchies were populated. Another issue was that all controllers 3402bound to a hierarchy were forced to have exactly the same view of the 3403hierarchy. It wasn't possible to vary the granularity depending on 3404the specific controller. 3405 3406In practice, these issues heavily limited which controllers could be 3407put on the same hierarchy and most configurations resorted to putting 3408each controller on its own hierarchy. Only closely related ones, such 3409as the cpu and cpuacct controllers, made sense to be put on the same 3410hierarchy. This often meant that userland ended up managing multiple 3411similar hierarchies repeating the same steps on each hierarchy 3412whenever a hierarchy management operation was necessary. 3413 3414Furthermore, support for multiple hierarchies came at a steep cost. 3415It greatly complicated cgroup core implementation but more importantly 3416the support for multiple hierarchies restricted how cgroup could be 3417used in general and what controllers was able to do. 3418 3419There was no limit on how many hierarchies there might be, which meant 3420that a thread's cgroup membership couldn't be described in finite 3421length. The key might contain any number of entries and was unlimited 3422in length, which made it highly awkward to manipulate and led to 3423addition of controllers which existed only to identify membership, 3424which in turn exacerbated the original problem of proliferating number 3425of hierarchies. 3426 3427Also, as a controller couldn't have any expectation regarding the 3428topologies of hierarchies other controllers might be on, each 3429controller had to assume that all other controllers were attached to 3430completely orthogonal hierarchies. This made it impossible, or at 3431least very cumbersome, for controllers to cooperate with each other. 3432 3433In most use cases, putting controllers on hierarchies which are 3434completely orthogonal to each other isn't necessary. What usually is 3435called for is the ability to have differing levels of granularity 3436depending on the specific controller. In other words, hierarchy may 3437be collapsed from leaf towards root when viewed from specific 3438controllers. For example, a given configuration might not care about 3439how memory is distributed beyond a certain level while still wanting 3440to control how CPU cycles are distributed. 3441 3442 3443Thread Granularity 3444------------------ 3445 3446cgroup v1 allowed threads of a process to belong to different cgroups. 3447This didn't make sense for some controllers and those controllers 3448ended up implementing different ways to ignore such situations but 3449much more importantly it blurred the line between API exposed to 3450individual applications and system management interface. 3451 3452Generally, in-process knowledge is available only to the process 3453itself; thus, unlike service-level organization of processes, 3454categorizing threads of a process requires active participation from 3455the application which owns the target process. 3456 3457cgroup v1 had an ambiguously defined delegation model which got abused 3458in combination with thread granularity. cgroups were delegated to 3459individual applications so that they can create and manage their own 3460sub-hierarchies and control resource distributions along them. This 3461effectively raised cgroup to the status of a syscall-like API exposed 3462to lay programs. 3463 3464First of all, cgroup has a fundamentally inadequate interface to be 3465exposed this way. For a process to access its own knobs, it has to 3466extract the path on the target hierarchy from /proc/self/cgroup, 3467construct the path by appending the name of the knob to the path, open 3468and then read and/or write to it. This is not only extremely clunky 3469and unusual but also inherently racy. There is no conventional way to 3470define transaction across the required steps and nothing can guarantee 3471that the process would actually be operating on its own sub-hierarchy. 3472 3473cgroup controllers implemented a number of knobs which would never be 3474accepted as public APIs because they were just adding control knobs to 3475system-management pseudo filesystem. cgroup ended up with interface 3476knobs which were not properly abstracted or refined and directly 3477revealed kernel internal details. These knobs got exposed to 3478individual applications through the ill-defined delegation mechanism 3479effectively abusing cgroup as a shortcut to implementing public APIs 3480without going through the required scrutiny. 3481 3482This was painful for both userland and kernel. Userland ended up with 3483misbehaving and poorly abstracted interfaces and kernel exposing and 3484locked into constructs inadvertently. 3485 3486 3487Competition Between Inner Nodes and Threads 3488------------------------------------------- 3489 3490cgroup v1 allowed threads to be in any cgroups which created an 3491interesting problem where threads belonging to a parent cgroup and its 3492children cgroups competed for resources. This was nasty as two 3493different types of entities competed and there was no obvious way to 3494settle it. Different controllers did different things. 3495 3496The cpu controller considered threads and cgroups as equivalents and 3497mapped nice levels to cgroup weights. This worked for some cases but 3498fell flat when children wanted to be allocated specific ratios of CPU 3499cycles and the number of internal threads fluctuated - the ratios 3500constantly changed as the number of competing entities fluctuated. 3501There also were other issues. The mapping from nice level to weight 3502wasn't obvious or universal, and there were various other knobs which 3503simply weren't available for threads. 3504 3505The io controller implicitly created a hidden leaf node for each 3506cgroup to host the threads. The hidden leaf had its own copies of all 3507the knobs with ``leaf_`` prefixed. While this allowed equivalent 3508control over internal threads, it was with serious drawbacks. It 3509always added an extra layer of nesting which wouldn't be necessary 3510otherwise, made the interface messy and significantly complicated the 3511implementation. 3512 3513The memory controller didn't have a way to control what happened 3514between internal tasks and child cgroups and the behavior was not 3515clearly defined. There were attempts to add ad-hoc behaviors and 3516knobs to tailor the behavior to specific workloads which would have 3517led to problems extremely difficult to resolve in the long term. 3518 3519Multiple controllers struggled with internal tasks and came up with 3520different ways to deal with it; unfortunately, all the approaches were 3521severely flawed and, furthermore, the widely different behaviors 3522made cgroup as a whole highly inconsistent. 3523 3524This clearly is a problem which needs to be addressed from cgroup core 3525in a uniform way. 3526 3527 3528Other Interface Issues 3529---------------------- 3530 3531cgroup v1 grew without oversight and developed a large number of 3532idiosyncrasies and inconsistencies. One issue on the cgroup core side 3533was how an empty cgroup was notified - a userland helper binary was 3534forked and executed for each event. The event delivery wasn't 3535recursive or delegatable. The limitations of the mechanism also led 3536to in-kernel event delivery filtering mechanism further complicating 3537the interface. 3538 3539Controller interfaces were problematic too. An extreme example is 3540controllers completely ignoring hierarchical organization and treating 3541all cgroups as if they were all located directly under the root 3542cgroup. Some controllers exposed a large amount of inconsistent 3543implementation details to userland. 3544 3545There also was no consistency across controllers. When a new cgroup 3546was created, some controllers defaulted to not imposing extra 3547restrictions while others disallowed any resource usage until 3548explicitly configured. Configuration knobs for the same type of 3549control used widely differing naming schemes and formats. Statistics 3550and information knobs were named arbitrarily and used different 3551formats and units even in the same controller. 3552 3553cgroup v2 establishes common conventions where appropriate and updates 3554controllers so that they expose minimal and consistent interfaces. 3555 3556 3557Controller Issues and Remedies 3558------------------------------ 3559 3560Memory 3561~~~~~~ 3562 3563The original lower boundary, the soft limit, is defined as a limit 3564that is per default unset. As a result, the set of cgroups that 3565global reclaim prefers is opt-in, rather than opt-out. The costs for 3566optimizing these mostly negative lookups are so high that the 3567implementation, despite its enormous size, does not even provide the 3568basic desirable behavior. First off, the soft limit has no 3569hierarchical meaning. All configured groups are organized in a global 3570rbtree and treated like equal peers, regardless where they are located 3571in the hierarchy. This makes subtree delegation impossible. Second, 3572the soft limit reclaim pass is so aggressive that it not just 3573introduces high allocation latencies into the system, but also impacts 3574system performance due to overreclaim, to the point where the feature 3575becomes self-defeating. 3576 3577The memory.low boundary on the other hand is a top-down allocated 3578reserve. A cgroup enjoys reclaim protection when it's within its 3579effective low, which makes delegation of subtrees possible. It also 3580enjoys having reclaim pressure proportional to its overage when 3581above its effective low. 3582 3583The original high boundary, the hard limit, is defined as a strict 3584limit that can not budge, even if the OOM killer has to be called. 3585But this generally goes against the goal of making the most out of the 3586available memory. The memory consumption of workloads varies during 3587runtime, and that requires users to overcommit. But doing that with a 3588strict upper limit requires either a fairly accurate prediction of the 3589working set size or adding slack to the limit. Since working set size 3590estimation is hard and error prone, and getting it wrong results in 3591OOM kills, most users tend to err on the side of a looser limit and 3592end up wasting precious resources. 3593 3594The memory.high boundary on the other hand can be set much more 3595conservatively. When hit, it throttles allocations by forcing them 3596into direct reclaim to work off the excess, but it never invokes the 3597OOM killer. As a result, a high boundary that is chosen too 3598aggressively will not terminate the processes, but instead it will 3599lead to gradual performance degradation. The user can monitor this 3600and make corrections until the minimal memory footprint that still 3601gives acceptable performance is found. 3602 3603In extreme cases, with many concurrent allocations and a complete 3604breakdown of reclaim progress within the group, the high boundary can 3605be exceeded. But even then it's mostly better to satisfy the 3606allocation from the slack available in other groups or the rest of the 3607system than killing the group. Otherwise, memory.max is there to 3608limit this type of spillover and ultimately contain buggy or even 3609malicious applications. 3610 3611Setting the original memory.limit_in_bytes below the current usage was 3612subject to a race condition, where concurrent charges could cause the 3613limit setting to fail. memory.max on the other hand will first set the 3614limit to prevent new charges, and then reclaim and OOM kill until the 3615new limit is met - or the task writing to memory.max is killed. 3616 3617The combined memory+swap accounting and limiting is replaced by real 3618control over swap space. 3619 3620The main argument for a combined memory+swap facility in the original 3621cgroup design was that global or parental pressure would always be 3622able to swap all anonymous memory of a child group, regardless of the 3623child's own (possibly untrusted) configuration. However, untrusted 3624groups can sabotage swapping by other means - such as referencing its 3625anonymous memory in a tight loop - and an admin can not assume full 3626swappability when overcommitting untrusted jobs. 3627 3628For trusted jobs, on the other hand, a combined counter is not an 3629intuitive userspace interface, and it flies in the face of the idea 3630that cgroup controllers should account and limit specific physical 3631resources. Swap space is a resource like all others in the system, 3632and that's why unified hierarchy allows distributing it separately. 3633