1================ 2Control Group v2 3================ 4 5:Date: October, 2015 6:Author: Tejun Heo <tj@kernel.org> 7 8This is the authoritative documentation on the design, interface and 9conventions of cgroup v2. It describes all userland-visible aspects 10of cgroup including core and specific controller behaviors. All 11future changes must be reflected in this document. Documentation for 12v1 is available under :ref:`Documentation/admin-guide/cgroup-v1/index.rst <cgroup-v1>`. 13 14.. CONTENTS 15 16 1. Introduction 17 1-1. Terminology 18 1-2. What is cgroup? 19 2. Basic Operations 20 2-1. Mounting 21 2-2. Organizing Processes and Threads 22 2-2-1. Processes 23 2-2-2. Threads 24 2-3. [Un]populated Notification 25 2-4. Controlling Controllers 26 2-4-1. Enabling and Disabling 27 2-4-2. Top-down Constraint 28 2-4-3. No Internal Process Constraint 29 2-5. Delegation 30 2-5-1. Model of Delegation 31 2-5-2. Delegation Containment 32 2-6. Guidelines 33 2-6-1. Organize Once and Control 34 2-6-2. Avoid Name Collisions 35 3. Resource Distribution Models 36 3-1. Weights 37 3-2. Limits 38 3-3. Protections 39 3-4. Allocations 40 4. Interface Files 41 4-1. Format 42 4-2. Conventions 43 4-3. Core Interface Files 44 5. Controllers 45 5-1. CPU 46 5-1-1. CPU Interface Files 47 5-2. Memory 48 5-2-1. Memory Interface Files 49 5-2-2. Usage Guidelines 50 5-2-3. Memory Ownership 51 5-3. IO 52 5-3-1. IO Interface Files 53 5-3-2. Writeback 54 5-3-3. IO Latency 55 5-3-3-1. How IO Latency Throttling Works 56 5-3-3-2. IO Latency Interface Files 57 5-4. PID 58 5-4-1. PID Interface Files 59 5-5. Cpuset 60 5.5-1. Cpuset Interface Files 61 5-6. Device 62 5-7. RDMA 63 5-7-1. RDMA Interface Files 64 5-8. HugeTLB 65 5.8-1. HugeTLB Interface Files 66 5-8. Misc 67 5-8-1. perf_event 68 5-N. Non-normative information 69 5-N-1. CPU controller root cgroup process behaviour 70 5-N-2. IO controller root cgroup process behaviour 71 6. Namespace 72 6-1. Basics 73 6-2. The Root and Views 74 6-3. Migration and setns(2) 75 6-4. Interaction with Other Namespaces 76 P. Information on Kernel Programming 77 P-1. Filesystem Support for Writeback 78 D. Deprecated v1 Core Features 79 R. Issues with v1 and Rationales for v2 80 R-1. Multiple Hierarchies 81 R-2. Thread Granularity 82 R-3. Competition Between Inner Nodes and Threads 83 R-4. Other Interface Issues 84 R-5. Controller Issues and Remedies 85 R-5-1. Memory 86 87 88Introduction 89============ 90 91Terminology 92----------- 93 94"cgroup" stands for "control group" and is never capitalized. The 95singular form is used to designate the whole feature and also as a 96qualifier as in "cgroup controllers". When explicitly referring to 97multiple individual control groups, the plural form "cgroups" is used. 98 99 100What is cgroup? 101--------------- 102 103cgroup is a mechanism to organize processes hierarchically and 104distribute system resources along the hierarchy in a controlled and 105configurable manner. 106 107cgroup is largely composed of two parts - the core and controllers. 108cgroup core is primarily responsible for hierarchically organizing 109processes. A cgroup controller is usually responsible for 110distributing a specific type of system resource along the hierarchy 111although there are utility controllers which serve purposes other than 112resource distribution. 113 114cgroups form a tree structure and every process in the system belongs 115to one and only one cgroup. All threads of a process belong to the 116same cgroup. On creation, all processes are put in the cgroup that 117the parent process belongs to at the time. A process can be migrated 118to another cgroup. Migration of a process doesn't affect already 119existing descendant processes. 120 121Following certain structural constraints, controllers may be enabled or 122disabled selectively on a cgroup. All controller behaviors are 123hierarchical - if a controller is enabled on a cgroup, it affects all 124processes which belong to the cgroups consisting the inclusive 125sub-hierarchy of the cgroup. When a controller is enabled on a nested 126cgroup, it always restricts the resource distribution further. The 127restrictions set closer to the root in the hierarchy can not be 128overridden from further away. 129 130 131Basic Operations 132================ 133 134Mounting 135-------- 136 137Unlike v1, cgroup v2 has only single hierarchy. The cgroup v2 138hierarchy can be mounted with the following mount command:: 139 140 # mount -t cgroup2 none $MOUNT_POINT 141 142cgroup2 filesystem has the magic number 0x63677270 ("cgrp"). All 143controllers which support v2 and are not bound to a v1 hierarchy are 144automatically bound to the v2 hierarchy and show up at the root. 145Controllers which are not in active use in the v2 hierarchy can be 146bound to other hierarchies. This allows mixing v2 hierarchy with the 147legacy v1 multiple hierarchies in a fully backward compatible way. 148 149A controller can be moved across hierarchies only after the controller 150is no longer referenced in its current hierarchy. Because per-cgroup 151controller states are destroyed asynchronously and controllers may 152have lingering references, a controller may not show up immediately on 153the v2 hierarchy after the final umount of the previous hierarchy. 154Similarly, a controller should be fully disabled to be moved out of 155the unified hierarchy and it may take some time for the disabled 156controller to become available for other hierarchies; furthermore, due 157to inter-controller dependencies, other controllers may need to be 158disabled too. 159 160While useful for development and manual configurations, moving 161controllers dynamically between the v2 and other hierarchies is 162strongly discouraged for production use. It is recommended to decide 163the hierarchies and controller associations before starting using the 164controllers after system boot. 165 166During transition to v2, system management software might still 167automount the v1 cgroup filesystem and so hijack all controllers 168during boot, before manual intervention is possible. To make testing 169and experimenting easier, the kernel parameter cgroup_no_v1= allows 170disabling controllers in v1 and make them always available in v2. 171 172cgroup v2 currently supports the following mount options. 173 174 nsdelegate 175 176 Consider cgroup namespaces as delegation boundaries. This 177 option is system wide and can only be set on mount or modified 178 through remount from the init namespace. The mount option is 179 ignored on non-init namespace mounts. Please refer to the 180 Delegation section for details. 181 182 memory_localevents 183 184 Only populate memory.events with data for the current cgroup, 185 and not any subtrees. This is legacy behaviour, the default 186 behaviour without this option is to include subtree counts. 187 This option is system wide and can only be set on mount or 188 modified through remount from the init namespace. The mount 189 option is ignored on non-init namespace mounts. 190 191 memory_recursiveprot 192 193 Recursively apply memory.min and memory.low protection to 194 entire subtrees, without requiring explicit downward 195 propagation into leaf cgroups. This allows protecting entire 196 subtrees from one another, while retaining free competition 197 within those subtrees. This should have been the default 198 behavior but is a mount-option to avoid regressing setups 199 relying on the original semantics (e.g. specifying bogusly 200 high 'bypass' protection values at higher tree levels). 201 202 203Organizing Processes and Threads 204-------------------------------- 205 206Processes 207~~~~~~~~~ 208 209Initially, only the root cgroup exists to which all processes belong. 210A child cgroup can be created by creating a sub-directory:: 211 212 # mkdir $CGROUP_NAME 213 214A given cgroup may have multiple child cgroups forming a tree 215structure. Each cgroup has a read-writable interface file 216"cgroup.procs". When read, it lists the PIDs of all processes which 217belong to the cgroup one-per-line. The PIDs are not ordered and the 218same PID may show up more than once if the process got moved to 219another cgroup and then back or the PID got recycled while reading. 220 221A process can be migrated into a cgroup by writing its PID to the 222target cgroup's "cgroup.procs" file. Only one process can be migrated 223on a single write(2) call. If a process is composed of multiple 224threads, writing the PID of any thread migrates all threads of the 225process. 226 227When a process forks a child process, the new process is born into the 228cgroup that the forking process belongs to at the time of the 229operation. After exit, a process stays associated with the cgroup 230that it belonged to at the time of exit until it's reaped; however, a 231zombie process does not appear in "cgroup.procs" and thus can't be 232moved to another cgroup. 233 234A cgroup which doesn't have any children or live processes can be 235destroyed by removing the directory. Note that a cgroup which doesn't 236have any children and is associated only with zombie processes is 237considered empty and can be removed:: 238 239 # rmdir $CGROUP_NAME 240 241"/proc/$PID/cgroup" lists a process's cgroup membership. If legacy 242cgroup is in use in the system, this file may contain multiple lines, 243one for each hierarchy. The entry for cgroup v2 is always in the 244format "0::$PATH":: 245 246 # cat /proc/842/cgroup 247 ... 248 0::/test-cgroup/test-cgroup-nested 249 250If the process becomes a zombie and the cgroup it was associated with 251is removed subsequently, " (deleted)" is appended to the path:: 252 253 # cat /proc/842/cgroup 254 ... 255 0::/test-cgroup/test-cgroup-nested (deleted) 256 257 258Threads 259~~~~~~~ 260 261cgroup v2 supports thread granularity for a subset of controllers to 262support use cases requiring hierarchical resource distribution across 263the threads of a group of processes. By default, all threads of a 264process belong to the same cgroup, which also serves as the resource 265domain to host resource consumptions which are not specific to a 266process or thread. The thread mode allows threads to be spread across 267a subtree while still maintaining the common resource domain for them. 268 269Controllers which support thread mode are called threaded controllers. 270The ones which don't are called domain controllers. 271 272Marking a cgroup threaded makes it join the resource domain of its 273parent as a threaded cgroup. The parent may be another threaded 274cgroup whose resource domain is further up in the hierarchy. The root 275of a threaded subtree, that is, the nearest ancestor which is not 276threaded, is called threaded domain or thread root interchangeably and 277serves as the resource domain for the entire subtree. 278 279Inside a threaded subtree, threads of a process can be put in 280different cgroups and are not subject to the no internal process 281constraint - threaded controllers can be enabled on non-leaf cgroups 282whether they have threads in them or not. 283 284As the threaded domain cgroup hosts all the domain resource 285consumptions of the subtree, it is considered to have internal 286resource consumptions whether there are processes in it or not and 287can't have populated child cgroups which aren't threaded. Because the 288root cgroup is not subject to no internal process constraint, it can 289serve both as a threaded domain and a parent to domain cgroups. 290 291The current operation mode or type of the cgroup is shown in the 292"cgroup.type" file which indicates whether the cgroup is a normal 293domain, a domain which is serving as the domain of a threaded subtree, 294or a threaded cgroup. 295 296On creation, a cgroup is always a domain cgroup and can be made 297threaded by writing "threaded" to the "cgroup.type" file. The 298operation is single direction:: 299 300 # echo threaded > cgroup.type 301 302Once threaded, the cgroup can't be made a domain again. To enable the 303thread mode, the following conditions must be met. 304 305- As the cgroup will join the parent's resource domain. The parent 306 must either be a valid (threaded) domain or a threaded cgroup. 307 308- When the parent is an unthreaded domain, it must not have any domain 309 controllers enabled or populated domain children. The root is 310 exempt from this requirement. 311 312Topology-wise, a cgroup can be in an invalid state. Please consider 313the following topology:: 314 315 A (threaded domain) - B (threaded) - C (domain, just created) 316 317C is created as a domain but isn't connected to a parent which can 318host child domains. C can't be used until it is turned into a 319threaded cgroup. "cgroup.type" file will report "domain (invalid)" in 320these cases. Operations which fail due to invalid topology use 321EOPNOTSUPP as the errno. 322 323A domain cgroup is turned into a threaded domain when one of its child 324cgroup becomes threaded or threaded controllers are enabled in the 325"cgroup.subtree_control" file while there are processes in the cgroup. 326A threaded domain reverts to a normal domain when the conditions 327clear. 328 329When read, "cgroup.threads" contains the list of the thread IDs of all 330threads in the cgroup. Except that the operations are per-thread 331instead of per-process, "cgroup.threads" has the same format and 332behaves the same way as "cgroup.procs". While "cgroup.threads" can be 333written to in any cgroup, as it can only move threads inside the same 334threaded domain, its operations are confined inside each threaded 335subtree. 336 337The threaded domain cgroup serves as the resource domain for the whole 338subtree, and, while the threads can be scattered across the subtree, 339all the processes are considered to be in the threaded domain cgroup. 340"cgroup.procs" in a threaded domain cgroup contains the PIDs of all 341processes in the subtree and is not readable in the subtree proper. 342However, "cgroup.procs" can be written to from anywhere in the subtree 343to migrate all threads of the matching process to the cgroup. 344 345Only threaded controllers can be enabled in a threaded subtree. When 346a threaded controller is enabled inside a threaded subtree, it only 347accounts for and controls resource consumptions associated with the 348threads in the cgroup and its descendants. All consumptions which 349aren't tied to a specific thread belong to the threaded domain cgroup. 350 351Because a threaded subtree is exempt from no internal process 352constraint, a threaded controller must be able to handle competition 353between threads in a non-leaf cgroup and its child cgroups. Each 354threaded controller defines how such competitions are handled. 355 356 357[Un]populated Notification 358-------------------------- 359 360Each non-root cgroup has a "cgroup.events" file which contains 361"populated" field indicating whether the cgroup's sub-hierarchy has 362live processes in it. Its value is 0 if there is no live process in 363the cgroup and its descendants; otherwise, 1. poll and [id]notify 364events are triggered when the value changes. This can be used, for 365example, to start a clean-up operation after all processes of a given 366sub-hierarchy have exited. The populated state updates and 367notifications are recursive. Consider the following sub-hierarchy 368where the numbers in the parentheses represent the numbers of processes 369in each cgroup:: 370 371 A(4) - B(0) - C(1) 372 \ D(0) 373 374A, B and C's "populated" fields would be 1 while D's 0. After the one 375process in C exits, B and C's "populated" fields would flip to "0" and 376file modified events will be generated on the "cgroup.events" files of 377both cgroups. 378 379 380Controlling Controllers 381----------------------- 382 383Enabling and Disabling 384~~~~~~~~~~~~~~~~~~~~~~ 385 386Each cgroup has a "cgroup.controllers" file which lists all 387controllers available for the cgroup to enable:: 388 389 # cat cgroup.controllers 390 cpu io memory 391 392No controller is enabled by default. Controllers can be enabled and 393disabled by writing to the "cgroup.subtree_control" file:: 394 395 # echo "+cpu +memory -io" > cgroup.subtree_control 396 397Only controllers which are listed in "cgroup.controllers" can be 398enabled. When multiple operations are specified as above, either they 399all succeed or fail. If multiple operations on the same controller 400are specified, the last one is effective. 401 402Enabling a controller in a cgroup indicates that the distribution of 403the target resource across its immediate children will be controlled. 404Consider the following sub-hierarchy. The enabled controllers are 405listed in parentheses:: 406 407 A(cpu,memory) - B(memory) - C() 408 \ D() 409 410As A has "cpu" and "memory" enabled, A will control the distribution 411of CPU cycles and memory to its children, in this case, B. As B has 412"memory" enabled but not "CPU", C and D will compete freely on CPU 413cycles but their division of memory available to B will be controlled. 414 415As a controller regulates the distribution of the target resource to 416the cgroup's children, enabling it creates the controller's interface 417files in the child cgroups. In the above example, enabling "cpu" on B 418would create the "cpu." prefixed controller interface files in C and 419D. Likewise, disabling "memory" from B would remove the "memory." 420prefixed controller interface files from C and D. This means that the 421controller interface files - anything which doesn't start with 422"cgroup." are owned by the parent rather than the cgroup itself. 423 424 425Top-down Constraint 426~~~~~~~~~~~~~~~~~~~ 427 428Resources are distributed top-down and a cgroup can further distribute 429a resource only if the resource has been distributed to it from the 430parent. This means that all non-root "cgroup.subtree_control" files 431can only contain controllers which are enabled in the parent's 432"cgroup.subtree_control" file. A controller can be enabled only if 433the parent has the controller enabled and a controller can't be 434disabled if one or more children have it enabled. 435 436 437No Internal Process Constraint 438~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 439 440Non-root cgroups can distribute domain resources to their children 441only when they don't have any processes of their own. In other words, 442only domain cgroups which don't contain any processes can have domain 443controllers enabled in their "cgroup.subtree_control" files. 444 445This guarantees that, when a domain controller is looking at the part 446of the hierarchy which has it enabled, processes are always only on 447the leaves. This rules out situations where child cgroups compete 448against internal processes of the parent. 449 450The root cgroup is exempt from this restriction. Root contains 451processes and anonymous resource consumption which can't be associated 452with any other cgroups and requires special treatment from most 453controllers. How resource consumption in the root cgroup is governed 454is up to each controller (for more information on this topic please 455refer to the Non-normative information section in the Controllers 456chapter). 457 458Note that the restriction doesn't get in the way if there is no 459enabled controller in the cgroup's "cgroup.subtree_control". This is 460important as otherwise it wouldn't be possible to create children of a 461populated cgroup. To control resource distribution of a cgroup, the 462cgroup must create children and transfer all its processes to the 463children before enabling controllers in its "cgroup.subtree_control" 464file. 465 466 467Delegation 468---------- 469 470Model of Delegation 471~~~~~~~~~~~~~~~~~~~ 472 473A cgroup can be delegated in two ways. First, to a less privileged 474user by granting write access of the directory and its "cgroup.procs", 475"cgroup.threads" and "cgroup.subtree_control" files to the user. 476Second, if the "nsdelegate" mount option is set, automatically to a 477cgroup namespace on namespace creation. 478 479Because the resource control interface files in a given directory 480control the distribution of the parent's resources, the delegatee 481shouldn't be allowed to write to them. For the first method, this is 482achieved by not granting access to these files. For the second, the 483kernel rejects writes to all files other than "cgroup.procs" and 484"cgroup.subtree_control" on a namespace root from inside the 485namespace. 486 487The end results are equivalent for both delegation types. Once 488delegated, the user can build sub-hierarchy under the directory, 489organize processes inside it as it sees fit and further distribute the 490resources it received from the parent. The limits and other settings 491of all resource controllers are hierarchical and regardless of what 492happens in the delegated sub-hierarchy, nothing can escape the 493resource restrictions imposed by the parent. 494 495Currently, cgroup doesn't impose any restrictions on the number of 496cgroups in or nesting depth of a delegated sub-hierarchy; however, 497this may be limited explicitly in the future. 498 499 500Delegation Containment 501~~~~~~~~~~~~~~~~~~~~~~ 502 503A delegated sub-hierarchy is contained in the sense that processes 504can't be moved into or out of the sub-hierarchy by the delegatee. 505 506For delegations to a less privileged user, this is achieved by 507requiring the following conditions for a process with a non-root euid 508to migrate a target process into a cgroup by writing its PID to the 509"cgroup.procs" file. 510 511- The writer must have write access to the "cgroup.procs" file. 512 513- The writer must have write access to the "cgroup.procs" file of the 514 common ancestor of the source and destination cgroups. 515 516The above two constraints ensure that while a delegatee may migrate 517processes around freely in the delegated sub-hierarchy it can't pull 518in from or push out to outside the sub-hierarchy. 519 520For an example, let's assume cgroups C0 and C1 have been delegated to 521user U0 who created C00, C01 under C0 and C10 under C1 as follows and 522all processes under C0 and C1 belong to U0:: 523 524 ~~~~~~~~~~~~~ - C0 - C00 525 ~ cgroup ~ \ C01 526 ~ hierarchy ~ 527 ~~~~~~~~~~~~~ - C1 - C10 528 529Let's also say U0 wants to write the PID of a process which is 530currently in C10 into "C00/cgroup.procs". U0 has write access to the 531file; however, the common ancestor of the source cgroup C10 and the 532destination cgroup C00 is above the points of delegation and U0 would 533not have write access to its "cgroup.procs" files and thus the write 534will be denied with -EACCES. 535 536For delegations to namespaces, containment is achieved by requiring 537that both the source and destination cgroups are reachable from the 538namespace of the process which is attempting the migration. If either 539is not reachable, the migration is rejected with -ENOENT. 540 541 542Guidelines 543---------- 544 545Organize Once and Control 546~~~~~~~~~~~~~~~~~~~~~~~~~ 547 548Migrating a process across cgroups is a relatively expensive operation 549and stateful resources such as memory are not moved together with the 550process. This is an explicit design decision as there often exist 551inherent trade-offs between migration and various hot paths in terms 552of synchronization cost. 553 554As such, migrating processes across cgroups frequently as a means to 555apply different resource restrictions is discouraged. A workload 556should be assigned to a cgroup according to the system's logical and 557resource structure once on start-up. Dynamic adjustments to resource 558distribution can be made by changing controller configuration through 559the interface files. 560 561 562Avoid Name Collisions 563~~~~~~~~~~~~~~~~~~~~~ 564 565Interface files for a cgroup and its children cgroups occupy the same 566directory and it is possible to create children cgroups which collide 567with interface files. 568 569All cgroup core interface files are prefixed with "cgroup." and each 570controller's interface files are prefixed with the controller name and 571a dot. A controller's name is composed of lower case alphabets and 572'_'s but never begins with an '_' so it can be used as the prefix 573character for collision avoidance. Also, interface file names won't 574start or end with terms which are often used in categorizing workloads 575such as job, service, slice, unit or workload. 576 577cgroup doesn't do anything to prevent name collisions and it's the 578user's responsibility to avoid them. 579 580 581Resource Distribution Models 582============================ 583 584cgroup controllers implement several resource distribution schemes 585depending on the resource type and expected use cases. This section 586describes major schemes in use along with their expected behaviors. 587 588 589Weights 590------- 591 592A parent's resource is distributed by adding up the weights of all 593active children and giving each the fraction matching the ratio of its 594weight against the sum. As only children which can make use of the 595resource at the moment participate in the distribution, this is 596work-conserving. Due to the dynamic nature, this model is usually 597used for stateless resources. 598 599All weights are in the range [1, 10000] with the default at 100. This 600allows symmetric multiplicative biases in both directions at fine 601enough granularity while staying in the intuitive range. 602 603As long as the weight is in range, all configuration combinations are 604valid and there is no reason to reject configuration changes or 605process migrations. 606 607"cpu.weight" proportionally distributes CPU cycles to active children 608and is an example of this type. 609 610 611Limits 612------ 613 614A child can only consume upto the configured amount of the resource. 615Limits can be over-committed - the sum of the limits of children can 616exceed the amount of resource available to the parent. 617 618Limits are in the range [0, max] and defaults to "max", which is noop. 619 620As limits can be over-committed, all configuration combinations are 621valid and there is no reason to reject configuration changes or 622process migrations. 623 624"io.max" limits the maximum BPS and/or IOPS that a cgroup can consume 625on an IO device and is an example of this type. 626 627 628Protections 629----------- 630 631A cgroup is protected upto the configured amount of the resource 632as long as the usages of all its ancestors are under their 633protected levels. Protections can be hard guarantees or best effort 634soft boundaries. Protections can also be over-committed in which case 635only upto the amount available to the parent is protected among 636children. 637 638Protections are in the range [0, max] and defaults to 0, which is 639noop. 640 641As protections can be over-committed, all configuration combinations 642are valid and there is no reason to reject configuration changes or 643process migrations. 644 645"memory.low" implements best-effort memory protection and is an 646example of this type. 647 648 649Allocations 650----------- 651 652A cgroup is exclusively allocated a certain amount of a finite 653resource. Allocations can't be over-committed - the sum of the 654allocations of children can not exceed the amount of resource 655available to the parent. 656 657Allocations are in the range [0, max] and defaults to 0, which is no 658resource. 659 660As allocations can't be over-committed, some configuration 661combinations are invalid and should be rejected. Also, if the 662resource is mandatory for execution of processes, process migrations 663may be rejected. 664 665"cpu.rt.max" hard-allocates realtime slices and is an example of this 666type. 667 668 669Interface Files 670=============== 671 672Format 673------ 674 675All interface files should be in one of the following formats whenever 676possible:: 677 678 New-line separated values 679 (when only one value can be written at once) 680 681 VAL0\n 682 VAL1\n 683 ... 684 685 Space separated values 686 (when read-only or multiple values can be written at once) 687 688 VAL0 VAL1 ...\n 689 690 Flat keyed 691 692 KEY0 VAL0\n 693 KEY1 VAL1\n 694 ... 695 696 Nested keyed 697 698 KEY0 SUB_KEY0=VAL00 SUB_KEY1=VAL01... 699 KEY1 SUB_KEY0=VAL10 SUB_KEY1=VAL11... 700 ... 701 702For a writable file, the format for writing should generally match 703reading; however, controllers may allow omitting later fields or 704implement restricted shortcuts for most common use cases. 705 706For both flat and nested keyed files, only the values for a single key 707can be written at a time. For nested keyed files, the sub key pairs 708may be specified in any order and not all pairs have to be specified. 709 710 711Conventions 712----------- 713 714- Settings for a single feature should be contained in a single file. 715 716- The root cgroup should be exempt from resource control and thus 717 shouldn't have resource control interface files. 718 719- The default time unit is microseconds. If a different unit is ever 720 used, an explicit unit suffix must be present. 721 722- A parts-per quantity should use a percentage decimal with at least 723 two digit fractional part - e.g. 13.40. 724 725- If a controller implements weight based resource distribution, its 726 interface file should be named "weight" and have the range [1, 727 10000] with 100 as the default. The values are chosen to allow 728 enough and symmetric bias in both directions while keeping it 729 intuitive (the default is 100%). 730 731- If a controller implements an absolute resource guarantee and/or 732 limit, the interface files should be named "min" and "max" 733 respectively. If a controller implements best effort resource 734 guarantee and/or limit, the interface files should be named "low" 735 and "high" respectively. 736 737 In the above four control files, the special token "max" should be 738 used to represent upward infinity for both reading and writing. 739 740- If a setting has a configurable default value and keyed specific 741 overrides, the default entry should be keyed with "default" and 742 appear as the first entry in the file. 743 744 The default value can be updated by writing either "default $VAL" or 745 "$VAL". 746 747 When writing to update a specific override, "default" can be used as 748 the value to indicate removal of the override. Override entries 749 with "default" as the value must not appear when read. 750 751 For example, a setting which is keyed by major:minor device numbers 752 with integer values may look like the following:: 753 754 # cat cgroup-example-interface-file 755 default 150 756 8:0 300 757 758 The default value can be updated by:: 759 760 # echo 125 > cgroup-example-interface-file 761 762 or:: 763 764 # echo "default 125" > cgroup-example-interface-file 765 766 An override can be set by:: 767 768 # echo "8:16 170" > cgroup-example-interface-file 769 770 and cleared by:: 771 772 # echo "8:0 default" > cgroup-example-interface-file 773 # cat cgroup-example-interface-file 774 default 125 775 8:16 170 776 777- For events which are not very high frequency, an interface file 778 "events" should be created which lists event key value pairs. 779 Whenever a notifiable event happens, file modified event should be 780 generated on the file. 781 782 783Core Interface Files 784-------------------- 785 786All cgroup core files are prefixed with "cgroup." 787 788 cgroup.type 789 790 A read-write single value file which exists on non-root 791 cgroups. 792 793 When read, it indicates the current type of the cgroup, which 794 can be one of the following values. 795 796 - "domain" : A normal valid domain cgroup. 797 798 - "domain threaded" : A threaded domain cgroup which is 799 serving as the root of a threaded subtree. 800 801 - "domain invalid" : A cgroup which is in an invalid state. 802 It can't be populated or have controllers enabled. It may 803 be allowed to become a threaded cgroup. 804 805 - "threaded" : A threaded cgroup which is a member of a 806 threaded subtree. 807 808 A cgroup can be turned into a threaded cgroup by writing 809 "threaded" to this file. 810 811 cgroup.procs 812 A read-write new-line separated values file which exists on 813 all cgroups. 814 815 When read, it lists the PIDs of all processes which belong to 816 the cgroup one-per-line. The PIDs are not ordered and the 817 same PID may show up more than once if the process got moved 818 to another cgroup and then back or the PID got recycled while 819 reading. 820 821 A PID can be written to migrate the process associated with 822 the PID to the cgroup. The writer should match all of the 823 following conditions. 824 825 - It must have write access to the "cgroup.procs" file. 826 827 - It must have write access to the "cgroup.procs" file of the 828 common ancestor of the source and destination cgroups. 829 830 When delegating a sub-hierarchy, write access to this file 831 should be granted along with the containing directory. 832 833 In a threaded cgroup, reading this file fails with EOPNOTSUPP 834 as all the processes belong to the thread root. Writing is 835 supported and moves every thread of the process to the cgroup. 836 837 cgroup.threads 838 A read-write new-line separated values file which exists on 839 all cgroups. 840 841 When read, it lists the TIDs of all threads which belong to 842 the cgroup one-per-line. The TIDs are not ordered and the 843 same TID may show up more than once if the thread got moved to 844 another cgroup and then back or the TID got recycled while 845 reading. 846 847 A TID can be written to migrate the thread associated with the 848 TID to the cgroup. The writer should match all of the 849 following conditions. 850 851 - It must have write access to the "cgroup.threads" file. 852 853 - The cgroup that the thread is currently in must be in the 854 same resource domain as the destination cgroup. 855 856 - It must have write access to the "cgroup.procs" file of the 857 common ancestor of the source and destination cgroups. 858 859 When delegating a sub-hierarchy, write access to this file 860 should be granted along with the containing directory. 861 862 cgroup.controllers 863 A read-only space separated values file which exists on all 864 cgroups. 865 866 It shows space separated list of all controllers available to 867 the cgroup. The controllers are not ordered. 868 869 cgroup.subtree_control 870 A read-write space separated values file which exists on all 871 cgroups. Starts out empty. 872 873 When read, it shows space separated list of the controllers 874 which are enabled to control resource distribution from the 875 cgroup to its children. 876 877 Space separated list of controllers prefixed with '+' or '-' 878 can be written to enable or disable controllers. A controller 879 name prefixed with '+' enables the controller and '-' 880 disables. If a controller appears more than once on the list, 881 the last one is effective. When multiple enable and disable 882 operations are specified, either all succeed or all fail. 883 884 cgroup.events 885 A read-only flat-keyed file which exists on non-root cgroups. 886 The following entries are defined. Unless specified 887 otherwise, a value change in this file generates a file 888 modified event. 889 890 populated 891 1 if the cgroup or its descendants contains any live 892 processes; otherwise, 0. 893 frozen 894 1 if the cgroup is frozen; otherwise, 0. 895 896 cgroup.max.descendants 897 A read-write single value files. The default is "max". 898 899 Maximum allowed number of descent cgroups. 900 If the actual number of descendants is equal or larger, 901 an attempt to create a new cgroup in the hierarchy will fail. 902 903 cgroup.max.depth 904 A read-write single value files. The default is "max". 905 906 Maximum allowed descent depth below the current cgroup. 907 If the actual descent depth is equal or larger, 908 an attempt to create a new child cgroup will fail. 909 910 cgroup.stat 911 A read-only flat-keyed file with the following entries: 912 913 nr_descendants 914 Total number of visible descendant cgroups. 915 916 nr_dying_descendants 917 Total number of dying descendant cgroups. A cgroup becomes 918 dying after being deleted by a user. The cgroup will remain 919 in dying state for some time undefined time (which can depend 920 on system load) before being completely destroyed. 921 922 A process can't enter a dying cgroup under any circumstances, 923 a dying cgroup can't revive. 924 925 A dying cgroup can consume system resources not exceeding 926 limits, which were active at the moment of cgroup deletion. 927 928 cgroup.freeze 929 A read-write single value file which exists on non-root cgroups. 930 Allowed values are "0" and "1". The default is "0". 931 932 Writing "1" to the file causes freezing of the cgroup and all 933 descendant cgroups. This means that all belonging processes will 934 be stopped and will not run until the cgroup will be explicitly 935 unfrozen. Freezing of the cgroup may take some time; when this action 936 is completed, the "frozen" value in the cgroup.events control file 937 will be updated to "1" and the corresponding notification will be 938 issued. 939 940 A cgroup can be frozen either by its own settings, or by settings 941 of any ancestor cgroups. If any of ancestor cgroups is frozen, the 942 cgroup will remain frozen. 943 944 Processes in the frozen cgroup can be killed by a fatal signal. 945 They also can enter and leave a frozen cgroup: either by an explicit 946 move by a user, or if freezing of the cgroup races with fork(). 947 If a process is moved to a frozen cgroup, it stops. If a process is 948 moved out of a frozen cgroup, it becomes running. 949 950 Frozen status of a cgroup doesn't affect any cgroup tree operations: 951 it's possible to delete a frozen (and empty) cgroup, as well as 952 create new sub-cgroups. 953 954Controllers 955=========== 956 957CPU 958--- 959 960The "cpu" controllers regulates distribution of CPU cycles. This 961controller implements weight and absolute bandwidth limit models for 962normal scheduling policy and absolute bandwidth allocation model for 963realtime scheduling policy. 964 965In all the above models, cycles distribution is defined only on a temporal 966base and it does not account for the frequency at which tasks are executed. 967The (optional) utilization clamping support allows to hint the schedutil 968cpufreq governor about the minimum desired frequency which should always be 969provided by a CPU, as well as the maximum desired frequency, which should not 970be exceeded by a CPU. 971 972WARNING: cgroup2 doesn't yet support control of realtime processes and 973the cpu controller can only be enabled when all RT processes are in 974the root cgroup. Be aware that system management software may already 975have placed RT processes into nonroot cgroups during the system boot 976process, and these processes may need to be moved to the root cgroup 977before the cpu controller can be enabled. 978 979 980CPU Interface Files 981~~~~~~~~~~~~~~~~~~~ 982 983All time durations are in microseconds. 984 985 cpu.stat 986 A read-only flat-keyed file. 987 This file exists whether the controller is enabled or not. 988 989 It always reports the following three stats: 990 991 - usage_usec 992 - user_usec 993 - system_usec 994 995 and the following three when the controller is enabled: 996 997 - nr_periods 998 - nr_throttled 999 - throttled_usec 1000 1001 cpu.weight 1002 A read-write single value file which exists on non-root 1003 cgroups. The default is "100". 1004 1005 The weight in the range [1, 10000]. 1006 1007 cpu.weight.nice 1008 A read-write single value file which exists on non-root 1009 cgroups. The default is "0". 1010 1011 The nice value is in the range [-20, 19]. 1012 1013 This interface file is an alternative interface for 1014 "cpu.weight" and allows reading and setting weight using the 1015 same values used by nice(2). Because the range is smaller and 1016 granularity is coarser for the nice values, the read value is 1017 the closest approximation of the current weight. 1018 1019 cpu.max 1020 A read-write two value file which exists on non-root cgroups. 1021 The default is "max 100000". 1022 1023 The maximum bandwidth limit. It's in the following format:: 1024 1025 $MAX $PERIOD 1026 1027 which indicates that the group may consume upto $MAX in each 1028 $PERIOD duration. "max" for $MAX indicates no limit. If only 1029 one number is written, $MAX is updated. 1030 1031 cpu.pressure 1032 A read-only nested-key file which exists on non-root cgroups. 1033 1034 Shows pressure stall information for CPU. See 1035 :ref:`Documentation/accounting/psi.rst <psi>` for details. 1036 1037 cpu.uclamp.min 1038 A read-write single value file which exists on non-root cgroups. 1039 The default is "0", i.e. no utilization boosting. 1040 1041 The requested minimum utilization (protection) as a percentage 1042 rational number, e.g. 12.34 for 12.34%. 1043 1044 This interface allows reading and setting minimum utilization clamp 1045 values similar to the sched_setattr(2). This minimum utilization 1046 value is used to clamp the task specific minimum utilization clamp. 1047 1048 The requested minimum utilization (protection) is always capped by 1049 the current value for the maximum utilization (limit), i.e. 1050 `cpu.uclamp.max`. 1051 1052 cpu.uclamp.max 1053 A read-write single value file which exists on non-root cgroups. 1054 The default is "max". i.e. no utilization capping 1055 1056 The requested maximum utilization (limit) as a percentage rational 1057 number, e.g. 98.76 for 98.76%. 1058 1059 This interface allows reading and setting maximum utilization clamp 1060 values similar to the sched_setattr(2). This maximum utilization 1061 value is used to clamp the task specific maximum utilization clamp. 1062 1063 1064 1065Memory 1066------ 1067 1068The "memory" controller regulates distribution of memory. Memory is 1069stateful and implements both limit and protection models. Due to the 1070intertwining between memory usage and reclaim pressure and the 1071stateful nature of memory, the distribution model is relatively 1072complex. 1073 1074While not completely water-tight, all major memory usages by a given 1075cgroup are tracked so that the total memory consumption can be 1076accounted and controlled to a reasonable extent. Currently, the 1077following types of memory usages are tracked. 1078 1079- Userland memory - page cache and anonymous memory. 1080 1081- Kernel data structures such as dentries and inodes. 1082 1083- TCP socket buffers. 1084 1085The above list may expand in the future for better coverage. 1086 1087 1088Memory Interface Files 1089~~~~~~~~~~~~~~~~~~~~~~ 1090 1091All memory amounts are in bytes. If a value which is not aligned to 1092PAGE_SIZE is written, the value may be rounded up to the closest 1093PAGE_SIZE multiple when read back. 1094 1095 memory.current 1096 A read-only single value file which exists on non-root 1097 cgroups. 1098 1099 The total amount of memory currently being used by the cgroup 1100 and its descendants. 1101 1102 memory.min 1103 A read-write single value file which exists on non-root 1104 cgroups. The default is "0". 1105 1106 Hard memory protection. If the memory usage of a cgroup 1107 is within its effective min boundary, the cgroup's memory 1108 won't be reclaimed under any conditions. If there is no 1109 unprotected reclaimable memory available, OOM killer 1110 is invoked. Above the effective min boundary (or 1111 effective low boundary if it is higher), pages are reclaimed 1112 proportionally to the overage, reducing reclaim pressure for 1113 smaller overages. 1114 1115 Effective min boundary is limited by memory.min values of 1116 all ancestor cgroups. If there is memory.min overcommitment 1117 (child cgroup or cgroups are requiring more protected memory 1118 than parent will allow), then each child cgroup will get 1119 the part of parent's protection proportional to its 1120 actual memory usage below memory.min. 1121 1122 Putting more memory than generally available under this 1123 protection is discouraged and may lead to constant OOMs. 1124 1125 If a memory cgroup is not populated with processes, 1126 its memory.min is ignored. 1127 1128 memory.low 1129 A read-write single value file which exists on non-root 1130 cgroups. The default is "0". 1131 1132 Best-effort memory protection. If the memory usage of a 1133 cgroup is within its effective low boundary, the cgroup's 1134 memory won't be reclaimed unless there is no reclaimable 1135 memory available in unprotected cgroups. 1136 Above the effective low boundary (or 1137 effective min boundary if it is higher), pages are reclaimed 1138 proportionally to the overage, reducing reclaim pressure for 1139 smaller overages. 1140 1141 Effective low boundary is limited by memory.low values of 1142 all ancestor cgroups. If there is memory.low overcommitment 1143 (child cgroup or cgroups are requiring more protected memory 1144 than parent will allow), then each child cgroup will get 1145 the part of parent's protection proportional to its 1146 actual memory usage below memory.low. 1147 1148 Putting more memory than generally available under this 1149 protection is discouraged. 1150 1151 memory.high 1152 A read-write single value file which exists on non-root 1153 cgroups. The default is "max". 1154 1155 Memory usage throttle limit. This is the main mechanism to 1156 control memory usage of a cgroup. If a cgroup's usage goes 1157 over the high boundary, the processes of the cgroup are 1158 throttled and put under heavy reclaim pressure. 1159 1160 Going over the high limit never invokes the OOM killer and 1161 under extreme conditions the limit may be breached. 1162 1163 memory.max 1164 A read-write single value file which exists on non-root 1165 cgroups. The default is "max". 1166 1167 Memory usage hard limit. This is the final protection 1168 mechanism. If a cgroup's memory usage reaches this limit and 1169 can't be reduced, the OOM killer is invoked in the cgroup. 1170 Under certain circumstances, the usage may go over the limit 1171 temporarily. 1172 1173 In default configuration regular 0-order allocations always 1174 succeed unless OOM killer chooses current task as a victim. 1175 1176 Some kinds of allocations don't invoke the OOM killer. 1177 Caller could retry them differently, return into userspace 1178 as -ENOMEM or silently ignore in cases like disk readahead. 1179 1180 This is the ultimate protection mechanism. As long as the 1181 high limit is used and monitored properly, this limit's 1182 utility is limited to providing the final safety net. 1183 1184 memory.oom.group 1185 A read-write single value file which exists on non-root 1186 cgroups. The default value is "0". 1187 1188 Determines whether the cgroup should be treated as 1189 an indivisible workload by the OOM killer. If set, 1190 all tasks belonging to the cgroup or to its descendants 1191 (if the memory cgroup is not a leaf cgroup) are killed 1192 together or not at all. This can be used to avoid 1193 partial kills to guarantee workload integrity. 1194 1195 Tasks with the OOM protection (oom_score_adj set to -1000) 1196 are treated as an exception and are never killed. 1197 1198 If the OOM killer is invoked in a cgroup, it's not going 1199 to kill any tasks outside of this cgroup, regardless 1200 memory.oom.group values of ancestor cgroups. 1201 1202 memory.events 1203 A read-only flat-keyed file which exists on non-root cgroups. 1204 The following entries are defined. Unless specified 1205 otherwise, a value change in this file generates a file 1206 modified event. 1207 1208 Note that all fields in this file are hierarchical and the 1209 file modified event can be generated due to an event down the 1210 hierarchy. For for the local events at the cgroup level see 1211 memory.events.local. 1212 1213 low 1214 The number of times the cgroup is reclaimed due to 1215 high memory pressure even though its usage is under 1216 the low boundary. This usually indicates that the low 1217 boundary is over-committed. 1218 1219 high 1220 The number of times processes of the cgroup are 1221 throttled and routed to perform direct memory reclaim 1222 because the high memory boundary was exceeded. For a 1223 cgroup whose memory usage is capped by the high limit 1224 rather than global memory pressure, this event's 1225 occurrences are expected. 1226 1227 max 1228 The number of times the cgroup's memory usage was 1229 about to go over the max boundary. If direct reclaim 1230 fails to bring it down, the cgroup goes to OOM state. 1231 1232 oom 1233 The number of time the cgroup's memory usage was 1234 reached the limit and allocation was about to fail. 1235 1236 This event is not raised if the OOM killer is not 1237 considered as an option, e.g. for failed high-order 1238 allocations or if caller asked to not retry attempts. 1239 1240 oom_kill 1241 The number of processes belonging to this cgroup 1242 killed by any kind of OOM killer. 1243 1244 memory.events.local 1245 Similar to memory.events but the fields in the file are local 1246 to the cgroup i.e. not hierarchical. The file modified event 1247 generated on this file reflects only the local events. 1248 1249 memory.stat 1250 A read-only flat-keyed file which exists on non-root cgroups. 1251 1252 This breaks down the cgroup's memory footprint into different 1253 types of memory, type-specific details, and other information 1254 on the state and past events of the memory management system. 1255 1256 All memory amounts are in bytes. 1257 1258 The entries are ordered to be human readable, and new entries 1259 can show up in the middle. Don't rely on items remaining in a 1260 fixed position; use the keys to look up specific values! 1261 1262 anon 1263 Amount of memory used in anonymous mappings such as 1264 brk(), sbrk(), and mmap(MAP_ANONYMOUS) 1265 1266 file 1267 Amount of memory used to cache filesystem data, 1268 including tmpfs and shared memory. 1269 1270 kernel_stack 1271 Amount of memory allocated to kernel stacks. 1272 1273 slab 1274 Amount of memory used for storing in-kernel data 1275 structures. 1276 1277 sock 1278 Amount of memory used in network transmission buffers 1279 1280 shmem 1281 Amount of cached filesystem data that is swap-backed, 1282 such as tmpfs, shm segments, shared anonymous mmap()s 1283 1284 file_mapped 1285 Amount of cached filesystem data mapped with mmap() 1286 1287 file_dirty 1288 Amount of cached filesystem data that was modified but 1289 not yet written back to disk 1290 1291 file_writeback 1292 Amount of cached filesystem data that was modified and 1293 is currently being written back to disk 1294 1295 anon_thp 1296 Amount of memory used in anonymous mappings backed by 1297 transparent hugepages 1298 1299 inactive_anon, active_anon, inactive_file, active_file, unevictable 1300 Amount of memory, swap-backed and filesystem-backed, 1301 on the internal memory management lists used by the 1302 page reclaim algorithm. 1303 1304 As these represent internal list state (eg. shmem pages are on anon 1305 memory management lists), inactive_foo + active_foo may not be equal to 1306 the value for the foo counter, since the foo counter is type-based, not 1307 list-based. 1308 1309 slab_reclaimable 1310 Part of "slab" that might be reclaimed, such as 1311 dentries and inodes. 1312 1313 slab_unreclaimable 1314 Part of "slab" that cannot be reclaimed on memory 1315 pressure. 1316 1317 pgfault 1318 Total number of page faults incurred 1319 1320 pgmajfault 1321 Number of major page faults incurred 1322 1323 workingset_refault 1324 Number of refaults of previously evicted pages 1325 1326 workingset_activate 1327 Number of refaulted pages that were immediately activated 1328 1329 workingset_restore 1330 Number of restored pages which have been detected as an active 1331 workingset before they got reclaimed. 1332 1333 workingset_nodereclaim 1334 Number of times a shadow node has been reclaimed 1335 1336 pgrefill 1337 Amount of scanned pages (in an active LRU list) 1338 1339 pgscan 1340 Amount of scanned pages (in an inactive LRU list) 1341 1342 pgsteal 1343 Amount of reclaimed pages 1344 1345 pgactivate 1346 Amount of pages moved to the active LRU list 1347 1348 pgdeactivate 1349 Amount of pages moved to the inactive LRU list 1350 1351 pglazyfree 1352 Amount of pages postponed to be freed under memory pressure 1353 1354 pglazyfreed 1355 Amount of reclaimed lazyfree pages 1356 1357 thp_fault_alloc 1358 Number of transparent hugepages which were allocated to satisfy 1359 a page fault, including COW faults. This counter is not present 1360 when CONFIG_TRANSPARENT_HUGEPAGE is not set. 1361 1362 thp_collapse_alloc 1363 Number of transparent hugepages which were allocated to allow 1364 collapsing an existing range of pages. This counter is not 1365 present when CONFIG_TRANSPARENT_HUGEPAGE is not set. 1366 1367 memory.swap.current 1368 A read-only single value file which exists on non-root 1369 cgroups. 1370 1371 The total amount of swap currently being used by the cgroup 1372 and its descendants. 1373 1374 memory.swap.high 1375 A read-write single value file which exists on non-root 1376 cgroups. The default is "max". 1377 1378 Swap usage throttle limit. If a cgroup's swap usage exceeds 1379 this limit, all its further allocations will be throttled to 1380 allow userspace to implement custom out-of-memory procedures. 1381 1382 This limit marks a point of no return for the cgroup. It is NOT 1383 designed to manage the amount of swapping a workload does 1384 during regular operation. Compare to memory.swap.max, which 1385 prohibits swapping past a set amount, but lets the cgroup 1386 continue unimpeded as long as other memory can be reclaimed. 1387 1388 Healthy workloads are not expected to reach this limit. 1389 1390 memory.swap.max 1391 A read-write single value file which exists on non-root 1392 cgroups. The default is "max". 1393 1394 Swap usage hard limit. If a cgroup's swap usage reaches this 1395 limit, anonymous memory of the cgroup will not be swapped out. 1396 1397 memory.swap.events 1398 A read-only flat-keyed file which exists on non-root cgroups. 1399 The following entries are defined. Unless specified 1400 otherwise, a value change in this file generates a file 1401 modified event. 1402 1403 high 1404 The number of times the cgroup's swap usage was over 1405 the high threshold. 1406 1407 max 1408 The number of times the cgroup's swap usage was about 1409 to go over the max boundary and swap allocation 1410 failed. 1411 1412 fail 1413 The number of times swap allocation failed either 1414 because of running out of swap system-wide or max 1415 limit. 1416 1417 When reduced under the current usage, the existing swap 1418 entries are reclaimed gradually and the swap usage may stay 1419 higher than the limit for an extended period of time. This 1420 reduces the impact on the workload and memory management. 1421 1422 memory.pressure 1423 A read-only nested-key file which exists on non-root cgroups. 1424 1425 Shows pressure stall information for memory. See 1426 :ref:`Documentation/accounting/psi.rst <psi>` for details. 1427 1428 1429Usage Guidelines 1430~~~~~~~~~~~~~~~~ 1431 1432"memory.high" is the main mechanism to control memory usage. 1433Over-committing on high limit (sum of high limits > available memory) 1434and letting global memory pressure to distribute memory according to 1435usage is a viable strategy. 1436 1437Because breach of the high limit doesn't trigger the OOM killer but 1438throttles the offending cgroup, a management agent has ample 1439opportunities to monitor and take appropriate actions such as granting 1440more memory or terminating the workload. 1441 1442Determining whether a cgroup has enough memory is not trivial as 1443memory usage doesn't indicate whether the workload can benefit from 1444more memory. For example, a workload which writes data received from 1445network to a file can use all available memory but can also operate as 1446performant with a small amount of memory. A measure of memory 1447pressure - how much the workload is being impacted due to lack of 1448memory - is necessary to determine whether a workload needs more 1449memory; unfortunately, memory pressure monitoring mechanism isn't 1450implemented yet. 1451 1452 1453Memory Ownership 1454~~~~~~~~~~~~~~~~ 1455 1456A memory area is charged to the cgroup which instantiated it and stays 1457charged to the cgroup until the area is released. Migrating a process 1458to a different cgroup doesn't move the memory usages that it 1459instantiated while in the previous cgroup to the new cgroup. 1460 1461A memory area may be used by processes belonging to different cgroups. 1462To which cgroup the area will be charged is in-deterministic; however, 1463over time, the memory area is likely to end up in a cgroup which has 1464enough memory allowance to avoid high reclaim pressure. 1465 1466If a cgroup sweeps a considerable amount of memory which is expected 1467to be accessed repeatedly by other cgroups, it may make sense to use 1468POSIX_FADV_DONTNEED to relinquish the ownership of memory areas 1469belonging to the affected files to ensure correct memory ownership. 1470 1471 1472IO 1473-- 1474 1475The "io" controller regulates the distribution of IO resources. This 1476controller implements both weight based and absolute bandwidth or IOPS 1477limit distribution; however, weight based distribution is available 1478only if cfq-iosched is in use and neither scheme is available for 1479blk-mq devices. 1480 1481 1482IO Interface Files 1483~~~~~~~~~~~~~~~~~~ 1484 1485 io.stat 1486 A read-only nested-keyed file which exists on non-root 1487 cgroups. 1488 1489 Lines are keyed by $MAJ:$MIN device numbers and not ordered. 1490 The following nested keys are defined. 1491 1492 ====== ===================== 1493 rbytes Bytes read 1494 wbytes Bytes written 1495 rios Number of read IOs 1496 wios Number of write IOs 1497 dbytes Bytes discarded 1498 dios Number of discard IOs 1499 ====== ===================== 1500 1501 An example read output follows:: 1502 1503 8:16 rbytes=1459200 wbytes=314773504 rios=192 wios=353 dbytes=0 dios=0 1504 8:0 rbytes=90430464 wbytes=299008000 rios=8950 wios=1252 dbytes=50331648 dios=3021 1505 1506 io.cost.qos 1507 A read-write nested-keyed file with exists only on the root 1508 cgroup. 1509 1510 This file configures the Quality of Service of the IO cost 1511 model based controller (CONFIG_BLK_CGROUP_IOCOST) which 1512 currently implements "io.weight" proportional control. Lines 1513 are keyed by $MAJ:$MIN device numbers and not ordered. The 1514 line for a given device is populated on the first write for 1515 the device on "io.cost.qos" or "io.cost.model". The following 1516 nested keys are defined. 1517 1518 ====== ===================================== 1519 enable Weight-based control enable 1520 ctrl "auto" or "user" 1521 rpct Read latency percentile [0, 100] 1522 rlat Read latency threshold 1523 wpct Write latency percentile [0, 100] 1524 wlat Write latency threshold 1525 min Minimum scaling percentage [1, 10000] 1526 max Maximum scaling percentage [1, 10000] 1527 ====== ===================================== 1528 1529 The controller is disabled by default and can be enabled by 1530 setting "enable" to 1. "rpct" and "wpct" parameters default 1531 to zero and the controller uses internal device saturation 1532 state to adjust the overall IO rate between "min" and "max". 1533 1534 When a better control quality is needed, latency QoS 1535 parameters can be configured. For example:: 1536 1537 8:16 enable=1 ctrl=auto rpct=95.00 rlat=75000 wpct=95.00 wlat=150000 min=50.00 max=150.0 1538 1539 shows that on sdb, the controller is enabled, will consider 1540 the device saturated if the 95th percentile of read completion 1541 latencies is above 75ms or write 150ms, and adjust the overall 1542 IO issue rate between 50% and 150% accordingly. 1543 1544 The lower the saturation point, the better the latency QoS at 1545 the cost of aggregate bandwidth. The narrower the allowed 1546 adjustment range between "min" and "max", the more conformant 1547 to the cost model the IO behavior. Note that the IO issue 1548 base rate may be far off from 100% and setting "min" and "max" 1549 blindly can lead to a significant loss of device capacity or 1550 control quality. "min" and "max" are useful for regulating 1551 devices which show wide temporary behavior changes - e.g. a 1552 ssd which accepts writes at the line speed for a while and 1553 then completely stalls for multiple seconds. 1554 1555 When "ctrl" is "auto", the parameters are controlled by the 1556 kernel and may change automatically. Setting "ctrl" to "user" 1557 or setting any of the percentile and latency parameters puts 1558 it into "user" mode and disables the automatic changes. The 1559 automatic mode can be restored by setting "ctrl" to "auto". 1560 1561 io.cost.model 1562 A read-write nested-keyed file with exists only on the root 1563 cgroup. 1564 1565 This file configures the cost model of the IO cost model based 1566 controller (CONFIG_BLK_CGROUP_IOCOST) which currently 1567 implements "io.weight" proportional control. Lines are keyed 1568 by $MAJ:$MIN device numbers and not ordered. The line for a 1569 given device is populated on the first write for the device on 1570 "io.cost.qos" or "io.cost.model". The following nested keys 1571 are defined. 1572 1573 ===== ================================ 1574 ctrl "auto" or "user" 1575 model The cost model in use - "linear" 1576 ===== ================================ 1577 1578 When "ctrl" is "auto", the kernel may change all parameters 1579 dynamically. When "ctrl" is set to "user" or any other 1580 parameters are written to, "ctrl" become "user" and the 1581 automatic changes are disabled. 1582 1583 When "model" is "linear", the following model parameters are 1584 defined. 1585 1586 ============= ======================================== 1587 [r|w]bps The maximum sequential IO throughput 1588 [r|w]seqiops The maximum 4k sequential IOs per second 1589 [r|w]randiops The maximum 4k random IOs per second 1590 ============= ======================================== 1591 1592 From the above, the builtin linear model determines the base 1593 costs of a sequential and random IO and the cost coefficient 1594 for the IO size. While simple, this model can cover most 1595 common device classes acceptably. 1596 1597 The IO cost model isn't expected to be accurate in absolute 1598 sense and is scaled to the device behavior dynamically. 1599 1600 If needed, tools/cgroup/iocost_coef_gen.py can be used to 1601 generate device-specific coefficients. 1602 1603 io.weight 1604 A read-write flat-keyed file which exists on non-root cgroups. 1605 The default is "default 100". 1606 1607 The first line is the default weight applied to devices 1608 without specific override. The rest are overrides keyed by 1609 $MAJ:$MIN device numbers and not ordered. The weights are in 1610 the range [1, 10000] and specifies the relative amount IO time 1611 the cgroup can use in relation to its siblings. 1612 1613 The default weight can be updated by writing either "default 1614 $WEIGHT" or simply "$WEIGHT". Overrides can be set by writing 1615 "$MAJ:$MIN $WEIGHT" and unset by writing "$MAJ:$MIN default". 1616 1617 An example read output follows:: 1618 1619 default 100 1620 8:16 200 1621 8:0 50 1622 1623 io.max 1624 A read-write nested-keyed file which exists on non-root 1625 cgroups. 1626 1627 BPS and IOPS based IO limit. Lines are keyed by $MAJ:$MIN 1628 device numbers and not ordered. The following nested keys are 1629 defined. 1630 1631 ===== ================================== 1632 rbps Max read bytes per second 1633 wbps Max write bytes per second 1634 riops Max read IO operations per second 1635 wiops Max write IO operations per second 1636 ===== ================================== 1637 1638 When writing, any number of nested key-value pairs can be 1639 specified in any order. "max" can be specified as the value 1640 to remove a specific limit. If the same key is specified 1641 multiple times, the outcome is undefined. 1642 1643 BPS and IOPS are measured in each IO direction and IOs are 1644 delayed if limit is reached. Temporary bursts are allowed. 1645 1646 Setting read limit at 2M BPS and write at 120 IOPS for 8:16:: 1647 1648 echo "8:16 rbps=2097152 wiops=120" > io.max 1649 1650 Reading returns the following:: 1651 1652 8:16 rbps=2097152 wbps=max riops=max wiops=120 1653 1654 Write IOPS limit can be removed by writing the following:: 1655 1656 echo "8:16 wiops=max" > io.max 1657 1658 Reading now returns the following:: 1659 1660 8:16 rbps=2097152 wbps=max riops=max wiops=max 1661 1662 io.pressure 1663 A read-only nested-key file which exists on non-root cgroups. 1664 1665 Shows pressure stall information for IO. See 1666 :ref:`Documentation/accounting/psi.rst <psi>` for details. 1667 1668 1669Writeback 1670~~~~~~~~~ 1671 1672Page cache is dirtied through buffered writes and shared mmaps and 1673written asynchronously to the backing filesystem by the writeback 1674mechanism. Writeback sits between the memory and IO domains and 1675regulates the proportion of dirty memory by balancing dirtying and 1676write IOs. 1677 1678The io controller, in conjunction with the memory controller, 1679implements control of page cache writeback IOs. The memory controller 1680defines the memory domain that dirty memory ratio is calculated and 1681maintained for and the io controller defines the io domain which 1682writes out dirty pages for the memory domain. Both system-wide and 1683per-cgroup dirty memory states are examined and the more restrictive 1684of the two is enforced. 1685 1686cgroup writeback requires explicit support from the underlying 1687filesystem. Currently, cgroup writeback is implemented on ext2, ext4 1688and btrfs. On other filesystems, all writeback IOs are attributed to 1689the root cgroup. 1690 1691There are inherent differences in memory and writeback management 1692which affects how cgroup ownership is tracked. Memory is tracked per 1693page while writeback per inode. For the purpose of writeback, an 1694inode is assigned to a cgroup and all IO requests to write dirty pages 1695from the inode are attributed to that cgroup. 1696 1697As cgroup ownership for memory is tracked per page, there can be pages 1698which are associated with different cgroups than the one the inode is 1699associated with. These are called foreign pages. The writeback 1700constantly keeps track of foreign pages and, if a particular foreign 1701cgroup becomes the majority over a certain period of time, switches 1702the ownership of the inode to that cgroup. 1703 1704While this model is enough for most use cases where a given inode is 1705mostly dirtied by a single cgroup even when the main writing cgroup 1706changes over time, use cases where multiple cgroups write to a single 1707inode simultaneously are not supported well. In such circumstances, a 1708significant portion of IOs are likely to be attributed incorrectly. 1709As memory controller assigns page ownership on the first use and 1710doesn't update it until the page is released, even if writeback 1711strictly follows page ownership, multiple cgroups dirtying overlapping 1712areas wouldn't work as expected. It's recommended to avoid such usage 1713patterns. 1714 1715The sysctl knobs which affect writeback behavior are applied to cgroup 1716writeback as follows. 1717 1718 vm.dirty_background_ratio, vm.dirty_ratio 1719 These ratios apply the same to cgroup writeback with the 1720 amount of available memory capped by limits imposed by the 1721 memory controller and system-wide clean memory. 1722 1723 vm.dirty_background_bytes, vm.dirty_bytes 1724 For cgroup writeback, this is calculated into ratio against 1725 total available memory and applied the same way as 1726 vm.dirty[_background]_ratio. 1727 1728 1729IO Latency 1730~~~~~~~~~~ 1731 1732This is a cgroup v2 controller for IO workload protection. You provide a group 1733with a latency target, and if the average latency exceeds that target the 1734controller will throttle any peers that have a lower latency target than the 1735protected workload. 1736 1737The limits are only applied at the peer level in the hierarchy. This means that 1738in the diagram below, only groups A, B, and C will influence each other, and 1739groups D and F will influence each other. Group G will influence nobody:: 1740 1741 [root] 1742 / | \ 1743 A B C 1744 / \ | 1745 D F G 1746 1747 1748So the ideal way to configure this is to set io.latency in groups A, B, and C. 1749Generally you do not want to set a value lower than the latency your device 1750supports. Experiment to find the value that works best for your workload. 1751Start at higher than the expected latency for your device and watch the 1752avg_lat value in io.stat for your workload group to get an idea of the 1753latency you see during normal operation. Use the avg_lat value as a basis for 1754your real setting, setting at 10-15% higher than the value in io.stat. 1755 1756How IO Latency Throttling Works 1757~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1758 1759io.latency is work conserving; so as long as everybody is meeting their latency 1760target the controller doesn't do anything. Once a group starts missing its 1761target it begins throttling any peer group that has a higher target than itself. 1762This throttling takes 2 forms: 1763 1764- Queue depth throttling. This is the number of outstanding IO's a group is 1765 allowed to have. We will clamp down relatively quickly, starting at no limit 1766 and going all the way down to 1 IO at a time. 1767 1768- Artificial delay induction. There are certain types of IO that cannot be 1769 throttled without possibly adversely affecting higher priority groups. This 1770 includes swapping and metadata IO. These types of IO are allowed to occur 1771 normally, however they are "charged" to the originating group. If the 1772 originating group is being throttled you will see the use_delay and delay 1773 fields in io.stat increase. The delay value is how many microseconds that are 1774 being added to any process that runs in this group. Because this number can 1775 grow quite large if there is a lot of swapping or metadata IO occurring we 1776 limit the individual delay events to 1 second at a time. 1777 1778Once the victimized group starts meeting its latency target again it will start 1779unthrottling any peer groups that were throttled previously. If the victimized 1780group simply stops doing IO the global counter will unthrottle appropriately. 1781 1782IO Latency Interface Files 1783~~~~~~~~~~~~~~~~~~~~~~~~~~ 1784 1785 io.latency 1786 This takes a similar format as the other controllers. 1787 1788 "MAJOR:MINOR target=<target time in microseconds" 1789 1790 io.stat 1791 If the controller is enabled you will see extra stats in io.stat in 1792 addition to the normal ones. 1793 1794 depth 1795 This is the current queue depth for the group. 1796 1797 avg_lat 1798 This is an exponential moving average with a decay rate of 1/exp 1799 bound by the sampling interval. The decay rate interval can be 1800 calculated by multiplying the win value in io.stat by the 1801 corresponding number of samples based on the win value. 1802 1803 win 1804 The sampling window size in milliseconds. This is the minimum 1805 duration of time between evaluation events. Windows only elapse 1806 with IO activity. Idle periods extend the most recent window. 1807 1808PID 1809--- 1810 1811The process number controller is used to allow a cgroup to stop any 1812new tasks from being fork()'d or clone()'d after a specified limit is 1813reached. 1814 1815The number of tasks in a cgroup can be exhausted in ways which other 1816controllers cannot prevent, thus warranting its own controller. For 1817example, a fork bomb is likely to exhaust the number of tasks before 1818hitting memory restrictions. 1819 1820Note that PIDs used in this controller refer to TIDs, process IDs as 1821used by the kernel. 1822 1823 1824PID Interface Files 1825~~~~~~~~~~~~~~~~~~~ 1826 1827 pids.max 1828 A read-write single value file which exists on non-root 1829 cgroups. The default is "max". 1830 1831 Hard limit of number of processes. 1832 1833 pids.current 1834 A read-only single value file which exists on all cgroups. 1835 1836 The number of processes currently in the cgroup and its 1837 descendants. 1838 1839Organisational operations are not blocked by cgroup policies, so it is 1840possible to have pids.current > pids.max. This can be done by either 1841setting the limit to be smaller than pids.current, or attaching enough 1842processes to the cgroup such that pids.current is larger than 1843pids.max. However, it is not possible to violate a cgroup PID policy 1844through fork() or clone(). These will return -EAGAIN if the creation 1845of a new process would cause a cgroup policy to be violated. 1846 1847 1848Cpuset 1849------ 1850 1851The "cpuset" controller provides a mechanism for constraining 1852the CPU and memory node placement of tasks to only the resources 1853specified in the cpuset interface files in a task's current cgroup. 1854This is especially valuable on large NUMA systems where placing jobs 1855on properly sized subsets of the systems with careful processor and 1856memory placement to reduce cross-node memory access and contention 1857can improve overall system performance. 1858 1859The "cpuset" controller is hierarchical. That means the controller 1860cannot use CPUs or memory nodes not allowed in its parent. 1861 1862 1863Cpuset Interface Files 1864~~~~~~~~~~~~~~~~~~~~~~ 1865 1866 cpuset.cpus 1867 A read-write multiple values file which exists on non-root 1868 cpuset-enabled cgroups. 1869 1870 It lists the requested CPUs to be used by tasks within this 1871 cgroup. The actual list of CPUs to be granted, however, is 1872 subjected to constraints imposed by its parent and can differ 1873 from the requested CPUs. 1874 1875 The CPU numbers are comma-separated numbers or ranges. 1876 For example:: 1877 1878 # cat cpuset.cpus 1879 0-4,6,8-10 1880 1881 An empty value indicates that the cgroup is using the same 1882 setting as the nearest cgroup ancestor with a non-empty 1883 "cpuset.cpus" or all the available CPUs if none is found. 1884 1885 The value of "cpuset.cpus" stays constant until the next update 1886 and won't be affected by any CPU hotplug events. 1887 1888 cpuset.cpus.effective 1889 A read-only multiple values file which exists on all 1890 cpuset-enabled cgroups. 1891 1892 It lists the onlined CPUs that are actually granted to this 1893 cgroup by its parent. These CPUs are allowed to be used by 1894 tasks within the current cgroup. 1895 1896 If "cpuset.cpus" is empty, the "cpuset.cpus.effective" file shows 1897 all the CPUs from the parent cgroup that can be available to 1898 be used by this cgroup. Otherwise, it should be a subset of 1899 "cpuset.cpus" unless none of the CPUs listed in "cpuset.cpus" 1900 can be granted. In this case, it will be treated just like an 1901 empty "cpuset.cpus". 1902 1903 Its value will be affected by CPU hotplug events. 1904 1905 cpuset.mems 1906 A read-write multiple values file which exists on non-root 1907 cpuset-enabled cgroups. 1908 1909 It lists the requested memory nodes to be used by tasks within 1910 this cgroup. The actual list of memory nodes granted, however, 1911 is subjected to constraints imposed by its parent and can differ 1912 from the requested memory nodes. 1913 1914 The memory node numbers are comma-separated numbers or ranges. 1915 For example:: 1916 1917 # cat cpuset.mems 1918 0-1,3 1919 1920 An empty value indicates that the cgroup is using the same 1921 setting as the nearest cgroup ancestor with a non-empty 1922 "cpuset.mems" or all the available memory nodes if none 1923 is found. 1924 1925 The value of "cpuset.mems" stays constant until the next update 1926 and won't be affected by any memory nodes hotplug events. 1927 1928 cpuset.mems.effective 1929 A read-only multiple values file which exists on all 1930 cpuset-enabled cgroups. 1931 1932 It lists the onlined memory nodes that are actually granted to 1933 this cgroup by its parent. These memory nodes are allowed to 1934 be used by tasks within the current cgroup. 1935 1936 If "cpuset.mems" is empty, it shows all the memory nodes from the 1937 parent cgroup that will be available to be used by this cgroup. 1938 Otherwise, it should be a subset of "cpuset.mems" unless none of 1939 the memory nodes listed in "cpuset.mems" can be granted. In this 1940 case, it will be treated just like an empty "cpuset.mems". 1941 1942 Its value will be affected by memory nodes hotplug events. 1943 1944 cpuset.cpus.partition 1945 A read-write single value file which exists on non-root 1946 cpuset-enabled cgroups. This flag is owned by the parent cgroup 1947 and is not delegatable. 1948 1949 It accepts only the following input values when written to. 1950 1951 "root" - a partition root 1952 "member" - a non-root member of a partition 1953 1954 When set to be a partition root, the current cgroup is the 1955 root of a new partition or scheduling domain that comprises 1956 itself and all its descendants except those that are separate 1957 partition roots themselves and their descendants. The root 1958 cgroup is always a partition root. 1959 1960 There are constraints on where a partition root can be set. 1961 It can only be set in a cgroup if all the following conditions 1962 are true. 1963 1964 1) The "cpuset.cpus" is not empty and the list of CPUs are 1965 exclusive, i.e. they are not shared by any of its siblings. 1966 2) The parent cgroup is a partition root. 1967 3) The "cpuset.cpus" is also a proper subset of the parent's 1968 "cpuset.cpus.effective". 1969 4) There is no child cgroups with cpuset enabled. This is for 1970 eliminating corner cases that have to be handled if such a 1971 condition is allowed. 1972 1973 Setting it to partition root will take the CPUs away from the 1974 effective CPUs of the parent cgroup. Once it is set, this 1975 file cannot be reverted back to "member" if there are any child 1976 cgroups with cpuset enabled. 1977 1978 A parent partition cannot distribute all its CPUs to its 1979 child partitions. There must be at least one cpu left in the 1980 parent partition. 1981 1982 Once becoming a partition root, changes to "cpuset.cpus" is 1983 generally allowed as long as the first condition above is true, 1984 the change will not take away all the CPUs from the parent 1985 partition and the new "cpuset.cpus" value is a superset of its 1986 children's "cpuset.cpus" values. 1987 1988 Sometimes, external factors like changes to ancestors' 1989 "cpuset.cpus" or cpu hotplug can cause the state of the partition 1990 root to change. On read, the "cpuset.sched.partition" file 1991 can show the following values. 1992 1993 "member" Non-root member of a partition 1994 "root" Partition root 1995 "root invalid" Invalid partition root 1996 1997 It is a partition root if the first 2 partition root conditions 1998 above are true and at least one CPU from "cpuset.cpus" is 1999 granted by the parent cgroup. 2000 2001 A partition root can become invalid if none of CPUs requested 2002 in "cpuset.cpus" can be granted by the parent cgroup or the 2003 parent cgroup is no longer a partition root itself. In this 2004 case, it is not a real partition even though the restriction 2005 of the first partition root condition above will still apply. 2006 The cpu affinity of all the tasks in the cgroup will then be 2007 associated with CPUs in the nearest ancestor partition. 2008 2009 An invalid partition root can be transitioned back to a 2010 real partition root if at least one of the requested CPUs 2011 can now be granted by its parent. In this case, the cpu 2012 affinity of all the tasks in the formerly invalid partition 2013 will be associated to the CPUs of the newly formed partition. 2014 Changing the partition state of an invalid partition root to 2015 "member" is always allowed even if child cpusets are present. 2016 2017 2018Device controller 2019----------------- 2020 2021Device controller manages access to device files. It includes both 2022creation of new device files (using mknod), and access to the 2023existing device files. 2024 2025Cgroup v2 device controller has no interface files and is implemented 2026on top of cgroup BPF. To control access to device files, a user may 2027create bpf programs of the BPF_CGROUP_DEVICE type and attach them 2028to cgroups. On an attempt to access a device file, corresponding 2029BPF programs will be executed, and depending on the return value 2030the attempt will succeed or fail with -EPERM. 2031 2032A BPF_CGROUP_DEVICE program takes a pointer to the bpf_cgroup_dev_ctx 2033structure, which describes the device access attempt: access type 2034(mknod/read/write) and device (type, major and minor numbers). 2035If the program returns 0, the attempt fails with -EPERM, otherwise 2036it succeeds. 2037 2038An example of BPF_CGROUP_DEVICE program may be found in the kernel 2039source tree in the tools/testing/selftests/bpf/dev_cgroup.c file. 2040 2041 2042RDMA 2043---- 2044 2045The "rdma" controller regulates the distribution and accounting of 2046of RDMA resources. 2047 2048RDMA Interface Files 2049~~~~~~~~~~~~~~~~~~~~ 2050 2051 rdma.max 2052 A readwrite nested-keyed file that exists for all the cgroups 2053 except root that describes current configured resource limit 2054 for a RDMA/IB device. 2055 2056 Lines are keyed by device name and are not ordered. 2057 Each line contains space separated resource name and its configured 2058 limit that can be distributed. 2059 2060 The following nested keys are defined. 2061 2062 ========== ============================= 2063 hca_handle Maximum number of HCA Handles 2064 hca_object Maximum number of HCA Objects 2065 ========== ============================= 2066 2067 An example for mlx4 and ocrdma device follows:: 2068 2069 mlx4_0 hca_handle=2 hca_object=2000 2070 ocrdma1 hca_handle=3 hca_object=max 2071 2072 rdma.current 2073 A read-only file that describes current resource usage. 2074 It exists for all the cgroup except root. 2075 2076 An example for mlx4 and ocrdma device follows:: 2077 2078 mlx4_0 hca_handle=1 hca_object=20 2079 ocrdma1 hca_handle=1 hca_object=23 2080 2081HugeTLB 2082------- 2083 2084The HugeTLB controller allows to limit the HugeTLB usage per control group and 2085enforces the controller limit during page fault. 2086 2087HugeTLB Interface Files 2088~~~~~~~~~~~~~~~~~~~~~~~ 2089 2090 hugetlb.<hugepagesize>.current 2091 Show current usage for "hugepagesize" hugetlb. It exists for all 2092 the cgroup except root. 2093 2094 hugetlb.<hugepagesize>.max 2095 Set/show the hard limit of "hugepagesize" hugetlb usage. 2096 The default value is "max". It exists for all the cgroup except root. 2097 2098 hugetlb.<hugepagesize>.events 2099 A read-only flat-keyed file which exists on non-root cgroups. 2100 2101 max 2102 The number of allocation failure due to HugeTLB limit 2103 2104 hugetlb.<hugepagesize>.events.local 2105 Similar to hugetlb.<hugepagesize>.events but the fields in the file 2106 are local to the cgroup i.e. not hierarchical. The file modified event 2107 generated on this file reflects only the local events. 2108 2109Misc 2110---- 2111 2112perf_event 2113~~~~~~~~~~ 2114 2115perf_event controller, if not mounted on a legacy hierarchy, is 2116automatically enabled on the v2 hierarchy so that perf events can 2117always be filtered by cgroup v2 path. The controller can still be 2118moved to a legacy hierarchy after v2 hierarchy is populated. 2119 2120 2121Non-normative information 2122------------------------- 2123 2124This section contains information that isn't considered to be a part of 2125the stable kernel API and so is subject to change. 2126 2127 2128CPU controller root cgroup process behaviour 2129~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2130 2131When distributing CPU cycles in the root cgroup each thread in this 2132cgroup is treated as if it was hosted in a separate child cgroup of the 2133root cgroup. This child cgroup weight is dependent on its thread nice 2134level. 2135 2136For details of this mapping see sched_prio_to_weight array in 2137kernel/sched/core.c file (values from this array should be scaled 2138appropriately so the neutral - nice 0 - value is 100 instead of 1024). 2139 2140 2141IO controller root cgroup process behaviour 2142~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2143 2144Root cgroup processes are hosted in an implicit leaf child node. 2145When distributing IO resources this implicit child node is taken into 2146account as if it was a normal child cgroup of the root cgroup with a 2147weight value of 200. 2148 2149 2150Namespace 2151========= 2152 2153Basics 2154------ 2155 2156cgroup namespace provides a mechanism to virtualize the view of the 2157"/proc/$PID/cgroup" file and cgroup mounts. The CLONE_NEWCGROUP clone 2158flag can be used with clone(2) and unshare(2) to create a new cgroup 2159namespace. The process running inside the cgroup namespace will have 2160its "/proc/$PID/cgroup" output restricted to cgroupns root. The 2161cgroupns root is the cgroup of the process at the time of creation of 2162the cgroup namespace. 2163 2164Without cgroup namespace, the "/proc/$PID/cgroup" file shows the 2165complete path of the cgroup of a process. In a container setup where 2166a set of cgroups and namespaces are intended to isolate processes the 2167"/proc/$PID/cgroup" file may leak potential system level information 2168to the isolated processes. For Example:: 2169 2170 # cat /proc/self/cgroup 2171 0::/batchjobs/container_id1 2172 2173The path '/batchjobs/container_id1' can be considered as system-data 2174and undesirable to expose to the isolated processes. cgroup namespace 2175can be used to restrict visibility of this path. For example, before 2176creating a cgroup namespace, one would see:: 2177 2178 # ls -l /proc/self/ns/cgroup 2179 lrwxrwxrwx 1 root root 0 2014-07-15 10:37 /proc/self/ns/cgroup -> cgroup:[4026531835] 2180 # cat /proc/self/cgroup 2181 0::/batchjobs/container_id1 2182 2183After unsharing a new namespace, the view changes:: 2184 2185 # ls -l /proc/self/ns/cgroup 2186 lrwxrwxrwx 1 root root 0 2014-07-15 10:35 /proc/self/ns/cgroup -> cgroup:[4026532183] 2187 # cat /proc/self/cgroup 2188 0::/ 2189 2190When some thread from a multi-threaded process unshares its cgroup 2191namespace, the new cgroupns gets applied to the entire process (all 2192the threads). This is natural for the v2 hierarchy; however, for the 2193legacy hierarchies, this may be unexpected. 2194 2195A cgroup namespace is alive as long as there are processes inside or 2196mounts pinning it. When the last usage goes away, the cgroup 2197namespace is destroyed. The cgroupns root and the actual cgroups 2198remain. 2199 2200 2201The Root and Views 2202------------------ 2203 2204The 'cgroupns root' for a cgroup namespace is the cgroup in which the 2205process calling unshare(2) is running. For example, if a process in 2206/batchjobs/container_id1 cgroup calls unshare, cgroup 2207/batchjobs/container_id1 becomes the cgroupns root. For the 2208init_cgroup_ns, this is the real root ('/') cgroup. 2209 2210The cgroupns root cgroup does not change even if the namespace creator 2211process later moves to a different cgroup:: 2212 2213 # ~/unshare -c # unshare cgroupns in some cgroup 2214 # cat /proc/self/cgroup 2215 0::/ 2216 # mkdir sub_cgrp_1 2217 # echo 0 > sub_cgrp_1/cgroup.procs 2218 # cat /proc/self/cgroup 2219 0::/sub_cgrp_1 2220 2221Each process gets its namespace-specific view of "/proc/$PID/cgroup" 2222 2223Processes running inside the cgroup namespace will be able to see 2224cgroup paths (in /proc/self/cgroup) only inside their root cgroup. 2225From within an unshared cgroupns:: 2226 2227 # sleep 100000 & 2228 [1] 7353 2229 # echo 7353 > sub_cgrp_1/cgroup.procs 2230 # cat /proc/7353/cgroup 2231 0::/sub_cgrp_1 2232 2233From the initial cgroup namespace, the real cgroup path will be 2234visible:: 2235 2236 $ cat /proc/7353/cgroup 2237 0::/batchjobs/container_id1/sub_cgrp_1 2238 2239From a sibling cgroup namespace (that is, a namespace rooted at a 2240different cgroup), the cgroup path relative to its own cgroup 2241namespace root will be shown. For instance, if PID 7353's cgroup 2242namespace root is at '/batchjobs/container_id2', then it will see:: 2243 2244 # cat /proc/7353/cgroup 2245 0::/../container_id2/sub_cgrp_1 2246 2247Note that the relative path always starts with '/' to indicate that 2248its relative to the cgroup namespace root of the caller. 2249 2250 2251Migration and setns(2) 2252---------------------- 2253 2254Processes inside a cgroup namespace can move into and out of the 2255namespace root if they have proper access to external cgroups. For 2256example, from inside a namespace with cgroupns root at 2257/batchjobs/container_id1, and assuming that the global hierarchy is 2258still accessible inside cgroupns:: 2259 2260 # cat /proc/7353/cgroup 2261 0::/sub_cgrp_1 2262 # echo 7353 > batchjobs/container_id2/cgroup.procs 2263 # cat /proc/7353/cgroup 2264 0::/../container_id2 2265 2266Note that this kind of setup is not encouraged. A task inside cgroup 2267namespace should only be exposed to its own cgroupns hierarchy. 2268 2269setns(2) to another cgroup namespace is allowed when: 2270 2271(a) the process has CAP_SYS_ADMIN against its current user namespace 2272(b) the process has CAP_SYS_ADMIN against the target cgroup 2273 namespace's userns 2274 2275No implicit cgroup changes happen with attaching to another cgroup 2276namespace. It is expected that the someone moves the attaching 2277process under the target cgroup namespace root. 2278 2279 2280Interaction with Other Namespaces 2281--------------------------------- 2282 2283Namespace specific cgroup hierarchy can be mounted by a process 2284running inside a non-init cgroup namespace:: 2285 2286 # mount -t cgroup2 none $MOUNT_POINT 2287 2288This will mount the unified cgroup hierarchy with cgroupns root as the 2289filesystem root. The process needs CAP_SYS_ADMIN against its user and 2290mount namespaces. 2291 2292The virtualization of /proc/self/cgroup file combined with restricting 2293the view of cgroup hierarchy by namespace-private cgroupfs mount 2294provides a properly isolated cgroup view inside the container. 2295 2296 2297Information on Kernel Programming 2298================================= 2299 2300This section contains kernel programming information in the areas 2301where interacting with cgroup is necessary. cgroup core and 2302controllers are not covered. 2303 2304 2305Filesystem Support for Writeback 2306-------------------------------- 2307 2308A filesystem can support cgroup writeback by updating 2309address_space_operations->writepage[s]() to annotate bio's using the 2310following two functions. 2311 2312 wbc_init_bio(@wbc, @bio) 2313 Should be called for each bio carrying writeback data and 2314 associates the bio with the inode's owner cgroup and the 2315 corresponding request queue. This must be called after 2316 a queue (device) has been associated with the bio and 2317 before submission. 2318 2319 wbc_account_cgroup_owner(@wbc, @page, @bytes) 2320 Should be called for each data segment being written out. 2321 While this function doesn't care exactly when it's called 2322 during the writeback session, it's the easiest and most 2323 natural to call it as data segments are added to a bio. 2324 2325With writeback bio's annotated, cgroup support can be enabled per 2326super_block by setting SB_I_CGROUPWB in ->s_iflags. This allows for 2327selective disabling of cgroup writeback support which is helpful when 2328certain filesystem features, e.g. journaled data mode, are 2329incompatible. 2330 2331wbc_init_bio() binds the specified bio to its cgroup. Depending on 2332the configuration, the bio may be executed at a lower priority and if 2333the writeback session is holding shared resources, e.g. a journal 2334entry, may lead to priority inversion. There is no one easy solution 2335for the problem. Filesystems can try to work around specific problem 2336cases by skipping wbc_init_bio() and using bio_associate_blkg() 2337directly. 2338 2339 2340Deprecated v1 Core Features 2341=========================== 2342 2343- Multiple hierarchies including named ones are not supported. 2344 2345- All v1 mount options are not supported. 2346 2347- The "tasks" file is removed and "cgroup.procs" is not sorted. 2348 2349- "cgroup.clone_children" is removed. 2350 2351- /proc/cgroups is meaningless for v2. Use "cgroup.controllers" file 2352 at the root instead. 2353 2354 2355Issues with v1 and Rationales for v2 2356==================================== 2357 2358Multiple Hierarchies 2359-------------------- 2360 2361cgroup v1 allowed an arbitrary number of hierarchies and each 2362hierarchy could host any number of controllers. While this seemed to 2363provide a high level of flexibility, it wasn't useful in practice. 2364 2365For example, as there is only one instance of each controller, utility 2366type controllers such as freezer which can be useful in all 2367hierarchies could only be used in one. The issue is exacerbated by 2368the fact that controllers couldn't be moved to another hierarchy once 2369hierarchies were populated. Another issue was that all controllers 2370bound to a hierarchy were forced to have exactly the same view of the 2371hierarchy. It wasn't possible to vary the granularity depending on 2372the specific controller. 2373 2374In practice, these issues heavily limited which controllers could be 2375put on the same hierarchy and most configurations resorted to putting 2376each controller on its own hierarchy. Only closely related ones, such 2377as the cpu and cpuacct controllers, made sense to be put on the same 2378hierarchy. This often meant that userland ended up managing multiple 2379similar hierarchies repeating the same steps on each hierarchy 2380whenever a hierarchy management operation was necessary. 2381 2382Furthermore, support for multiple hierarchies came at a steep cost. 2383It greatly complicated cgroup core implementation but more importantly 2384the support for multiple hierarchies restricted how cgroup could be 2385used in general and what controllers was able to do. 2386 2387There was no limit on how many hierarchies there might be, which meant 2388that a thread's cgroup membership couldn't be described in finite 2389length. The key might contain any number of entries and was unlimited 2390in length, which made it highly awkward to manipulate and led to 2391addition of controllers which existed only to identify membership, 2392which in turn exacerbated the original problem of proliferating number 2393of hierarchies. 2394 2395Also, as a controller couldn't have any expectation regarding the 2396topologies of hierarchies other controllers might be on, each 2397controller had to assume that all other controllers were attached to 2398completely orthogonal hierarchies. This made it impossible, or at 2399least very cumbersome, for controllers to cooperate with each other. 2400 2401In most use cases, putting controllers on hierarchies which are 2402completely orthogonal to each other isn't necessary. What usually is 2403called for is the ability to have differing levels of granularity 2404depending on the specific controller. In other words, hierarchy may 2405be collapsed from leaf towards root when viewed from specific 2406controllers. For example, a given configuration might not care about 2407how memory is distributed beyond a certain level while still wanting 2408to control how CPU cycles are distributed. 2409 2410 2411Thread Granularity 2412------------------ 2413 2414cgroup v1 allowed threads of a process to belong to different cgroups. 2415This didn't make sense for some controllers and those controllers 2416ended up implementing different ways to ignore such situations but 2417much more importantly it blurred the line between API exposed to 2418individual applications and system management interface. 2419 2420Generally, in-process knowledge is available only to the process 2421itself; thus, unlike service-level organization of processes, 2422categorizing threads of a process requires active participation from 2423the application which owns the target process. 2424 2425cgroup v1 had an ambiguously defined delegation model which got abused 2426in combination with thread granularity. cgroups were delegated to 2427individual applications so that they can create and manage their own 2428sub-hierarchies and control resource distributions along them. This 2429effectively raised cgroup to the status of a syscall-like API exposed 2430to lay programs. 2431 2432First of all, cgroup has a fundamentally inadequate interface to be 2433exposed this way. For a process to access its own knobs, it has to 2434extract the path on the target hierarchy from /proc/self/cgroup, 2435construct the path by appending the name of the knob to the path, open 2436and then read and/or write to it. This is not only extremely clunky 2437and unusual but also inherently racy. There is no conventional way to 2438define transaction across the required steps and nothing can guarantee 2439that the process would actually be operating on its own sub-hierarchy. 2440 2441cgroup controllers implemented a number of knobs which would never be 2442accepted as public APIs because they were just adding control knobs to 2443system-management pseudo filesystem. cgroup ended up with interface 2444knobs which were not properly abstracted or refined and directly 2445revealed kernel internal details. These knobs got exposed to 2446individual applications through the ill-defined delegation mechanism 2447effectively abusing cgroup as a shortcut to implementing public APIs 2448without going through the required scrutiny. 2449 2450This was painful for both userland and kernel. Userland ended up with 2451misbehaving and poorly abstracted interfaces and kernel exposing and 2452locked into constructs inadvertently. 2453 2454 2455Competition Between Inner Nodes and Threads 2456------------------------------------------- 2457 2458cgroup v1 allowed threads to be in any cgroups which created an 2459interesting problem where threads belonging to a parent cgroup and its 2460children cgroups competed for resources. This was nasty as two 2461different types of entities competed and there was no obvious way to 2462settle it. Different controllers did different things. 2463 2464The cpu controller considered threads and cgroups as equivalents and 2465mapped nice levels to cgroup weights. This worked for some cases but 2466fell flat when children wanted to be allocated specific ratios of CPU 2467cycles and the number of internal threads fluctuated - the ratios 2468constantly changed as the number of competing entities fluctuated. 2469There also were other issues. The mapping from nice level to weight 2470wasn't obvious or universal, and there were various other knobs which 2471simply weren't available for threads. 2472 2473The io controller implicitly created a hidden leaf node for each 2474cgroup to host the threads. The hidden leaf had its own copies of all 2475the knobs with ``leaf_`` prefixed. While this allowed equivalent 2476control over internal threads, it was with serious drawbacks. It 2477always added an extra layer of nesting which wouldn't be necessary 2478otherwise, made the interface messy and significantly complicated the 2479implementation. 2480 2481The memory controller didn't have a way to control what happened 2482between internal tasks and child cgroups and the behavior was not 2483clearly defined. There were attempts to add ad-hoc behaviors and 2484knobs to tailor the behavior to specific workloads which would have 2485led to problems extremely difficult to resolve in the long term. 2486 2487Multiple controllers struggled with internal tasks and came up with 2488different ways to deal with it; unfortunately, all the approaches were 2489severely flawed and, furthermore, the widely different behaviors 2490made cgroup as a whole highly inconsistent. 2491 2492This clearly is a problem which needs to be addressed from cgroup core 2493in a uniform way. 2494 2495 2496Other Interface Issues 2497---------------------- 2498 2499cgroup v1 grew without oversight and developed a large number of 2500idiosyncrasies and inconsistencies. One issue on the cgroup core side 2501was how an empty cgroup was notified - a userland helper binary was 2502forked and executed for each event. The event delivery wasn't 2503recursive or delegatable. The limitations of the mechanism also led 2504to in-kernel event delivery filtering mechanism further complicating 2505the interface. 2506 2507Controller interfaces were problematic too. An extreme example is 2508controllers completely ignoring hierarchical organization and treating 2509all cgroups as if they were all located directly under the root 2510cgroup. Some controllers exposed a large amount of inconsistent 2511implementation details to userland. 2512 2513There also was no consistency across controllers. When a new cgroup 2514was created, some controllers defaulted to not imposing extra 2515restrictions while others disallowed any resource usage until 2516explicitly configured. Configuration knobs for the same type of 2517control used widely differing naming schemes and formats. Statistics 2518and information knobs were named arbitrarily and used different 2519formats and units even in the same controller. 2520 2521cgroup v2 establishes common conventions where appropriate and updates 2522controllers so that they expose minimal and consistent interfaces. 2523 2524 2525Controller Issues and Remedies 2526------------------------------ 2527 2528Memory 2529~~~~~~ 2530 2531The original lower boundary, the soft limit, is defined as a limit 2532that is per default unset. As a result, the set of cgroups that 2533global reclaim prefers is opt-in, rather than opt-out. The costs for 2534optimizing these mostly negative lookups are so high that the 2535implementation, despite its enormous size, does not even provide the 2536basic desirable behavior. First off, the soft limit has no 2537hierarchical meaning. All configured groups are organized in a global 2538rbtree and treated like equal peers, regardless where they are located 2539in the hierarchy. This makes subtree delegation impossible. Second, 2540the soft limit reclaim pass is so aggressive that it not just 2541introduces high allocation latencies into the system, but also impacts 2542system performance due to overreclaim, to the point where the feature 2543becomes self-defeating. 2544 2545The memory.low boundary on the other hand is a top-down allocated 2546reserve. A cgroup enjoys reclaim protection when it's within its 2547effective low, which makes delegation of subtrees possible. It also 2548enjoys having reclaim pressure proportional to its overage when 2549above its effective low. 2550 2551The original high boundary, the hard limit, is defined as a strict 2552limit that can not budge, even if the OOM killer has to be called. 2553But this generally goes against the goal of making the most out of the 2554available memory. The memory consumption of workloads varies during 2555runtime, and that requires users to overcommit. But doing that with a 2556strict upper limit requires either a fairly accurate prediction of the 2557working set size or adding slack to the limit. Since working set size 2558estimation is hard and error prone, and getting it wrong results in 2559OOM kills, most users tend to err on the side of a looser limit and 2560end up wasting precious resources. 2561 2562The memory.high boundary on the other hand can be set much more 2563conservatively. When hit, it throttles allocations by forcing them 2564into direct reclaim to work off the excess, but it never invokes the 2565OOM killer. As a result, a high boundary that is chosen too 2566aggressively will not terminate the processes, but instead it will 2567lead to gradual performance degradation. The user can monitor this 2568and make corrections until the minimal memory footprint that still 2569gives acceptable performance is found. 2570 2571In extreme cases, with many concurrent allocations and a complete 2572breakdown of reclaim progress within the group, the high boundary can 2573be exceeded. But even then it's mostly better to satisfy the 2574allocation from the slack available in other groups or the rest of the 2575system than killing the group. Otherwise, memory.max is there to 2576limit this type of spillover and ultimately contain buggy or even 2577malicious applications. 2578 2579Setting the original memory.limit_in_bytes below the current usage was 2580subject to a race condition, where concurrent charges could cause the 2581limit setting to fail. memory.max on the other hand will first set the 2582limit to prevent new charges, and then reclaim and OOM kill until the 2583new limit is met - or the task writing to memory.max is killed. 2584 2585The combined memory+swap accounting and limiting is replaced by real 2586control over swap space. 2587 2588The main argument for a combined memory+swap facility in the original 2589cgroup design was that global or parental pressure would always be 2590able to swap all anonymous memory of a child group, regardless of the 2591child's own (possibly untrusted) configuration. However, untrusted 2592groups can sabotage swapping by other means - such as referencing its 2593anonymous memory in a tight loop - and an admin can not assume full 2594swappability when overcommitting untrusted jobs. 2595 2596For trusted jobs, on the other hand, a combined counter is not an 2597intuitive userspace interface, and it flies in the face of the idea 2598that cgroup controllers should account and limit specific physical 2599resources. Swap space is a resource like all others in the system, 2600and that's why unified hierarchy allows distributing it separately. 2601