1.. _cgroup-v2: 2 3================ 4Control Group v2 5================ 6 7:Date: October, 2015 8:Author: Tejun Heo <tj@kernel.org> 9 10This is the authoritative documentation on the design, interface and 11conventions of cgroup v2. It describes all userland-visible aspects 12of cgroup including core and specific controller behaviors. All 13future changes must be reflected in this document. Documentation for 14v1 is available under :ref:`Documentation/admin-guide/cgroup-v1/index.rst <cgroup-v1>`. 15 16.. CONTENTS 17 18 1. Introduction 19 1-1. Terminology 20 1-2. What is cgroup? 21 2. Basic Operations 22 2-1. Mounting 23 2-2. Organizing Processes and Threads 24 2-2-1. Processes 25 2-2-2. Threads 26 2-3. [Un]populated Notification 27 2-4. Controlling Controllers 28 2-4-1. Enabling and Disabling 29 2-4-2. Top-down Constraint 30 2-4-3. No Internal Process Constraint 31 2-5. Delegation 32 2-5-1. Model of Delegation 33 2-5-2. Delegation Containment 34 2-6. Guidelines 35 2-6-1. Organize Once and Control 36 2-6-2. Avoid Name Collisions 37 3. Resource Distribution Models 38 3-1. Weights 39 3-2. Limits 40 3-3. Protections 41 3-4. Allocations 42 4. Interface Files 43 4-1. Format 44 4-2. Conventions 45 4-3. Core Interface Files 46 5. Controllers 47 5-1. CPU 48 5-1-1. CPU Interface Files 49 5-2. Memory 50 5-2-1. Memory Interface Files 51 5-2-2. Usage Guidelines 52 5-2-3. Memory Ownership 53 5-3. IO 54 5-3-1. IO Interface Files 55 5-3-2. Writeback 56 5-3-3. IO Latency 57 5-3-3-1. How IO Latency Throttling Works 58 5-3-3-2. IO Latency Interface Files 59 5-3-4. IO Priority 60 5-4. PID 61 5-4-1. PID Interface Files 62 5-5. Cpuset 63 5.5-1. Cpuset Interface Files 64 5-6. Device 65 5-7. RDMA 66 5-7-1. RDMA Interface Files 67 5-8. HugeTLB 68 5.8-1. HugeTLB Interface Files 69 5-9. Misc 70 5.9-1 Miscellaneous cgroup Interface Files 71 5.9-2 Migration and Ownership 72 5-10. Others 73 5-10-1. perf_event 74 5-N. Non-normative information 75 5-N-1. CPU controller root cgroup process behaviour 76 5-N-2. IO controller root cgroup process behaviour 77 6. Namespace 78 6-1. Basics 79 6-2. The Root and Views 80 6-3. Migration and setns(2) 81 6-4. Interaction with Other Namespaces 82 P. Information on Kernel Programming 83 P-1. Filesystem Support for Writeback 84 D. Deprecated v1 Core Features 85 R. Issues with v1 and Rationales for v2 86 R-1. Multiple Hierarchies 87 R-2. Thread Granularity 88 R-3. Competition Between Inner Nodes and Threads 89 R-4. Other Interface Issues 90 R-5. Controller Issues and Remedies 91 R-5-1. Memory 92 93 94Introduction 95============ 96 97Terminology 98----------- 99 100"cgroup" stands for "control group" and is never capitalized. The 101singular form is used to designate the whole feature and also as a 102qualifier as in "cgroup controllers". When explicitly referring to 103multiple individual control groups, the plural form "cgroups" is used. 104 105 106What is cgroup? 107--------------- 108 109cgroup is a mechanism to organize processes hierarchically and 110distribute system resources along the hierarchy in a controlled and 111configurable manner. 112 113cgroup is largely composed of two parts - the core and controllers. 114cgroup core is primarily responsible for hierarchically organizing 115processes. A cgroup controller is usually responsible for 116distributing a specific type of system resource along the hierarchy 117although there are utility controllers which serve purposes other than 118resource distribution. 119 120cgroups form a tree structure and every process in the system belongs 121to one and only one cgroup. All threads of a process belong to the 122same cgroup. On creation, all processes are put in the cgroup that 123the parent process belongs to at the time. A process can be migrated 124to another cgroup. Migration of a process doesn't affect already 125existing descendant processes. 126 127Following certain structural constraints, controllers may be enabled or 128disabled selectively on a cgroup. All controller behaviors are 129hierarchical - if a controller is enabled on a cgroup, it affects all 130processes which belong to the cgroups consisting the inclusive 131sub-hierarchy of the cgroup. When a controller is enabled on a nested 132cgroup, it always restricts the resource distribution further. The 133restrictions set closer to the root in the hierarchy can not be 134overridden from further away. 135 136 137Basic Operations 138================ 139 140Mounting 141-------- 142 143Unlike v1, cgroup v2 has only single hierarchy. The cgroup v2 144hierarchy can be mounted with the following mount command:: 145 146 # mount -t cgroup2 none $MOUNT_POINT 147 148cgroup2 filesystem has the magic number 0x63677270 ("cgrp"). All 149controllers which support v2 and are not bound to a v1 hierarchy are 150automatically bound to the v2 hierarchy and show up at the root. 151Controllers which are not in active use in the v2 hierarchy can be 152bound to other hierarchies. This allows mixing v2 hierarchy with the 153legacy v1 multiple hierarchies in a fully backward compatible way. 154 155A controller can be moved across hierarchies only after the controller 156is no longer referenced in its current hierarchy. Because per-cgroup 157controller states are destroyed asynchronously and controllers may 158have lingering references, a controller may not show up immediately on 159the v2 hierarchy after the final umount of the previous hierarchy. 160Similarly, a controller should be fully disabled to be moved out of 161the unified hierarchy and it may take some time for the disabled 162controller to become available for other hierarchies; furthermore, due 163to inter-controller dependencies, other controllers may need to be 164disabled too. 165 166While useful for development and manual configurations, moving 167controllers dynamically between the v2 and other hierarchies is 168strongly discouraged for production use. It is recommended to decide 169the hierarchies and controller associations before starting using the 170controllers after system boot. 171 172During transition to v2, system management software might still 173automount the v1 cgroup filesystem and so hijack all controllers 174during boot, before manual intervention is possible. To make testing 175and experimenting easier, the kernel parameter cgroup_no_v1= allows 176disabling controllers in v1 and make them always available in v2. 177 178cgroup v2 currently supports the following mount options. 179 180 nsdelegate 181 Consider cgroup namespaces as delegation boundaries. This 182 option is system wide and can only be set on mount or modified 183 through remount from the init namespace. The mount option is 184 ignored on non-init namespace mounts. Please refer to the 185 Delegation section for details. 186 187 memory_localevents 188 Only populate memory.events with data for the current cgroup, 189 and not any subtrees. This is legacy behaviour, the default 190 behaviour without this option is to include subtree counts. 191 This option is system wide and can only be set on mount or 192 modified through remount from the init namespace. The mount 193 option is ignored on non-init namespace mounts. 194 195 memory_recursiveprot 196 Recursively apply memory.min and memory.low protection to 197 entire subtrees, without requiring explicit downward 198 propagation into leaf cgroups. This allows protecting entire 199 subtrees from one another, while retaining free competition 200 within those subtrees. This should have been the default 201 behavior but is a mount-option to avoid regressing setups 202 relying on the original semantics (e.g. specifying bogusly 203 high 'bypass' protection values at higher tree levels). 204 205 206Organizing Processes and Threads 207-------------------------------- 208 209Processes 210~~~~~~~~~ 211 212Initially, only the root cgroup exists to which all processes belong. 213A child cgroup can be created by creating a sub-directory:: 214 215 # mkdir $CGROUP_NAME 216 217A given cgroup may have multiple child cgroups forming a tree 218structure. Each cgroup has a read-writable interface file 219"cgroup.procs". When read, it lists the PIDs of all processes which 220belong to the cgroup one-per-line. The PIDs are not ordered and the 221same PID may show up more than once if the process got moved to 222another cgroup and then back or the PID got recycled while reading. 223 224A process can be migrated into a cgroup by writing its PID to the 225target cgroup's "cgroup.procs" file. Only one process can be migrated 226on a single write(2) call. If a process is composed of multiple 227threads, writing the PID of any thread migrates all threads of the 228process. 229 230When a process forks a child process, the new process is born into the 231cgroup that the forking process belongs to at the time of the 232operation. After exit, a process stays associated with the cgroup 233that it belonged to at the time of exit until it's reaped; however, a 234zombie process does not appear in "cgroup.procs" and thus can't be 235moved to another cgroup. 236 237A cgroup which doesn't have any children or live processes can be 238destroyed by removing the directory. Note that a cgroup which doesn't 239have any children and is associated only with zombie processes is 240considered empty and can be removed:: 241 242 # rmdir $CGROUP_NAME 243 244"/proc/$PID/cgroup" lists a process's cgroup membership. If legacy 245cgroup is in use in the system, this file may contain multiple lines, 246one for each hierarchy. The entry for cgroup v2 is always in the 247format "0::$PATH":: 248 249 # cat /proc/842/cgroup 250 ... 251 0::/test-cgroup/test-cgroup-nested 252 253If the process becomes a zombie and the cgroup it was associated with 254is removed subsequently, " (deleted)" is appended to the path:: 255 256 # cat /proc/842/cgroup 257 ... 258 0::/test-cgroup/test-cgroup-nested (deleted) 259 260 261Threads 262~~~~~~~ 263 264cgroup v2 supports thread granularity for a subset of controllers to 265support use cases requiring hierarchical resource distribution across 266the threads of a group of processes. By default, all threads of a 267process belong to the same cgroup, which also serves as the resource 268domain to host resource consumptions which are not specific to a 269process or thread. The thread mode allows threads to be spread across 270a subtree while still maintaining the common resource domain for them. 271 272Controllers which support thread mode are called threaded controllers. 273The ones which don't are called domain controllers. 274 275Marking a cgroup threaded makes it join the resource domain of its 276parent as a threaded cgroup. The parent may be another threaded 277cgroup whose resource domain is further up in the hierarchy. The root 278of a threaded subtree, that is, the nearest ancestor which is not 279threaded, is called threaded domain or thread root interchangeably and 280serves as the resource domain for the entire subtree. 281 282Inside a threaded subtree, threads of a process can be put in 283different cgroups and are not subject to the no internal process 284constraint - threaded controllers can be enabled on non-leaf cgroups 285whether they have threads in them or not. 286 287As the threaded domain cgroup hosts all the domain resource 288consumptions of the subtree, it is considered to have internal 289resource consumptions whether there are processes in it or not and 290can't have populated child cgroups which aren't threaded. Because the 291root cgroup is not subject to no internal process constraint, it can 292serve both as a threaded domain and a parent to domain cgroups. 293 294The current operation mode or type of the cgroup is shown in the 295"cgroup.type" file which indicates whether the cgroup is a normal 296domain, a domain which is serving as the domain of a threaded subtree, 297or a threaded cgroup. 298 299On creation, a cgroup is always a domain cgroup and can be made 300threaded by writing "threaded" to the "cgroup.type" file. The 301operation is single direction:: 302 303 # echo threaded > cgroup.type 304 305Once threaded, the cgroup can't be made a domain again. To enable the 306thread mode, the following conditions must be met. 307 308- As the cgroup will join the parent's resource domain. The parent 309 must either be a valid (threaded) domain or a threaded cgroup. 310 311- When the parent is an unthreaded domain, it must not have any domain 312 controllers enabled or populated domain children. The root is 313 exempt from this requirement. 314 315Topology-wise, a cgroup can be in an invalid state. Please consider 316the following topology:: 317 318 A (threaded domain) - B (threaded) - C (domain, just created) 319 320C is created as a domain but isn't connected to a parent which can 321host child domains. C can't be used until it is turned into a 322threaded cgroup. "cgroup.type" file will report "domain (invalid)" in 323these cases. Operations which fail due to invalid topology use 324EOPNOTSUPP as the errno. 325 326A domain cgroup is turned into a threaded domain when one of its child 327cgroup becomes threaded or threaded controllers are enabled in the 328"cgroup.subtree_control" file while there are processes in the cgroup. 329A threaded domain reverts to a normal domain when the conditions 330clear. 331 332When read, "cgroup.threads" contains the list of the thread IDs of all 333threads in the cgroup. Except that the operations are per-thread 334instead of per-process, "cgroup.threads" has the same format and 335behaves the same way as "cgroup.procs". While "cgroup.threads" can be 336written to in any cgroup, as it can only move threads inside the same 337threaded domain, its operations are confined inside each threaded 338subtree. 339 340The threaded domain cgroup serves as the resource domain for the whole 341subtree, and, while the threads can be scattered across the subtree, 342all the processes are considered to be in the threaded domain cgroup. 343"cgroup.procs" in a threaded domain cgroup contains the PIDs of all 344processes in the subtree and is not readable in the subtree proper. 345However, "cgroup.procs" can be written to from anywhere in the subtree 346to migrate all threads of the matching process to the cgroup. 347 348Only threaded controllers can be enabled in a threaded subtree. When 349a threaded controller is enabled inside a threaded subtree, it only 350accounts for and controls resource consumptions associated with the 351threads in the cgroup and its descendants. All consumptions which 352aren't tied to a specific thread belong to the threaded domain cgroup. 353 354Because a threaded subtree is exempt from no internal process 355constraint, a threaded controller must be able to handle competition 356between threads in a non-leaf cgroup and its child cgroups. Each 357threaded controller defines how such competitions are handled. 358 359 360[Un]populated Notification 361-------------------------- 362 363Each non-root cgroup has a "cgroup.events" file which contains 364"populated" field indicating whether the cgroup's sub-hierarchy has 365live processes in it. Its value is 0 if there is no live process in 366the cgroup and its descendants; otherwise, 1. poll and [id]notify 367events are triggered when the value changes. This can be used, for 368example, to start a clean-up operation after all processes of a given 369sub-hierarchy have exited. The populated state updates and 370notifications are recursive. Consider the following sub-hierarchy 371where the numbers in the parentheses represent the numbers of processes 372in each cgroup:: 373 374 A(4) - B(0) - C(1) 375 \ D(0) 376 377A, B and C's "populated" fields would be 1 while D's 0. After the one 378process in C exits, B and C's "populated" fields would flip to "0" and 379file modified events will be generated on the "cgroup.events" files of 380both cgroups. 381 382 383Controlling Controllers 384----------------------- 385 386Enabling and Disabling 387~~~~~~~~~~~~~~~~~~~~~~ 388 389Each cgroup has a "cgroup.controllers" file which lists all 390controllers available for the cgroup to enable:: 391 392 # cat cgroup.controllers 393 cpu io memory 394 395No controller is enabled by default. Controllers can be enabled and 396disabled by writing to the "cgroup.subtree_control" file:: 397 398 # echo "+cpu +memory -io" > cgroup.subtree_control 399 400Only controllers which are listed in "cgroup.controllers" can be 401enabled. When multiple operations are specified as above, either they 402all succeed or fail. If multiple operations on the same controller 403are specified, the last one is effective. 404 405Enabling a controller in a cgroup indicates that the distribution of 406the target resource across its immediate children will be controlled. 407Consider the following sub-hierarchy. The enabled controllers are 408listed in parentheses:: 409 410 A(cpu,memory) - B(memory) - C() 411 \ D() 412 413As A has "cpu" and "memory" enabled, A will control the distribution 414of CPU cycles and memory to its children, in this case, B. As B has 415"memory" enabled but not "CPU", C and D will compete freely on CPU 416cycles but their division of memory available to B will be controlled. 417 418As a controller regulates the distribution of the target resource to 419the cgroup's children, enabling it creates the controller's interface 420files in the child cgroups. In the above example, enabling "cpu" on B 421would create the "cpu." prefixed controller interface files in C and 422D. Likewise, disabling "memory" from B would remove the "memory." 423prefixed controller interface files from C and D. This means that the 424controller interface files - anything which doesn't start with 425"cgroup." are owned by the parent rather than the cgroup itself. 426 427 428Top-down Constraint 429~~~~~~~~~~~~~~~~~~~ 430 431Resources are distributed top-down and a cgroup can further distribute 432a resource only if the resource has been distributed to it from the 433parent. This means that all non-root "cgroup.subtree_control" files 434can only contain controllers which are enabled in the parent's 435"cgroup.subtree_control" file. A controller can be enabled only if 436the parent has the controller enabled and a controller can't be 437disabled if one or more children have it enabled. 438 439 440No Internal Process Constraint 441~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 442 443Non-root cgroups can distribute domain resources to their children 444only when they don't have any processes of their own. In other words, 445only domain cgroups which don't contain any processes can have domain 446controllers enabled in their "cgroup.subtree_control" files. 447 448This guarantees that, when a domain controller is looking at the part 449of the hierarchy which has it enabled, processes are always only on 450the leaves. This rules out situations where child cgroups compete 451against internal processes of the parent. 452 453The root cgroup is exempt from this restriction. Root contains 454processes and anonymous resource consumption which can't be associated 455with any other cgroups and requires special treatment from most 456controllers. How resource consumption in the root cgroup is governed 457is up to each controller (for more information on this topic please 458refer to the Non-normative information section in the Controllers 459chapter). 460 461Note that the restriction doesn't get in the way if there is no 462enabled controller in the cgroup's "cgroup.subtree_control". This is 463important as otherwise it wouldn't be possible to create children of a 464populated cgroup. To control resource distribution of a cgroup, the 465cgroup must create children and transfer all its processes to the 466children before enabling controllers in its "cgroup.subtree_control" 467file. 468 469 470Delegation 471---------- 472 473Model of Delegation 474~~~~~~~~~~~~~~~~~~~ 475 476A cgroup can be delegated in two ways. First, to a less privileged 477user by granting write access of the directory and its "cgroup.procs", 478"cgroup.threads" and "cgroup.subtree_control" files to the user. 479Second, if the "nsdelegate" mount option is set, automatically to a 480cgroup namespace on namespace creation. 481 482Because the resource control interface files in a given directory 483control the distribution of the parent's resources, the delegatee 484shouldn't be allowed to write to them. For the first method, this is 485achieved by not granting access to these files. For the second, the 486kernel rejects writes to all files other than "cgroup.procs" and 487"cgroup.subtree_control" on a namespace root from inside the 488namespace. 489 490The end results are equivalent for both delegation types. Once 491delegated, the user can build sub-hierarchy under the directory, 492organize processes inside it as it sees fit and further distribute the 493resources it received from the parent. The limits and other settings 494of all resource controllers are hierarchical and regardless of what 495happens in the delegated sub-hierarchy, nothing can escape the 496resource restrictions imposed by the parent. 497 498Currently, cgroup doesn't impose any restrictions on the number of 499cgroups in or nesting depth of a delegated sub-hierarchy; however, 500this may be limited explicitly in the future. 501 502 503Delegation Containment 504~~~~~~~~~~~~~~~~~~~~~~ 505 506A delegated sub-hierarchy is contained in the sense that processes 507can't be moved into or out of the sub-hierarchy by the delegatee. 508 509For delegations to a less privileged user, this is achieved by 510requiring the following conditions for a process with a non-root euid 511to migrate a target process into a cgroup by writing its PID to the 512"cgroup.procs" file. 513 514- The writer must have write access to the "cgroup.procs" file. 515 516- The writer must have write access to the "cgroup.procs" file of the 517 common ancestor of the source and destination cgroups. 518 519The above two constraints ensure that while a delegatee may migrate 520processes around freely in the delegated sub-hierarchy it can't pull 521in from or push out to outside the sub-hierarchy. 522 523For an example, let's assume cgroups C0 and C1 have been delegated to 524user U0 who created C00, C01 under C0 and C10 under C1 as follows and 525all processes under C0 and C1 belong to U0:: 526 527 ~~~~~~~~~~~~~ - C0 - C00 528 ~ cgroup ~ \ C01 529 ~ hierarchy ~ 530 ~~~~~~~~~~~~~ - C1 - C10 531 532Let's also say U0 wants to write the PID of a process which is 533currently in C10 into "C00/cgroup.procs". U0 has write access to the 534file; however, the common ancestor of the source cgroup C10 and the 535destination cgroup C00 is above the points of delegation and U0 would 536not have write access to its "cgroup.procs" files and thus the write 537will be denied with -EACCES. 538 539For delegations to namespaces, containment is achieved by requiring 540that both the source and destination cgroups are reachable from the 541namespace of the process which is attempting the migration. If either 542is not reachable, the migration is rejected with -ENOENT. 543 544 545Guidelines 546---------- 547 548Organize Once and Control 549~~~~~~~~~~~~~~~~~~~~~~~~~ 550 551Migrating a process across cgroups is a relatively expensive operation 552and stateful resources such as memory are not moved together with the 553process. This is an explicit design decision as there often exist 554inherent trade-offs between migration and various hot paths in terms 555of synchronization cost. 556 557As such, migrating processes across cgroups frequently as a means to 558apply different resource restrictions is discouraged. A workload 559should be assigned to a cgroup according to the system's logical and 560resource structure once on start-up. Dynamic adjustments to resource 561distribution can be made by changing controller configuration through 562the interface files. 563 564 565Avoid Name Collisions 566~~~~~~~~~~~~~~~~~~~~~ 567 568Interface files for a cgroup and its children cgroups occupy the same 569directory and it is possible to create children cgroups which collide 570with interface files. 571 572All cgroup core interface files are prefixed with "cgroup." and each 573controller's interface files are prefixed with the controller name and 574a dot. A controller's name is composed of lower case alphabets and 575'_'s but never begins with an '_' so it can be used as the prefix 576character for collision avoidance. Also, interface file names won't 577start or end with terms which are often used in categorizing workloads 578such as job, service, slice, unit or workload. 579 580cgroup doesn't do anything to prevent name collisions and it's the 581user's responsibility to avoid them. 582 583 584Resource Distribution Models 585============================ 586 587cgroup controllers implement several resource distribution schemes 588depending on the resource type and expected use cases. This section 589describes major schemes in use along with their expected behaviors. 590 591 592Weights 593------- 594 595A parent's resource is distributed by adding up the weights of all 596active children and giving each the fraction matching the ratio of its 597weight against the sum. As only children which can make use of the 598resource at the moment participate in the distribution, this is 599work-conserving. Due to the dynamic nature, this model is usually 600used for stateless resources. 601 602All weights are in the range [1, 10000] with the default at 100. This 603allows symmetric multiplicative biases in both directions at fine 604enough granularity while staying in the intuitive range. 605 606As long as the weight is in range, all configuration combinations are 607valid and there is no reason to reject configuration changes or 608process migrations. 609 610"cpu.weight" proportionally distributes CPU cycles to active children 611and is an example of this type. 612 613 614Limits 615------ 616 617A child can only consume upto the configured amount of the resource. 618Limits can be over-committed - the sum of the limits of children can 619exceed the amount of resource available to the parent. 620 621Limits are in the range [0, max] and defaults to "max", which is noop. 622 623As limits can be over-committed, all configuration combinations are 624valid and there is no reason to reject configuration changes or 625process migrations. 626 627"io.max" limits the maximum BPS and/or IOPS that a cgroup can consume 628on an IO device and is an example of this type. 629 630 631Protections 632----------- 633 634A cgroup is protected upto the configured amount of the resource 635as long as the usages of all its ancestors are under their 636protected levels. Protections can be hard guarantees or best effort 637soft boundaries. Protections can also be over-committed in which case 638only upto the amount available to the parent is protected among 639children. 640 641Protections are in the range [0, max] and defaults to 0, which is 642noop. 643 644As protections can be over-committed, all configuration combinations 645are valid and there is no reason to reject configuration changes or 646process migrations. 647 648"memory.low" implements best-effort memory protection and is an 649example of this type. 650 651 652Allocations 653----------- 654 655A cgroup is exclusively allocated a certain amount of a finite 656resource. Allocations can't be over-committed - the sum of the 657allocations of children can not exceed the amount of resource 658available to the parent. 659 660Allocations are in the range [0, max] and defaults to 0, which is no 661resource. 662 663As allocations can't be over-committed, some configuration 664combinations are invalid and should be rejected. Also, if the 665resource is mandatory for execution of processes, process migrations 666may be rejected. 667 668"cpu.rt.max" hard-allocates realtime slices and is an example of this 669type. 670 671 672Interface Files 673=============== 674 675Format 676------ 677 678All interface files should be in one of the following formats whenever 679possible:: 680 681 New-line separated values 682 (when only one value can be written at once) 683 684 VAL0\n 685 VAL1\n 686 ... 687 688 Space separated values 689 (when read-only or multiple values can be written at once) 690 691 VAL0 VAL1 ...\n 692 693 Flat keyed 694 695 KEY0 VAL0\n 696 KEY1 VAL1\n 697 ... 698 699 Nested keyed 700 701 KEY0 SUB_KEY0=VAL00 SUB_KEY1=VAL01... 702 KEY1 SUB_KEY0=VAL10 SUB_KEY1=VAL11... 703 ... 704 705For a writable file, the format for writing should generally match 706reading; however, controllers may allow omitting later fields or 707implement restricted shortcuts for most common use cases. 708 709For both flat and nested keyed files, only the values for a single key 710can be written at a time. For nested keyed files, the sub key pairs 711may be specified in any order and not all pairs have to be specified. 712 713 714Conventions 715----------- 716 717- Settings for a single feature should be contained in a single file. 718 719- The root cgroup should be exempt from resource control and thus 720 shouldn't have resource control interface files. 721 722- The default time unit is microseconds. If a different unit is ever 723 used, an explicit unit suffix must be present. 724 725- A parts-per quantity should use a percentage decimal with at least 726 two digit fractional part - e.g. 13.40. 727 728- If a controller implements weight based resource distribution, its 729 interface file should be named "weight" and have the range [1, 730 10000] with 100 as the default. The values are chosen to allow 731 enough and symmetric bias in both directions while keeping it 732 intuitive (the default is 100%). 733 734- If a controller implements an absolute resource guarantee and/or 735 limit, the interface files should be named "min" and "max" 736 respectively. If a controller implements best effort resource 737 guarantee and/or limit, the interface files should be named "low" 738 and "high" respectively. 739 740 In the above four control files, the special token "max" should be 741 used to represent upward infinity for both reading and writing. 742 743- If a setting has a configurable default value and keyed specific 744 overrides, the default entry should be keyed with "default" and 745 appear as the first entry in the file. 746 747 The default value can be updated by writing either "default $VAL" or 748 "$VAL". 749 750 When writing to update a specific override, "default" can be used as 751 the value to indicate removal of the override. Override entries 752 with "default" as the value must not appear when read. 753 754 For example, a setting which is keyed by major:minor device numbers 755 with integer values may look like the following:: 756 757 # cat cgroup-example-interface-file 758 default 150 759 8:0 300 760 761 The default value can be updated by:: 762 763 # echo 125 > cgroup-example-interface-file 764 765 or:: 766 767 # echo "default 125" > cgroup-example-interface-file 768 769 An override can be set by:: 770 771 # echo "8:16 170" > cgroup-example-interface-file 772 773 and cleared by:: 774 775 # echo "8:0 default" > cgroup-example-interface-file 776 # cat cgroup-example-interface-file 777 default 125 778 8:16 170 779 780- For events which are not very high frequency, an interface file 781 "events" should be created which lists event key value pairs. 782 Whenever a notifiable event happens, file modified event should be 783 generated on the file. 784 785 786Core Interface Files 787-------------------- 788 789All cgroup core files are prefixed with "cgroup." 790 791 cgroup.type 792 A read-write single value file which exists on non-root 793 cgroups. 794 795 When read, it indicates the current type of the cgroup, which 796 can be one of the following values. 797 798 - "domain" : A normal valid domain cgroup. 799 800 - "domain threaded" : A threaded domain cgroup which is 801 serving as the root of a threaded subtree. 802 803 - "domain invalid" : A cgroup which is in an invalid state. 804 It can't be populated or have controllers enabled. It may 805 be allowed to become a threaded cgroup. 806 807 - "threaded" : A threaded cgroup which is a member of a 808 threaded subtree. 809 810 A cgroup can be turned into a threaded cgroup by writing 811 "threaded" to this file. 812 813 cgroup.procs 814 A read-write new-line separated values file which exists on 815 all cgroups. 816 817 When read, it lists the PIDs of all processes which belong to 818 the cgroup one-per-line. The PIDs are not ordered and the 819 same PID may show up more than once if the process got moved 820 to another cgroup and then back or the PID got recycled while 821 reading. 822 823 A PID can be written to migrate the process associated with 824 the PID to the cgroup. The writer should match all of the 825 following conditions. 826 827 - It must have write access to the "cgroup.procs" file. 828 829 - It must have write access to the "cgroup.procs" file of the 830 common ancestor of the source and destination cgroups. 831 832 When delegating a sub-hierarchy, write access to this file 833 should be granted along with the containing directory. 834 835 In a threaded cgroup, reading this file fails with EOPNOTSUPP 836 as all the processes belong to the thread root. Writing is 837 supported and moves every thread of the process to the cgroup. 838 839 cgroup.threads 840 A read-write new-line separated values file which exists on 841 all cgroups. 842 843 When read, it lists the TIDs of all threads which belong to 844 the cgroup one-per-line. The TIDs are not ordered and the 845 same TID may show up more than once if the thread got moved to 846 another cgroup and then back or the TID got recycled while 847 reading. 848 849 A TID can be written to migrate the thread associated with the 850 TID to the cgroup. The writer should match all of the 851 following conditions. 852 853 - It must have write access to the "cgroup.threads" file. 854 855 - The cgroup that the thread is currently in must be in the 856 same resource domain as the destination cgroup. 857 858 - It must have write access to the "cgroup.procs" file of the 859 common ancestor of the source and destination cgroups. 860 861 When delegating a sub-hierarchy, write access to this file 862 should be granted along with the containing directory. 863 864 cgroup.controllers 865 A read-only space separated values file which exists on all 866 cgroups. 867 868 It shows space separated list of all controllers available to 869 the cgroup. The controllers are not ordered. 870 871 cgroup.subtree_control 872 A read-write space separated values file which exists on all 873 cgroups. Starts out empty. 874 875 When read, it shows space separated list of the controllers 876 which are enabled to control resource distribution from the 877 cgroup to its children. 878 879 Space separated list of controllers prefixed with '+' or '-' 880 can be written to enable or disable controllers. A controller 881 name prefixed with '+' enables the controller and '-' 882 disables. If a controller appears more than once on the list, 883 the last one is effective. When multiple enable and disable 884 operations are specified, either all succeed or all fail. 885 886 cgroup.events 887 A read-only flat-keyed file which exists on non-root cgroups. 888 The following entries are defined. Unless specified 889 otherwise, a value change in this file generates a file 890 modified event. 891 892 populated 893 1 if the cgroup or its descendants contains any live 894 processes; otherwise, 0. 895 frozen 896 1 if the cgroup is frozen; otherwise, 0. 897 898 cgroup.max.descendants 899 A read-write single value files. The default is "max". 900 901 Maximum allowed number of descent cgroups. 902 If the actual number of descendants is equal or larger, 903 an attempt to create a new cgroup in the hierarchy will fail. 904 905 cgroup.max.depth 906 A read-write single value files. The default is "max". 907 908 Maximum allowed descent depth below the current cgroup. 909 If the actual descent depth is equal or larger, 910 an attempt to create a new child cgroup will fail. 911 912 cgroup.stat 913 A read-only flat-keyed file with the following entries: 914 915 nr_descendants 916 Total number of visible descendant cgroups. 917 918 nr_dying_descendants 919 Total number of dying descendant cgroups. A cgroup becomes 920 dying after being deleted by a user. The cgroup will remain 921 in dying state for some time undefined time (which can depend 922 on system load) before being completely destroyed. 923 924 A process can't enter a dying cgroup under any circumstances, 925 a dying cgroup can't revive. 926 927 A dying cgroup can consume system resources not exceeding 928 limits, which were active at the moment of cgroup deletion. 929 930 cgroup.freeze 931 A read-write single value file which exists on non-root cgroups. 932 Allowed values are "0" and "1". The default is "0". 933 934 Writing "1" to the file causes freezing of the cgroup and all 935 descendant cgroups. This means that all belonging processes will 936 be stopped and will not run until the cgroup will be explicitly 937 unfrozen. Freezing of the cgroup may take some time; when this action 938 is completed, the "frozen" value in the cgroup.events control file 939 will be updated to "1" and the corresponding notification will be 940 issued. 941 942 A cgroup can be frozen either by its own settings, or by settings 943 of any ancestor cgroups. If any of ancestor cgroups is frozen, the 944 cgroup will remain frozen. 945 946 Processes in the frozen cgroup can be killed by a fatal signal. 947 They also can enter and leave a frozen cgroup: either by an explicit 948 move by a user, or if freezing of the cgroup races with fork(). 949 If a process is moved to a frozen cgroup, it stops. If a process is 950 moved out of a frozen cgroup, it becomes running. 951 952 Frozen status of a cgroup doesn't affect any cgroup tree operations: 953 it's possible to delete a frozen (and empty) cgroup, as well as 954 create new sub-cgroups. 955 956 cgroup.kill 957 A write-only single value file which exists in non-root cgroups. 958 The only allowed value is "1". 959 960 Writing "1" to the file causes the cgroup and all descendant cgroups to 961 be killed. This means that all processes located in the affected cgroup 962 tree will be killed via SIGKILL. 963 964 Killing a cgroup tree will deal with concurrent forks appropriately and 965 is protected against migrations. 966 967 In a threaded cgroup, writing this file fails with EOPNOTSUPP as 968 killing cgroups is a process directed operation, i.e. it affects 969 the whole thread-group. 970 971Controllers 972=========== 973 974.. _cgroup-v2-cpu: 975 976CPU 977--- 978 979The "cpu" controllers regulates distribution of CPU cycles. This 980controller implements weight and absolute bandwidth limit models for 981normal scheduling policy and absolute bandwidth allocation model for 982realtime scheduling policy. 983 984In all the above models, cycles distribution is defined only on a temporal 985base and it does not account for the frequency at which tasks are executed. 986The (optional) utilization clamping support allows to hint the schedutil 987cpufreq governor about the minimum desired frequency which should always be 988provided by a CPU, as well as the maximum desired frequency, which should not 989be exceeded by a CPU. 990 991WARNING: cgroup2 doesn't yet support control of realtime processes and 992the cpu controller can only be enabled when all RT processes are in 993the root cgroup. Be aware that system management software may already 994have placed RT processes into nonroot cgroups during the system boot 995process, and these processes may need to be moved to the root cgroup 996before the cpu controller can be enabled. 997 998 999CPU Interface Files 1000~~~~~~~~~~~~~~~~~~~ 1001 1002All time durations are in microseconds. 1003 1004 cpu.stat 1005 A read-only flat-keyed file. 1006 This file exists whether the controller is enabled or not. 1007 1008 It always reports the following three stats: 1009 1010 - usage_usec 1011 - user_usec 1012 - system_usec 1013 1014 and the following three when the controller is enabled: 1015 1016 - nr_periods 1017 - nr_throttled 1018 - throttled_usec 1019 - nr_bursts 1020 - burst_usec 1021 1022 cpu.weight 1023 A read-write single value file which exists on non-root 1024 cgroups. The default is "100". 1025 1026 The weight in the range [1, 10000]. 1027 1028 cpu.weight.nice 1029 A read-write single value file which exists on non-root 1030 cgroups. The default is "0". 1031 1032 The nice value is in the range [-20, 19]. 1033 1034 This interface file is an alternative interface for 1035 "cpu.weight" and allows reading and setting weight using the 1036 same values used by nice(2). Because the range is smaller and 1037 granularity is coarser for the nice values, the read value is 1038 the closest approximation of the current weight. 1039 1040 cpu.max 1041 A read-write two value file which exists on non-root cgroups. 1042 The default is "max 100000". 1043 1044 The maximum bandwidth limit. It's in the following format:: 1045 1046 $MAX $PERIOD 1047 1048 which indicates that the group may consume upto $MAX in each 1049 $PERIOD duration. "max" for $MAX indicates no limit. If only 1050 one number is written, $MAX is updated. 1051 1052 cpu.max.burst 1053 A read-write single value file which exists on non-root 1054 cgroups. The default is "0". 1055 1056 The burst in the range [0, $MAX]. 1057 1058 cpu.pressure 1059 A read-write nested-keyed file. 1060 1061 Shows pressure stall information for CPU. See 1062 :ref:`Documentation/accounting/psi.rst <psi>` for details. 1063 1064 cpu.uclamp.min 1065 A read-write single value file which exists on non-root cgroups. 1066 The default is "0", i.e. no utilization boosting. 1067 1068 The requested minimum utilization (protection) as a percentage 1069 rational number, e.g. 12.34 for 12.34%. 1070 1071 This interface allows reading and setting minimum utilization clamp 1072 values similar to the sched_setattr(2). This minimum utilization 1073 value is used to clamp the task specific minimum utilization clamp. 1074 1075 The requested minimum utilization (protection) is always capped by 1076 the current value for the maximum utilization (limit), i.e. 1077 `cpu.uclamp.max`. 1078 1079 cpu.uclamp.max 1080 A read-write single value file which exists on non-root cgroups. 1081 The default is "max". i.e. no utilization capping 1082 1083 The requested maximum utilization (limit) as a percentage rational 1084 number, e.g. 98.76 for 98.76%. 1085 1086 This interface allows reading and setting maximum utilization clamp 1087 values similar to the sched_setattr(2). This maximum utilization 1088 value is used to clamp the task specific maximum utilization clamp. 1089 1090 1091 1092Memory 1093------ 1094 1095The "memory" controller regulates distribution of memory. Memory is 1096stateful and implements both limit and protection models. Due to the 1097intertwining between memory usage and reclaim pressure and the 1098stateful nature of memory, the distribution model is relatively 1099complex. 1100 1101While not completely water-tight, all major memory usages by a given 1102cgroup are tracked so that the total memory consumption can be 1103accounted and controlled to a reasonable extent. Currently, the 1104following types of memory usages are tracked. 1105 1106- Userland memory - page cache and anonymous memory. 1107 1108- Kernel data structures such as dentries and inodes. 1109 1110- TCP socket buffers. 1111 1112The above list may expand in the future for better coverage. 1113 1114 1115Memory Interface Files 1116~~~~~~~~~~~~~~~~~~~~~~ 1117 1118All memory amounts are in bytes. If a value which is not aligned to 1119PAGE_SIZE is written, the value may be rounded up to the closest 1120PAGE_SIZE multiple when read back. 1121 1122 memory.current 1123 A read-only single value file which exists on non-root 1124 cgroups. 1125 1126 The total amount of memory currently being used by the cgroup 1127 and its descendants. 1128 1129 memory.min 1130 A read-write single value file which exists on non-root 1131 cgroups. The default is "0". 1132 1133 Hard memory protection. If the memory usage of a cgroup 1134 is within its effective min boundary, the cgroup's memory 1135 won't be reclaimed under any conditions. If there is no 1136 unprotected reclaimable memory available, OOM killer 1137 is invoked. Above the effective min boundary (or 1138 effective low boundary if it is higher), pages are reclaimed 1139 proportionally to the overage, reducing reclaim pressure for 1140 smaller overages. 1141 1142 Effective min boundary is limited by memory.min values of 1143 all ancestor cgroups. If there is memory.min overcommitment 1144 (child cgroup or cgroups are requiring more protected memory 1145 than parent will allow), then each child cgroup will get 1146 the part of parent's protection proportional to its 1147 actual memory usage below memory.min. 1148 1149 Putting more memory than generally available under this 1150 protection is discouraged and may lead to constant OOMs. 1151 1152 If a memory cgroup is not populated with processes, 1153 its memory.min is ignored. 1154 1155 memory.low 1156 A read-write single value file which exists on non-root 1157 cgroups. The default is "0". 1158 1159 Best-effort memory protection. If the memory usage of a 1160 cgroup is within its effective low boundary, the cgroup's 1161 memory won't be reclaimed unless there is no reclaimable 1162 memory available in unprotected cgroups. 1163 Above the effective low boundary (or 1164 effective min boundary if it is higher), pages are reclaimed 1165 proportionally to the overage, reducing reclaim pressure for 1166 smaller overages. 1167 1168 Effective low boundary is limited by memory.low values of 1169 all ancestor cgroups. If there is memory.low overcommitment 1170 (child cgroup or cgroups are requiring more protected memory 1171 than parent will allow), then each child cgroup will get 1172 the part of parent's protection proportional to its 1173 actual memory usage below memory.low. 1174 1175 Putting more memory than generally available under this 1176 protection is discouraged. 1177 1178 memory.high 1179 A read-write single value file which exists on non-root 1180 cgroups. The default is "max". 1181 1182 Memory usage throttle limit. This is the main mechanism to 1183 control memory usage of a cgroup. If a cgroup's usage goes 1184 over the high boundary, the processes of the cgroup are 1185 throttled and put under heavy reclaim pressure. 1186 1187 Going over the high limit never invokes the OOM killer and 1188 under extreme conditions the limit may be breached. 1189 1190 memory.max 1191 A read-write single value file which exists on non-root 1192 cgroups. The default is "max". 1193 1194 Memory usage hard limit. This is the final protection 1195 mechanism. If a cgroup's memory usage reaches this limit and 1196 can't be reduced, the OOM killer is invoked in the cgroup. 1197 Under certain circumstances, the usage may go over the limit 1198 temporarily. 1199 1200 In default configuration regular 0-order allocations always 1201 succeed unless OOM killer chooses current task as a victim. 1202 1203 Some kinds of allocations don't invoke the OOM killer. 1204 Caller could retry them differently, return into userspace 1205 as -ENOMEM or silently ignore in cases like disk readahead. 1206 1207 This is the ultimate protection mechanism. As long as the 1208 high limit is used and monitored properly, this limit's 1209 utility is limited to providing the final safety net. 1210 1211 memory.oom.group 1212 A read-write single value file which exists on non-root 1213 cgroups. The default value is "0". 1214 1215 Determines whether the cgroup should be treated as 1216 an indivisible workload by the OOM killer. If set, 1217 all tasks belonging to the cgroup or to its descendants 1218 (if the memory cgroup is not a leaf cgroup) are killed 1219 together or not at all. This can be used to avoid 1220 partial kills to guarantee workload integrity. 1221 1222 Tasks with the OOM protection (oom_score_adj set to -1000) 1223 are treated as an exception and are never killed. 1224 1225 If the OOM killer is invoked in a cgroup, it's not going 1226 to kill any tasks outside of this cgroup, regardless 1227 memory.oom.group values of ancestor cgroups. 1228 1229 memory.events 1230 A read-only flat-keyed file which exists on non-root cgroups. 1231 The following entries are defined. Unless specified 1232 otherwise, a value change in this file generates a file 1233 modified event. 1234 1235 Note that all fields in this file are hierarchical and the 1236 file modified event can be generated due to an event down the 1237 hierarchy. For the local events at the cgroup level see 1238 memory.events.local. 1239 1240 low 1241 The number of times the cgroup is reclaimed due to 1242 high memory pressure even though its usage is under 1243 the low boundary. This usually indicates that the low 1244 boundary is over-committed. 1245 1246 high 1247 The number of times processes of the cgroup are 1248 throttled and routed to perform direct memory reclaim 1249 because the high memory boundary was exceeded. For a 1250 cgroup whose memory usage is capped by the high limit 1251 rather than global memory pressure, this event's 1252 occurrences are expected. 1253 1254 max 1255 The number of times the cgroup's memory usage was 1256 about to go over the max boundary. If direct reclaim 1257 fails to bring it down, the cgroup goes to OOM state. 1258 1259 oom 1260 The number of time the cgroup's memory usage was 1261 reached the limit and allocation was about to fail. 1262 1263 This event is not raised if the OOM killer is not 1264 considered as an option, e.g. for failed high-order 1265 allocations or if caller asked to not retry attempts. 1266 1267 oom_kill 1268 The number of processes belonging to this cgroup 1269 killed by any kind of OOM killer. 1270 1271 oom_group_kill 1272 The number of times a group OOM has occurred. 1273 1274 memory.events.local 1275 Similar to memory.events but the fields in the file are local 1276 to the cgroup i.e. not hierarchical. The file modified event 1277 generated on this file reflects only the local events. 1278 1279 memory.stat 1280 A read-only flat-keyed file which exists on non-root cgroups. 1281 1282 This breaks down the cgroup's memory footprint into different 1283 types of memory, type-specific details, and other information 1284 on the state and past events of the memory management system. 1285 1286 All memory amounts are in bytes. 1287 1288 The entries are ordered to be human readable, and new entries 1289 can show up in the middle. Don't rely on items remaining in a 1290 fixed position; use the keys to look up specific values! 1291 1292 If the entry has no per-node counter (or not show in the 1293 memory.numa_stat). We use 'npn' (non-per-node) as the tag 1294 to indicate that it will not show in the memory.numa_stat. 1295 1296 anon 1297 Amount of memory used in anonymous mappings such as 1298 brk(), sbrk(), and mmap(MAP_ANONYMOUS) 1299 1300 file 1301 Amount of memory used to cache filesystem data, 1302 including tmpfs and shared memory. 1303 1304 kernel_stack 1305 Amount of memory allocated to kernel stacks. 1306 1307 pagetables 1308 Amount of memory allocated for page tables. 1309 1310 percpu (npn) 1311 Amount of memory used for storing per-cpu kernel 1312 data structures. 1313 1314 sock (npn) 1315 Amount of memory used in network transmission buffers 1316 1317 vmalloc (npn) 1318 Amount of memory used for vmap backed memory. 1319 1320 shmem 1321 Amount of cached filesystem data that is swap-backed, 1322 such as tmpfs, shm segments, shared anonymous mmap()s 1323 1324 file_mapped 1325 Amount of cached filesystem data mapped with mmap() 1326 1327 file_dirty 1328 Amount of cached filesystem data that was modified but 1329 not yet written back to disk 1330 1331 file_writeback 1332 Amount of cached filesystem data that was modified and 1333 is currently being written back to disk 1334 1335 swapcached 1336 Amount of swap cached in memory. The swapcache is accounted 1337 against both memory and swap usage. 1338 1339 anon_thp 1340 Amount of memory used in anonymous mappings backed by 1341 transparent hugepages 1342 1343 file_thp 1344 Amount of cached filesystem data backed by transparent 1345 hugepages 1346 1347 shmem_thp 1348 Amount of shm, tmpfs, shared anonymous mmap()s backed by 1349 transparent hugepages 1350 1351 inactive_anon, active_anon, inactive_file, active_file, unevictable 1352 Amount of memory, swap-backed and filesystem-backed, 1353 on the internal memory management lists used by the 1354 page reclaim algorithm. 1355 1356 As these represent internal list state (eg. shmem pages are on anon 1357 memory management lists), inactive_foo + active_foo may not be equal to 1358 the value for the foo counter, since the foo counter is type-based, not 1359 list-based. 1360 1361 slab_reclaimable 1362 Part of "slab" that might be reclaimed, such as 1363 dentries and inodes. 1364 1365 slab_unreclaimable 1366 Part of "slab" that cannot be reclaimed on memory 1367 pressure. 1368 1369 slab (npn) 1370 Amount of memory used for storing in-kernel data 1371 structures. 1372 1373 workingset_refault_anon 1374 Number of refaults of previously evicted anonymous pages. 1375 1376 workingset_refault_file 1377 Number of refaults of previously evicted file pages. 1378 1379 workingset_activate_anon 1380 Number of refaulted anonymous pages that were immediately 1381 activated. 1382 1383 workingset_activate_file 1384 Number of refaulted file pages that were immediately activated. 1385 1386 workingset_restore_anon 1387 Number of restored anonymous pages which have been detected as 1388 an active workingset before they got reclaimed. 1389 1390 workingset_restore_file 1391 Number of restored file pages which have been detected as an 1392 active workingset before they got reclaimed. 1393 1394 workingset_nodereclaim 1395 Number of times a shadow node has been reclaimed 1396 1397 pgfault (npn) 1398 Total number of page faults incurred 1399 1400 pgmajfault (npn) 1401 Number of major page faults incurred 1402 1403 pgrefill (npn) 1404 Amount of scanned pages (in an active LRU list) 1405 1406 pgscan (npn) 1407 Amount of scanned pages (in an inactive LRU list) 1408 1409 pgsteal (npn) 1410 Amount of reclaimed pages 1411 1412 pgactivate (npn) 1413 Amount of pages moved to the active LRU list 1414 1415 pgdeactivate (npn) 1416 Amount of pages moved to the inactive LRU list 1417 1418 pglazyfree (npn) 1419 Amount of pages postponed to be freed under memory pressure 1420 1421 pglazyfreed (npn) 1422 Amount of reclaimed lazyfree pages 1423 1424 thp_fault_alloc (npn) 1425 Number of transparent hugepages which were allocated to satisfy 1426 a page fault. This counter is not present when CONFIG_TRANSPARENT_HUGEPAGE 1427 is not set. 1428 1429 thp_collapse_alloc (npn) 1430 Number of transparent hugepages which were allocated to allow 1431 collapsing an existing range of pages. This counter is not 1432 present when CONFIG_TRANSPARENT_HUGEPAGE is not set. 1433 1434 memory.numa_stat 1435 A read-only nested-keyed file which exists on non-root cgroups. 1436 1437 This breaks down the cgroup's memory footprint into different 1438 types of memory, type-specific details, and other information 1439 per node on the state of the memory management system. 1440 1441 This is useful for providing visibility into the NUMA locality 1442 information within an memcg since the pages are allowed to be 1443 allocated from any physical node. One of the use case is evaluating 1444 application performance by combining this information with the 1445 application's CPU allocation. 1446 1447 All memory amounts are in bytes. 1448 1449 The output format of memory.numa_stat is:: 1450 1451 type N0=<bytes in node 0> N1=<bytes in node 1> ... 1452 1453 The entries are ordered to be human readable, and new entries 1454 can show up in the middle. Don't rely on items remaining in a 1455 fixed position; use the keys to look up specific values! 1456 1457 The entries can refer to the memory.stat. 1458 1459 memory.swap.current 1460 A read-only single value file which exists on non-root 1461 cgroups. 1462 1463 The total amount of swap currently being used by the cgroup 1464 and its descendants. 1465 1466 memory.swap.high 1467 A read-write single value file which exists on non-root 1468 cgroups. The default is "max". 1469 1470 Swap usage throttle limit. If a cgroup's swap usage exceeds 1471 this limit, all its further allocations will be throttled to 1472 allow userspace to implement custom out-of-memory procedures. 1473 1474 This limit marks a point of no return for the cgroup. It is NOT 1475 designed to manage the amount of swapping a workload does 1476 during regular operation. Compare to memory.swap.max, which 1477 prohibits swapping past a set amount, but lets the cgroup 1478 continue unimpeded as long as other memory can be reclaimed. 1479 1480 Healthy workloads are not expected to reach this limit. 1481 1482 memory.swap.max 1483 A read-write single value file which exists on non-root 1484 cgroups. The default is "max". 1485 1486 Swap usage hard limit. If a cgroup's swap usage reaches this 1487 limit, anonymous memory of the cgroup will not be swapped out. 1488 1489 memory.swap.events 1490 A read-only flat-keyed file which exists on non-root cgroups. 1491 The following entries are defined. Unless specified 1492 otherwise, a value change in this file generates a file 1493 modified event. 1494 1495 high 1496 The number of times the cgroup's swap usage was over 1497 the high threshold. 1498 1499 max 1500 The number of times the cgroup's swap usage was about 1501 to go over the max boundary and swap allocation 1502 failed. 1503 1504 fail 1505 The number of times swap allocation failed either 1506 because of running out of swap system-wide or max 1507 limit. 1508 1509 When reduced under the current usage, the existing swap 1510 entries are reclaimed gradually and the swap usage may stay 1511 higher than the limit for an extended period of time. This 1512 reduces the impact on the workload and memory management. 1513 1514 memory.pressure 1515 A read-only nested-keyed file. 1516 1517 Shows pressure stall information for memory. See 1518 :ref:`Documentation/accounting/psi.rst <psi>` for details. 1519 1520 1521Usage Guidelines 1522~~~~~~~~~~~~~~~~ 1523 1524"memory.high" is the main mechanism to control memory usage. 1525Over-committing on high limit (sum of high limits > available memory) 1526and letting global memory pressure to distribute memory according to 1527usage is a viable strategy. 1528 1529Because breach of the high limit doesn't trigger the OOM killer but 1530throttles the offending cgroup, a management agent has ample 1531opportunities to monitor and take appropriate actions such as granting 1532more memory or terminating the workload. 1533 1534Determining whether a cgroup has enough memory is not trivial as 1535memory usage doesn't indicate whether the workload can benefit from 1536more memory. For example, a workload which writes data received from 1537network to a file can use all available memory but can also operate as 1538performant with a small amount of memory. A measure of memory 1539pressure - how much the workload is being impacted due to lack of 1540memory - is necessary to determine whether a workload needs more 1541memory; unfortunately, memory pressure monitoring mechanism isn't 1542implemented yet. 1543 1544 1545Memory Ownership 1546~~~~~~~~~~~~~~~~ 1547 1548A memory area is charged to the cgroup which instantiated it and stays 1549charged to the cgroup until the area is released. Migrating a process 1550to a different cgroup doesn't move the memory usages that it 1551instantiated while in the previous cgroup to the new cgroup. 1552 1553A memory area may be used by processes belonging to different cgroups. 1554To which cgroup the area will be charged is in-deterministic; however, 1555over time, the memory area is likely to end up in a cgroup which has 1556enough memory allowance to avoid high reclaim pressure. 1557 1558If a cgroup sweeps a considerable amount of memory which is expected 1559to be accessed repeatedly by other cgroups, it may make sense to use 1560POSIX_FADV_DONTNEED to relinquish the ownership of memory areas 1561belonging to the affected files to ensure correct memory ownership. 1562 1563 1564IO 1565-- 1566 1567The "io" controller regulates the distribution of IO resources. This 1568controller implements both weight based and absolute bandwidth or IOPS 1569limit distribution; however, weight based distribution is available 1570only if cfq-iosched is in use and neither scheme is available for 1571blk-mq devices. 1572 1573 1574IO Interface Files 1575~~~~~~~~~~~~~~~~~~ 1576 1577 io.stat 1578 A read-only nested-keyed file. 1579 1580 Lines are keyed by $MAJ:$MIN device numbers and not ordered. 1581 The following nested keys are defined. 1582 1583 ====== ===================== 1584 rbytes Bytes read 1585 wbytes Bytes written 1586 rios Number of read IOs 1587 wios Number of write IOs 1588 dbytes Bytes discarded 1589 dios Number of discard IOs 1590 ====== ===================== 1591 1592 An example read output follows:: 1593 1594 8:16 rbytes=1459200 wbytes=314773504 rios=192 wios=353 dbytes=0 dios=0 1595 8:0 rbytes=90430464 wbytes=299008000 rios=8950 wios=1252 dbytes=50331648 dios=3021 1596 1597 io.cost.qos 1598 A read-write nested-keyed file which exists only on the root 1599 cgroup. 1600 1601 This file configures the Quality of Service of the IO cost 1602 model based controller (CONFIG_BLK_CGROUP_IOCOST) which 1603 currently implements "io.weight" proportional control. Lines 1604 are keyed by $MAJ:$MIN device numbers and not ordered. The 1605 line for a given device is populated on the first write for 1606 the device on "io.cost.qos" or "io.cost.model". The following 1607 nested keys are defined. 1608 1609 ====== ===================================== 1610 enable Weight-based control enable 1611 ctrl "auto" or "user" 1612 rpct Read latency percentile [0, 100] 1613 rlat Read latency threshold 1614 wpct Write latency percentile [0, 100] 1615 wlat Write latency threshold 1616 min Minimum scaling percentage [1, 10000] 1617 max Maximum scaling percentage [1, 10000] 1618 ====== ===================================== 1619 1620 The controller is disabled by default and can be enabled by 1621 setting "enable" to 1. "rpct" and "wpct" parameters default 1622 to zero and the controller uses internal device saturation 1623 state to adjust the overall IO rate between "min" and "max". 1624 1625 When a better control quality is needed, latency QoS 1626 parameters can be configured. For example:: 1627 1628 8:16 enable=1 ctrl=auto rpct=95.00 rlat=75000 wpct=95.00 wlat=150000 min=50.00 max=150.0 1629 1630 shows that on sdb, the controller is enabled, will consider 1631 the device saturated if the 95th percentile of read completion 1632 latencies is above 75ms or write 150ms, and adjust the overall 1633 IO issue rate between 50% and 150% accordingly. 1634 1635 The lower the saturation point, the better the latency QoS at 1636 the cost of aggregate bandwidth. The narrower the allowed 1637 adjustment range between "min" and "max", the more conformant 1638 to the cost model the IO behavior. Note that the IO issue 1639 base rate may be far off from 100% and setting "min" and "max" 1640 blindly can lead to a significant loss of device capacity or 1641 control quality. "min" and "max" are useful for regulating 1642 devices which show wide temporary behavior changes - e.g. a 1643 ssd which accepts writes at the line speed for a while and 1644 then completely stalls for multiple seconds. 1645 1646 When "ctrl" is "auto", the parameters are controlled by the 1647 kernel and may change automatically. Setting "ctrl" to "user" 1648 or setting any of the percentile and latency parameters puts 1649 it into "user" mode and disables the automatic changes. The 1650 automatic mode can be restored by setting "ctrl" to "auto". 1651 1652 io.cost.model 1653 A read-write nested-keyed file which exists only on the root 1654 cgroup. 1655 1656 This file configures the cost model of the IO cost model based 1657 controller (CONFIG_BLK_CGROUP_IOCOST) which currently 1658 implements "io.weight" proportional control. Lines are keyed 1659 by $MAJ:$MIN device numbers and not ordered. The line for a 1660 given device is populated on the first write for the device on 1661 "io.cost.qos" or "io.cost.model". The following nested keys 1662 are defined. 1663 1664 ===== ================================ 1665 ctrl "auto" or "user" 1666 model The cost model in use - "linear" 1667 ===== ================================ 1668 1669 When "ctrl" is "auto", the kernel may change all parameters 1670 dynamically. When "ctrl" is set to "user" or any other 1671 parameters are written to, "ctrl" become "user" and the 1672 automatic changes are disabled. 1673 1674 When "model" is "linear", the following model parameters are 1675 defined. 1676 1677 ============= ======================================== 1678 [r|w]bps The maximum sequential IO throughput 1679 [r|w]seqiops The maximum 4k sequential IOs per second 1680 [r|w]randiops The maximum 4k random IOs per second 1681 ============= ======================================== 1682 1683 From the above, the builtin linear model determines the base 1684 costs of a sequential and random IO and the cost coefficient 1685 for the IO size. While simple, this model can cover most 1686 common device classes acceptably. 1687 1688 The IO cost model isn't expected to be accurate in absolute 1689 sense and is scaled to the device behavior dynamically. 1690 1691 If needed, tools/cgroup/iocost_coef_gen.py can be used to 1692 generate device-specific coefficients. 1693 1694 io.weight 1695 A read-write flat-keyed file which exists on non-root cgroups. 1696 The default is "default 100". 1697 1698 The first line is the default weight applied to devices 1699 without specific override. The rest are overrides keyed by 1700 $MAJ:$MIN device numbers and not ordered. The weights are in 1701 the range [1, 10000] and specifies the relative amount IO time 1702 the cgroup can use in relation to its siblings. 1703 1704 The default weight can be updated by writing either "default 1705 $WEIGHT" or simply "$WEIGHT". Overrides can be set by writing 1706 "$MAJ:$MIN $WEIGHT" and unset by writing "$MAJ:$MIN default". 1707 1708 An example read output follows:: 1709 1710 default 100 1711 8:16 200 1712 8:0 50 1713 1714 io.max 1715 A read-write nested-keyed file which exists on non-root 1716 cgroups. 1717 1718 BPS and IOPS based IO limit. Lines are keyed by $MAJ:$MIN 1719 device numbers and not ordered. The following nested keys are 1720 defined. 1721 1722 ===== ================================== 1723 rbps Max read bytes per second 1724 wbps Max write bytes per second 1725 riops Max read IO operations per second 1726 wiops Max write IO operations per second 1727 ===== ================================== 1728 1729 When writing, any number of nested key-value pairs can be 1730 specified in any order. "max" can be specified as the value 1731 to remove a specific limit. If the same key is specified 1732 multiple times, the outcome is undefined. 1733 1734 BPS and IOPS are measured in each IO direction and IOs are 1735 delayed if limit is reached. Temporary bursts are allowed. 1736 1737 Setting read limit at 2M BPS and write at 120 IOPS for 8:16:: 1738 1739 echo "8:16 rbps=2097152 wiops=120" > io.max 1740 1741 Reading returns the following:: 1742 1743 8:16 rbps=2097152 wbps=max riops=max wiops=120 1744 1745 Write IOPS limit can be removed by writing the following:: 1746 1747 echo "8:16 wiops=max" > io.max 1748 1749 Reading now returns the following:: 1750 1751 8:16 rbps=2097152 wbps=max riops=max wiops=max 1752 1753 io.pressure 1754 A read-only nested-keyed file. 1755 1756 Shows pressure stall information for IO. See 1757 :ref:`Documentation/accounting/psi.rst <psi>` for details. 1758 1759 1760Writeback 1761~~~~~~~~~ 1762 1763Page cache is dirtied through buffered writes and shared mmaps and 1764written asynchronously to the backing filesystem by the writeback 1765mechanism. Writeback sits between the memory and IO domains and 1766regulates the proportion of dirty memory by balancing dirtying and 1767write IOs. 1768 1769The io controller, in conjunction with the memory controller, 1770implements control of page cache writeback IOs. The memory controller 1771defines the memory domain that dirty memory ratio is calculated and 1772maintained for and the io controller defines the io domain which 1773writes out dirty pages for the memory domain. Both system-wide and 1774per-cgroup dirty memory states are examined and the more restrictive 1775of the two is enforced. 1776 1777cgroup writeback requires explicit support from the underlying 1778filesystem. Currently, cgroup writeback is implemented on ext2, ext4, 1779btrfs, f2fs, and xfs. On other filesystems, all writeback IOs are 1780attributed to the root cgroup. 1781 1782There are inherent differences in memory and writeback management 1783which affects how cgroup ownership is tracked. Memory is tracked per 1784page while writeback per inode. For the purpose of writeback, an 1785inode is assigned to a cgroup and all IO requests to write dirty pages 1786from the inode are attributed to that cgroup. 1787 1788As cgroup ownership for memory is tracked per page, there can be pages 1789which are associated with different cgroups than the one the inode is 1790associated with. These are called foreign pages. The writeback 1791constantly keeps track of foreign pages and, if a particular foreign 1792cgroup becomes the majority over a certain period of time, switches 1793the ownership of the inode to that cgroup. 1794 1795While this model is enough for most use cases where a given inode is 1796mostly dirtied by a single cgroup even when the main writing cgroup 1797changes over time, use cases where multiple cgroups write to a single 1798inode simultaneously are not supported well. In such circumstances, a 1799significant portion of IOs are likely to be attributed incorrectly. 1800As memory controller assigns page ownership on the first use and 1801doesn't update it until the page is released, even if writeback 1802strictly follows page ownership, multiple cgroups dirtying overlapping 1803areas wouldn't work as expected. It's recommended to avoid such usage 1804patterns. 1805 1806The sysctl knobs which affect writeback behavior are applied to cgroup 1807writeback as follows. 1808 1809 vm.dirty_background_ratio, vm.dirty_ratio 1810 These ratios apply the same to cgroup writeback with the 1811 amount of available memory capped by limits imposed by the 1812 memory controller and system-wide clean memory. 1813 1814 vm.dirty_background_bytes, vm.dirty_bytes 1815 For cgroup writeback, this is calculated into ratio against 1816 total available memory and applied the same way as 1817 vm.dirty[_background]_ratio. 1818 1819 1820IO Latency 1821~~~~~~~~~~ 1822 1823This is a cgroup v2 controller for IO workload protection. You provide a group 1824with a latency target, and if the average latency exceeds that target the 1825controller will throttle any peers that have a lower latency target than the 1826protected workload. 1827 1828The limits are only applied at the peer level in the hierarchy. This means that 1829in the diagram below, only groups A, B, and C will influence each other, and 1830groups D and F will influence each other. Group G will influence nobody:: 1831 1832 [root] 1833 / | \ 1834 A B C 1835 / \ | 1836 D F G 1837 1838 1839So the ideal way to configure this is to set io.latency in groups A, B, and C. 1840Generally you do not want to set a value lower than the latency your device 1841supports. Experiment to find the value that works best for your workload. 1842Start at higher than the expected latency for your device and watch the 1843avg_lat value in io.stat for your workload group to get an idea of the 1844latency you see during normal operation. Use the avg_lat value as a basis for 1845your real setting, setting at 10-15% higher than the value in io.stat. 1846 1847How IO Latency Throttling Works 1848~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1849 1850io.latency is work conserving; so as long as everybody is meeting their latency 1851target the controller doesn't do anything. Once a group starts missing its 1852target it begins throttling any peer group that has a higher target than itself. 1853This throttling takes 2 forms: 1854 1855- Queue depth throttling. This is the number of outstanding IO's a group is 1856 allowed to have. We will clamp down relatively quickly, starting at no limit 1857 and going all the way down to 1 IO at a time. 1858 1859- Artificial delay induction. There are certain types of IO that cannot be 1860 throttled without possibly adversely affecting higher priority groups. This 1861 includes swapping and metadata IO. These types of IO are allowed to occur 1862 normally, however they are "charged" to the originating group. If the 1863 originating group is being throttled you will see the use_delay and delay 1864 fields in io.stat increase. The delay value is how many microseconds that are 1865 being added to any process that runs in this group. Because this number can 1866 grow quite large if there is a lot of swapping or metadata IO occurring we 1867 limit the individual delay events to 1 second at a time. 1868 1869Once the victimized group starts meeting its latency target again it will start 1870unthrottling any peer groups that were throttled previously. If the victimized 1871group simply stops doing IO the global counter will unthrottle appropriately. 1872 1873IO Latency Interface Files 1874~~~~~~~~~~~~~~~~~~~~~~~~~~ 1875 1876 io.latency 1877 This takes a similar format as the other controllers. 1878 1879 "MAJOR:MINOR target=<target time in microseconds" 1880 1881 io.stat 1882 If the controller is enabled you will see extra stats in io.stat in 1883 addition to the normal ones. 1884 1885 depth 1886 This is the current queue depth for the group. 1887 1888 avg_lat 1889 This is an exponential moving average with a decay rate of 1/exp 1890 bound by the sampling interval. The decay rate interval can be 1891 calculated by multiplying the win value in io.stat by the 1892 corresponding number of samples based on the win value. 1893 1894 win 1895 The sampling window size in milliseconds. This is the minimum 1896 duration of time between evaluation events. Windows only elapse 1897 with IO activity. Idle periods extend the most recent window. 1898 1899IO Priority 1900~~~~~~~~~~~ 1901 1902A single attribute controls the behavior of the I/O priority cgroup policy, 1903namely the blkio.prio.class attribute. The following values are accepted for 1904that attribute: 1905 1906 no-change 1907 Do not modify the I/O priority class. 1908 1909 none-to-rt 1910 For requests that do not have an I/O priority class (NONE), 1911 change the I/O priority class into RT. Do not modify 1912 the I/O priority class of other requests. 1913 1914 restrict-to-be 1915 For requests that do not have an I/O priority class or that have I/O 1916 priority class RT, change it into BE. Do not modify the I/O priority 1917 class of requests that have priority class IDLE. 1918 1919 idle 1920 Change the I/O priority class of all requests into IDLE, the lowest 1921 I/O priority class. 1922 1923The following numerical values are associated with the I/O priority policies: 1924 1925+-------------+---+ 1926| no-change | 0 | 1927+-------------+---+ 1928| none-to-rt | 1 | 1929+-------------+---+ 1930| rt-to-be | 2 | 1931+-------------+---+ 1932| all-to-idle | 3 | 1933+-------------+---+ 1934 1935The numerical value that corresponds to each I/O priority class is as follows: 1936 1937+-------------------------------+---+ 1938| IOPRIO_CLASS_NONE | 0 | 1939+-------------------------------+---+ 1940| IOPRIO_CLASS_RT (real-time) | 1 | 1941+-------------------------------+---+ 1942| IOPRIO_CLASS_BE (best effort) | 2 | 1943+-------------------------------+---+ 1944| IOPRIO_CLASS_IDLE | 3 | 1945+-------------------------------+---+ 1946 1947The algorithm to set the I/O priority class for a request is as follows: 1948 1949- Translate the I/O priority class policy into a number. 1950- Change the request I/O priority class into the maximum of the I/O priority 1951 class policy number and the numerical I/O priority class. 1952 1953PID 1954--- 1955 1956The process number controller is used to allow a cgroup to stop any 1957new tasks from being fork()'d or clone()'d after a specified limit is 1958reached. 1959 1960The number of tasks in a cgroup can be exhausted in ways which other 1961controllers cannot prevent, thus warranting its own controller. For 1962example, a fork bomb is likely to exhaust the number of tasks before 1963hitting memory restrictions. 1964 1965Note that PIDs used in this controller refer to TIDs, process IDs as 1966used by the kernel. 1967 1968 1969PID Interface Files 1970~~~~~~~~~~~~~~~~~~~ 1971 1972 pids.max 1973 A read-write single value file which exists on non-root 1974 cgroups. The default is "max". 1975 1976 Hard limit of number of processes. 1977 1978 pids.current 1979 A read-only single value file which exists on all cgroups. 1980 1981 The number of processes currently in the cgroup and its 1982 descendants. 1983 1984Organisational operations are not blocked by cgroup policies, so it is 1985possible to have pids.current > pids.max. This can be done by either 1986setting the limit to be smaller than pids.current, or attaching enough 1987processes to the cgroup such that pids.current is larger than 1988pids.max. However, it is not possible to violate a cgroup PID policy 1989through fork() or clone(). These will return -EAGAIN if the creation 1990of a new process would cause a cgroup policy to be violated. 1991 1992 1993Cpuset 1994------ 1995 1996The "cpuset" controller provides a mechanism for constraining 1997the CPU and memory node placement of tasks to only the resources 1998specified in the cpuset interface files in a task's current cgroup. 1999This is especially valuable on large NUMA systems where placing jobs 2000on properly sized subsets of the systems with careful processor and 2001memory placement to reduce cross-node memory access and contention 2002can improve overall system performance. 2003 2004The "cpuset" controller is hierarchical. That means the controller 2005cannot use CPUs or memory nodes not allowed in its parent. 2006 2007 2008Cpuset Interface Files 2009~~~~~~~~~~~~~~~~~~~~~~ 2010 2011 cpuset.cpus 2012 A read-write multiple values file which exists on non-root 2013 cpuset-enabled cgroups. 2014 2015 It lists the requested CPUs to be used by tasks within this 2016 cgroup. The actual list of CPUs to be granted, however, is 2017 subjected to constraints imposed by its parent and can differ 2018 from the requested CPUs. 2019 2020 The CPU numbers are comma-separated numbers or ranges. 2021 For example:: 2022 2023 # cat cpuset.cpus 2024 0-4,6,8-10 2025 2026 An empty value indicates that the cgroup is using the same 2027 setting as the nearest cgroup ancestor with a non-empty 2028 "cpuset.cpus" or all the available CPUs if none is found. 2029 2030 The value of "cpuset.cpus" stays constant until the next update 2031 and won't be affected by any CPU hotplug events. 2032 2033 cpuset.cpus.effective 2034 A read-only multiple values file which exists on all 2035 cpuset-enabled cgroups. 2036 2037 It lists the onlined CPUs that are actually granted to this 2038 cgroup by its parent. These CPUs are allowed to be used by 2039 tasks within the current cgroup. 2040 2041 If "cpuset.cpus" is empty, the "cpuset.cpus.effective" file shows 2042 all the CPUs from the parent cgroup that can be available to 2043 be used by this cgroup. Otherwise, it should be a subset of 2044 "cpuset.cpus" unless none of the CPUs listed in "cpuset.cpus" 2045 can be granted. In this case, it will be treated just like an 2046 empty "cpuset.cpus". 2047 2048 Its value will be affected by CPU hotplug events. 2049 2050 cpuset.mems 2051 A read-write multiple values file which exists on non-root 2052 cpuset-enabled cgroups. 2053 2054 It lists the requested memory nodes to be used by tasks within 2055 this cgroup. The actual list of memory nodes granted, however, 2056 is subjected to constraints imposed by its parent and can differ 2057 from the requested memory nodes. 2058 2059 The memory node numbers are comma-separated numbers or ranges. 2060 For example:: 2061 2062 # cat cpuset.mems 2063 0-1,3 2064 2065 An empty value indicates that the cgroup is using the same 2066 setting as the nearest cgroup ancestor with a non-empty 2067 "cpuset.mems" or all the available memory nodes if none 2068 is found. 2069 2070 The value of "cpuset.mems" stays constant until the next update 2071 and won't be affected by any memory nodes hotplug events. 2072 2073 Setting a non-empty value to "cpuset.mems" causes memory of 2074 tasks within the cgroup to be migrated to the designated nodes if 2075 they are currently using memory outside of the designated nodes. 2076 2077 There is a cost for this memory migration. The migration 2078 may not be complete and some memory pages may be left behind. 2079 So it is recommended that "cpuset.mems" should be set properly 2080 before spawning new tasks into the cpuset. Even if there is 2081 a need to change "cpuset.mems" with active tasks, it shouldn't 2082 be done frequently. 2083 2084 cpuset.mems.effective 2085 A read-only multiple values file which exists on all 2086 cpuset-enabled cgroups. 2087 2088 It lists the onlined memory nodes that are actually granted to 2089 this cgroup by its parent. These memory nodes are allowed to 2090 be used by tasks within the current cgroup. 2091 2092 If "cpuset.mems" is empty, it shows all the memory nodes from the 2093 parent cgroup that will be available to be used by this cgroup. 2094 Otherwise, it should be a subset of "cpuset.mems" unless none of 2095 the memory nodes listed in "cpuset.mems" can be granted. In this 2096 case, it will be treated just like an empty "cpuset.mems". 2097 2098 Its value will be affected by memory nodes hotplug events. 2099 2100 cpuset.cpus.partition 2101 A read-write single value file which exists on non-root 2102 cpuset-enabled cgroups. This flag is owned by the parent cgroup 2103 and is not delegatable. 2104 2105 It accepts only the following input values when written to. 2106 2107 ======== ================================ 2108 "root" a partition root 2109 "member" a non-root member of a partition 2110 ======== ================================ 2111 2112 When set to be a partition root, the current cgroup is the 2113 root of a new partition or scheduling domain that comprises 2114 itself and all its descendants except those that are separate 2115 partition roots themselves and their descendants. The root 2116 cgroup is always a partition root. 2117 2118 There are constraints on where a partition root can be set. 2119 It can only be set in a cgroup if all the following conditions 2120 are true. 2121 2122 1) The "cpuset.cpus" is not empty and the list of CPUs are 2123 exclusive, i.e. they are not shared by any of its siblings. 2124 2) The parent cgroup is a partition root. 2125 3) The "cpuset.cpus" is also a proper subset of the parent's 2126 "cpuset.cpus.effective". 2127 4) There is no child cgroups with cpuset enabled. This is for 2128 eliminating corner cases that have to be handled if such a 2129 condition is allowed. 2130 2131 Setting it to partition root will take the CPUs away from the 2132 effective CPUs of the parent cgroup. Once it is set, this 2133 file cannot be reverted back to "member" if there are any child 2134 cgroups with cpuset enabled. 2135 2136 A parent partition cannot distribute all its CPUs to its 2137 child partitions. There must be at least one cpu left in the 2138 parent partition. 2139 2140 Once becoming a partition root, changes to "cpuset.cpus" is 2141 generally allowed as long as the first condition above is true, 2142 the change will not take away all the CPUs from the parent 2143 partition and the new "cpuset.cpus" value is a superset of its 2144 children's "cpuset.cpus" values. 2145 2146 Sometimes, external factors like changes to ancestors' 2147 "cpuset.cpus" or cpu hotplug can cause the state of the partition 2148 root to change. On read, the "cpuset.sched.partition" file 2149 can show the following values. 2150 2151 ============== ============================== 2152 "member" Non-root member of a partition 2153 "root" Partition root 2154 "root invalid" Invalid partition root 2155 ============== ============================== 2156 2157 It is a partition root if the first 2 partition root conditions 2158 above are true and at least one CPU from "cpuset.cpus" is 2159 granted by the parent cgroup. 2160 2161 A partition root can become invalid if none of CPUs requested 2162 in "cpuset.cpus" can be granted by the parent cgroup or the 2163 parent cgroup is no longer a partition root itself. In this 2164 case, it is not a real partition even though the restriction 2165 of the first partition root condition above will still apply. 2166 The cpu affinity of all the tasks in the cgroup will then be 2167 associated with CPUs in the nearest ancestor partition. 2168 2169 An invalid partition root can be transitioned back to a 2170 real partition root if at least one of the requested CPUs 2171 can now be granted by its parent. In this case, the cpu 2172 affinity of all the tasks in the formerly invalid partition 2173 will be associated to the CPUs of the newly formed partition. 2174 Changing the partition state of an invalid partition root to 2175 "member" is always allowed even if child cpusets are present. 2176 2177 2178Device controller 2179----------------- 2180 2181Device controller manages access to device files. It includes both 2182creation of new device files (using mknod), and access to the 2183existing device files. 2184 2185Cgroup v2 device controller has no interface files and is implemented 2186on top of cgroup BPF. To control access to device files, a user may 2187create bpf programs of type BPF_PROG_TYPE_CGROUP_DEVICE and attach 2188them to cgroups with BPF_CGROUP_DEVICE flag. On an attempt to access a 2189device file, corresponding BPF programs will be executed, and depending 2190on the return value the attempt will succeed or fail with -EPERM. 2191 2192A BPF_PROG_TYPE_CGROUP_DEVICE program takes a pointer to the 2193bpf_cgroup_dev_ctx structure, which describes the device access attempt: 2194access type (mknod/read/write) and device (type, major and minor numbers). 2195If the program returns 0, the attempt fails with -EPERM, otherwise it 2196succeeds. 2197 2198An example of BPF_PROG_TYPE_CGROUP_DEVICE program may be found in 2199tools/testing/selftests/bpf/progs/dev_cgroup.c in the kernel source tree. 2200 2201 2202RDMA 2203---- 2204 2205The "rdma" controller regulates the distribution and accounting of 2206RDMA resources. 2207 2208RDMA Interface Files 2209~~~~~~~~~~~~~~~~~~~~ 2210 2211 rdma.max 2212 A readwrite nested-keyed file that exists for all the cgroups 2213 except root that describes current configured resource limit 2214 for a RDMA/IB device. 2215 2216 Lines are keyed by device name and are not ordered. 2217 Each line contains space separated resource name and its configured 2218 limit that can be distributed. 2219 2220 The following nested keys are defined. 2221 2222 ========== ============================= 2223 hca_handle Maximum number of HCA Handles 2224 hca_object Maximum number of HCA Objects 2225 ========== ============================= 2226 2227 An example for mlx4 and ocrdma device follows:: 2228 2229 mlx4_0 hca_handle=2 hca_object=2000 2230 ocrdma1 hca_handle=3 hca_object=max 2231 2232 rdma.current 2233 A read-only file that describes current resource usage. 2234 It exists for all the cgroup except root. 2235 2236 An example for mlx4 and ocrdma device follows:: 2237 2238 mlx4_0 hca_handle=1 hca_object=20 2239 ocrdma1 hca_handle=1 hca_object=23 2240 2241HugeTLB 2242------- 2243 2244The HugeTLB controller allows to limit the HugeTLB usage per control group and 2245enforces the controller limit during page fault. 2246 2247HugeTLB Interface Files 2248~~~~~~~~~~~~~~~~~~~~~~~ 2249 2250 hugetlb.<hugepagesize>.current 2251 Show current usage for "hugepagesize" hugetlb. It exists for all 2252 the cgroup except root. 2253 2254 hugetlb.<hugepagesize>.max 2255 Set/show the hard limit of "hugepagesize" hugetlb usage. 2256 The default value is "max". It exists for all the cgroup except root. 2257 2258 hugetlb.<hugepagesize>.events 2259 A read-only flat-keyed file which exists on non-root cgroups. 2260 2261 max 2262 The number of allocation failure due to HugeTLB limit 2263 2264 hugetlb.<hugepagesize>.events.local 2265 Similar to hugetlb.<hugepagesize>.events but the fields in the file 2266 are local to the cgroup i.e. not hierarchical. The file modified event 2267 generated on this file reflects only the local events. 2268 2269 hugetlb.<hugepagesize>.numa_stat 2270 Similar to memory.numa_stat, it shows the numa information of the 2271 hugetlb pages of <hugepagesize> in this cgroup. Only active in 2272 use hugetlb pages are included. The per-node values are in bytes. 2273 2274Misc 2275---- 2276 2277The Miscellaneous cgroup provides the resource limiting and tracking 2278mechanism for the scalar resources which cannot be abstracted like the other 2279cgroup resources. Controller is enabled by the CONFIG_CGROUP_MISC config 2280option. 2281 2282A resource can be added to the controller via enum misc_res_type{} in the 2283include/linux/misc_cgroup.h file and the corresponding name via misc_res_name[] 2284in the kernel/cgroup/misc.c file. Provider of the resource must set its 2285capacity prior to using the resource by calling misc_cg_set_capacity(). 2286 2287Once a capacity is set then the resource usage can be updated using charge and 2288uncharge APIs. All of the APIs to interact with misc controller are in 2289include/linux/misc_cgroup.h. 2290 2291Misc Interface Files 2292~~~~~~~~~~~~~~~~~~~~ 2293 2294Miscellaneous controller provides 3 interface files. If two misc resources (res_a and res_b) are registered then: 2295 2296 misc.capacity 2297 A read-only flat-keyed file shown only in the root cgroup. It shows 2298 miscellaneous scalar resources available on the platform along with 2299 their quantities:: 2300 2301 $ cat misc.capacity 2302 res_a 50 2303 res_b 10 2304 2305 misc.current 2306 A read-only flat-keyed file shown in the non-root cgroups. It shows 2307 the current usage of the resources in the cgroup and its children.:: 2308 2309 $ cat misc.current 2310 res_a 3 2311 res_b 0 2312 2313 misc.max 2314 A read-write flat-keyed file shown in the non root cgroups. Allowed 2315 maximum usage of the resources in the cgroup and its children.:: 2316 2317 $ cat misc.max 2318 res_a max 2319 res_b 4 2320 2321 Limit can be set by:: 2322 2323 # echo res_a 1 > misc.max 2324 2325 Limit can be set to max by:: 2326 2327 # echo res_a max > misc.max 2328 2329 Limits can be set higher than the capacity value in the misc.capacity 2330 file. 2331 2332 misc.events 2333 A read-only flat-keyed file which exists on non-root cgroups. The 2334 following entries are defined. Unless specified otherwise, a value 2335 change in this file generates a file modified event. All fields in 2336 this file are hierarchical. 2337 2338 max 2339 The number of times the cgroup's resource usage was 2340 about to go over the max boundary. 2341 2342Migration and Ownership 2343~~~~~~~~~~~~~~~~~~~~~~~ 2344 2345A miscellaneous scalar resource is charged to the cgroup in which it is used 2346first, and stays charged to that cgroup until that resource is freed. Migrating 2347a process to a different cgroup does not move the charge to the destination 2348cgroup where the process has moved. 2349 2350Others 2351------ 2352 2353perf_event 2354~~~~~~~~~~ 2355 2356perf_event controller, if not mounted on a legacy hierarchy, is 2357automatically enabled on the v2 hierarchy so that perf events can 2358always be filtered by cgroup v2 path. The controller can still be 2359moved to a legacy hierarchy after v2 hierarchy is populated. 2360 2361 2362Non-normative information 2363------------------------- 2364 2365This section contains information that isn't considered to be a part of 2366the stable kernel API and so is subject to change. 2367 2368 2369CPU controller root cgroup process behaviour 2370~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2371 2372When distributing CPU cycles in the root cgroup each thread in this 2373cgroup is treated as if it was hosted in a separate child cgroup of the 2374root cgroup. This child cgroup weight is dependent on its thread nice 2375level. 2376 2377For details of this mapping see sched_prio_to_weight array in 2378kernel/sched/core.c file (values from this array should be scaled 2379appropriately so the neutral - nice 0 - value is 100 instead of 1024). 2380 2381 2382IO controller root cgroup process behaviour 2383~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2384 2385Root cgroup processes are hosted in an implicit leaf child node. 2386When distributing IO resources this implicit child node is taken into 2387account as if it was a normal child cgroup of the root cgroup with a 2388weight value of 200. 2389 2390 2391Namespace 2392========= 2393 2394Basics 2395------ 2396 2397cgroup namespace provides a mechanism to virtualize the view of the 2398"/proc/$PID/cgroup" file and cgroup mounts. The CLONE_NEWCGROUP clone 2399flag can be used with clone(2) and unshare(2) to create a new cgroup 2400namespace. The process running inside the cgroup namespace will have 2401its "/proc/$PID/cgroup" output restricted to cgroupns root. The 2402cgroupns root is the cgroup of the process at the time of creation of 2403the cgroup namespace. 2404 2405Without cgroup namespace, the "/proc/$PID/cgroup" file shows the 2406complete path of the cgroup of a process. In a container setup where 2407a set of cgroups and namespaces are intended to isolate processes the 2408"/proc/$PID/cgroup" file may leak potential system level information 2409to the isolated processes. For example:: 2410 2411 # cat /proc/self/cgroup 2412 0::/batchjobs/container_id1 2413 2414The path '/batchjobs/container_id1' can be considered as system-data 2415and undesirable to expose to the isolated processes. cgroup namespace 2416can be used to restrict visibility of this path. For example, before 2417creating a cgroup namespace, one would see:: 2418 2419 # ls -l /proc/self/ns/cgroup 2420 lrwxrwxrwx 1 root root 0 2014-07-15 10:37 /proc/self/ns/cgroup -> cgroup:[4026531835] 2421 # cat /proc/self/cgroup 2422 0::/batchjobs/container_id1 2423 2424After unsharing a new namespace, the view changes:: 2425 2426 # ls -l /proc/self/ns/cgroup 2427 lrwxrwxrwx 1 root root 0 2014-07-15 10:35 /proc/self/ns/cgroup -> cgroup:[4026532183] 2428 # cat /proc/self/cgroup 2429 0::/ 2430 2431When some thread from a multi-threaded process unshares its cgroup 2432namespace, the new cgroupns gets applied to the entire process (all 2433the threads). This is natural for the v2 hierarchy; however, for the 2434legacy hierarchies, this may be unexpected. 2435 2436A cgroup namespace is alive as long as there are processes inside or 2437mounts pinning it. When the last usage goes away, the cgroup 2438namespace is destroyed. The cgroupns root and the actual cgroups 2439remain. 2440 2441 2442The Root and Views 2443------------------ 2444 2445The 'cgroupns root' for a cgroup namespace is the cgroup in which the 2446process calling unshare(2) is running. For example, if a process in 2447/batchjobs/container_id1 cgroup calls unshare, cgroup 2448/batchjobs/container_id1 becomes the cgroupns root. For the 2449init_cgroup_ns, this is the real root ('/') cgroup. 2450 2451The cgroupns root cgroup does not change even if the namespace creator 2452process later moves to a different cgroup:: 2453 2454 # ~/unshare -c # unshare cgroupns in some cgroup 2455 # cat /proc/self/cgroup 2456 0::/ 2457 # mkdir sub_cgrp_1 2458 # echo 0 > sub_cgrp_1/cgroup.procs 2459 # cat /proc/self/cgroup 2460 0::/sub_cgrp_1 2461 2462Each process gets its namespace-specific view of "/proc/$PID/cgroup" 2463 2464Processes running inside the cgroup namespace will be able to see 2465cgroup paths (in /proc/self/cgroup) only inside their root cgroup. 2466From within an unshared cgroupns:: 2467 2468 # sleep 100000 & 2469 [1] 7353 2470 # echo 7353 > sub_cgrp_1/cgroup.procs 2471 # cat /proc/7353/cgroup 2472 0::/sub_cgrp_1 2473 2474From the initial cgroup namespace, the real cgroup path will be 2475visible:: 2476 2477 $ cat /proc/7353/cgroup 2478 0::/batchjobs/container_id1/sub_cgrp_1 2479 2480From a sibling cgroup namespace (that is, a namespace rooted at a 2481different cgroup), the cgroup path relative to its own cgroup 2482namespace root will be shown. For instance, if PID 7353's cgroup 2483namespace root is at '/batchjobs/container_id2', then it will see:: 2484 2485 # cat /proc/7353/cgroup 2486 0::/../container_id2/sub_cgrp_1 2487 2488Note that the relative path always starts with '/' to indicate that 2489its relative to the cgroup namespace root of the caller. 2490 2491 2492Migration and setns(2) 2493---------------------- 2494 2495Processes inside a cgroup namespace can move into and out of the 2496namespace root if they have proper access to external cgroups. For 2497example, from inside a namespace with cgroupns root at 2498/batchjobs/container_id1, and assuming that the global hierarchy is 2499still accessible inside cgroupns:: 2500 2501 # cat /proc/7353/cgroup 2502 0::/sub_cgrp_1 2503 # echo 7353 > batchjobs/container_id2/cgroup.procs 2504 # cat /proc/7353/cgroup 2505 0::/../container_id2 2506 2507Note that this kind of setup is not encouraged. A task inside cgroup 2508namespace should only be exposed to its own cgroupns hierarchy. 2509 2510setns(2) to another cgroup namespace is allowed when: 2511 2512(a) the process has CAP_SYS_ADMIN against its current user namespace 2513(b) the process has CAP_SYS_ADMIN against the target cgroup 2514 namespace's userns 2515 2516No implicit cgroup changes happen with attaching to another cgroup 2517namespace. It is expected that the someone moves the attaching 2518process under the target cgroup namespace root. 2519 2520 2521Interaction with Other Namespaces 2522--------------------------------- 2523 2524Namespace specific cgroup hierarchy can be mounted by a process 2525running inside a non-init cgroup namespace:: 2526 2527 # mount -t cgroup2 none $MOUNT_POINT 2528 2529This will mount the unified cgroup hierarchy with cgroupns root as the 2530filesystem root. The process needs CAP_SYS_ADMIN against its user and 2531mount namespaces. 2532 2533The virtualization of /proc/self/cgroup file combined with restricting 2534the view of cgroup hierarchy by namespace-private cgroupfs mount 2535provides a properly isolated cgroup view inside the container. 2536 2537 2538Information on Kernel Programming 2539================================= 2540 2541This section contains kernel programming information in the areas 2542where interacting with cgroup is necessary. cgroup core and 2543controllers are not covered. 2544 2545 2546Filesystem Support for Writeback 2547-------------------------------- 2548 2549A filesystem can support cgroup writeback by updating 2550address_space_operations->writepage[s]() to annotate bio's using the 2551following two functions. 2552 2553 wbc_init_bio(@wbc, @bio) 2554 Should be called for each bio carrying writeback data and 2555 associates the bio with the inode's owner cgroup and the 2556 corresponding request queue. This must be called after 2557 a queue (device) has been associated with the bio and 2558 before submission. 2559 2560 wbc_account_cgroup_owner(@wbc, @page, @bytes) 2561 Should be called for each data segment being written out. 2562 While this function doesn't care exactly when it's called 2563 during the writeback session, it's the easiest and most 2564 natural to call it as data segments are added to a bio. 2565 2566With writeback bio's annotated, cgroup support can be enabled per 2567super_block by setting SB_I_CGROUPWB in ->s_iflags. This allows for 2568selective disabling of cgroup writeback support which is helpful when 2569certain filesystem features, e.g. journaled data mode, are 2570incompatible. 2571 2572wbc_init_bio() binds the specified bio to its cgroup. Depending on 2573the configuration, the bio may be executed at a lower priority and if 2574the writeback session is holding shared resources, e.g. a journal 2575entry, may lead to priority inversion. There is no one easy solution 2576for the problem. Filesystems can try to work around specific problem 2577cases by skipping wbc_init_bio() and using bio_associate_blkg() 2578directly. 2579 2580 2581Deprecated v1 Core Features 2582=========================== 2583 2584- Multiple hierarchies including named ones are not supported. 2585 2586- All v1 mount options are not supported. 2587 2588- The "tasks" file is removed and "cgroup.procs" is not sorted. 2589 2590- "cgroup.clone_children" is removed. 2591 2592- /proc/cgroups is meaningless for v2. Use "cgroup.controllers" file 2593 at the root instead. 2594 2595 2596Issues with v1 and Rationales for v2 2597==================================== 2598 2599Multiple Hierarchies 2600-------------------- 2601 2602cgroup v1 allowed an arbitrary number of hierarchies and each 2603hierarchy could host any number of controllers. While this seemed to 2604provide a high level of flexibility, it wasn't useful in practice. 2605 2606For example, as there is only one instance of each controller, utility 2607type controllers such as freezer which can be useful in all 2608hierarchies could only be used in one. The issue is exacerbated by 2609the fact that controllers couldn't be moved to another hierarchy once 2610hierarchies were populated. Another issue was that all controllers 2611bound to a hierarchy were forced to have exactly the same view of the 2612hierarchy. It wasn't possible to vary the granularity depending on 2613the specific controller. 2614 2615In practice, these issues heavily limited which controllers could be 2616put on the same hierarchy and most configurations resorted to putting 2617each controller on its own hierarchy. Only closely related ones, such 2618as the cpu and cpuacct controllers, made sense to be put on the same 2619hierarchy. This often meant that userland ended up managing multiple 2620similar hierarchies repeating the same steps on each hierarchy 2621whenever a hierarchy management operation was necessary. 2622 2623Furthermore, support for multiple hierarchies came at a steep cost. 2624It greatly complicated cgroup core implementation but more importantly 2625the support for multiple hierarchies restricted how cgroup could be 2626used in general and what controllers was able to do. 2627 2628There was no limit on how many hierarchies there might be, which meant 2629that a thread's cgroup membership couldn't be described in finite 2630length. The key might contain any number of entries and was unlimited 2631in length, which made it highly awkward to manipulate and led to 2632addition of controllers which existed only to identify membership, 2633which in turn exacerbated the original problem of proliferating number 2634of hierarchies. 2635 2636Also, as a controller couldn't have any expectation regarding the 2637topologies of hierarchies other controllers might be on, each 2638controller had to assume that all other controllers were attached to 2639completely orthogonal hierarchies. This made it impossible, or at 2640least very cumbersome, for controllers to cooperate with each other. 2641 2642In most use cases, putting controllers on hierarchies which are 2643completely orthogonal to each other isn't necessary. What usually is 2644called for is the ability to have differing levels of granularity 2645depending on the specific controller. In other words, hierarchy may 2646be collapsed from leaf towards root when viewed from specific 2647controllers. For example, a given configuration might not care about 2648how memory is distributed beyond a certain level while still wanting 2649to control how CPU cycles are distributed. 2650 2651 2652Thread Granularity 2653------------------ 2654 2655cgroup v1 allowed threads of a process to belong to different cgroups. 2656This didn't make sense for some controllers and those controllers 2657ended up implementing different ways to ignore such situations but 2658much more importantly it blurred the line between API exposed to 2659individual applications and system management interface. 2660 2661Generally, in-process knowledge is available only to the process 2662itself; thus, unlike service-level organization of processes, 2663categorizing threads of a process requires active participation from 2664the application which owns the target process. 2665 2666cgroup v1 had an ambiguously defined delegation model which got abused 2667in combination with thread granularity. cgroups were delegated to 2668individual applications so that they can create and manage their own 2669sub-hierarchies and control resource distributions along them. This 2670effectively raised cgroup to the status of a syscall-like API exposed 2671to lay programs. 2672 2673First of all, cgroup has a fundamentally inadequate interface to be 2674exposed this way. For a process to access its own knobs, it has to 2675extract the path on the target hierarchy from /proc/self/cgroup, 2676construct the path by appending the name of the knob to the path, open 2677and then read and/or write to it. This is not only extremely clunky 2678and unusual but also inherently racy. There is no conventional way to 2679define transaction across the required steps and nothing can guarantee 2680that the process would actually be operating on its own sub-hierarchy. 2681 2682cgroup controllers implemented a number of knobs which would never be 2683accepted as public APIs because they were just adding control knobs to 2684system-management pseudo filesystem. cgroup ended up with interface 2685knobs which were not properly abstracted or refined and directly 2686revealed kernel internal details. These knobs got exposed to 2687individual applications through the ill-defined delegation mechanism 2688effectively abusing cgroup as a shortcut to implementing public APIs 2689without going through the required scrutiny. 2690 2691This was painful for both userland and kernel. Userland ended up with 2692misbehaving and poorly abstracted interfaces and kernel exposing and 2693locked into constructs inadvertently. 2694 2695 2696Competition Between Inner Nodes and Threads 2697------------------------------------------- 2698 2699cgroup v1 allowed threads to be in any cgroups which created an 2700interesting problem where threads belonging to a parent cgroup and its 2701children cgroups competed for resources. This was nasty as two 2702different types of entities competed and there was no obvious way to 2703settle it. Different controllers did different things. 2704 2705The cpu controller considered threads and cgroups as equivalents and 2706mapped nice levels to cgroup weights. This worked for some cases but 2707fell flat when children wanted to be allocated specific ratios of CPU 2708cycles and the number of internal threads fluctuated - the ratios 2709constantly changed as the number of competing entities fluctuated. 2710There also were other issues. The mapping from nice level to weight 2711wasn't obvious or universal, and there were various other knobs which 2712simply weren't available for threads. 2713 2714The io controller implicitly created a hidden leaf node for each 2715cgroup to host the threads. The hidden leaf had its own copies of all 2716the knobs with ``leaf_`` prefixed. While this allowed equivalent 2717control over internal threads, it was with serious drawbacks. It 2718always added an extra layer of nesting which wouldn't be necessary 2719otherwise, made the interface messy and significantly complicated the 2720implementation. 2721 2722The memory controller didn't have a way to control what happened 2723between internal tasks and child cgroups and the behavior was not 2724clearly defined. There were attempts to add ad-hoc behaviors and 2725knobs to tailor the behavior to specific workloads which would have 2726led to problems extremely difficult to resolve in the long term. 2727 2728Multiple controllers struggled with internal tasks and came up with 2729different ways to deal with it; unfortunately, all the approaches were 2730severely flawed and, furthermore, the widely different behaviors 2731made cgroup as a whole highly inconsistent. 2732 2733This clearly is a problem which needs to be addressed from cgroup core 2734in a uniform way. 2735 2736 2737Other Interface Issues 2738---------------------- 2739 2740cgroup v1 grew without oversight and developed a large number of 2741idiosyncrasies and inconsistencies. One issue on the cgroup core side 2742was how an empty cgroup was notified - a userland helper binary was 2743forked and executed for each event. The event delivery wasn't 2744recursive or delegatable. The limitations of the mechanism also led 2745to in-kernel event delivery filtering mechanism further complicating 2746the interface. 2747 2748Controller interfaces were problematic too. An extreme example is 2749controllers completely ignoring hierarchical organization and treating 2750all cgroups as if they were all located directly under the root 2751cgroup. Some controllers exposed a large amount of inconsistent 2752implementation details to userland. 2753 2754There also was no consistency across controllers. When a new cgroup 2755was created, some controllers defaulted to not imposing extra 2756restrictions while others disallowed any resource usage until 2757explicitly configured. Configuration knobs for the same type of 2758control used widely differing naming schemes and formats. Statistics 2759and information knobs were named arbitrarily and used different 2760formats and units even in the same controller. 2761 2762cgroup v2 establishes common conventions where appropriate and updates 2763controllers so that they expose minimal and consistent interfaces. 2764 2765 2766Controller Issues and Remedies 2767------------------------------ 2768 2769Memory 2770~~~~~~ 2771 2772The original lower boundary, the soft limit, is defined as a limit 2773that is per default unset. As a result, the set of cgroups that 2774global reclaim prefers is opt-in, rather than opt-out. The costs for 2775optimizing these mostly negative lookups are so high that the 2776implementation, despite its enormous size, does not even provide the 2777basic desirable behavior. First off, the soft limit has no 2778hierarchical meaning. All configured groups are organized in a global 2779rbtree and treated like equal peers, regardless where they are located 2780in the hierarchy. This makes subtree delegation impossible. Second, 2781the soft limit reclaim pass is so aggressive that it not just 2782introduces high allocation latencies into the system, but also impacts 2783system performance due to overreclaim, to the point where the feature 2784becomes self-defeating. 2785 2786The memory.low boundary on the other hand is a top-down allocated 2787reserve. A cgroup enjoys reclaim protection when it's within its 2788effective low, which makes delegation of subtrees possible. It also 2789enjoys having reclaim pressure proportional to its overage when 2790above its effective low. 2791 2792The original high boundary, the hard limit, is defined as a strict 2793limit that can not budge, even if the OOM killer has to be called. 2794But this generally goes against the goal of making the most out of the 2795available memory. The memory consumption of workloads varies during 2796runtime, and that requires users to overcommit. But doing that with a 2797strict upper limit requires either a fairly accurate prediction of the 2798working set size or adding slack to the limit. Since working set size 2799estimation is hard and error prone, and getting it wrong results in 2800OOM kills, most users tend to err on the side of a looser limit and 2801end up wasting precious resources. 2802 2803The memory.high boundary on the other hand can be set much more 2804conservatively. When hit, it throttles allocations by forcing them 2805into direct reclaim to work off the excess, but it never invokes the 2806OOM killer. As a result, a high boundary that is chosen too 2807aggressively will not terminate the processes, but instead it will 2808lead to gradual performance degradation. The user can monitor this 2809and make corrections until the minimal memory footprint that still 2810gives acceptable performance is found. 2811 2812In extreme cases, with many concurrent allocations and a complete 2813breakdown of reclaim progress within the group, the high boundary can 2814be exceeded. But even then it's mostly better to satisfy the 2815allocation from the slack available in other groups or the rest of the 2816system than killing the group. Otherwise, memory.max is there to 2817limit this type of spillover and ultimately contain buggy or even 2818malicious applications. 2819 2820Setting the original memory.limit_in_bytes below the current usage was 2821subject to a race condition, where concurrent charges could cause the 2822limit setting to fail. memory.max on the other hand will first set the 2823limit to prevent new charges, and then reclaim and OOM kill until the 2824new limit is met - or the task writing to memory.max is killed. 2825 2826The combined memory+swap accounting and limiting is replaced by real 2827control over swap space. 2828 2829The main argument for a combined memory+swap facility in the original 2830cgroup design was that global or parental pressure would always be 2831able to swap all anonymous memory of a child group, regardless of the 2832child's own (possibly untrusted) configuration. However, untrusted 2833groups can sabotage swapping by other means - such as referencing its 2834anonymous memory in a tight loop - and an admin can not assume full 2835swappability when overcommitting untrusted jobs. 2836 2837For trusted jobs, on the other hand, a combined counter is not an 2838intuitive userspace interface, and it flies in the face of the idea 2839that cgroup controllers should account and limit specific physical 2840resources. Swap space is a resource like all others in the system, 2841and that's why unified hierarchy allows distributing it separately. 2842