1================ 2Control Group v2 3================ 4 5:Date: October, 2015 6:Author: Tejun Heo <tj@kernel.org> 7 8This is the authoritative documentation on the design, interface and 9conventions of cgroup v2. It describes all userland-visible aspects 10of cgroup including core and specific controller behaviors. All 11future changes must be reflected in this document. Documentation for 12v1 is available under Documentation/admin-guide/cgroup-v1/. 13 14.. CONTENTS 15 16 1. Introduction 17 1-1. Terminology 18 1-2. What is cgroup? 19 2. Basic Operations 20 2-1. Mounting 21 2-2. Organizing Processes and Threads 22 2-2-1. Processes 23 2-2-2. Threads 24 2-3. [Un]populated Notification 25 2-4. Controlling Controllers 26 2-4-1. Enabling and Disabling 27 2-4-2. Top-down Constraint 28 2-4-3. No Internal Process Constraint 29 2-5. Delegation 30 2-5-1. Model of Delegation 31 2-5-2. Delegation Containment 32 2-6. Guidelines 33 2-6-1. Organize Once and Control 34 2-6-2. Avoid Name Collisions 35 3. Resource Distribution Models 36 3-1. Weights 37 3-2. Limits 38 3-3. Protections 39 3-4. Allocations 40 4. Interface Files 41 4-1. Format 42 4-2. Conventions 43 4-3. Core Interface Files 44 5. Controllers 45 5-1. CPU 46 5-1-1. CPU Interface Files 47 5-2. Memory 48 5-2-1. Memory Interface Files 49 5-2-2. Usage Guidelines 50 5-2-3. Memory Ownership 51 5-3. IO 52 5-3-1. IO Interface Files 53 5-3-2. Writeback 54 5-3-3. IO Latency 55 5-3-3-1. How IO Latency Throttling Works 56 5-3-3-2. IO Latency Interface Files 57 5-4. PID 58 5-4-1. PID Interface Files 59 5-5. Cpuset 60 5.5-1. Cpuset Interface Files 61 5-6. Device 62 5-7. RDMA 63 5-7-1. RDMA Interface Files 64 5-8. HugeTLB 65 5.8-1. HugeTLB Interface Files 66 5-8. Misc 67 5-8-1. perf_event 68 5-N. Non-normative information 69 5-N-1. CPU controller root cgroup process behaviour 70 5-N-2. IO controller root cgroup process behaviour 71 6. Namespace 72 6-1. Basics 73 6-2. The Root and Views 74 6-3. Migration and setns(2) 75 6-4. Interaction with Other Namespaces 76 P. Information on Kernel Programming 77 P-1. Filesystem Support for Writeback 78 D. Deprecated v1 Core Features 79 R. Issues with v1 and Rationales for v2 80 R-1. Multiple Hierarchies 81 R-2. Thread Granularity 82 R-3. Competition Between Inner Nodes and Threads 83 R-4. Other Interface Issues 84 R-5. Controller Issues and Remedies 85 R-5-1. Memory 86 87 88Introduction 89============ 90 91Terminology 92----------- 93 94"cgroup" stands for "control group" and is never capitalized. The 95singular form is used to designate the whole feature and also as a 96qualifier as in "cgroup controllers". When explicitly referring to 97multiple individual control groups, the plural form "cgroups" is used. 98 99 100What is cgroup? 101--------------- 102 103cgroup is a mechanism to organize processes hierarchically and 104distribute system resources along the hierarchy in a controlled and 105configurable manner. 106 107cgroup is largely composed of two parts - the core and controllers. 108cgroup core is primarily responsible for hierarchically organizing 109processes. A cgroup controller is usually responsible for 110distributing a specific type of system resource along the hierarchy 111although there are utility controllers which serve purposes other than 112resource distribution. 113 114cgroups form a tree structure and every process in the system belongs 115to one and only one cgroup. All threads of a process belong to the 116same cgroup. On creation, all processes are put in the cgroup that 117the parent process belongs to at the time. A process can be migrated 118to another cgroup. Migration of a process doesn't affect already 119existing descendant processes. 120 121Following certain structural constraints, controllers may be enabled or 122disabled selectively on a cgroup. All controller behaviors are 123hierarchical - if a controller is enabled on a cgroup, it affects all 124processes which belong to the cgroups consisting the inclusive 125sub-hierarchy of the cgroup. When a controller is enabled on a nested 126cgroup, it always restricts the resource distribution further. The 127restrictions set closer to the root in the hierarchy can not be 128overridden from further away. 129 130 131Basic Operations 132================ 133 134Mounting 135-------- 136 137Unlike v1, cgroup v2 has only single hierarchy. The cgroup v2 138hierarchy can be mounted with the following mount command:: 139 140 # mount -t cgroup2 none $MOUNT_POINT 141 142cgroup2 filesystem has the magic number 0x63677270 ("cgrp"). All 143controllers which support v2 and are not bound to a v1 hierarchy are 144automatically bound to the v2 hierarchy and show up at the root. 145Controllers which are not in active use in the v2 hierarchy can be 146bound to other hierarchies. This allows mixing v2 hierarchy with the 147legacy v1 multiple hierarchies in a fully backward compatible way. 148 149A controller can be moved across hierarchies only after the controller 150is no longer referenced in its current hierarchy. Because per-cgroup 151controller states are destroyed asynchronously and controllers may 152have lingering references, a controller may not show up immediately on 153the v2 hierarchy after the final umount of the previous hierarchy. 154Similarly, a controller should be fully disabled to be moved out of 155the unified hierarchy and it may take some time for the disabled 156controller to become available for other hierarchies; furthermore, due 157to inter-controller dependencies, other controllers may need to be 158disabled too. 159 160While useful for development and manual configurations, moving 161controllers dynamically between the v2 and other hierarchies is 162strongly discouraged for production use. It is recommended to decide 163the hierarchies and controller associations before starting using the 164controllers after system boot. 165 166During transition to v2, system management software might still 167automount the v1 cgroup filesystem and so hijack all controllers 168during boot, before manual intervention is possible. To make testing 169and experimenting easier, the kernel parameter cgroup_no_v1= allows 170disabling controllers in v1 and make them always available in v2. 171 172cgroup v2 currently supports the following mount options. 173 174 nsdelegate 175 176 Consider cgroup namespaces as delegation boundaries. This 177 option is system wide and can only be set on mount or modified 178 through remount from the init namespace. The mount option is 179 ignored on non-init namespace mounts. Please refer to the 180 Delegation section for details. 181 182 memory_localevents 183 184 Only populate memory.events with data for the current cgroup, 185 and not any subtrees. This is legacy behaviour, the default 186 behaviour without this option is to include subtree counts. 187 This option is system wide and can only be set on mount or 188 modified through remount from the init namespace. The mount 189 option is ignored on non-init namespace mounts. 190 191 192Organizing Processes and Threads 193-------------------------------- 194 195Processes 196~~~~~~~~~ 197 198Initially, only the root cgroup exists to which all processes belong. 199A child cgroup can be created by creating a sub-directory:: 200 201 # mkdir $CGROUP_NAME 202 203A given cgroup may have multiple child cgroups forming a tree 204structure. Each cgroup has a read-writable interface file 205"cgroup.procs". When read, it lists the PIDs of all processes which 206belong to the cgroup one-per-line. The PIDs are not ordered and the 207same PID may show up more than once if the process got moved to 208another cgroup and then back or the PID got recycled while reading. 209 210A process can be migrated into a cgroup by writing its PID to the 211target cgroup's "cgroup.procs" file. Only one process can be migrated 212on a single write(2) call. If a process is composed of multiple 213threads, writing the PID of any thread migrates all threads of the 214process. 215 216When a process forks a child process, the new process is born into the 217cgroup that the forking process belongs to at the time of the 218operation. After exit, a process stays associated with the cgroup 219that it belonged to at the time of exit until it's reaped; however, a 220zombie process does not appear in "cgroup.procs" and thus can't be 221moved to another cgroup. 222 223A cgroup which doesn't have any children or live processes can be 224destroyed by removing the directory. Note that a cgroup which doesn't 225have any children and is associated only with zombie processes is 226considered empty and can be removed:: 227 228 # rmdir $CGROUP_NAME 229 230"/proc/$PID/cgroup" lists a process's cgroup membership. If legacy 231cgroup is in use in the system, this file may contain multiple lines, 232one for each hierarchy. The entry for cgroup v2 is always in the 233format "0::$PATH":: 234 235 # cat /proc/842/cgroup 236 ... 237 0::/test-cgroup/test-cgroup-nested 238 239If the process becomes a zombie and the cgroup it was associated with 240is removed subsequently, " (deleted)" is appended to the path:: 241 242 # cat /proc/842/cgroup 243 ... 244 0::/test-cgroup/test-cgroup-nested (deleted) 245 246 247Threads 248~~~~~~~ 249 250cgroup v2 supports thread granularity for a subset of controllers to 251support use cases requiring hierarchical resource distribution across 252the threads of a group of processes. By default, all threads of a 253process belong to the same cgroup, which also serves as the resource 254domain to host resource consumptions which are not specific to a 255process or thread. The thread mode allows threads to be spread across 256a subtree while still maintaining the common resource domain for them. 257 258Controllers which support thread mode are called threaded controllers. 259The ones which don't are called domain controllers. 260 261Marking a cgroup threaded makes it join the resource domain of its 262parent as a threaded cgroup. The parent may be another threaded 263cgroup whose resource domain is further up in the hierarchy. The root 264of a threaded subtree, that is, the nearest ancestor which is not 265threaded, is called threaded domain or thread root interchangeably and 266serves as the resource domain for the entire subtree. 267 268Inside a threaded subtree, threads of a process can be put in 269different cgroups and are not subject to the no internal process 270constraint - threaded controllers can be enabled on non-leaf cgroups 271whether they have threads in them or not. 272 273As the threaded domain cgroup hosts all the domain resource 274consumptions of the subtree, it is considered to have internal 275resource consumptions whether there are processes in it or not and 276can't have populated child cgroups which aren't threaded. Because the 277root cgroup is not subject to no internal process constraint, it can 278serve both as a threaded domain and a parent to domain cgroups. 279 280The current operation mode or type of the cgroup is shown in the 281"cgroup.type" file which indicates whether the cgroup is a normal 282domain, a domain which is serving as the domain of a threaded subtree, 283or a threaded cgroup. 284 285On creation, a cgroup is always a domain cgroup and can be made 286threaded by writing "threaded" to the "cgroup.type" file. The 287operation is single direction:: 288 289 # echo threaded > cgroup.type 290 291Once threaded, the cgroup can't be made a domain again. To enable the 292thread mode, the following conditions must be met. 293 294- As the cgroup will join the parent's resource domain. The parent 295 must either be a valid (threaded) domain or a threaded cgroup. 296 297- When the parent is an unthreaded domain, it must not have any domain 298 controllers enabled or populated domain children. The root is 299 exempt from this requirement. 300 301Topology-wise, a cgroup can be in an invalid state. Please consider 302the following topology:: 303 304 A (threaded domain) - B (threaded) - C (domain, just created) 305 306C is created as a domain but isn't connected to a parent which can 307host child domains. C can't be used until it is turned into a 308threaded cgroup. "cgroup.type" file will report "domain (invalid)" in 309these cases. Operations which fail due to invalid topology use 310EOPNOTSUPP as the errno. 311 312A domain cgroup is turned into a threaded domain when one of its child 313cgroup becomes threaded or threaded controllers are enabled in the 314"cgroup.subtree_control" file while there are processes in the cgroup. 315A threaded domain reverts to a normal domain when the conditions 316clear. 317 318When read, "cgroup.threads" contains the list of the thread IDs of all 319threads in the cgroup. Except that the operations are per-thread 320instead of per-process, "cgroup.threads" has the same format and 321behaves the same way as "cgroup.procs". While "cgroup.threads" can be 322written to in any cgroup, as it can only move threads inside the same 323threaded domain, its operations are confined inside each threaded 324subtree. 325 326The threaded domain cgroup serves as the resource domain for the whole 327subtree, and, while the threads can be scattered across the subtree, 328all the processes are considered to be in the threaded domain cgroup. 329"cgroup.procs" in a threaded domain cgroup contains the PIDs of all 330processes in the subtree and is not readable in the subtree proper. 331However, "cgroup.procs" can be written to from anywhere in the subtree 332to migrate all threads of the matching process to the cgroup. 333 334Only threaded controllers can be enabled in a threaded subtree. When 335a threaded controller is enabled inside a threaded subtree, it only 336accounts for and controls resource consumptions associated with the 337threads in the cgroup and its descendants. All consumptions which 338aren't tied to a specific thread belong to the threaded domain cgroup. 339 340Because a threaded subtree is exempt from no internal process 341constraint, a threaded controller must be able to handle competition 342between threads in a non-leaf cgroup and its child cgroups. Each 343threaded controller defines how such competitions are handled. 344 345 346[Un]populated Notification 347-------------------------- 348 349Each non-root cgroup has a "cgroup.events" file which contains 350"populated" field indicating whether the cgroup's sub-hierarchy has 351live processes in it. Its value is 0 if there is no live process in 352the cgroup and its descendants; otherwise, 1. poll and [id]notify 353events are triggered when the value changes. This can be used, for 354example, to start a clean-up operation after all processes of a given 355sub-hierarchy have exited. The populated state updates and 356notifications are recursive. Consider the following sub-hierarchy 357where the numbers in the parentheses represent the numbers of processes 358in each cgroup:: 359 360 A(4) - B(0) - C(1) 361 \ D(0) 362 363A, B and C's "populated" fields would be 1 while D's 0. After the one 364process in C exits, B and C's "populated" fields would flip to "0" and 365file modified events will be generated on the "cgroup.events" files of 366both cgroups. 367 368 369Controlling Controllers 370----------------------- 371 372Enabling and Disabling 373~~~~~~~~~~~~~~~~~~~~~~ 374 375Each cgroup has a "cgroup.controllers" file which lists all 376controllers available for the cgroup to enable:: 377 378 # cat cgroup.controllers 379 cpu io memory 380 381No controller is enabled by default. Controllers can be enabled and 382disabled by writing to the "cgroup.subtree_control" file:: 383 384 # echo "+cpu +memory -io" > cgroup.subtree_control 385 386Only controllers which are listed in "cgroup.controllers" can be 387enabled. When multiple operations are specified as above, either they 388all succeed or fail. If multiple operations on the same controller 389are specified, the last one is effective. 390 391Enabling a controller in a cgroup indicates that the distribution of 392the target resource across its immediate children will be controlled. 393Consider the following sub-hierarchy. The enabled controllers are 394listed in parentheses:: 395 396 A(cpu,memory) - B(memory) - C() 397 \ D() 398 399As A has "cpu" and "memory" enabled, A will control the distribution 400of CPU cycles and memory to its children, in this case, B. As B has 401"memory" enabled but not "CPU", C and D will compete freely on CPU 402cycles but their division of memory available to B will be controlled. 403 404As a controller regulates the distribution of the target resource to 405the cgroup's children, enabling it creates the controller's interface 406files in the child cgroups. In the above example, enabling "cpu" on B 407would create the "cpu." prefixed controller interface files in C and 408D. Likewise, disabling "memory" from B would remove the "memory." 409prefixed controller interface files from C and D. This means that the 410controller interface files - anything which doesn't start with 411"cgroup." are owned by the parent rather than the cgroup itself. 412 413 414Top-down Constraint 415~~~~~~~~~~~~~~~~~~~ 416 417Resources are distributed top-down and a cgroup can further distribute 418a resource only if the resource has been distributed to it from the 419parent. This means that all non-root "cgroup.subtree_control" files 420can only contain controllers which are enabled in the parent's 421"cgroup.subtree_control" file. A controller can be enabled only if 422the parent has the controller enabled and a controller can't be 423disabled if one or more children have it enabled. 424 425 426No Internal Process Constraint 427~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 428 429Non-root cgroups can distribute domain resources to their children 430only when they don't have any processes of their own. In other words, 431only domain cgroups which don't contain any processes can have domain 432controllers enabled in their "cgroup.subtree_control" files. 433 434This guarantees that, when a domain controller is looking at the part 435of the hierarchy which has it enabled, processes are always only on 436the leaves. This rules out situations where child cgroups compete 437against internal processes of the parent. 438 439The root cgroup is exempt from this restriction. Root contains 440processes and anonymous resource consumption which can't be associated 441with any other cgroups and requires special treatment from most 442controllers. How resource consumption in the root cgroup is governed 443is up to each controller (for more information on this topic please 444refer to the Non-normative information section in the Controllers 445chapter). 446 447Note that the restriction doesn't get in the way if there is no 448enabled controller in the cgroup's "cgroup.subtree_control". This is 449important as otherwise it wouldn't be possible to create children of a 450populated cgroup. To control resource distribution of a cgroup, the 451cgroup must create children and transfer all its processes to the 452children before enabling controllers in its "cgroup.subtree_control" 453file. 454 455 456Delegation 457---------- 458 459Model of Delegation 460~~~~~~~~~~~~~~~~~~~ 461 462A cgroup can be delegated in two ways. First, to a less privileged 463user by granting write access of the directory and its "cgroup.procs", 464"cgroup.threads" and "cgroup.subtree_control" files to the user. 465Second, if the "nsdelegate" mount option is set, automatically to a 466cgroup namespace on namespace creation. 467 468Because the resource control interface files in a given directory 469control the distribution of the parent's resources, the delegatee 470shouldn't be allowed to write to them. For the first method, this is 471achieved by not granting access to these files. For the second, the 472kernel rejects writes to all files other than "cgroup.procs" and 473"cgroup.subtree_control" on a namespace root from inside the 474namespace. 475 476The end results are equivalent for both delegation types. Once 477delegated, the user can build sub-hierarchy under the directory, 478organize processes inside it as it sees fit and further distribute the 479resources it received from the parent. The limits and other settings 480of all resource controllers are hierarchical and regardless of what 481happens in the delegated sub-hierarchy, nothing can escape the 482resource restrictions imposed by the parent. 483 484Currently, cgroup doesn't impose any restrictions on the number of 485cgroups in or nesting depth of a delegated sub-hierarchy; however, 486this may be limited explicitly in the future. 487 488 489Delegation Containment 490~~~~~~~~~~~~~~~~~~~~~~ 491 492A delegated sub-hierarchy is contained in the sense that processes 493can't be moved into or out of the sub-hierarchy by the delegatee. 494 495For delegations to a less privileged user, this is achieved by 496requiring the following conditions for a process with a non-root euid 497to migrate a target process into a cgroup by writing its PID to the 498"cgroup.procs" file. 499 500- The writer must have write access to the "cgroup.procs" file. 501 502- The writer must have write access to the "cgroup.procs" file of the 503 common ancestor of the source and destination cgroups. 504 505The above two constraints ensure that while a delegatee may migrate 506processes around freely in the delegated sub-hierarchy it can't pull 507in from or push out to outside the sub-hierarchy. 508 509For an example, let's assume cgroups C0 and C1 have been delegated to 510user U0 who created C00, C01 under C0 and C10 under C1 as follows and 511all processes under C0 and C1 belong to U0:: 512 513 ~~~~~~~~~~~~~ - C0 - C00 514 ~ cgroup ~ \ C01 515 ~ hierarchy ~ 516 ~~~~~~~~~~~~~ - C1 - C10 517 518Let's also say U0 wants to write the PID of a process which is 519currently in C10 into "C00/cgroup.procs". U0 has write access to the 520file; however, the common ancestor of the source cgroup C10 and the 521destination cgroup C00 is above the points of delegation and U0 would 522not have write access to its "cgroup.procs" files and thus the write 523will be denied with -EACCES. 524 525For delegations to namespaces, containment is achieved by requiring 526that both the source and destination cgroups are reachable from the 527namespace of the process which is attempting the migration. If either 528is not reachable, the migration is rejected with -ENOENT. 529 530 531Guidelines 532---------- 533 534Organize Once and Control 535~~~~~~~~~~~~~~~~~~~~~~~~~ 536 537Migrating a process across cgroups is a relatively expensive operation 538and stateful resources such as memory are not moved together with the 539process. This is an explicit design decision as there often exist 540inherent trade-offs between migration and various hot paths in terms 541of synchronization cost. 542 543As such, migrating processes across cgroups frequently as a means to 544apply different resource restrictions is discouraged. A workload 545should be assigned to a cgroup according to the system's logical and 546resource structure once on start-up. Dynamic adjustments to resource 547distribution can be made by changing controller configuration through 548the interface files. 549 550 551Avoid Name Collisions 552~~~~~~~~~~~~~~~~~~~~~ 553 554Interface files for a cgroup and its children cgroups occupy the same 555directory and it is possible to create children cgroups which collide 556with interface files. 557 558All cgroup core interface files are prefixed with "cgroup." and each 559controller's interface files are prefixed with the controller name and 560a dot. A controller's name is composed of lower case alphabets and 561'_'s but never begins with an '_' so it can be used as the prefix 562character for collision avoidance. Also, interface file names won't 563start or end with terms which are often used in categorizing workloads 564such as job, service, slice, unit or workload. 565 566cgroup doesn't do anything to prevent name collisions and it's the 567user's responsibility to avoid them. 568 569 570Resource Distribution Models 571============================ 572 573cgroup controllers implement several resource distribution schemes 574depending on the resource type and expected use cases. This section 575describes major schemes in use along with their expected behaviors. 576 577 578Weights 579------- 580 581A parent's resource is distributed by adding up the weights of all 582active children and giving each the fraction matching the ratio of its 583weight against the sum. As only children which can make use of the 584resource at the moment participate in the distribution, this is 585work-conserving. Due to the dynamic nature, this model is usually 586used for stateless resources. 587 588All weights are in the range [1, 10000] with the default at 100. This 589allows symmetric multiplicative biases in both directions at fine 590enough granularity while staying in the intuitive range. 591 592As long as the weight is in range, all configuration combinations are 593valid and there is no reason to reject configuration changes or 594process migrations. 595 596"cpu.weight" proportionally distributes CPU cycles to active children 597and is an example of this type. 598 599 600Limits 601------ 602 603A child can only consume upto the configured amount of the resource. 604Limits can be over-committed - the sum of the limits of children can 605exceed the amount of resource available to the parent. 606 607Limits are in the range [0, max] and defaults to "max", which is noop. 608 609As limits can be over-committed, all configuration combinations are 610valid and there is no reason to reject configuration changes or 611process migrations. 612 613"io.max" limits the maximum BPS and/or IOPS that a cgroup can consume 614on an IO device and is an example of this type. 615 616 617Protections 618----------- 619 620A cgroup is protected upto the configured amount of the resource 621as long as the usages of all its ancestors are under their 622protected levels. Protections can be hard guarantees or best effort 623soft boundaries. Protections can also be over-committed in which case 624only upto the amount available to the parent is protected among 625children. 626 627Protections are in the range [0, max] and defaults to 0, which is 628noop. 629 630As protections can be over-committed, all configuration combinations 631are valid and there is no reason to reject configuration changes or 632process migrations. 633 634"memory.low" implements best-effort memory protection and is an 635example of this type. 636 637 638Allocations 639----------- 640 641A cgroup is exclusively allocated a certain amount of a finite 642resource. Allocations can't be over-committed - the sum of the 643allocations of children can not exceed the amount of resource 644available to the parent. 645 646Allocations are in the range [0, max] and defaults to 0, which is no 647resource. 648 649As allocations can't be over-committed, some configuration 650combinations are invalid and should be rejected. Also, if the 651resource is mandatory for execution of processes, process migrations 652may be rejected. 653 654"cpu.rt.max" hard-allocates realtime slices and is an example of this 655type. 656 657 658Interface Files 659=============== 660 661Format 662------ 663 664All interface files should be in one of the following formats whenever 665possible:: 666 667 New-line separated values 668 (when only one value can be written at once) 669 670 VAL0\n 671 VAL1\n 672 ... 673 674 Space separated values 675 (when read-only or multiple values can be written at once) 676 677 VAL0 VAL1 ...\n 678 679 Flat keyed 680 681 KEY0 VAL0\n 682 KEY1 VAL1\n 683 ... 684 685 Nested keyed 686 687 KEY0 SUB_KEY0=VAL00 SUB_KEY1=VAL01... 688 KEY1 SUB_KEY0=VAL10 SUB_KEY1=VAL11... 689 ... 690 691For a writable file, the format for writing should generally match 692reading; however, controllers may allow omitting later fields or 693implement restricted shortcuts for most common use cases. 694 695For both flat and nested keyed files, only the values for a single key 696can be written at a time. For nested keyed files, the sub key pairs 697may be specified in any order and not all pairs have to be specified. 698 699 700Conventions 701----------- 702 703- Settings for a single feature should be contained in a single file. 704 705- The root cgroup should be exempt from resource control and thus 706 shouldn't have resource control interface files. Also, 707 informational files on the root cgroup which end up showing global 708 information available elsewhere shouldn't exist. 709 710- The default time unit is microseconds. If a different unit is ever 711 used, an explicit unit suffix must be present. 712 713- A parts-per quantity should use a percentage decimal with at least 714 two digit fractional part - e.g. 13.40. 715 716- If a controller implements weight based resource distribution, its 717 interface file should be named "weight" and have the range [1, 718 10000] with 100 as the default. The values are chosen to allow 719 enough and symmetric bias in both directions while keeping it 720 intuitive (the default is 100%). 721 722- If a controller implements an absolute resource guarantee and/or 723 limit, the interface files should be named "min" and "max" 724 respectively. If a controller implements best effort resource 725 guarantee and/or limit, the interface files should be named "low" 726 and "high" respectively. 727 728 In the above four control files, the special token "max" should be 729 used to represent upward infinity for both reading and writing. 730 731- If a setting has a configurable default value and keyed specific 732 overrides, the default entry should be keyed with "default" and 733 appear as the first entry in the file. 734 735 The default value can be updated by writing either "default $VAL" or 736 "$VAL". 737 738 When writing to update a specific override, "default" can be used as 739 the value to indicate removal of the override. Override entries 740 with "default" as the value must not appear when read. 741 742 For example, a setting which is keyed by major:minor device numbers 743 with integer values may look like the following:: 744 745 # cat cgroup-example-interface-file 746 default 150 747 8:0 300 748 749 The default value can be updated by:: 750 751 # echo 125 > cgroup-example-interface-file 752 753 or:: 754 755 # echo "default 125" > cgroup-example-interface-file 756 757 An override can be set by:: 758 759 # echo "8:16 170" > cgroup-example-interface-file 760 761 and cleared by:: 762 763 # echo "8:0 default" > cgroup-example-interface-file 764 # cat cgroup-example-interface-file 765 default 125 766 8:16 170 767 768- For events which are not very high frequency, an interface file 769 "events" should be created which lists event key value pairs. 770 Whenever a notifiable event happens, file modified event should be 771 generated on the file. 772 773 774Core Interface Files 775-------------------- 776 777All cgroup core files are prefixed with "cgroup." 778 779 cgroup.type 780 781 A read-write single value file which exists on non-root 782 cgroups. 783 784 When read, it indicates the current type of the cgroup, which 785 can be one of the following values. 786 787 - "domain" : A normal valid domain cgroup. 788 789 - "domain threaded" : A threaded domain cgroup which is 790 serving as the root of a threaded subtree. 791 792 - "domain invalid" : A cgroup which is in an invalid state. 793 It can't be populated or have controllers enabled. It may 794 be allowed to become a threaded cgroup. 795 796 - "threaded" : A threaded cgroup which is a member of a 797 threaded subtree. 798 799 A cgroup can be turned into a threaded cgroup by writing 800 "threaded" to this file. 801 802 cgroup.procs 803 A read-write new-line separated values file which exists on 804 all cgroups. 805 806 When read, it lists the PIDs of all processes which belong to 807 the cgroup one-per-line. The PIDs are not ordered and the 808 same PID may show up more than once if the process got moved 809 to another cgroup and then back or the PID got recycled while 810 reading. 811 812 A PID can be written to migrate the process associated with 813 the PID to the cgroup. The writer should match all of the 814 following conditions. 815 816 - It must have write access to the "cgroup.procs" file. 817 818 - It must have write access to the "cgroup.procs" file of the 819 common ancestor of the source and destination cgroups. 820 821 When delegating a sub-hierarchy, write access to this file 822 should be granted along with the containing directory. 823 824 In a threaded cgroup, reading this file fails with EOPNOTSUPP 825 as all the processes belong to the thread root. Writing is 826 supported and moves every thread of the process to the cgroup. 827 828 cgroup.threads 829 A read-write new-line separated values file which exists on 830 all cgroups. 831 832 When read, it lists the TIDs of all threads which belong to 833 the cgroup one-per-line. The TIDs are not ordered and the 834 same TID may show up more than once if the thread got moved to 835 another cgroup and then back or the TID got recycled while 836 reading. 837 838 A TID can be written to migrate the thread associated with the 839 TID to the cgroup. The writer should match all of the 840 following conditions. 841 842 - It must have write access to the "cgroup.threads" file. 843 844 - The cgroup that the thread is currently in must be in the 845 same resource domain as the destination cgroup. 846 847 - It must have write access to the "cgroup.procs" file of the 848 common ancestor of the source and destination cgroups. 849 850 When delegating a sub-hierarchy, write access to this file 851 should be granted along with the containing directory. 852 853 cgroup.controllers 854 A read-only space separated values file which exists on all 855 cgroups. 856 857 It shows space separated list of all controllers available to 858 the cgroup. The controllers are not ordered. 859 860 cgroup.subtree_control 861 A read-write space separated values file which exists on all 862 cgroups. Starts out empty. 863 864 When read, it shows space separated list of the controllers 865 which are enabled to control resource distribution from the 866 cgroup to its children. 867 868 Space separated list of controllers prefixed with '+' or '-' 869 can be written to enable or disable controllers. A controller 870 name prefixed with '+' enables the controller and '-' 871 disables. If a controller appears more than once on the list, 872 the last one is effective. When multiple enable and disable 873 operations are specified, either all succeed or all fail. 874 875 cgroup.events 876 A read-only flat-keyed file which exists on non-root cgroups. 877 The following entries are defined. Unless specified 878 otherwise, a value change in this file generates a file 879 modified event. 880 881 populated 882 1 if the cgroup or its descendants contains any live 883 processes; otherwise, 0. 884 frozen 885 1 if the cgroup is frozen; otherwise, 0. 886 887 cgroup.max.descendants 888 A read-write single value files. The default is "max". 889 890 Maximum allowed number of descent cgroups. 891 If the actual number of descendants is equal or larger, 892 an attempt to create a new cgroup in the hierarchy will fail. 893 894 cgroup.max.depth 895 A read-write single value files. The default is "max". 896 897 Maximum allowed descent depth below the current cgroup. 898 If the actual descent depth is equal or larger, 899 an attempt to create a new child cgroup will fail. 900 901 cgroup.stat 902 A read-only flat-keyed file with the following entries: 903 904 nr_descendants 905 Total number of visible descendant cgroups. 906 907 nr_dying_descendants 908 Total number of dying descendant cgroups. A cgroup becomes 909 dying after being deleted by a user. The cgroup will remain 910 in dying state for some time undefined time (which can depend 911 on system load) before being completely destroyed. 912 913 A process can't enter a dying cgroup under any circumstances, 914 a dying cgroup can't revive. 915 916 A dying cgroup can consume system resources not exceeding 917 limits, which were active at the moment of cgroup deletion. 918 919 cgroup.freeze 920 A read-write single value file which exists on non-root cgroups. 921 Allowed values are "0" and "1". The default is "0". 922 923 Writing "1" to the file causes freezing of the cgroup and all 924 descendant cgroups. This means that all belonging processes will 925 be stopped and will not run until the cgroup will be explicitly 926 unfrozen. Freezing of the cgroup may take some time; when this action 927 is completed, the "frozen" value in the cgroup.events control file 928 will be updated to "1" and the corresponding notification will be 929 issued. 930 931 A cgroup can be frozen either by its own settings, or by settings 932 of any ancestor cgroups. If any of ancestor cgroups is frozen, the 933 cgroup will remain frozen. 934 935 Processes in the frozen cgroup can be killed by a fatal signal. 936 They also can enter and leave a frozen cgroup: either by an explicit 937 move by a user, or if freezing of the cgroup races with fork(). 938 If a process is moved to a frozen cgroup, it stops. If a process is 939 moved out of a frozen cgroup, it becomes running. 940 941 Frozen status of a cgroup doesn't affect any cgroup tree operations: 942 it's possible to delete a frozen (and empty) cgroup, as well as 943 create new sub-cgroups. 944 945Controllers 946=========== 947 948CPU 949--- 950 951The "cpu" controllers regulates distribution of CPU cycles. This 952controller implements weight and absolute bandwidth limit models for 953normal scheduling policy and absolute bandwidth allocation model for 954realtime scheduling policy. 955 956In all the above models, cycles distribution is defined only on a temporal 957base and it does not account for the frequency at which tasks are executed. 958The (optional) utilization clamping support allows to hint the schedutil 959cpufreq governor about the minimum desired frequency which should always be 960provided by a CPU, as well as the maximum desired frequency, which should not 961be exceeded by a CPU. 962 963WARNING: cgroup2 doesn't yet support control of realtime processes and 964the cpu controller can only be enabled when all RT processes are in 965the root cgroup. Be aware that system management software may already 966have placed RT processes into nonroot cgroups during the system boot 967process, and these processes may need to be moved to the root cgroup 968before the cpu controller can be enabled. 969 970 971CPU Interface Files 972~~~~~~~~~~~~~~~~~~~ 973 974All time durations are in microseconds. 975 976 cpu.stat 977 A read-only flat-keyed file which exists on non-root cgroups. 978 This file exists whether the controller is enabled or not. 979 980 It always reports the following three stats: 981 982 - usage_usec 983 - user_usec 984 - system_usec 985 986 and the following three when the controller is enabled: 987 988 - nr_periods 989 - nr_throttled 990 - throttled_usec 991 992 cpu.weight 993 A read-write single value file which exists on non-root 994 cgroups. The default is "100". 995 996 The weight in the range [1, 10000]. 997 998 cpu.weight.nice 999 A read-write single value file which exists on non-root 1000 cgroups. The default is "0". 1001 1002 The nice value is in the range [-20, 19]. 1003 1004 This interface file is an alternative interface for 1005 "cpu.weight" and allows reading and setting weight using the 1006 same values used by nice(2). Because the range is smaller and 1007 granularity is coarser for the nice values, the read value is 1008 the closest approximation of the current weight. 1009 1010 cpu.max 1011 A read-write two value file which exists on non-root cgroups. 1012 The default is "max 100000". 1013 1014 The maximum bandwidth limit. It's in the following format:: 1015 1016 $MAX $PERIOD 1017 1018 which indicates that the group may consume upto $MAX in each 1019 $PERIOD duration. "max" for $MAX indicates no limit. If only 1020 one number is written, $MAX is updated. 1021 1022 cpu.pressure 1023 A read-only nested-key file which exists on non-root cgroups. 1024 1025 Shows pressure stall information for CPU. See 1026 Documentation/accounting/psi.rst for details. 1027 1028 cpu.uclamp.min 1029 A read-write single value file which exists on non-root cgroups. 1030 The default is "0", i.e. no utilization boosting. 1031 1032 The requested minimum utilization (protection) as a percentage 1033 rational number, e.g. 12.34 for 12.34%. 1034 1035 This interface allows reading and setting minimum utilization clamp 1036 values similar to the sched_setattr(2). This minimum utilization 1037 value is used to clamp the task specific minimum utilization clamp. 1038 1039 The requested minimum utilization (protection) is always capped by 1040 the current value for the maximum utilization (limit), i.e. 1041 `cpu.uclamp.max`. 1042 1043 cpu.uclamp.max 1044 A read-write single value file which exists on non-root cgroups. 1045 The default is "max". i.e. no utilization capping 1046 1047 The requested maximum utilization (limit) as a percentage rational 1048 number, e.g. 98.76 for 98.76%. 1049 1050 This interface allows reading and setting maximum utilization clamp 1051 values similar to the sched_setattr(2). This maximum utilization 1052 value is used to clamp the task specific maximum utilization clamp. 1053 1054 1055 1056Memory 1057------ 1058 1059The "memory" controller regulates distribution of memory. Memory is 1060stateful and implements both limit and protection models. Due to the 1061intertwining between memory usage and reclaim pressure and the 1062stateful nature of memory, the distribution model is relatively 1063complex. 1064 1065While not completely water-tight, all major memory usages by a given 1066cgroup are tracked so that the total memory consumption can be 1067accounted and controlled to a reasonable extent. Currently, the 1068following types of memory usages are tracked. 1069 1070- Userland memory - page cache and anonymous memory. 1071 1072- Kernel data structures such as dentries and inodes. 1073 1074- TCP socket buffers. 1075 1076The above list may expand in the future for better coverage. 1077 1078 1079Memory Interface Files 1080~~~~~~~~~~~~~~~~~~~~~~ 1081 1082All memory amounts are in bytes. If a value which is not aligned to 1083PAGE_SIZE is written, the value may be rounded up to the closest 1084PAGE_SIZE multiple when read back. 1085 1086 memory.current 1087 A read-only single value file which exists on non-root 1088 cgroups. 1089 1090 The total amount of memory currently being used by the cgroup 1091 and its descendants. 1092 1093 memory.min 1094 A read-write single value file which exists on non-root 1095 cgroups. The default is "0". 1096 1097 Hard memory protection. If the memory usage of a cgroup 1098 is within its effective min boundary, the cgroup's memory 1099 won't be reclaimed under any conditions. If there is no 1100 unprotected reclaimable memory available, OOM killer 1101 is invoked. Above the effective min boundary (or 1102 effective low boundary if it is higher), pages are reclaimed 1103 proportionally to the overage, reducing reclaim pressure for 1104 smaller overages. 1105 1106 Effective min boundary is limited by memory.min values of 1107 all ancestor cgroups. If there is memory.min overcommitment 1108 (child cgroup or cgroups are requiring more protected memory 1109 than parent will allow), then each child cgroup will get 1110 the part of parent's protection proportional to its 1111 actual memory usage below memory.min. 1112 1113 Putting more memory than generally available under this 1114 protection is discouraged and may lead to constant OOMs. 1115 1116 If a memory cgroup is not populated with processes, 1117 its memory.min is ignored. 1118 1119 memory.low 1120 A read-write single value file which exists on non-root 1121 cgroups. The default is "0". 1122 1123 Best-effort memory protection. If the memory usage of a 1124 cgroup is within its effective low boundary, the cgroup's 1125 memory won't be reclaimed unless there is no reclaimable 1126 memory available in unprotected cgroups. 1127 Above the effective low boundary (or 1128 effective min boundary if it is higher), pages are reclaimed 1129 proportionally to the overage, reducing reclaim pressure for 1130 smaller overages. 1131 1132 Effective low boundary is limited by memory.low values of 1133 all ancestor cgroups. If there is memory.low overcommitment 1134 (child cgroup or cgroups are requiring more protected memory 1135 than parent will allow), then each child cgroup will get 1136 the part of parent's protection proportional to its 1137 actual memory usage below memory.low. 1138 1139 Putting more memory than generally available under this 1140 protection is discouraged. 1141 1142 memory.high 1143 A read-write single value file which exists on non-root 1144 cgroups. The default is "max". 1145 1146 Memory usage throttle limit. This is the main mechanism to 1147 control memory usage of a cgroup. If a cgroup's usage goes 1148 over the high boundary, the processes of the cgroup are 1149 throttled and put under heavy reclaim pressure. 1150 1151 Going over the high limit never invokes the OOM killer and 1152 under extreme conditions the limit may be breached. 1153 1154 memory.max 1155 A read-write single value file which exists on non-root 1156 cgroups. The default is "max". 1157 1158 Memory usage hard limit. This is the final protection 1159 mechanism. If a cgroup's memory usage reaches this limit and 1160 can't be reduced, the OOM killer is invoked in the cgroup. 1161 Under certain circumstances, the usage may go over the limit 1162 temporarily. 1163 1164 This is the ultimate protection mechanism. As long as the 1165 high limit is used and monitored properly, this limit's 1166 utility is limited to providing the final safety net. 1167 1168 memory.oom.group 1169 A read-write single value file which exists on non-root 1170 cgroups. The default value is "0". 1171 1172 Determines whether the cgroup should be treated as 1173 an indivisible workload by the OOM killer. If set, 1174 all tasks belonging to the cgroup or to its descendants 1175 (if the memory cgroup is not a leaf cgroup) are killed 1176 together or not at all. This can be used to avoid 1177 partial kills to guarantee workload integrity. 1178 1179 Tasks with the OOM protection (oom_score_adj set to -1000) 1180 are treated as an exception and are never killed. 1181 1182 If the OOM killer is invoked in a cgroup, it's not going 1183 to kill any tasks outside of this cgroup, regardless 1184 memory.oom.group values of ancestor cgroups. 1185 1186 memory.events 1187 A read-only flat-keyed file which exists on non-root cgroups. 1188 The following entries are defined. Unless specified 1189 otherwise, a value change in this file generates a file 1190 modified event. 1191 1192 Note that all fields in this file are hierarchical and the 1193 file modified event can be generated due to an event down the 1194 hierarchy. For for the local events at the cgroup level see 1195 memory.events.local. 1196 1197 low 1198 The number of times the cgroup is reclaimed due to 1199 high memory pressure even though its usage is under 1200 the low boundary. This usually indicates that the low 1201 boundary is over-committed. 1202 1203 high 1204 The number of times processes of the cgroup are 1205 throttled and routed to perform direct memory reclaim 1206 because the high memory boundary was exceeded. For a 1207 cgroup whose memory usage is capped by the high limit 1208 rather than global memory pressure, this event's 1209 occurrences are expected. 1210 1211 max 1212 The number of times the cgroup's memory usage was 1213 about to go over the max boundary. If direct reclaim 1214 fails to bring it down, the cgroup goes to OOM state. 1215 1216 oom 1217 The number of time the cgroup's memory usage was 1218 reached the limit and allocation was about to fail. 1219 1220 Depending on context result could be invocation of OOM 1221 killer and retrying allocation or failing allocation. 1222 1223 Failed allocation in its turn could be returned into 1224 userspace as -ENOMEM or silently ignored in cases like 1225 disk readahead. For now OOM in memory cgroup kills 1226 tasks iff shortage has happened inside page fault. 1227 1228 This event is not raised if the OOM killer is not 1229 considered as an option, e.g. for failed high-order 1230 allocations. 1231 1232 oom_kill 1233 The number of processes belonging to this cgroup 1234 killed by any kind of OOM killer. 1235 1236 memory.events.local 1237 Similar to memory.events but the fields in the file are local 1238 to the cgroup i.e. not hierarchical. The file modified event 1239 generated on this file reflects only the local events. 1240 1241 memory.stat 1242 A read-only flat-keyed file which exists on non-root cgroups. 1243 1244 This breaks down the cgroup's memory footprint into different 1245 types of memory, type-specific details, and other information 1246 on the state and past events of the memory management system. 1247 1248 All memory amounts are in bytes. 1249 1250 The entries are ordered to be human readable, and new entries 1251 can show up in the middle. Don't rely on items remaining in a 1252 fixed position; use the keys to look up specific values! 1253 1254 anon 1255 Amount of memory used in anonymous mappings such as 1256 brk(), sbrk(), and mmap(MAP_ANONYMOUS) 1257 1258 file 1259 Amount of memory used to cache filesystem data, 1260 including tmpfs and shared memory. 1261 1262 kernel_stack 1263 Amount of memory allocated to kernel stacks. 1264 1265 slab 1266 Amount of memory used for storing in-kernel data 1267 structures. 1268 1269 sock 1270 Amount of memory used in network transmission buffers 1271 1272 shmem 1273 Amount of cached filesystem data that is swap-backed, 1274 such as tmpfs, shm segments, shared anonymous mmap()s 1275 1276 file_mapped 1277 Amount of cached filesystem data mapped with mmap() 1278 1279 file_dirty 1280 Amount of cached filesystem data that was modified but 1281 not yet written back to disk 1282 1283 file_writeback 1284 Amount of cached filesystem data that was modified and 1285 is currently being written back to disk 1286 1287 anon_thp 1288 Amount of memory used in anonymous mappings backed by 1289 transparent hugepages 1290 1291 inactive_anon, active_anon, inactive_file, active_file, unevictable 1292 Amount of memory, swap-backed and filesystem-backed, 1293 on the internal memory management lists used by the 1294 page reclaim algorithm. 1295 1296 As these represent internal list state (eg. shmem pages are on anon 1297 memory management lists), inactive_foo + active_foo may not be equal to 1298 the value for the foo counter, since the foo counter is type-based, not 1299 list-based. 1300 1301 slab_reclaimable 1302 Part of "slab" that might be reclaimed, such as 1303 dentries and inodes. 1304 1305 slab_unreclaimable 1306 Part of "slab" that cannot be reclaimed on memory 1307 pressure. 1308 1309 pgfault 1310 Total number of page faults incurred 1311 1312 pgmajfault 1313 Number of major page faults incurred 1314 1315 workingset_refault 1316 1317 Number of refaults of previously evicted pages 1318 1319 workingset_activate 1320 1321 Number of refaulted pages that were immediately activated 1322 1323 workingset_nodereclaim 1324 1325 Number of times a shadow node has been reclaimed 1326 1327 pgrefill 1328 1329 Amount of scanned pages (in an active LRU list) 1330 1331 pgscan 1332 1333 Amount of scanned pages (in an inactive LRU list) 1334 1335 pgsteal 1336 1337 Amount of reclaimed pages 1338 1339 pgactivate 1340 1341 Amount of pages moved to the active LRU list 1342 1343 pgdeactivate 1344 1345 Amount of pages moved to the inactive LRU list 1346 1347 pglazyfree 1348 1349 Amount of pages postponed to be freed under memory pressure 1350 1351 pglazyfreed 1352 1353 Amount of reclaimed lazyfree pages 1354 1355 thp_fault_alloc 1356 1357 Number of transparent hugepages which were allocated to satisfy 1358 a page fault, including COW faults. This counter is not present 1359 when CONFIG_TRANSPARENT_HUGEPAGE is not set. 1360 1361 thp_collapse_alloc 1362 1363 Number of transparent hugepages which were allocated to allow 1364 collapsing an existing range of pages. This counter is not 1365 present when CONFIG_TRANSPARENT_HUGEPAGE is not set. 1366 1367 memory.swap.current 1368 A read-only single value file which exists on non-root 1369 cgroups. 1370 1371 The total amount of swap currently being used by the cgroup 1372 and its descendants. 1373 1374 memory.swap.max 1375 A read-write single value file which exists on non-root 1376 cgroups. The default is "max". 1377 1378 Swap usage hard limit. If a cgroup's swap usage reaches this 1379 limit, anonymous memory of the cgroup will not be swapped out. 1380 1381 memory.swap.events 1382 A read-only flat-keyed file which exists on non-root cgroups. 1383 The following entries are defined. Unless specified 1384 otherwise, a value change in this file generates a file 1385 modified event. 1386 1387 max 1388 The number of times the cgroup's swap usage was about 1389 to go over the max boundary and swap allocation 1390 failed. 1391 1392 fail 1393 The number of times swap allocation failed either 1394 because of running out of swap system-wide or max 1395 limit. 1396 1397 When reduced under the current usage, the existing swap 1398 entries are reclaimed gradually and the swap usage may stay 1399 higher than the limit for an extended period of time. This 1400 reduces the impact on the workload and memory management. 1401 1402 memory.pressure 1403 A read-only nested-key file which exists on non-root cgroups. 1404 1405 Shows pressure stall information for memory. See 1406 Documentation/accounting/psi.rst for details. 1407 1408 1409Usage Guidelines 1410~~~~~~~~~~~~~~~~ 1411 1412"memory.high" is the main mechanism to control memory usage. 1413Over-committing on high limit (sum of high limits > available memory) 1414and letting global memory pressure to distribute memory according to 1415usage is a viable strategy. 1416 1417Because breach of the high limit doesn't trigger the OOM killer but 1418throttles the offending cgroup, a management agent has ample 1419opportunities to monitor and take appropriate actions such as granting 1420more memory or terminating the workload. 1421 1422Determining whether a cgroup has enough memory is not trivial as 1423memory usage doesn't indicate whether the workload can benefit from 1424more memory. For example, a workload which writes data received from 1425network to a file can use all available memory but can also operate as 1426performant with a small amount of memory. A measure of memory 1427pressure - how much the workload is being impacted due to lack of 1428memory - is necessary to determine whether a workload needs more 1429memory; unfortunately, memory pressure monitoring mechanism isn't 1430implemented yet. 1431 1432 1433Memory Ownership 1434~~~~~~~~~~~~~~~~ 1435 1436A memory area is charged to the cgroup which instantiated it and stays 1437charged to the cgroup until the area is released. Migrating a process 1438to a different cgroup doesn't move the memory usages that it 1439instantiated while in the previous cgroup to the new cgroup. 1440 1441A memory area may be used by processes belonging to different cgroups. 1442To which cgroup the area will be charged is in-deterministic; however, 1443over time, the memory area is likely to end up in a cgroup which has 1444enough memory allowance to avoid high reclaim pressure. 1445 1446If a cgroup sweeps a considerable amount of memory which is expected 1447to be accessed repeatedly by other cgroups, it may make sense to use 1448POSIX_FADV_DONTNEED to relinquish the ownership of memory areas 1449belonging to the affected files to ensure correct memory ownership. 1450 1451 1452IO 1453-- 1454 1455The "io" controller regulates the distribution of IO resources. This 1456controller implements both weight based and absolute bandwidth or IOPS 1457limit distribution; however, weight based distribution is available 1458only if cfq-iosched is in use and neither scheme is available for 1459blk-mq devices. 1460 1461 1462IO Interface Files 1463~~~~~~~~~~~~~~~~~~ 1464 1465 io.stat 1466 A read-only nested-keyed file which exists on non-root 1467 cgroups. 1468 1469 Lines are keyed by $MAJ:$MIN device numbers and not ordered. 1470 The following nested keys are defined. 1471 1472 ====== ===================== 1473 rbytes Bytes read 1474 wbytes Bytes written 1475 rios Number of read IOs 1476 wios Number of write IOs 1477 dbytes Bytes discarded 1478 dios Number of discard IOs 1479 ====== ===================== 1480 1481 An example read output follows: 1482 1483 8:16 rbytes=1459200 wbytes=314773504 rios=192 wios=353 dbytes=0 dios=0 1484 8:0 rbytes=90430464 wbytes=299008000 rios=8950 wios=1252 dbytes=50331648 dios=3021 1485 1486 io.cost.qos 1487 A read-write nested-keyed file with exists only on the root 1488 cgroup. 1489 1490 This file configures the Quality of Service of the IO cost 1491 model based controller (CONFIG_BLK_CGROUP_IOCOST) which 1492 currently implements "io.weight" proportional control. Lines 1493 are keyed by $MAJ:$MIN device numbers and not ordered. The 1494 line for a given device is populated on the first write for 1495 the device on "io.cost.qos" or "io.cost.model". The following 1496 nested keys are defined. 1497 1498 ====== ===================================== 1499 enable Weight-based control enable 1500 ctrl "auto" or "user" 1501 rpct Read latency percentile [0, 100] 1502 rlat Read latency threshold 1503 wpct Write latency percentile [0, 100] 1504 wlat Write latency threshold 1505 min Minimum scaling percentage [1, 10000] 1506 max Maximum scaling percentage [1, 10000] 1507 ====== ===================================== 1508 1509 The controller is disabled by default and can be enabled by 1510 setting "enable" to 1. "rpct" and "wpct" parameters default 1511 to zero and the controller uses internal device saturation 1512 state to adjust the overall IO rate between "min" and "max". 1513 1514 When a better control quality is needed, latency QoS 1515 parameters can be configured. For example:: 1516 1517 8:16 enable=1 ctrl=auto rpct=95.00 rlat=75000 wpct=95.00 wlat=150000 min=50.00 max=150.0 1518 1519 shows that on sdb, the controller is enabled, will consider 1520 the device saturated if the 95th percentile of read completion 1521 latencies is above 75ms or write 150ms, and adjust the overall 1522 IO issue rate between 50% and 150% accordingly. 1523 1524 The lower the saturation point, the better the latency QoS at 1525 the cost of aggregate bandwidth. The narrower the allowed 1526 adjustment range between "min" and "max", the more conformant 1527 to the cost model the IO behavior. Note that the IO issue 1528 base rate may be far off from 100% and setting "min" and "max" 1529 blindly can lead to a significant loss of device capacity or 1530 control quality. "min" and "max" are useful for regulating 1531 devices which show wide temporary behavior changes - e.g. a 1532 ssd which accepts writes at the line speed for a while and 1533 then completely stalls for multiple seconds. 1534 1535 When "ctrl" is "auto", the parameters are controlled by the 1536 kernel and may change automatically. Setting "ctrl" to "user" 1537 or setting any of the percentile and latency parameters puts 1538 it into "user" mode and disables the automatic changes. The 1539 automatic mode can be restored by setting "ctrl" to "auto". 1540 1541 io.cost.model 1542 A read-write nested-keyed file with exists only on the root 1543 cgroup. 1544 1545 This file configures the cost model of the IO cost model based 1546 controller (CONFIG_BLK_CGROUP_IOCOST) which currently 1547 implements "io.weight" proportional control. Lines are keyed 1548 by $MAJ:$MIN device numbers and not ordered. The line for a 1549 given device is populated on the first write for the device on 1550 "io.cost.qos" or "io.cost.model". The following nested keys 1551 are defined. 1552 1553 ===== ================================ 1554 ctrl "auto" or "user" 1555 model The cost model in use - "linear" 1556 ===== ================================ 1557 1558 When "ctrl" is "auto", the kernel may change all parameters 1559 dynamically. When "ctrl" is set to "user" or any other 1560 parameters are written to, "ctrl" become "user" and the 1561 automatic changes are disabled. 1562 1563 When "model" is "linear", the following model parameters are 1564 defined. 1565 1566 ============= ======================================== 1567 [r|w]bps The maximum sequential IO throughput 1568 [r|w]seqiops The maximum 4k sequential IOs per second 1569 [r|w]randiops The maximum 4k random IOs per second 1570 ============= ======================================== 1571 1572 From the above, the builtin linear model determines the base 1573 costs of a sequential and random IO and the cost coefficient 1574 for the IO size. While simple, this model can cover most 1575 common device classes acceptably. 1576 1577 The IO cost model isn't expected to be accurate in absolute 1578 sense and is scaled to the device behavior dynamically. 1579 1580 If needed, tools/cgroup/iocost_coef_gen.py can be used to 1581 generate device-specific coefficients. 1582 1583 io.weight 1584 A read-write flat-keyed file which exists on non-root cgroups. 1585 The default is "default 100". 1586 1587 The first line is the default weight applied to devices 1588 without specific override. The rest are overrides keyed by 1589 $MAJ:$MIN device numbers and not ordered. The weights are in 1590 the range [1, 10000] and specifies the relative amount IO time 1591 the cgroup can use in relation to its siblings. 1592 1593 The default weight can be updated by writing either "default 1594 $WEIGHT" or simply "$WEIGHT". Overrides can be set by writing 1595 "$MAJ:$MIN $WEIGHT" and unset by writing "$MAJ:$MIN default". 1596 1597 An example read output follows:: 1598 1599 default 100 1600 8:16 200 1601 8:0 50 1602 1603 io.max 1604 A read-write nested-keyed file which exists on non-root 1605 cgroups. 1606 1607 BPS and IOPS based IO limit. Lines are keyed by $MAJ:$MIN 1608 device numbers and not ordered. The following nested keys are 1609 defined. 1610 1611 ===== ================================== 1612 rbps Max read bytes per second 1613 wbps Max write bytes per second 1614 riops Max read IO operations per second 1615 wiops Max write IO operations per second 1616 ===== ================================== 1617 1618 When writing, any number of nested key-value pairs can be 1619 specified in any order. "max" can be specified as the value 1620 to remove a specific limit. If the same key is specified 1621 multiple times, the outcome is undefined. 1622 1623 BPS and IOPS are measured in each IO direction and IOs are 1624 delayed if limit is reached. Temporary bursts are allowed. 1625 1626 Setting read limit at 2M BPS and write at 120 IOPS for 8:16:: 1627 1628 echo "8:16 rbps=2097152 wiops=120" > io.max 1629 1630 Reading returns the following:: 1631 1632 8:16 rbps=2097152 wbps=max riops=max wiops=120 1633 1634 Write IOPS limit can be removed by writing the following:: 1635 1636 echo "8:16 wiops=max" > io.max 1637 1638 Reading now returns the following:: 1639 1640 8:16 rbps=2097152 wbps=max riops=max wiops=max 1641 1642 io.pressure 1643 A read-only nested-key file which exists on non-root cgroups. 1644 1645 Shows pressure stall information for IO. See 1646 Documentation/accounting/psi.rst for details. 1647 1648 1649Writeback 1650~~~~~~~~~ 1651 1652Page cache is dirtied through buffered writes and shared mmaps and 1653written asynchronously to the backing filesystem by the writeback 1654mechanism. Writeback sits between the memory and IO domains and 1655regulates the proportion of dirty memory by balancing dirtying and 1656write IOs. 1657 1658The io controller, in conjunction with the memory controller, 1659implements control of page cache writeback IOs. The memory controller 1660defines the memory domain that dirty memory ratio is calculated and 1661maintained for and the io controller defines the io domain which 1662writes out dirty pages for the memory domain. Both system-wide and 1663per-cgroup dirty memory states are examined and the more restrictive 1664of the two is enforced. 1665 1666cgroup writeback requires explicit support from the underlying 1667filesystem. Currently, cgroup writeback is implemented on ext2, ext4 1668and btrfs. On other filesystems, all writeback IOs are attributed to 1669the root cgroup. 1670 1671There are inherent differences in memory and writeback management 1672which affects how cgroup ownership is tracked. Memory is tracked per 1673page while writeback per inode. For the purpose of writeback, an 1674inode is assigned to a cgroup and all IO requests to write dirty pages 1675from the inode are attributed to that cgroup. 1676 1677As cgroup ownership for memory is tracked per page, there can be pages 1678which are associated with different cgroups than the one the inode is 1679associated with. These are called foreign pages. The writeback 1680constantly keeps track of foreign pages and, if a particular foreign 1681cgroup becomes the majority over a certain period of time, switches 1682the ownership of the inode to that cgroup. 1683 1684While this model is enough for most use cases where a given inode is 1685mostly dirtied by a single cgroup even when the main writing cgroup 1686changes over time, use cases where multiple cgroups write to a single 1687inode simultaneously are not supported well. In such circumstances, a 1688significant portion of IOs are likely to be attributed incorrectly. 1689As memory controller assigns page ownership on the first use and 1690doesn't update it until the page is released, even if writeback 1691strictly follows page ownership, multiple cgroups dirtying overlapping 1692areas wouldn't work as expected. It's recommended to avoid such usage 1693patterns. 1694 1695The sysctl knobs which affect writeback behavior are applied to cgroup 1696writeback as follows. 1697 1698 vm.dirty_background_ratio, vm.dirty_ratio 1699 These ratios apply the same to cgroup writeback with the 1700 amount of available memory capped by limits imposed by the 1701 memory controller and system-wide clean memory. 1702 1703 vm.dirty_background_bytes, vm.dirty_bytes 1704 For cgroup writeback, this is calculated into ratio against 1705 total available memory and applied the same way as 1706 vm.dirty[_background]_ratio. 1707 1708 1709IO Latency 1710~~~~~~~~~~ 1711 1712This is a cgroup v2 controller for IO workload protection. You provide a group 1713with a latency target, and if the average latency exceeds that target the 1714controller will throttle any peers that have a lower latency target than the 1715protected workload. 1716 1717The limits are only applied at the peer level in the hierarchy. This means that 1718in the diagram below, only groups A, B, and C will influence each other, and 1719groups D and F will influence each other. Group G will influence nobody:: 1720 1721 [root] 1722 / | \ 1723 A B C 1724 / \ | 1725 D F G 1726 1727 1728So the ideal way to configure this is to set io.latency in groups A, B, and C. 1729Generally you do not want to set a value lower than the latency your device 1730supports. Experiment to find the value that works best for your workload. 1731Start at higher than the expected latency for your device and watch the 1732avg_lat value in io.stat for your workload group to get an idea of the 1733latency you see during normal operation. Use the avg_lat value as a basis for 1734your real setting, setting at 10-15% higher than the value in io.stat. 1735 1736How IO Latency Throttling Works 1737~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1738 1739io.latency is work conserving; so as long as everybody is meeting their latency 1740target the controller doesn't do anything. Once a group starts missing its 1741target it begins throttling any peer group that has a higher target than itself. 1742This throttling takes 2 forms: 1743 1744- Queue depth throttling. This is the number of outstanding IO's a group is 1745 allowed to have. We will clamp down relatively quickly, starting at no limit 1746 and going all the way down to 1 IO at a time. 1747 1748- Artificial delay induction. There are certain types of IO that cannot be 1749 throttled without possibly adversely affecting higher priority groups. This 1750 includes swapping and metadata IO. These types of IO are allowed to occur 1751 normally, however they are "charged" to the originating group. If the 1752 originating group is being throttled you will see the use_delay and delay 1753 fields in io.stat increase. The delay value is how many microseconds that are 1754 being added to any process that runs in this group. Because this number can 1755 grow quite large if there is a lot of swapping or metadata IO occurring we 1756 limit the individual delay events to 1 second at a time. 1757 1758Once the victimized group starts meeting its latency target again it will start 1759unthrottling any peer groups that were throttled previously. If the victimized 1760group simply stops doing IO the global counter will unthrottle appropriately. 1761 1762IO Latency Interface Files 1763~~~~~~~~~~~~~~~~~~~~~~~~~~ 1764 1765 io.latency 1766 This takes a similar format as the other controllers. 1767 1768 "MAJOR:MINOR target=<target time in microseconds" 1769 1770 io.stat 1771 If the controller is enabled you will see extra stats in io.stat in 1772 addition to the normal ones. 1773 1774 depth 1775 This is the current queue depth for the group. 1776 1777 avg_lat 1778 This is an exponential moving average with a decay rate of 1/exp 1779 bound by the sampling interval. The decay rate interval can be 1780 calculated by multiplying the win value in io.stat by the 1781 corresponding number of samples based on the win value. 1782 1783 win 1784 The sampling window size in milliseconds. This is the minimum 1785 duration of time between evaluation events. Windows only elapse 1786 with IO activity. Idle periods extend the most recent window. 1787 1788PID 1789--- 1790 1791The process number controller is used to allow a cgroup to stop any 1792new tasks from being fork()'d or clone()'d after a specified limit is 1793reached. 1794 1795The number of tasks in a cgroup can be exhausted in ways which other 1796controllers cannot prevent, thus warranting its own controller. For 1797example, a fork bomb is likely to exhaust the number of tasks before 1798hitting memory restrictions. 1799 1800Note that PIDs used in this controller refer to TIDs, process IDs as 1801used by the kernel. 1802 1803 1804PID Interface Files 1805~~~~~~~~~~~~~~~~~~~ 1806 1807 pids.max 1808 A read-write single value file which exists on non-root 1809 cgroups. The default is "max". 1810 1811 Hard limit of number of processes. 1812 1813 pids.current 1814 A read-only single value file which exists on all cgroups. 1815 1816 The number of processes currently in the cgroup and its 1817 descendants. 1818 1819Organisational operations are not blocked by cgroup policies, so it is 1820possible to have pids.current > pids.max. This can be done by either 1821setting the limit to be smaller than pids.current, or attaching enough 1822processes to the cgroup such that pids.current is larger than 1823pids.max. However, it is not possible to violate a cgroup PID policy 1824through fork() or clone(). These will return -EAGAIN if the creation 1825of a new process would cause a cgroup policy to be violated. 1826 1827 1828Cpuset 1829------ 1830 1831The "cpuset" controller provides a mechanism for constraining 1832the CPU and memory node placement of tasks to only the resources 1833specified in the cpuset interface files in a task's current cgroup. 1834This is especially valuable on large NUMA systems where placing jobs 1835on properly sized subsets of the systems with careful processor and 1836memory placement to reduce cross-node memory access and contention 1837can improve overall system performance. 1838 1839The "cpuset" controller is hierarchical. That means the controller 1840cannot use CPUs or memory nodes not allowed in its parent. 1841 1842 1843Cpuset Interface Files 1844~~~~~~~~~~~~~~~~~~~~~~ 1845 1846 cpuset.cpus 1847 A read-write multiple values file which exists on non-root 1848 cpuset-enabled cgroups. 1849 1850 It lists the requested CPUs to be used by tasks within this 1851 cgroup. The actual list of CPUs to be granted, however, is 1852 subjected to constraints imposed by its parent and can differ 1853 from the requested CPUs. 1854 1855 The CPU numbers are comma-separated numbers or ranges. 1856 For example: 1857 1858 # cat cpuset.cpus 1859 0-4,6,8-10 1860 1861 An empty value indicates that the cgroup is using the same 1862 setting as the nearest cgroup ancestor with a non-empty 1863 "cpuset.cpus" or all the available CPUs if none is found. 1864 1865 The value of "cpuset.cpus" stays constant until the next update 1866 and won't be affected by any CPU hotplug events. 1867 1868 cpuset.cpus.effective 1869 A read-only multiple values file which exists on all 1870 cpuset-enabled cgroups. 1871 1872 It lists the onlined CPUs that are actually granted to this 1873 cgroup by its parent. These CPUs are allowed to be used by 1874 tasks within the current cgroup. 1875 1876 If "cpuset.cpus" is empty, the "cpuset.cpus.effective" file shows 1877 all the CPUs from the parent cgroup that can be available to 1878 be used by this cgroup. Otherwise, it should be a subset of 1879 "cpuset.cpus" unless none of the CPUs listed in "cpuset.cpus" 1880 can be granted. In this case, it will be treated just like an 1881 empty "cpuset.cpus". 1882 1883 Its value will be affected by CPU hotplug events. 1884 1885 cpuset.mems 1886 A read-write multiple values file which exists on non-root 1887 cpuset-enabled cgroups. 1888 1889 It lists the requested memory nodes to be used by tasks within 1890 this cgroup. The actual list of memory nodes granted, however, 1891 is subjected to constraints imposed by its parent and can differ 1892 from the requested memory nodes. 1893 1894 The memory node numbers are comma-separated numbers or ranges. 1895 For example: 1896 1897 # cat cpuset.mems 1898 0-1,3 1899 1900 An empty value indicates that the cgroup is using the same 1901 setting as the nearest cgroup ancestor with a non-empty 1902 "cpuset.mems" or all the available memory nodes if none 1903 is found. 1904 1905 The value of "cpuset.mems" stays constant until the next update 1906 and won't be affected by any memory nodes hotplug events. 1907 1908 cpuset.mems.effective 1909 A read-only multiple values file which exists on all 1910 cpuset-enabled cgroups. 1911 1912 It lists the onlined memory nodes that are actually granted to 1913 this cgroup by its parent. These memory nodes are allowed to 1914 be used by tasks within the current cgroup. 1915 1916 If "cpuset.mems" is empty, it shows all the memory nodes from the 1917 parent cgroup that will be available to be used by this cgroup. 1918 Otherwise, it should be a subset of "cpuset.mems" unless none of 1919 the memory nodes listed in "cpuset.mems" can be granted. In this 1920 case, it will be treated just like an empty "cpuset.mems". 1921 1922 Its value will be affected by memory nodes hotplug events. 1923 1924 cpuset.cpus.partition 1925 A read-write single value file which exists on non-root 1926 cpuset-enabled cgroups. This flag is owned by the parent cgroup 1927 and is not delegatable. 1928 1929 It accepts only the following input values when written to. 1930 1931 "root" - a partition root 1932 "member" - a non-root member of a partition 1933 1934 When set to be a partition root, the current cgroup is the 1935 root of a new partition or scheduling domain that comprises 1936 itself and all its descendants except those that are separate 1937 partition roots themselves and their descendants. The root 1938 cgroup is always a partition root. 1939 1940 There are constraints on where a partition root can be set. 1941 It can only be set in a cgroup if all the following conditions 1942 are true. 1943 1944 1) The "cpuset.cpus" is not empty and the list of CPUs are 1945 exclusive, i.e. they are not shared by any of its siblings. 1946 2) The parent cgroup is a partition root. 1947 3) The "cpuset.cpus" is also a proper subset of the parent's 1948 "cpuset.cpus.effective". 1949 4) There is no child cgroups with cpuset enabled. This is for 1950 eliminating corner cases that have to be handled if such a 1951 condition is allowed. 1952 1953 Setting it to partition root will take the CPUs away from the 1954 effective CPUs of the parent cgroup. Once it is set, this 1955 file cannot be reverted back to "member" if there are any child 1956 cgroups with cpuset enabled. 1957 1958 A parent partition cannot distribute all its CPUs to its 1959 child partitions. There must be at least one cpu left in the 1960 parent partition. 1961 1962 Once becoming a partition root, changes to "cpuset.cpus" is 1963 generally allowed as long as the first condition above is true, 1964 the change will not take away all the CPUs from the parent 1965 partition and the new "cpuset.cpus" value is a superset of its 1966 children's "cpuset.cpus" values. 1967 1968 Sometimes, external factors like changes to ancestors' 1969 "cpuset.cpus" or cpu hotplug can cause the state of the partition 1970 root to change. On read, the "cpuset.sched.partition" file 1971 can show the following values. 1972 1973 "member" Non-root member of a partition 1974 "root" Partition root 1975 "root invalid" Invalid partition root 1976 1977 It is a partition root if the first 2 partition root conditions 1978 above are true and at least one CPU from "cpuset.cpus" is 1979 granted by the parent cgroup. 1980 1981 A partition root can become invalid if none of CPUs requested 1982 in "cpuset.cpus" can be granted by the parent cgroup or the 1983 parent cgroup is no longer a partition root itself. In this 1984 case, it is not a real partition even though the restriction 1985 of the first partition root condition above will still apply. 1986 The cpu affinity of all the tasks in the cgroup will then be 1987 associated with CPUs in the nearest ancestor partition. 1988 1989 An invalid partition root can be transitioned back to a 1990 real partition root if at least one of the requested CPUs 1991 can now be granted by its parent. In this case, the cpu 1992 affinity of all the tasks in the formerly invalid partition 1993 will be associated to the CPUs of the newly formed partition. 1994 Changing the partition state of an invalid partition root to 1995 "member" is always allowed even if child cpusets are present. 1996 1997 1998Device controller 1999----------------- 2000 2001Device controller manages access to device files. It includes both 2002creation of new device files (using mknod), and access to the 2003existing device files. 2004 2005Cgroup v2 device controller has no interface files and is implemented 2006on top of cgroup BPF. To control access to device files, a user may 2007create bpf programs of the BPF_CGROUP_DEVICE type and attach them 2008to cgroups. On an attempt to access a device file, corresponding 2009BPF programs will be executed, and depending on the return value 2010the attempt will succeed or fail with -EPERM. 2011 2012A BPF_CGROUP_DEVICE program takes a pointer to the bpf_cgroup_dev_ctx 2013structure, which describes the device access attempt: access type 2014(mknod/read/write) and device (type, major and minor numbers). 2015If the program returns 0, the attempt fails with -EPERM, otherwise 2016it succeeds. 2017 2018An example of BPF_CGROUP_DEVICE program may be found in the kernel 2019source tree in the tools/testing/selftests/bpf/dev_cgroup.c file. 2020 2021 2022RDMA 2023---- 2024 2025The "rdma" controller regulates the distribution and accounting of 2026of RDMA resources. 2027 2028RDMA Interface Files 2029~~~~~~~~~~~~~~~~~~~~ 2030 2031 rdma.max 2032 A readwrite nested-keyed file that exists for all the cgroups 2033 except root that describes current configured resource limit 2034 for a RDMA/IB device. 2035 2036 Lines are keyed by device name and are not ordered. 2037 Each line contains space separated resource name and its configured 2038 limit that can be distributed. 2039 2040 The following nested keys are defined. 2041 2042 ========== ============================= 2043 hca_handle Maximum number of HCA Handles 2044 hca_object Maximum number of HCA Objects 2045 ========== ============================= 2046 2047 An example for mlx4 and ocrdma device follows:: 2048 2049 mlx4_0 hca_handle=2 hca_object=2000 2050 ocrdma1 hca_handle=3 hca_object=max 2051 2052 rdma.current 2053 A read-only file that describes current resource usage. 2054 It exists for all the cgroup except root. 2055 2056 An example for mlx4 and ocrdma device follows:: 2057 2058 mlx4_0 hca_handle=1 hca_object=20 2059 ocrdma1 hca_handle=1 hca_object=23 2060 2061HugeTLB 2062------- 2063 2064The HugeTLB controller allows to limit the HugeTLB usage per control group and 2065enforces the controller limit during page fault. 2066 2067HugeTLB Interface Files 2068~~~~~~~~~~~~~~~~~~~~~~~ 2069 2070 hugetlb.<hugepagesize>.current 2071 Show current usage for "hugepagesize" hugetlb. It exists for all 2072 the cgroup except root. 2073 2074 hugetlb.<hugepagesize>.max 2075 Set/show the hard limit of "hugepagesize" hugetlb usage. 2076 The default value is "max". It exists for all the cgroup except root. 2077 2078 hugetlb.<hugepagesize>.events 2079 A read-only flat-keyed file which exists on non-root cgroups. 2080 2081 max 2082 The number of allocation failure due to HugeTLB limit 2083 2084 hugetlb.<hugepagesize>.events.local 2085 Similar to hugetlb.<hugepagesize>.events but the fields in the file 2086 are local to the cgroup i.e. not hierarchical. The file modified event 2087 generated on this file reflects only the local events. 2088 2089Misc 2090---- 2091 2092perf_event 2093~~~~~~~~~~ 2094 2095perf_event controller, if not mounted on a legacy hierarchy, is 2096automatically enabled on the v2 hierarchy so that perf events can 2097always be filtered by cgroup v2 path. The controller can still be 2098moved to a legacy hierarchy after v2 hierarchy is populated. 2099 2100 2101Non-normative information 2102------------------------- 2103 2104This section contains information that isn't considered to be a part of 2105the stable kernel API and so is subject to change. 2106 2107 2108CPU controller root cgroup process behaviour 2109~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2110 2111When distributing CPU cycles in the root cgroup each thread in this 2112cgroup is treated as if it was hosted in a separate child cgroup of the 2113root cgroup. This child cgroup weight is dependent on its thread nice 2114level. 2115 2116For details of this mapping see sched_prio_to_weight array in 2117kernel/sched/core.c file (values from this array should be scaled 2118appropriately so the neutral - nice 0 - value is 100 instead of 1024). 2119 2120 2121IO controller root cgroup process behaviour 2122~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2123 2124Root cgroup processes are hosted in an implicit leaf child node. 2125When distributing IO resources this implicit child node is taken into 2126account as if it was a normal child cgroup of the root cgroup with a 2127weight value of 200. 2128 2129 2130Namespace 2131========= 2132 2133Basics 2134------ 2135 2136cgroup namespace provides a mechanism to virtualize the view of the 2137"/proc/$PID/cgroup" file and cgroup mounts. The CLONE_NEWCGROUP clone 2138flag can be used with clone(2) and unshare(2) to create a new cgroup 2139namespace. The process running inside the cgroup namespace will have 2140its "/proc/$PID/cgroup" output restricted to cgroupns root. The 2141cgroupns root is the cgroup of the process at the time of creation of 2142the cgroup namespace. 2143 2144Without cgroup namespace, the "/proc/$PID/cgroup" file shows the 2145complete path of the cgroup of a process. In a container setup where 2146a set of cgroups and namespaces are intended to isolate processes the 2147"/proc/$PID/cgroup" file may leak potential system level information 2148to the isolated processes. For Example:: 2149 2150 # cat /proc/self/cgroup 2151 0::/batchjobs/container_id1 2152 2153The path '/batchjobs/container_id1' can be considered as system-data 2154and undesirable to expose to the isolated processes. cgroup namespace 2155can be used to restrict visibility of this path. For example, before 2156creating a cgroup namespace, one would see:: 2157 2158 # ls -l /proc/self/ns/cgroup 2159 lrwxrwxrwx 1 root root 0 2014-07-15 10:37 /proc/self/ns/cgroup -> cgroup:[4026531835] 2160 # cat /proc/self/cgroup 2161 0::/batchjobs/container_id1 2162 2163After unsharing a new namespace, the view changes:: 2164 2165 # ls -l /proc/self/ns/cgroup 2166 lrwxrwxrwx 1 root root 0 2014-07-15 10:35 /proc/self/ns/cgroup -> cgroup:[4026532183] 2167 # cat /proc/self/cgroup 2168 0::/ 2169 2170When some thread from a multi-threaded process unshares its cgroup 2171namespace, the new cgroupns gets applied to the entire process (all 2172the threads). This is natural for the v2 hierarchy; however, for the 2173legacy hierarchies, this may be unexpected. 2174 2175A cgroup namespace is alive as long as there are processes inside or 2176mounts pinning it. When the last usage goes away, the cgroup 2177namespace is destroyed. The cgroupns root and the actual cgroups 2178remain. 2179 2180 2181The Root and Views 2182------------------ 2183 2184The 'cgroupns root' for a cgroup namespace is the cgroup in which the 2185process calling unshare(2) is running. For example, if a process in 2186/batchjobs/container_id1 cgroup calls unshare, cgroup 2187/batchjobs/container_id1 becomes the cgroupns root. For the 2188init_cgroup_ns, this is the real root ('/') cgroup. 2189 2190The cgroupns root cgroup does not change even if the namespace creator 2191process later moves to a different cgroup:: 2192 2193 # ~/unshare -c # unshare cgroupns in some cgroup 2194 # cat /proc/self/cgroup 2195 0::/ 2196 # mkdir sub_cgrp_1 2197 # echo 0 > sub_cgrp_1/cgroup.procs 2198 # cat /proc/self/cgroup 2199 0::/sub_cgrp_1 2200 2201Each process gets its namespace-specific view of "/proc/$PID/cgroup" 2202 2203Processes running inside the cgroup namespace will be able to see 2204cgroup paths (in /proc/self/cgroup) only inside their root cgroup. 2205From within an unshared cgroupns:: 2206 2207 # sleep 100000 & 2208 [1] 7353 2209 # echo 7353 > sub_cgrp_1/cgroup.procs 2210 # cat /proc/7353/cgroup 2211 0::/sub_cgrp_1 2212 2213From the initial cgroup namespace, the real cgroup path will be 2214visible:: 2215 2216 $ cat /proc/7353/cgroup 2217 0::/batchjobs/container_id1/sub_cgrp_1 2218 2219From a sibling cgroup namespace (that is, a namespace rooted at a 2220different cgroup), the cgroup path relative to its own cgroup 2221namespace root will be shown. For instance, if PID 7353's cgroup 2222namespace root is at '/batchjobs/container_id2', then it will see:: 2223 2224 # cat /proc/7353/cgroup 2225 0::/../container_id2/sub_cgrp_1 2226 2227Note that the relative path always starts with '/' to indicate that 2228its relative to the cgroup namespace root of the caller. 2229 2230 2231Migration and setns(2) 2232---------------------- 2233 2234Processes inside a cgroup namespace can move into and out of the 2235namespace root if they have proper access to external cgroups. For 2236example, from inside a namespace with cgroupns root at 2237/batchjobs/container_id1, and assuming that the global hierarchy is 2238still accessible inside cgroupns:: 2239 2240 # cat /proc/7353/cgroup 2241 0::/sub_cgrp_1 2242 # echo 7353 > batchjobs/container_id2/cgroup.procs 2243 # cat /proc/7353/cgroup 2244 0::/../container_id2 2245 2246Note that this kind of setup is not encouraged. A task inside cgroup 2247namespace should only be exposed to its own cgroupns hierarchy. 2248 2249setns(2) to another cgroup namespace is allowed when: 2250 2251(a) the process has CAP_SYS_ADMIN against its current user namespace 2252(b) the process has CAP_SYS_ADMIN against the target cgroup 2253 namespace's userns 2254 2255No implicit cgroup changes happen with attaching to another cgroup 2256namespace. It is expected that the someone moves the attaching 2257process under the target cgroup namespace root. 2258 2259 2260Interaction with Other Namespaces 2261--------------------------------- 2262 2263Namespace specific cgroup hierarchy can be mounted by a process 2264running inside a non-init cgroup namespace:: 2265 2266 # mount -t cgroup2 none $MOUNT_POINT 2267 2268This will mount the unified cgroup hierarchy with cgroupns root as the 2269filesystem root. The process needs CAP_SYS_ADMIN against its user and 2270mount namespaces. 2271 2272The virtualization of /proc/self/cgroup file combined with restricting 2273the view of cgroup hierarchy by namespace-private cgroupfs mount 2274provides a properly isolated cgroup view inside the container. 2275 2276 2277Information on Kernel Programming 2278================================= 2279 2280This section contains kernel programming information in the areas 2281where interacting with cgroup is necessary. cgroup core and 2282controllers are not covered. 2283 2284 2285Filesystem Support for Writeback 2286-------------------------------- 2287 2288A filesystem can support cgroup writeback by updating 2289address_space_operations->writepage[s]() to annotate bio's using the 2290following two functions. 2291 2292 wbc_init_bio(@wbc, @bio) 2293 Should be called for each bio carrying writeback data and 2294 associates the bio with the inode's owner cgroup and the 2295 corresponding request queue. This must be called after 2296 a queue (device) has been associated with the bio and 2297 before submission. 2298 2299 wbc_account_cgroup_owner(@wbc, @page, @bytes) 2300 Should be called for each data segment being written out. 2301 While this function doesn't care exactly when it's called 2302 during the writeback session, it's the easiest and most 2303 natural to call it as data segments are added to a bio. 2304 2305With writeback bio's annotated, cgroup support can be enabled per 2306super_block by setting SB_I_CGROUPWB in ->s_iflags. This allows for 2307selective disabling of cgroup writeback support which is helpful when 2308certain filesystem features, e.g. journaled data mode, are 2309incompatible. 2310 2311wbc_init_bio() binds the specified bio to its cgroup. Depending on 2312the configuration, the bio may be executed at a lower priority and if 2313the writeback session is holding shared resources, e.g. a journal 2314entry, may lead to priority inversion. There is no one easy solution 2315for the problem. Filesystems can try to work around specific problem 2316cases by skipping wbc_init_bio() and using bio_associate_blkg() 2317directly. 2318 2319 2320Deprecated v1 Core Features 2321=========================== 2322 2323- Multiple hierarchies including named ones are not supported. 2324 2325- All v1 mount options are not supported. 2326 2327- The "tasks" file is removed and "cgroup.procs" is not sorted. 2328 2329- "cgroup.clone_children" is removed. 2330 2331- /proc/cgroups is meaningless for v2. Use "cgroup.controllers" file 2332 at the root instead. 2333 2334 2335Issues with v1 and Rationales for v2 2336==================================== 2337 2338Multiple Hierarchies 2339-------------------- 2340 2341cgroup v1 allowed an arbitrary number of hierarchies and each 2342hierarchy could host any number of controllers. While this seemed to 2343provide a high level of flexibility, it wasn't useful in practice. 2344 2345For example, as there is only one instance of each controller, utility 2346type controllers such as freezer which can be useful in all 2347hierarchies could only be used in one. The issue is exacerbated by 2348the fact that controllers couldn't be moved to another hierarchy once 2349hierarchies were populated. Another issue was that all controllers 2350bound to a hierarchy were forced to have exactly the same view of the 2351hierarchy. It wasn't possible to vary the granularity depending on 2352the specific controller. 2353 2354In practice, these issues heavily limited which controllers could be 2355put on the same hierarchy and most configurations resorted to putting 2356each controller on its own hierarchy. Only closely related ones, such 2357as the cpu and cpuacct controllers, made sense to be put on the same 2358hierarchy. This often meant that userland ended up managing multiple 2359similar hierarchies repeating the same steps on each hierarchy 2360whenever a hierarchy management operation was necessary. 2361 2362Furthermore, support for multiple hierarchies came at a steep cost. 2363It greatly complicated cgroup core implementation but more importantly 2364the support for multiple hierarchies restricted how cgroup could be 2365used in general and what controllers was able to do. 2366 2367There was no limit on how many hierarchies there might be, which meant 2368that a thread's cgroup membership couldn't be described in finite 2369length. The key might contain any number of entries and was unlimited 2370in length, which made it highly awkward to manipulate and led to 2371addition of controllers which existed only to identify membership, 2372which in turn exacerbated the original problem of proliferating number 2373of hierarchies. 2374 2375Also, as a controller couldn't have any expectation regarding the 2376topologies of hierarchies other controllers might be on, each 2377controller had to assume that all other controllers were attached to 2378completely orthogonal hierarchies. This made it impossible, or at 2379least very cumbersome, for controllers to cooperate with each other. 2380 2381In most use cases, putting controllers on hierarchies which are 2382completely orthogonal to each other isn't necessary. What usually is 2383called for is the ability to have differing levels of granularity 2384depending on the specific controller. In other words, hierarchy may 2385be collapsed from leaf towards root when viewed from specific 2386controllers. For example, a given configuration might not care about 2387how memory is distributed beyond a certain level while still wanting 2388to control how CPU cycles are distributed. 2389 2390 2391Thread Granularity 2392------------------ 2393 2394cgroup v1 allowed threads of a process to belong to different cgroups. 2395This didn't make sense for some controllers and those controllers 2396ended up implementing different ways to ignore such situations but 2397much more importantly it blurred the line between API exposed to 2398individual applications and system management interface. 2399 2400Generally, in-process knowledge is available only to the process 2401itself; thus, unlike service-level organization of processes, 2402categorizing threads of a process requires active participation from 2403the application which owns the target process. 2404 2405cgroup v1 had an ambiguously defined delegation model which got abused 2406in combination with thread granularity. cgroups were delegated to 2407individual applications so that they can create and manage their own 2408sub-hierarchies and control resource distributions along them. This 2409effectively raised cgroup to the status of a syscall-like API exposed 2410to lay programs. 2411 2412First of all, cgroup has a fundamentally inadequate interface to be 2413exposed this way. For a process to access its own knobs, it has to 2414extract the path on the target hierarchy from /proc/self/cgroup, 2415construct the path by appending the name of the knob to the path, open 2416and then read and/or write to it. This is not only extremely clunky 2417and unusual but also inherently racy. There is no conventional way to 2418define transaction across the required steps and nothing can guarantee 2419that the process would actually be operating on its own sub-hierarchy. 2420 2421cgroup controllers implemented a number of knobs which would never be 2422accepted as public APIs because they were just adding control knobs to 2423system-management pseudo filesystem. cgroup ended up with interface 2424knobs which were not properly abstracted or refined and directly 2425revealed kernel internal details. These knobs got exposed to 2426individual applications through the ill-defined delegation mechanism 2427effectively abusing cgroup as a shortcut to implementing public APIs 2428without going through the required scrutiny. 2429 2430This was painful for both userland and kernel. Userland ended up with 2431misbehaving and poorly abstracted interfaces and kernel exposing and 2432locked into constructs inadvertently. 2433 2434 2435Competition Between Inner Nodes and Threads 2436------------------------------------------- 2437 2438cgroup v1 allowed threads to be in any cgroups which created an 2439interesting problem where threads belonging to a parent cgroup and its 2440children cgroups competed for resources. This was nasty as two 2441different types of entities competed and there was no obvious way to 2442settle it. Different controllers did different things. 2443 2444The cpu controller considered threads and cgroups as equivalents and 2445mapped nice levels to cgroup weights. This worked for some cases but 2446fell flat when children wanted to be allocated specific ratios of CPU 2447cycles and the number of internal threads fluctuated - the ratios 2448constantly changed as the number of competing entities fluctuated. 2449There also were other issues. The mapping from nice level to weight 2450wasn't obvious or universal, and there were various other knobs which 2451simply weren't available for threads. 2452 2453The io controller implicitly created a hidden leaf node for each 2454cgroup to host the threads. The hidden leaf had its own copies of all 2455the knobs with ``leaf_`` prefixed. While this allowed equivalent 2456control over internal threads, it was with serious drawbacks. It 2457always added an extra layer of nesting which wouldn't be necessary 2458otherwise, made the interface messy and significantly complicated the 2459implementation. 2460 2461The memory controller didn't have a way to control what happened 2462between internal tasks and child cgroups and the behavior was not 2463clearly defined. There were attempts to add ad-hoc behaviors and 2464knobs to tailor the behavior to specific workloads which would have 2465led to problems extremely difficult to resolve in the long term. 2466 2467Multiple controllers struggled with internal tasks and came up with 2468different ways to deal with it; unfortunately, all the approaches were 2469severely flawed and, furthermore, the widely different behaviors 2470made cgroup as a whole highly inconsistent. 2471 2472This clearly is a problem which needs to be addressed from cgroup core 2473in a uniform way. 2474 2475 2476Other Interface Issues 2477---------------------- 2478 2479cgroup v1 grew without oversight and developed a large number of 2480idiosyncrasies and inconsistencies. One issue on the cgroup core side 2481was how an empty cgroup was notified - a userland helper binary was 2482forked and executed for each event. The event delivery wasn't 2483recursive or delegatable. The limitations of the mechanism also led 2484to in-kernel event delivery filtering mechanism further complicating 2485the interface. 2486 2487Controller interfaces were problematic too. An extreme example is 2488controllers completely ignoring hierarchical organization and treating 2489all cgroups as if they were all located directly under the root 2490cgroup. Some controllers exposed a large amount of inconsistent 2491implementation details to userland. 2492 2493There also was no consistency across controllers. When a new cgroup 2494was created, some controllers defaulted to not imposing extra 2495restrictions while others disallowed any resource usage until 2496explicitly configured. Configuration knobs for the same type of 2497control used widely differing naming schemes and formats. Statistics 2498and information knobs were named arbitrarily and used different 2499formats and units even in the same controller. 2500 2501cgroup v2 establishes common conventions where appropriate and updates 2502controllers so that they expose minimal and consistent interfaces. 2503 2504 2505Controller Issues and Remedies 2506------------------------------ 2507 2508Memory 2509~~~~~~ 2510 2511The original lower boundary, the soft limit, is defined as a limit 2512that is per default unset. As a result, the set of cgroups that 2513global reclaim prefers is opt-in, rather than opt-out. The costs for 2514optimizing these mostly negative lookups are so high that the 2515implementation, despite its enormous size, does not even provide the 2516basic desirable behavior. First off, the soft limit has no 2517hierarchical meaning. All configured groups are organized in a global 2518rbtree and treated like equal peers, regardless where they are located 2519in the hierarchy. This makes subtree delegation impossible. Second, 2520the soft limit reclaim pass is so aggressive that it not just 2521introduces high allocation latencies into the system, but also impacts 2522system performance due to overreclaim, to the point where the feature 2523becomes self-defeating. 2524 2525The memory.low boundary on the other hand is a top-down allocated 2526reserve. A cgroup enjoys reclaim protection when it's within its 2527effective low, which makes delegation of subtrees possible. It also 2528enjoys having reclaim pressure proportional to its overage when 2529above its effective low. 2530 2531The original high boundary, the hard limit, is defined as a strict 2532limit that can not budge, even if the OOM killer has to be called. 2533But this generally goes against the goal of making the most out of the 2534available memory. The memory consumption of workloads varies during 2535runtime, and that requires users to overcommit. But doing that with a 2536strict upper limit requires either a fairly accurate prediction of the 2537working set size or adding slack to the limit. Since working set size 2538estimation is hard and error prone, and getting it wrong results in 2539OOM kills, most users tend to err on the side of a looser limit and 2540end up wasting precious resources. 2541 2542The memory.high boundary on the other hand can be set much more 2543conservatively. When hit, it throttles allocations by forcing them 2544into direct reclaim to work off the excess, but it never invokes the 2545OOM killer. As a result, a high boundary that is chosen too 2546aggressively will not terminate the processes, but instead it will 2547lead to gradual performance degradation. The user can monitor this 2548and make corrections until the minimal memory footprint that still 2549gives acceptable performance is found. 2550 2551In extreme cases, with many concurrent allocations and a complete 2552breakdown of reclaim progress within the group, the high boundary can 2553be exceeded. But even then it's mostly better to satisfy the 2554allocation from the slack available in other groups or the rest of the 2555system than killing the group. Otherwise, memory.max is there to 2556limit this type of spillover and ultimately contain buggy or even 2557malicious applications. 2558 2559Setting the original memory.limit_in_bytes below the current usage was 2560subject to a race condition, where concurrent charges could cause the 2561limit setting to fail. memory.max on the other hand will first set the 2562limit to prevent new charges, and then reclaim and OOM kill until the 2563new limit is met - or the task writing to memory.max is killed. 2564 2565The combined memory+swap accounting and limiting is replaced by real 2566control over swap space. 2567 2568The main argument for a combined memory+swap facility in the original 2569cgroup design was that global or parental pressure would always be 2570able to swap all anonymous memory of a child group, regardless of the 2571child's own (possibly untrusted) configuration. However, untrusted 2572groups can sabotage swapping by other means - such as referencing its 2573anonymous memory in a tight loop - and an admin can not assume full 2574swappability when overcommitting untrusted jobs. 2575 2576For trusted jobs, on the other hand, a combined counter is not an 2577intuitive userspace interface, and it flies in the face of the idea 2578that cgroup controllers should account and limit specific physical 2579resources. Swap space is a resource like all others in the system, 2580and that's why unified hierarchy allows distributing it separately. 2581