1.. SPDX-License-Identifier: GPL-2.0 2.. include:: <isonum.txt> 3 4.. |struct cpuidle_state| replace:: :c:type:`struct cpuidle_state <cpuidle_state>` 5.. |cpufreq| replace:: :doc:`CPU Performance Scaling <cpufreq>` 6 7======================== 8CPU Idle Time Management 9======================== 10 11:Copyright: |copy| 2018 Intel Corporation 12 13:Author: Rafael J. Wysocki <rafael.j.wysocki@intel.com> 14 15 16Concepts 17======== 18 19Modern processors are generally able to enter states in which the execution of 20a program is suspended and instructions belonging to it are not fetched from 21memory or executed. Those states are the *idle* states of the processor. 22 23Since part of the processor hardware is not used in idle states, entering them 24generally allows power drawn by the processor to be reduced and, in consequence, 25it is an opportunity to save energy. 26 27CPU idle time management is an energy-efficiency feature concerned about using 28the idle states of processors for this purpose. 29 30Logical CPUs 31------------ 32 33CPU idle time management operates on CPUs as seen by the *CPU scheduler* (that 34is the part of the kernel responsible for the distribution of computational 35work in the system). In its view, CPUs are *logical* units. That is, they need 36not be separate physical entities and may just be interfaces appearing to 37software as individual single-core processors. In other words, a CPU is an 38entity which appears to be fetching instructions that belong to one sequence 39(program) from memory and executing them, but it need not work this way 40physically. Generally, three different cases can be consider here. 41 42First, if the whole processor can only follow one sequence of instructions (one 43program) at a time, it is a CPU. In that case, if the hardware is asked to 44enter an idle state, that applies to the processor as a whole. 45 46Second, if the processor is multi-core, each core in it is able to follow at 47least one program at a time. The cores need not be entirely independent of each 48other (for example, they may share caches), but still most of the time they 49work physically in parallel with each other, so if each of them executes only 50one program, those programs run mostly independently of each other at the same 51time. The entire cores are CPUs in that case and if the hardware is asked to 52enter an idle state, that applies to the core that asked for it in the first 53place, but it also may apply to a larger unit (say a "package" or a "cluster") 54that the core belongs to (in fact, it may apply to an entire hierarchy of larger 55units containing the core). Namely, if all of the cores in the larger unit 56except for one have been put into idle states at the "core level" and the 57remaining core asks the processor to enter an idle state, that may trigger it 58to put the whole larger unit into an idle state which also will affect the 59other cores in that unit. 60 61Finally, each core in a multi-core processor may be able to follow more than one 62program in the same time frame (that is, each core may be able to fetch 63instructions from multiple locations in memory and execute them in the same time 64frame, but not necessarily entirely in parallel with each other). In that case 65the cores present themselves to software as "bundles" each consisting of 66multiple individual single-core "processors", referred to as *hardware threads* 67(or hyper-threads specifically on Intel hardware), that each can follow one 68sequence of instructions. Then, the hardware threads are CPUs from the CPU idle 69time management perspective and if the processor is asked to enter an idle state 70by one of them, the hardware thread (or CPU) that asked for it is stopped, but 71nothing more happens, unless all of the other hardware threads within the same 72core also have asked the processor to enter an idle state. In that situation, 73the core may be put into an idle state individually or a larger unit containing 74it may be put into an idle state as a whole (if the other cores within the 75larger unit are in idle states already). 76 77Idle CPUs 78--------- 79 80Logical CPUs, simply referred to as "CPUs" in what follows, are regarded as 81*idle* by the Linux kernel when there are no tasks to run on them except for the 82special "idle" task. 83 84Tasks are the CPU scheduler's representation of work. Each task consists of a 85sequence of instructions to execute, or code, data to be manipulated while 86running that code, and some context information that needs to be loaded into the 87processor every time the task's code is run by a CPU. The CPU scheduler 88distributes work by assigning tasks to run to the CPUs present in the system. 89 90Tasks can be in various states. In particular, they are *runnable* if there are 91no specific conditions preventing their code from being run by a CPU as long as 92there is a CPU available for that (for example, they are not waiting for any 93events to occur or similar). When a task becomes runnable, the CPU scheduler 94assigns it to one of the available CPUs to run and if there are no more runnable 95tasks assigned to it, the CPU will load the given task's context and run its 96code (from the instruction following the last one executed so far, possibly by 97another CPU). [If there are multiple runnable tasks assigned to one CPU 98simultaneously, they will be subject to prioritization and time sharing in order 99to allow them to make some progress over time.] 100 101The special "idle" task becomes runnable if there are no other runnable tasks 102assigned to the given CPU and the CPU is then regarded as idle. In other words, 103in Linux idle CPUs run the code of the "idle" task called *the idle loop*. That 104code may cause the processor to be put into one of its idle states, if they are 105supported, in order to save energy, but if the processor does not support any 106idle states, or there is not enough time to spend in an idle state before the 107next wakeup event, or there are strict latency constraints preventing any of the 108available idle states from being used, the CPU will simply execute more or less 109useless instructions in a loop until it is assigned a new task to run. 110 111 112.. _idle-loop: 113 114The Idle Loop 115============= 116 117The idle loop code takes two major steps in every iteration of it. First, it 118calls into a code module referred to as the *governor* that belongs to the CPU 119idle time management subsystem called ``CPUIdle`` to select an idle state for 120the CPU to ask the hardware to enter. Second, it invokes another code module 121from the ``CPUIdle`` subsystem, called the *driver*, to actually ask the 122processor hardware to enter the idle state selected by the governor. 123 124The role of the governor is to find an idle state most suitable for the 125conditions at hand. For this purpose, idle states that the hardware can be 126asked to enter by logical CPUs are represented in an abstract way independent of 127the platform or the processor architecture and organized in a one-dimensional 128(linear) array. That array has to be prepared and supplied by the ``CPUIdle`` 129driver matching the platform the kernel is running on at the initialization 130time. This allows ``CPUIdle`` governors to be independent of the underlying 131hardware and to work with any platforms that the Linux kernel can run on. 132 133Each idle state present in that array is characterized by two parameters to be 134taken into account by the governor, the *target residency* and the (worst-case) 135*exit latency*. The target residency is the minimum time the hardware must 136spend in the given state, including the time needed to enter it (which may be 137substantial), in order to save more energy than it would save by entering one of 138the shallower idle states instead. [The "depth" of an idle state roughly 139corresponds to the power drawn by the processor in that state.] The exit 140latency, in turn, is the maximum time it will take a CPU asking the processor 141hardware to enter an idle state to start executing the first instruction after a 142wakeup from that state. Note that in general the exit latency also must cover 143the time needed to enter the given state in case the wakeup occurs when the 144hardware is entering it and it must be entered completely to be exited in an 145ordered manner. 146 147There are two types of information that can influence the governor's decisions. 148First of all, the governor knows the time until the closest timer event. That 149time is known exactly, because the kernel programs timers and it knows exactly 150when they will trigger, and it is the maximum time the hardware that the given 151CPU depends on can spend in an idle state, including the time necessary to enter 152and exit it. However, the CPU may be woken up by a non-timer event at any time 153(in particular, before the closest timer triggers) and it generally is not known 154when that may happen. The governor can only see how much time the CPU actually 155was idle after it has been woken up (that time will be referred to as the *idle 156duration* from now on) and it can use that information somehow along with the 157time until the closest timer to estimate the idle duration in future. How the 158governor uses that information depends on what algorithm is implemented by it 159and that is the primary reason for having more than one governor in the 160``CPUIdle`` subsystem. 161 162There are four ``CPUIdle`` governors available, ``menu``, `TEO <teo-gov_>`_, 163``ladder`` and ``haltpoll``. Which of them is used by default depends on the 164configuration of the kernel and in particular on whether or not the scheduler 165tick can be `stopped by the idle loop <idle-cpus-and-tick_>`_. Available 166governors can be read from the :file:`available_governors`, and the governor 167can be changed at runtime. The name of the ``CPUIdle`` governor currently 168used by the kernel can be read from the :file:`current_governor_ro` or 169:file:`current_governor` file under :file:`/sys/devices/system/cpu/cpuidle/` 170in ``sysfs``. 171 172Which ``CPUIdle`` driver is used, on the other hand, usually depends on the 173platform the kernel is running on, but there are platforms with more than one 174matching driver. For example, there are two drivers that can work with the 175majority of Intel platforms, ``intel_idle`` and ``acpi_idle``, one with 176hardcoded idle states information and the other able to read that information 177from the system's ACPI tables, respectively. Still, even in those cases, the 178driver chosen at the system initialization time cannot be replaced later, so the 179decision on which one of them to use has to be made early (on Intel platforms 180the ``acpi_idle`` driver will be used if ``intel_idle`` is disabled for some 181reason or if it does not recognize the processor). The name of the ``CPUIdle`` 182driver currently used by the kernel can be read from the :file:`current_driver` 183file under :file:`/sys/devices/system/cpu/cpuidle/` in ``sysfs``. 184 185 186.. _idle-cpus-and-tick: 187 188Idle CPUs and The Scheduler Tick 189================================ 190 191The scheduler tick is a timer that triggers periodically in order to implement 192the time sharing strategy of the CPU scheduler. Of course, if there are 193multiple runnable tasks assigned to one CPU at the same time, the only way to 194allow them to make reasonable progress in a given time frame is to make them 195share the available CPU time. Namely, in rough approximation, each task is 196given a slice of the CPU time to run its code, subject to the scheduling class, 197prioritization and so on and when that time slice is used up, the CPU should be 198switched over to running (the code of) another task. The currently running task 199may not want to give the CPU away voluntarily, however, and the scheduler tick 200is there to make the switch happen regardless. That is not the only role of the 201tick, but it is the primary reason for using it. 202 203The scheduler tick is problematic from the CPU idle time management perspective, 204because it triggers periodically and relatively often (depending on the kernel 205configuration, the length of the tick period is between 1 ms and 10 ms). 206Thus, if the tick is allowed to trigger on idle CPUs, it will not make sense 207for them to ask the hardware to enter idle states with target residencies above 208the tick period length. Moreover, in that case the idle duration of any CPU 209will never exceed the tick period length and the energy used for entering and 210exiting idle states due to the tick wakeups on idle CPUs will be wasted. 211 212Fortunately, it is not really necessary to allow the tick to trigger on idle 213CPUs, because (by definition) they have no tasks to run except for the special 214"idle" one. In other words, from the CPU scheduler perspective, the only user 215of the CPU time on them is the idle loop. Since the time of an idle CPU need 216not be shared between multiple runnable tasks, the primary reason for using the 217tick goes away if the given CPU is idle. Consequently, it is possible to stop 218the scheduler tick entirely on idle CPUs in principle, even though that may not 219always be worth the effort. 220 221Whether or not it makes sense to stop the scheduler tick in the idle loop 222depends on what is expected by the governor. First, if there is another 223(non-tick) timer due to trigger within the tick range, stopping the tick clearly 224would be a waste of time, even though the timer hardware may not need to be 225reprogrammed in that case. Second, if the governor is expecting a non-timer 226wakeup within the tick range, stopping the tick is not necessary and it may even 227be harmful. Namely, in that case the governor will select an idle state with 228the target residency within the time until the expected wakeup, so that state is 229going to be relatively shallow. The governor really cannot select a deep idle 230state then, as that would contradict its own expectation of a wakeup in short 231order. Now, if the wakeup really occurs shortly, stopping the tick would be a 232waste of time and in this case the timer hardware would need to be reprogrammed, 233which is expensive. On the other hand, if the tick is stopped and the wakeup 234does not occur any time soon, the hardware may spend indefinite amount of time 235in the shallow idle state selected by the governor, which will be a waste of 236energy. Hence, if the governor is expecting a wakeup of any kind within the 237tick range, it is better to allow the tick trigger. Otherwise, however, the 238governor will select a relatively deep idle state, so the tick should be stopped 239so that it does not wake up the CPU too early. 240 241In any case, the governor knows what it is expecting and the decision on whether 242or not to stop the scheduler tick belongs to it. Still, if the tick has been 243stopped already (in one of the previous iterations of the loop), it is better 244to leave it as is and the governor needs to take that into account. 245 246The kernel can be configured to disable stopping the scheduler tick in the idle 247loop altogether. That can be done through the build-time configuration of it 248(by unsetting the ``CONFIG_NO_HZ_IDLE`` configuration option) or by passing 249``nohz=off`` to it in the command line. In both cases, as the stopping of the 250scheduler tick is disabled, the governor's decisions regarding it are simply 251ignored by the idle loop code and the tick is never stopped. 252 253The systems that run kernels configured to allow the scheduler tick to be 254stopped on idle CPUs are referred to as *tickless* systems and they are 255generally regarded as more energy-efficient than the systems running kernels in 256which the tick cannot be stopped. If the given system is tickless, it will use 257the ``menu`` governor by default and if it is not tickless, the default 258``CPUIdle`` governor on it will be ``ladder``. 259 260 261.. _menu-gov: 262 263The ``menu`` Governor 264===================== 265 266The ``menu`` governor is the default ``CPUIdle`` governor for tickless systems. 267It is quite complex, but the basic principle of its design is straightforward. 268Namely, when invoked to select an idle state for a CPU (i.e. an idle state that 269the CPU will ask the processor hardware to enter), it attempts to predict the 270idle duration and uses the predicted value for idle state selection. 271 272It first uses a simple pattern recognition algorithm to obtain a preliminary 273idle duration prediction. Namely, it saves the last 8 observed idle duration 274values and, when predicting the idle duration next time, it computes the average 275and variance of them. If the variance is small (smaller than 400 square 276milliseconds) or it is small relative to the average (the average is greater 277that 6 times the standard deviation), the average is regarded as the "typical 278interval" value. Otherwise, the longest of the saved observed idle duration 279values is discarded and the computation is repeated for the remaining ones. 280Again, if the variance of them is small (in the above sense), the average is 281taken as the "typical interval" value and so on, until either the "typical 282interval" is determined or too many data points are disregarded, in which case 283the "typical interval" is assumed to equal "infinity" (the maximum unsigned 284integer value). 285 286If the "typical interval" computed this way is long enough, the governor obtains 287the time until the closest timer event with the assumption that the scheduler 288tick will be stopped. That time, referred to as the *sleep length* in what follows, 289is the upper bound on the time before the next CPU wakeup. It is used to determine 290the sleep length range, which in turn is needed to get the sleep length correction 291factor. 292 293The ``menu`` governor maintains an array containing several correction factor 294values that correspond to different sleep length ranges organized so that each 295range represented in the array is approximately 10 times wider than the previous 296one. 297 298The correction factor for the given sleep length range (determined before 299selecting the idle state for the CPU) is updated after the CPU has been woken 300up and the closer the sleep length is to the observed idle duration, the closer 301to 1 the correction factor becomes (it must fall between 0 and 1 inclusive). 302The sleep length is multiplied by the correction factor for the range that it 303falls into to obtain an approximation of the predicted idle duration that is 304compared to the "typical interval" determined previously and the minimum of 305the two is taken as the idle duration prediction. 306 307If the "typical interval" value is small, which means that the CPU is likely 308to be woken up soon enough, the sleep length computation is skipped as it may 309be costly and the idle duration is simply predicted to equal the "typical 310interval" value. 311 312Now, the governor is ready to walk the list of idle states and choose one of 313them. For this purpose, it compares the target residency of each state with 314the predicted idle duration and the exit latency of it with the with the latency 315limit coming from the power management quality of service, or `PM QoS <cpu-pm-qos_>`_, 316framework. It selects the state with the target residency closest to the predicted 317idle duration, but still below it, and exit latency that does not exceed the 318limit. 319 320In the final step the governor may still need to refine the idle state selection 321if it has not decided to `stop the scheduler tick <idle-cpus-and-tick_>`_. That 322happens if the idle duration predicted by it is less than the tick period and 323the tick has not been stopped already (in a previous iteration of the idle 324loop). Then, the sleep length used in the previous computations may not reflect 325the real time until the closest timer event and if it really is greater than 326that time, the governor may need to select a shallower state with a suitable 327target residency. 328 329 330.. _teo-gov: 331 332The Timer Events Oriented (TEO) Governor 333======================================== 334 335The timer events oriented (TEO) governor is an alternative ``CPUIdle`` governor 336for tickless systems. It follows the same basic strategy as the ``menu`` `one 337<menu-gov_>`_: it always tries to find the deepest idle state suitable for the 338given conditions. However, it applies a different approach to that problem. 339 340.. kernel-doc:: drivers/cpuidle/governors/teo.c 341 :doc: teo-description 342 343.. _idle-states-representation: 344 345Representation of Idle States 346============================= 347 348For the CPU idle time management purposes all of the physical idle states 349supported by the processor have to be represented as a one-dimensional array of 350|struct cpuidle_state| objects each allowing an individual (logical) CPU to ask 351the processor hardware to enter an idle state of certain properties. If there 352is a hierarchy of units in the processor, one |struct cpuidle_state| object can 353cover a combination of idle states supported by the units at different levels of 354the hierarchy. In that case, the `target residency and exit latency parameters 355of it <idle-loop_>`_, must reflect the properties of the idle state at the 356deepest level (i.e. the idle state of the unit containing all of the other 357units). 358 359For example, take a processor with two cores in a larger unit referred to as 360a "module" and suppose that asking the hardware to enter a specific idle state 361(say "X") at the "core" level by one core will trigger the module to try to 362enter a specific idle state of its own (say "MX") if the other core is in idle 363state "X" already. In other words, asking for idle state "X" at the "core" 364level gives the hardware a license to go as deep as to idle state "MX" at the 365"module" level, but there is no guarantee that this is going to happen (the core 366asking for idle state "X" may just end up in that state by itself instead). 367Then, the target residency of the |struct cpuidle_state| object representing 368idle state "X" must reflect the minimum time to spend in idle state "MX" of 369the module (including the time needed to enter it), because that is the minimum 370time the CPU needs to be idle to save any energy in case the hardware enters 371that state. Analogously, the exit latency parameter of that object must cover 372the exit time of idle state "MX" of the module (and usually its entry time too), 373because that is the maximum delay between a wakeup signal and the time the CPU 374will start to execute the first new instruction (assuming that both cores in the 375module will always be ready to execute instructions as soon as the module 376becomes operational as a whole). 377 378There are processors without direct coordination between different levels of the 379hierarchy of units inside them, however. In those cases asking for an idle 380state at the "core" level does not automatically affect the "module" level, for 381example, in any way and the ``CPUIdle`` driver is responsible for the entire 382handling of the hierarchy. Then, the definition of the idle state objects is 383entirely up to the driver, but still the physical properties of the idle state 384that the processor hardware finally goes into must always follow the parameters 385used by the governor for idle state selection (for instance, the actual exit 386latency of that idle state must not exceed the exit latency parameter of the 387idle state object selected by the governor). 388 389In addition to the target residency and exit latency idle state parameters 390discussed above, the objects representing idle states each contain a few other 391parameters describing the idle state and a pointer to the function to run in 392order to ask the hardware to enter that state. Also, for each 393|struct cpuidle_state| object, there is a corresponding 394:c:type:`struct cpuidle_state_usage <cpuidle_state_usage>` one containing usage 395statistics of the given idle state. That information is exposed by the kernel 396via ``sysfs``. 397 398For each CPU in the system, there is a :file:`/sys/devices/system/cpu/cpu<N>/cpuidle/` 399directory in ``sysfs``, where the number ``<N>`` is assigned to the given 400CPU at the initialization time. That directory contains a set of subdirectories 401called :file:`state0`, :file:`state1` and so on, up to the number of idle state 402objects defined for the given CPU minus one. Each of these directories 403corresponds to one idle state object and the larger the number in its name, the 404deeper the (effective) idle state represented by it. Each of them contains 405a number of files (attributes) representing the properties of the idle state 406object corresponding to it, as follows: 407 408``above`` 409 Total number of times this idle state had been asked for, but the 410 observed idle duration was certainly too short to match its target 411 residency. 412 413``below`` 414 Total number of times this idle state had been asked for, but certainly 415 a deeper idle state would have been a better match for the observed idle 416 duration. 417 418``desc`` 419 Description of the idle state. 420 421``disable`` 422 Whether or not this idle state is disabled. 423 424``default_status`` 425 The default status of this state, "enabled" or "disabled". 426 427``latency`` 428 Exit latency of the idle state in microseconds. 429 430``name`` 431 Name of the idle state. 432 433``power`` 434 Power drawn by hardware in this idle state in milliwatts (if specified, 435 0 otherwise). 436 437``residency`` 438 Target residency of the idle state in microseconds. 439 440``time`` 441 Total time spent in this idle state by the given CPU (as measured by the 442 kernel) in microseconds. 443 444``usage`` 445 Total number of times the hardware has been asked by the given CPU to 446 enter this idle state. 447 448``rejected`` 449 Total number of times a request to enter this idle state on the given 450 CPU was rejected. 451 452The :file:`desc` and :file:`name` files both contain strings. The difference 453between them is that the name is expected to be more concise, while the 454description may be longer and it may contain white space or special characters. 455The other files listed above contain integer numbers. 456 457The :file:`disable` attribute is the only writeable one. If it contains 1, the 458given idle state is disabled for this particular CPU, which means that the 459governor will never select it for this particular CPU and the ``CPUIdle`` 460driver will never ask the hardware to enter it for that CPU as a result. 461However, disabling an idle state for one CPU does not prevent it from being 462asked for by the other CPUs, so it must be disabled for all of them in order to 463never be asked for by any of them. [Note that, due to the way the ``ladder`` 464governor is implemented, disabling an idle state prevents that governor from 465selecting any idle states deeper than the disabled one too.] 466 467If the :file:`disable` attribute contains 0, the given idle state is enabled for 468this particular CPU, but it still may be disabled for some or all of the other 469CPUs in the system at the same time. Writing 1 to it causes the idle state to 470be disabled for this particular CPU and writing 0 to it allows the governor to 471take it into consideration for the given CPU and the driver to ask for it, 472unless that state was disabled globally in the driver (in which case it cannot 473be used at all). 474 475The :file:`power` attribute is not defined very well, especially for idle state 476objects representing combinations of idle states at different levels of the 477hierarchy of units in the processor, and it generally is hard to obtain idle 478state power numbers for complex hardware, so :file:`power` often contains 0 (not 479available) and if it contains a nonzero number, that number may not be very 480accurate and it should not be relied on for anything meaningful. 481 482The number in the :file:`time` file generally may be greater than the total time 483really spent by the given CPU in the given idle state, because it is measured by 484the kernel and it may not cover the cases in which the hardware refused to enter 485this idle state and entered a shallower one instead of it (or even it did not 486enter any idle state at all). The kernel can only measure the time span between 487asking the hardware to enter an idle state and the subsequent wakeup of the CPU 488and it cannot say what really happened in the meantime at the hardware level. 489Moreover, if the idle state object in question represents a combination of idle 490states at different levels of the hierarchy of units in the processor, 491the kernel can never say how deep the hardware went down the hierarchy in any 492particular case. For these reasons, the only reliable way to find out how 493much time has been spent by the hardware in different idle states supported by 494it is to use idle state residency counters in the hardware, if available. 495 496Generally, an interrupt received when trying to enter an idle state causes the 497idle state entry request to be rejected, in which case the ``CPUIdle`` driver 498may return an error code to indicate that this was the case. The :file:`usage` 499and :file:`rejected` files report the number of times the given idle state 500was entered successfully or rejected, respectively. 501 502.. _cpu-pm-qos: 503 504Power Management Quality of Service for CPUs 505============================================ 506 507The power management quality of service (PM QoS) framework in the Linux kernel 508allows kernel code and user space processes to set constraints on various 509energy-efficiency features of the kernel to prevent performance from dropping 510below a required level. 511 512CPU idle time management can be affected by PM QoS in two ways, through the 513global CPU latency limit and through the resume latency constraints for 514individual CPUs. Kernel code (e.g. device drivers) can set both of them with 515the help of special internal interfaces provided by the PM QoS framework. User 516space can modify the former by opening the :file:`cpu_dma_latency` special 517device file under :file:`/dev/` and writing a binary value (interpreted as a 518signed 32-bit integer) to it. In turn, the resume latency constraint for a CPU 519can be modified from user space by writing a string (representing a signed 52032-bit integer) to the :file:`power/pm_qos_resume_latency_us` file under 521:file:`/sys/devices/system/cpu/cpu<N>/` in ``sysfs``, where the CPU number 522``<N>`` is allocated at the system initialization time. Negative values 523will be rejected in both cases and, also in both cases, the written integer 524number will be interpreted as a requested PM QoS constraint in microseconds. 525 526The requested value is not automatically applied as a new constraint, however, 527as it may be less restrictive (greater in this particular case) than another 528constraint previously requested by someone else. For this reason, the PM QoS 529framework maintains a list of requests that have been made so far for the 530global CPU latency limit and for each individual CPU, aggregates them and 531applies the effective (minimum in this particular case) value as the new 532constraint. 533 534In fact, opening the :file:`cpu_dma_latency` special device file causes a new 535PM QoS request to be created and added to a global priority list of CPU latency 536limit requests and the file descriptor coming from the "open" operation 537represents that request. If that file descriptor is then used for writing, the 538number written to it will be associated with the PM QoS request represented by 539it as a new requested limit value. Next, the priority list mechanism will be 540used to determine the new effective value of the entire list of requests and 541that effective value will be set as a new CPU latency limit. Thus requesting a 542new limit value will only change the real limit if the effective "list" value is 543affected by it, which is the case if it is the minimum of the requested values 544in the list. 545 546The process holding a file descriptor obtained by opening the 547:file:`cpu_dma_latency` special device file controls the PM QoS request 548associated with that file descriptor, but it controls this particular PM QoS 549request only. 550 551Closing the :file:`cpu_dma_latency` special device file or, more precisely, the 552file descriptor obtained while opening it, causes the PM QoS request associated 553with that file descriptor to be removed from the global priority list of CPU 554latency limit requests and destroyed. If that happens, the priority list 555mechanism will be used again, to determine the new effective value for the whole 556list and that value will become the new limit. 557 558In turn, for each CPU there is one resume latency PM QoS request associated with 559the :file:`power/pm_qos_resume_latency_us` file under 560:file:`/sys/devices/system/cpu/cpu<N>/` in ``sysfs`` and writing to it causes 561this single PM QoS request to be updated regardless of which user space 562process does that. In other words, this PM QoS request is shared by the entire 563user space, so access to the file associated with it needs to be arbitrated 564to avoid confusion. [Arguably, the only legitimate use of this mechanism in 565practice is to pin a process to the CPU in question and let it use the 566``sysfs`` interface to control the resume latency constraint for it.] It is 567still only a request, however. It is an entry in a priority list used to 568determine the effective value to be set as the resume latency constraint for the 569CPU in question every time the list of requests is updated this way or another 570(there may be other requests coming from kernel code in that list). 571 572CPU idle time governors are expected to regard the minimum of the global 573(effective) CPU latency limit and the effective resume latency constraint for 574the given CPU as the upper limit for the exit latency of the idle states that 575they are allowed to select for that CPU. They should never select any idle 576states with exit latency beyond that limit. 577 578 579Idle States Control Via Kernel Command Line 580=========================================== 581 582In addition to the ``sysfs`` interface allowing individual idle states to be 583`disabled for individual CPUs <idle-states-representation_>`_, there are kernel 584command line parameters affecting CPU idle time management. 585 586The ``cpuidle.off=1`` kernel command line option can be used to disable the 587CPU idle time management entirely. It does not prevent the idle loop from 588running on idle CPUs, but it prevents the CPU idle time governors and drivers 589from being invoked. If it is added to the kernel command line, the idle loop 590will ask the hardware to enter idle states on idle CPUs via the CPU architecture 591support code that is expected to provide a default mechanism for this purpose. 592That default mechanism usually is the least common denominator for all of the 593processors implementing the architecture (i.e. CPU instruction set) in question, 594however, so it is rather crude and not very energy-efficient. For this reason, 595it is not recommended for production use. 596 597The ``cpuidle.governor=`` kernel command line switch allows the ``CPUIdle`` 598governor to use to be specified. It has to be appended with a string matching 599the name of an available governor (e.g. ``cpuidle.governor=menu``) and that 600governor will be used instead of the default one. It is possible to force 601the ``menu`` governor to be used on the systems that use the ``ladder`` governor 602by default this way, for example. 603 604The other kernel command line parameters controlling CPU idle time management 605described below are only relevant for the *x86* architecture and references 606to ``intel_idle`` affect Intel processors only. 607 608The *x86* architecture support code recognizes three kernel command line 609options related to CPU idle time management: ``idle=poll``, ``idle=halt``, 610and ``idle=nomwait``. The first two of them disable the ``acpi_idle`` and 611``intel_idle`` drivers altogether, which effectively causes the entire 612``CPUIdle`` subsystem to be disabled and makes the idle loop invoke the 613architecture support code to deal with idle CPUs. How it does that depends on 614which of the two parameters is added to the kernel command line. In the 615``idle=halt`` case, the architecture support code will use the ``HLT`` 616instruction of the CPUs (which, as a rule, suspends the execution of the program 617and causes the hardware to attempt to enter the shallowest available idle state) 618for this purpose, and if ``idle=poll`` is used, idle CPUs will execute a 619more or less "lightweight" sequence of instructions in a tight loop. [Note 620that using ``idle=poll`` is somewhat drastic in many cases, as preventing idle 621CPUs from saving almost any energy at all may not be the only effect of it. 622For example, on Intel hardware it effectively prevents CPUs from using 623P-states (see |cpufreq|) that require any number of CPUs in a package to be 624idle, so it very well may hurt single-thread computations performance as well as 625energy-efficiency. Thus using it for performance reasons may not be a good idea 626at all.] 627 628The ``idle=nomwait`` option prevents the use of ``MWAIT`` instruction of 629the CPU to enter idle states. When this option is used, the ``acpi_idle`` 630driver will use the ``HLT`` instruction instead of ``MWAIT``. On systems 631running Intel processors, this option disables the ``intel_idle`` driver 632and forces the use of the ``acpi_idle`` driver instead. Note that in either 633case, ``acpi_idle`` driver will function only if all the information needed 634by it is in the system's ACPI tables. 635 636In addition to the architecture-level kernel command line options affecting CPU 637idle time management, there are parameters affecting individual ``CPUIdle`` 638drivers that can be passed to them via the kernel command line. Specifically, 639the ``intel_idle.max_cstate=<n>`` and ``processor.max_cstate=<n>`` parameters, 640where ``<n>`` is an idle state index also used in the name of the given 641state's directory in ``sysfs`` (see 642`Representation of Idle States <idle-states-representation_>`_), causes the 643``intel_idle`` and ``acpi_idle`` drivers, respectively, to discard all of the 644idle states deeper than idle state ``<n>``. In that case, they will never ask 645for any of those idle states or expose them to the governor. [The behavior of 646the two drivers is different for ``<n>`` equal to ``0``. Adding 647``intel_idle.max_cstate=0`` to the kernel command line disables the 648``intel_idle`` driver and allows ``acpi_idle`` to be used, whereas 649``processor.max_cstate=0`` is equivalent to ``processor.max_cstate=1``. 650Also, the ``acpi_idle`` driver is part of the ``processor`` kernel module that 651can be loaded separately and ``max_cstate=<n>`` can be passed to it as a module 652parameter when it is loaded.] 653