1*06a49ef7SShrikanth Hegde.. SPDX-License-Identifier: GPL-2.0 2*06a49ef7SShrikanth Hegde.. _sched-paravirt: 3*06a49ef7SShrikanth Hegde 4*06a49ef7SShrikanth HegdePreferred CPUs 5*06a49ef7SShrikanth Hegde============== 6*06a49ef7SShrikanth Hegde 7*06a49ef7SShrikanth HegdeIn paravirtualized environments CPU overcommit is a common scenario. 8*06a49ef7SShrikanth Hegdei.e. the sum of virtual CPUs (vCPUs) of all VMs is greater than number of 9*06a49ef7SShrikanth Hegdephysical CPUs (pCPUs). Under such conditions when all or many VMs have 10*06a49ef7SShrikanth Hegdehigh utilization, hypervisor won't be able to satisfy the CPU requirement 11*06a49ef7SShrikanth Hegdeand has to context switch within or across VMs. The hypervisor needs to 12*06a49ef7SShrikanth Hegdepreempt one vCPU to run another. This is called vCPU preemption. 13*06a49ef7SShrikanth HegdeThis is more expensive compared to task context switch within a vCPU, since 14*06a49ef7SShrikanth Hegdehypervisor lacks vCPU context and could preempt a critical section which 15*06a49ef7SShrikanth Hegdeslows forward progress. 16*06a49ef7SShrikanth Hegde 17*06a49ef7SShrikanth HegdeIn such cases it is better that combined vCPU demand from all VMs is reduced 18*06a49ef7SShrikanth Hegdeby not using some of the vCPUs in each VM. vCPUs where workload can be safely 19*06a49ef7SShrikanth Hegdescheduled which won't increase any contention for pCPU are called 20*06a49ef7SShrikanth Hegde"Preferred CPUs". 21*06a49ef7SShrikanth Hegde 22*06a49ef7SShrikanth HegdeOne of the main design constructs is that preferred CPUs are always 23*06a49ef7SShrikanth Hegdea subset of active CPUs. In most cases preferred CPUs will be same as 24*06a49ef7SShrikanth Hegdeactive CPUs. When there is pCPU contention, Preferred CPUs will reduce 25*06a49ef7SShrikanth Hegdebased on the steal time. When the pCPU contention goes away as indicated 26*06a49ef7SShrikanth Hegdeby steal time, Preferred CPUs could become same as active CPUs again. 27*06a49ef7SShrikanth HegdeThe policy decisions are to be taken by driver. 28*06a49ef7SShrikanth HegdeFor example, steal_governor. Look at its documentation for more 29*06a49ef7SShrikanth Hegdedetails. (``drivers/virt/steal_governor.c``) 30*06a49ef7SShrikanth Hegde 31*06a49ef7SShrikanth HegdeScheduling decisions such as wakeup, pushing the task etc, need this 32*06a49ef7SShrikanth HegdeCPU state info. This is maintained in ``cpu_preferred_mask``. 33*06a49ef7SShrikanth HegdevCPUs which are not in ``cpu_preferred_mask`` should be treated as vCPUs which 34*06a49ef7SShrikanth Hegdeshould not be used at this moment provided it doesn't break user affinity. 35*06a49ef7SShrikanth Hegde 36*06a49ef7SShrikanth HegdeThis is achieved by: 37*06a49ef7SShrikanth Hegde 38*06a49ef7SShrikanth Hegde1. Selecting a preferred CPU at wakeup using fallback mechanism. 39*06a49ef7SShrikanth Hegde2. Pushing the task away from non-preferred CPU at tick. 40*06a49ef7SShrikanth Hegde3. Selecting only preferred CPUs for load balance. 41*06a49ef7SShrikanth Hegde 42*06a49ef7SShrikanth Hegde``/sys/devices/system/cpu/preferred`` prints the current ``cpu_preferred_mask`` 43*06a49ef7SShrikanth Hegdein cpulist format. 44*06a49ef7SShrikanth Hegde 45*06a49ef7SShrikanth HegdeNotes: 46*06a49ef7SShrikanth Hegde 47*06a49ef7SShrikanth Hegde1. This feature is available under ``CONFIG_PREFERRED_CPU``. Driver which 48*06a49ef7SShrikanth Hegde makes decisions should enable it. For example, steal_governor driver 49*06a49ef7SShrikanth Hegde (``CONFIG_STEAL_GOVERNOR``). On enabling the driver, CPU preferred state 50*06a49ef7SShrikanth Hegde can change based on steal time. Without the driver, preferred CPUs is 51*06a49ef7SShrikanth Hegde same as active CPUs. 52*06a49ef7SShrikanth Hegde 53*06a49ef7SShrikanth Hegde2. This feature works for the FAIR class only. 54*06a49ef7SShrikanth Hegde 55*06a49ef7SShrikanth Hegde3. A pinned task, which can't be moved to preferred CPUs will continue 56*06a49ef7SShrikanth Hegde to run based on its affinity. But no load balancing happens if it is affined 57*06a49ef7SShrikanth Hegde only on non-preferred CPUs. 58*06a49ef7SShrikanth Hegde 59*06a49ef7SShrikanth Hegde4. Decision to change the preferred CPU state is driven by the kernel. 60*06a49ef7SShrikanth Hegde Hence it shouldn't break user affinities. One of the main reasons why 61*06a49ef7SShrikanth Hegde CPU hotplug or Isolated cpuset partitions was not a solution. 62*06a49ef7SShrikanth Hegde 63*06a49ef7SShrikanth Hegde5. This feature works best only when all the Guest VMs enable the feature as 64*06a49ef7SShrikanth Hegde it is a co-operative scheme. If a specific VM doesn't enable this feature 65*06a49ef7SShrikanth Hegde it may end up with more CPUs than others, still should lead to better 66*06a49ef7SShrikanth Hegde performance when seen from system view. 67*06a49ef7SShrikanth Hegde Users who enable this driver must ensure it is enabled in all Guest VMs. 68