1af105c9cSLike Xu.. SPDX-License-Identifier: GPL-2.0 2af105c9cSLike Xu 32f5947dfSChristoph Hellwig================= 42f5947dfSChristoph HellwigKVM VCPU Requests 52f5947dfSChristoph Hellwig================= 62f5947dfSChristoph Hellwig 72f5947dfSChristoph HellwigOverview 82f5947dfSChristoph Hellwig======== 92f5947dfSChristoph Hellwig 102f5947dfSChristoph HellwigKVM supports an internal API enabling threads to request a VCPU thread to 112f5947dfSChristoph Hellwigperform some activity. For example, a thread may request a VCPU to flush 122f5947dfSChristoph Hellwigits TLB with a VCPU request. The API consists of the following functions:: 132f5947dfSChristoph Hellwig 142f5947dfSChristoph Hellwig /* Check if any requests are pending for VCPU @vcpu. */ 152f5947dfSChristoph Hellwig bool kvm_request_pending(struct kvm_vcpu *vcpu); 162f5947dfSChristoph Hellwig 172f5947dfSChristoph Hellwig /* Check if VCPU @vcpu has request @req pending. */ 182f5947dfSChristoph Hellwig bool kvm_test_request(int req, struct kvm_vcpu *vcpu); 192f5947dfSChristoph Hellwig 202f5947dfSChristoph Hellwig /* Clear request @req for VCPU @vcpu. */ 212f5947dfSChristoph Hellwig void kvm_clear_request(int req, struct kvm_vcpu *vcpu); 222f5947dfSChristoph Hellwig 232f5947dfSChristoph Hellwig /* 242f5947dfSChristoph Hellwig * Check if VCPU @vcpu has request @req pending. When the request is 252f5947dfSChristoph Hellwig * pending it will be cleared and a memory barrier, which pairs with 262f5947dfSChristoph Hellwig * another in kvm_make_request(), will be issued. 272f5947dfSChristoph Hellwig */ 282f5947dfSChristoph Hellwig bool kvm_check_request(int req, struct kvm_vcpu *vcpu); 292f5947dfSChristoph Hellwig 302f5947dfSChristoph Hellwig /* 312f5947dfSChristoph Hellwig * Make request @req of VCPU @vcpu. Issues a memory barrier, which pairs 322f5947dfSChristoph Hellwig * with another in kvm_check_request(), prior to setting the request. 332f5947dfSChristoph Hellwig */ 342f5947dfSChristoph Hellwig void kvm_make_request(int req, struct kvm_vcpu *vcpu); 352f5947dfSChristoph Hellwig 362f5947dfSChristoph Hellwig /* Make request @req of all VCPUs of the VM with struct kvm @kvm. */ 372f5947dfSChristoph Hellwig bool kvm_make_all_cpus_request(struct kvm *kvm, unsigned int req); 382f5947dfSChristoph Hellwig 392f5947dfSChristoph HellwigTypically a requester wants the VCPU to perform the activity as soon 402f5947dfSChristoph Hellwigas possible after making the request. This means most requests 412f5947dfSChristoph Hellwig(kvm_make_request() calls) are followed by a call to kvm_vcpu_kick(), 422f5947dfSChristoph Hellwigand kvm_make_all_cpus_request() has the kicking of all VCPUs built 432f5947dfSChristoph Hellwiginto it. 442f5947dfSChristoph Hellwig 452f5947dfSChristoph HellwigVCPU Kicks 462f5947dfSChristoph Hellwig---------- 472f5947dfSChristoph Hellwig 482f5947dfSChristoph HellwigThe goal of a VCPU kick is to bring a VCPU thread out of guest mode in 492f5947dfSChristoph Hellwigorder to perform some KVM maintenance. To do so, an IPI is sent, forcing 502f5947dfSChristoph Hellwiga guest mode exit. However, a VCPU thread may not be in guest mode at the 512f5947dfSChristoph Hellwigtime of the kick. Therefore, depending on the mode and state of the VCPU 522f5947dfSChristoph Hellwigthread, there are two other actions a kick may take. All three actions 532f5947dfSChristoph Hellwigare listed below: 542f5947dfSChristoph Hellwig 552f5947dfSChristoph Hellwig1) Send an IPI. This forces a guest mode exit. 562f5947dfSChristoph Hellwig2) Waking a sleeping VCPU. Sleeping VCPUs are VCPU threads outside guest 572f5947dfSChristoph Hellwig mode that wait on waitqueues. Waking them removes the threads from 582f5947dfSChristoph Hellwig the waitqueues, allowing the threads to run again. This behavior 592f5947dfSChristoph Hellwig may be suppressed, see KVM_REQUEST_NO_WAKEUP below. 602f5947dfSChristoph Hellwig3) Nothing. When the VCPU is not in guest mode and the VCPU thread is not 612f5947dfSChristoph Hellwig sleeping, then there is nothing to do. 622f5947dfSChristoph Hellwig 632f5947dfSChristoph HellwigVCPU Mode 642f5947dfSChristoph Hellwig--------- 652f5947dfSChristoph Hellwig 662f5947dfSChristoph HellwigVCPUs have a mode state, ``vcpu->mode``, that is used to track whether the 672f5947dfSChristoph Hellwigguest is running in guest mode or not, as well as some specific 682f5947dfSChristoph Hellwigoutside guest mode states. The architecture may use ``vcpu->mode`` to 692f5947dfSChristoph Hellwigensure VCPU requests are seen by VCPUs (see "Ensuring Requests Are Seen"), 702f5947dfSChristoph Hellwigas well as to avoid sending unnecessary IPIs (see "IPI Reduction"), and 712f5947dfSChristoph Hellwigeven to ensure IPI acknowledgements are waited upon (see "Waiting for 722f5947dfSChristoph HellwigAcknowledgements"). The following modes are defined: 732f5947dfSChristoph Hellwig 742f5947dfSChristoph HellwigOUTSIDE_GUEST_MODE 752f5947dfSChristoph Hellwig 762f5947dfSChristoph Hellwig The VCPU thread is outside guest mode. 772f5947dfSChristoph Hellwig 782f5947dfSChristoph HellwigIN_GUEST_MODE 792f5947dfSChristoph Hellwig 802f5947dfSChristoph Hellwig The VCPU thread is in guest mode. 812f5947dfSChristoph Hellwig 822f5947dfSChristoph HellwigEXITING_GUEST_MODE 832f5947dfSChristoph Hellwig 842f5947dfSChristoph Hellwig The VCPU thread is transitioning from IN_GUEST_MODE to 852f5947dfSChristoph Hellwig OUTSIDE_GUEST_MODE. 862f5947dfSChristoph Hellwig 872f5947dfSChristoph HellwigREADING_SHADOW_PAGE_TABLES 882f5947dfSChristoph Hellwig 892f5947dfSChristoph Hellwig The VCPU thread is outside guest mode, but it wants the sender of 902f5947dfSChristoph Hellwig certain VCPU requests, namely KVM_REQ_TLB_FLUSH, to wait until the VCPU 912f5947dfSChristoph Hellwig thread is done reading the page tables. 922f5947dfSChristoph Hellwig 932f5947dfSChristoph HellwigVCPU Request Internals 942f5947dfSChristoph Hellwig====================== 952f5947dfSChristoph Hellwig 962f5947dfSChristoph HellwigVCPU requests are simply bit indices of the ``vcpu->requests`` bitmap. 972f5947dfSChristoph HellwigThis means general bitops, like those documented in [atomic-ops]_ could 982f5947dfSChristoph Hellwigalso be used, e.g. :: 992f5947dfSChristoph Hellwig 100c59fb127SPaolo Bonzini clear_bit(KVM_REQ_UNBLOCK & KVM_REQUEST_MASK, &vcpu->requests); 1012f5947dfSChristoph Hellwig 1022f5947dfSChristoph HellwigHowever, VCPU request users should refrain from doing so, as it would 1032f5947dfSChristoph Hellwigbreak the abstraction. The first 8 bits are reserved for architecture 104*95b4d47aSRandy Dunlapindependent requests; all additional bits are available for architecture 1052f5947dfSChristoph Hellwigdependent requests. 1062f5947dfSChristoph Hellwig 1072f5947dfSChristoph HellwigArchitecture Independent Requests 1082f5947dfSChristoph Hellwig--------------------------------- 1092f5947dfSChristoph Hellwig 1102f5947dfSChristoph HellwigKVM_REQ_TLB_FLUSH 1112f5947dfSChristoph Hellwig 1122f5947dfSChristoph Hellwig KVM's common MMU notifier may need to flush all of a guest's TLB 1132f5947dfSChristoph Hellwig entries, calling kvm_flush_remote_tlbs() to do so. Architectures that 1142f5947dfSChristoph Hellwig choose to use the common kvm_flush_remote_tlbs() implementation will 1152f5947dfSChristoph Hellwig need to handle this VCPU request. 1162f5947dfSChristoph Hellwig 117e65a3b46SSean ChristophersonKVM_REQ_VM_DEAD 1182f5947dfSChristoph Hellwig 119e65a3b46SSean Christopherson This request informs all VCPUs that the VM is dead and unusable, e.g. due to 120e65a3b46SSean Christopherson fatal error or because the VM's state has been intentionally destroyed. 1212f5947dfSChristoph Hellwig 122084071d5SMarcelo TosattiKVM_REQ_UNBLOCK 1232f5947dfSChristoph Hellwig 124084071d5SMarcelo Tosatti This request informs the vCPU to exit kvm_vcpu_block. It is used for 125084071d5SMarcelo Tosatti example from timer handlers that run on the host on behalf of a vCPU, 126084071d5SMarcelo Tosatti or in order to update the interrupt routing and ensure that assigned 127084071d5SMarcelo Tosatti devices will wake up the vCPU. 1282f5947dfSChristoph Hellwig 129df06dae3SSean ChristophersonKVM_REQ_OUTSIDE_GUEST_MODE 130df06dae3SSean Christopherson 131df06dae3SSean Christopherson This "request" ensures the target vCPU has exited guest mode prior to the 132df06dae3SSean Christopherson sender of the request continuing on. No action needs be taken by the target, 133df06dae3SSean Christopherson and so no request is actually logged for the target. This request is similar 134df06dae3SSean Christopherson to a "kick", but unlike a kick it guarantees the vCPU has actually exited 135df06dae3SSean Christopherson guest mode. A kick only guarantees the vCPU will exit at some point in the 136df06dae3SSean Christopherson future, e.g. a previous kick may have started the process, but there's no 137df06dae3SSean Christopherson guarantee the to-be-kicked vCPU has fully exited guest mode. 138df06dae3SSean Christopherson 1392f5947dfSChristoph HellwigKVM_REQUEST_MASK 1402f5947dfSChristoph Hellwig---------------- 1412f5947dfSChristoph Hellwig 1422f5947dfSChristoph HellwigVCPU requests should be masked by KVM_REQUEST_MASK before using them with 1432f5947dfSChristoph Hellwigbitops. This is because only the lower 8 bits are used to represent the 1442f5947dfSChristoph Hellwigrequest's number. The upper bits are used as flags. Currently only two 1452f5947dfSChristoph Hellwigflags are defined. 1462f5947dfSChristoph Hellwig 1472f5947dfSChristoph HellwigVCPU Request Flags 1482f5947dfSChristoph Hellwig------------------ 1492f5947dfSChristoph Hellwig 1502f5947dfSChristoph HellwigKVM_REQUEST_NO_WAKEUP 1512f5947dfSChristoph Hellwig 1522f5947dfSChristoph Hellwig This flag is applied to requests that only need immediate attention 1532f5947dfSChristoph Hellwig from VCPUs running in guest mode. That is, sleeping VCPUs do not need 154*95b4d47aSRandy Dunlap to be awakened for these requests. Sleeping VCPUs will handle the 155*95b4d47aSRandy Dunlap requests when they are awakened later for some other reason. 1562f5947dfSChristoph Hellwig 1572f5947dfSChristoph HellwigKVM_REQUEST_WAIT 1582f5947dfSChristoph Hellwig 1592f5947dfSChristoph Hellwig When requests with this flag are made with kvm_make_all_cpus_request(), 1602f5947dfSChristoph Hellwig then the caller will wait for each VCPU to acknowledge its IPI before 1612f5947dfSChristoph Hellwig proceeding. This flag only applies to VCPUs that would receive IPIs. 1622f5947dfSChristoph Hellwig If, for example, the VCPU is sleeping, so no IPI is necessary, then 1632f5947dfSChristoph Hellwig the requesting thread does not wait. This means that this flag may be 1642f5947dfSChristoph Hellwig safely combined with KVM_REQUEST_NO_WAKEUP. See "Waiting for 1652f5947dfSChristoph Hellwig Acknowledgements" for more information about requests with 1662f5947dfSChristoph Hellwig KVM_REQUEST_WAIT. 1672f5947dfSChristoph Hellwig 1682f5947dfSChristoph HellwigVCPU Requests with Associated State 1692f5947dfSChristoph Hellwig=================================== 1702f5947dfSChristoph Hellwig 1712f5947dfSChristoph HellwigRequesters that want the receiving VCPU to handle new state need to ensure 1722f5947dfSChristoph Hellwigthe newly written state is observable to the receiving VCPU thread's CPU 1732f5947dfSChristoph Hellwigby the time it observes the request. This means a write memory barrier 1742f5947dfSChristoph Hellwigmust be inserted after writing the new state and before setting the VCPU 1752f5947dfSChristoph Hellwigrequest bit. Additionally, on the receiving VCPU thread's side, a 1762f5947dfSChristoph Hellwigcorresponding read barrier must be inserted after reading the request bit 1772f5947dfSChristoph Hellwigand before proceeding to read the new state associated with it. See 1782f5947dfSChristoph Hellwigscenario 3, Message and Flag, of [lwn-mb]_ and the kernel documentation 1792f5947dfSChristoph Hellwig[memory-barriers]_. 1802f5947dfSChristoph Hellwig 1812f5947dfSChristoph HellwigThe pair of functions, kvm_check_request() and kvm_make_request(), provide 1822f5947dfSChristoph Hellwigthe memory barriers, allowing this requirement to be handled internally by 1832f5947dfSChristoph Hellwigthe API. 1842f5947dfSChristoph Hellwig 1852f5947dfSChristoph HellwigEnsuring Requests Are Seen 1862f5947dfSChristoph Hellwig========================== 1872f5947dfSChristoph Hellwig 1882f5947dfSChristoph HellwigWhen making requests to VCPUs, we want to avoid the receiving VCPU 1892f5947dfSChristoph Hellwigexecuting in guest mode for an arbitrary long time without handling the 1902f5947dfSChristoph Hellwigrequest. We can be sure this won't happen as long as we ensure the VCPU 1912f5947dfSChristoph Hellwigthread checks kvm_request_pending() before entering guest mode and that a 1922f5947dfSChristoph Hellwigkick will send an IPI to force an exit from guest mode when necessary. 1932f5947dfSChristoph HellwigExtra care must be taken to cover the period after the VCPU thread's last 1942f5947dfSChristoph Hellwigkvm_request_pending() check and before it has entered guest mode, as kick 1952f5947dfSChristoph HellwigIPIs will only trigger guest mode exits for VCPU threads that are in guest 1962f5947dfSChristoph Hellwigmode or at least have already disabled interrupts in order to prepare to 1972f5947dfSChristoph Hellwigenter guest mode. This means that an optimized implementation (see "IPI 1982f5947dfSChristoph HellwigReduction") must be certain when it's safe to not send the IPI. One 1992f5947dfSChristoph Hellwigsolution, which all architectures except s390 apply, is to: 2002f5947dfSChristoph Hellwig 2012f5947dfSChristoph Hellwig- set ``vcpu->mode`` to IN_GUEST_MODE between disabling the interrupts and 2022f5947dfSChristoph Hellwig the last kvm_request_pending() check; 2032f5947dfSChristoph Hellwig- enable interrupts atomically when entering the guest. 2042f5947dfSChristoph Hellwig 2052f5947dfSChristoph HellwigThis solution also requires memory barriers to be placed carefully in both 2062f5947dfSChristoph Hellwigthe requesting thread and the receiving VCPU. With the memory barriers we 2072f5947dfSChristoph Hellwigcan exclude the possibility of a VCPU thread observing 2082f5947dfSChristoph Hellwig!kvm_request_pending() on its last check and then not receiving an IPI for 2092f5947dfSChristoph Hellwigthe next request made of it, even if the request is made immediately after 2102f5947dfSChristoph Hellwigthe check. This is done by way of the Dekker memory barrier pattern 2112f5947dfSChristoph Hellwig(scenario 10 of [lwn-mb]_). As the Dekker pattern requires two variables, 2122f5947dfSChristoph Hellwigthis solution pairs ``vcpu->mode`` with ``vcpu->requests``. Substituting 2132f5947dfSChristoph Hellwigthem into the pattern gives:: 2142f5947dfSChristoph Hellwig 2152f5947dfSChristoph Hellwig CPU1 CPU2 2162f5947dfSChristoph Hellwig ================= ================= 2172f5947dfSChristoph Hellwig local_irq_disable(); 2182f5947dfSChristoph Hellwig WRITE_ONCE(vcpu->mode, IN_GUEST_MODE); kvm_make_request(REQ, vcpu); 2192f5947dfSChristoph Hellwig smp_mb(); smp_mb(); 2202f5947dfSChristoph Hellwig if (kvm_request_pending(vcpu)) { if (READ_ONCE(vcpu->mode) == 2212f5947dfSChristoph Hellwig IN_GUEST_MODE) { 2222f5947dfSChristoph Hellwig ...abort guest entry... ...send IPI... 2232f5947dfSChristoph Hellwig } } 2242f5947dfSChristoph Hellwig 2252f5947dfSChristoph HellwigAs stated above, the IPI is only useful for VCPU threads in guest mode or 2262f5947dfSChristoph Hellwigthat have already disabled interrupts. This is why this specific case of 2272f5947dfSChristoph Hellwigthe Dekker pattern has been extended to disable interrupts before setting 2282f5947dfSChristoph Hellwig``vcpu->mode`` to IN_GUEST_MODE. WRITE_ONCE() and READ_ONCE() are used to 2292f5947dfSChristoph Hellwigpedantically implement the memory barrier pattern, guaranteeing the 2302f5947dfSChristoph Hellwigcompiler doesn't interfere with ``vcpu->mode``'s carefully planned 2312f5947dfSChristoph Hellwigaccesses. 2322f5947dfSChristoph Hellwig 2332f5947dfSChristoph HellwigIPI Reduction 2342f5947dfSChristoph Hellwig------------- 2352f5947dfSChristoph Hellwig 2362f5947dfSChristoph HellwigAs only one IPI is needed to get a VCPU to check for any/all requests, 2372f5947dfSChristoph Hellwigthen they may be coalesced. This is easily done by having the first IPI 2382f5947dfSChristoph Hellwigsending kick also change the VCPU mode to something !IN_GUEST_MODE. The 2392f5947dfSChristoph Hellwigtransitional state, EXITING_GUEST_MODE, is used for this purpose. 2402f5947dfSChristoph Hellwig 2412f5947dfSChristoph HellwigWaiting for Acknowledgements 2422f5947dfSChristoph Hellwig---------------------------- 2432f5947dfSChristoph Hellwig 2442f5947dfSChristoph HellwigSome requests, those with the KVM_REQUEST_WAIT flag set, require IPIs to 2452f5947dfSChristoph Hellwigbe sent, and the acknowledgements to be waited upon, even when the target 2462f5947dfSChristoph HellwigVCPU threads are in modes other than IN_GUEST_MODE. For example, one case 2472f5947dfSChristoph Hellwigis when a target VCPU thread is in READING_SHADOW_PAGE_TABLES mode, which 2482f5947dfSChristoph Hellwigis set after disabling interrupts. To support these cases, the 2492f5947dfSChristoph HellwigKVM_REQUEST_WAIT flag changes the condition for sending an IPI from 2502f5947dfSChristoph Hellwigchecking that the VCPU is IN_GUEST_MODE to checking that it is not 2512f5947dfSChristoph HellwigOUTSIDE_GUEST_MODE. 2522f5947dfSChristoph Hellwig 2532f5947dfSChristoph HellwigRequest-less VCPU Kicks 2542f5947dfSChristoph Hellwig----------------------- 2552f5947dfSChristoph Hellwig 2562f5947dfSChristoph HellwigAs the determination of whether or not to send an IPI depends on the 2572f5947dfSChristoph Hellwigtwo-variable Dekker memory barrier pattern, then it's clear that 2582f5947dfSChristoph Hellwigrequest-less VCPU kicks are almost never correct. Without the assurance 2592f5947dfSChristoph Hellwigthat a non-IPI generating kick will still result in an action by the 2602f5947dfSChristoph Hellwigreceiving VCPU, as the final kvm_request_pending() check does for 2612f5947dfSChristoph Hellwigrequest-accompanying kicks, then the kick may not do anything useful at 2622f5947dfSChristoph Hellwigall. If, for instance, a request-less kick was made to a VCPU that was 2632f5947dfSChristoph Hellwigjust about to set its mode to IN_GUEST_MODE, meaning no IPI is sent, then 2642f5947dfSChristoph Hellwigthe VCPU thread may continue its entry without actually having done 2652f5947dfSChristoph Hellwigwhatever it was the kick was meant to initiate. 2662f5947dfSChristoph Hellwig 2672f5947dfSChristoph HellwigOne exception is x86's posted interrupt mechanism. In this case, however, 2682f5947dfSChristoph Hellwigeven the request-less VCPU kick is coupled with the same 2692f5947dfSChristoph Hellwiglocal_irq_disable() + smp_mb() pattern described above; the ON bit 2702f5947dfSChristoph Hellwig(Outstanding Notification) in the posted interrupt descriptor takes the 2712f5947dfSChristoph Hellwigrole of ``vcpu->requests``. When sending a posted interrupt, PIR.ON is 2722f5947dfSChristoph Hellwigset before reading ``vcpu->mode``; dually, in the VCPU thread, 2732f5947dfSChristoph Hellwigvmx_sync_pir_to_irr() reads PIR after setting ``vcpu->mode`` to 2742f5947dfSChristoph HellwigIN_GUEST_MODE. 2752f5947dfSChristoph Hellwig 2762f5947dfSChristoph HellwigAdditional Considerations 2772f5947dfSChristoph Hellwig========================= 2782f5947dfSChristoph Hellwig 2792f5947dfSChristoph HellwigSleeping VCPUs 2802f5947dfSChristoph Hellwig-------------- 2812f5947dfSChristoph Hellwig 2822f5947dfSChristoph HellwigVCPU threads may need to consider requests before and/or after calling 2832f5947dfSChristoph Hellwigfunctions that may put them to sleep, e.g. kvm_vcpu_block(). Whether they 2842f5947dfSChristoph Hellwigdo or not, and, if they do, which requests need consideration, is 2852f5947dfSChristoph Hellwigarchitecture dependent. kvm_vcpu_block() calls kvm_arch_vcpu_runnable() 2862f5947dfSChristoph Hellwigto check if it should awaken. One reason to do so is to provide 2872f5947dfSChristoph Hellwigarchitectures a function where requests may be checked if necessary. 2882f5947dfSChristoph Hellwig 2892f5947dfSChristoph HellwigReferences 2902f5947dfSChristoph Hellwig========== 2912f5947dfSChristoph Hellwig 292e437c1a3SMauro Carvalho Chehab.. [atomic-ops] Documentation/atomic_bitops.txt and Documentation/atomic_t.txt 2932f5947dfSChristoph Hellwig.. [memory-barriers] Documentation/memory-barriers.txt 2942f5947dfSChristoph Hellwig.. [lwn-mb] https://lwn.net/Articles/573436/ 295