1Entry/exit handling for exceptions, interrupts, syscalls and KVM 2================================================================ 3 4All transitions between execution domains require state updates which are 5subject to strict ordering constraints. State updates are required for the 6following: 7 8 * Lockdep 9 * RCU / Context tracking 10 * Preemption counter 11 * Tracing 12 * Time accounting 13 14The update order depends on the transition type and is explained below in 15the transition type sections: `Syscalls`_, `KVM`_, `Interrupts and regular 16exceptions`_, `NMI and NMI-like exceptions`_. 17 18Non-instrumentable code - noinstr 19--------------------------------- 20 21Most instrumentation facilities depend on RCU, so instrumentation is prohibited 22for entry code before RCU starts watching and exit code after RCU stops 23watching. In addition, many architectures must save and restore register state, 24which means that (for example) a breakpoint in the breakpoint entry code would 25overwrite the debug registers of the initial breakpoint. 26 27Such code must be marked with the 'noinstr' attribute, placing that code into a 28special section inaccessible to instrumentation and debug facilities. Some 29functions are partially instrumentable, which is handled by marking them 30noinstr and using instrumentation_begin() and instrumentation_end() to flag the 31instrumentable ranges of code: 32 33.. code-block:: c 34 35 noinstr void entry(void) 36 { 37 handle_entry(); // <-- must be 'noinstr' or '__always_inline' 38 ... 39 40 instrumentation_begin(); 41 handle_context(); // <-- instrumentable code 42 instrumentation_end(); 43 44 ... 45 handle_exit(); // <-- must be 'noinstr' or '__always_inline' 46 } 47 48This allows verification of the 'noinstr' restrictions via objtool on 49supported architectures. 50 51Invoking non-instrumentable functions from instrumentable context has no 52restrictions and is useful to protect e.g. state switching which would 53cause malfunction if instrumented. 54 55All non-instrumentable entry/exit code sections before and after the RCU 56state transitions must run with interrupts disabled. 57 58Syscalls 59-------- 60 61Syscall-entry code starts in assembly code and calls out into low-level C 62code after establishing low-level architecture-specific state and stack 63frames. This low-level C code must not be instrumented. The recommended 64syscall handling function invoked from low-level assembly code looks like 65this: 66 67.. code-block:: c 68 69 noinstr void syscall(struct pt_regs *regs, long nr) 70 { 71 arch_syscall_enter(regs); 72 result_reg(regs) = -ENOSYS; 73 if (syscall_enter_from_user_mode_randomize_stack(regs, &nr)) { 74 instrumentation_begin(); 75 if (valid(nr) 76 result_reg(regs) = invoke_syscall(regs, nr); 77 instrumentation_end(); 78 } 79 syscall_exit_to_user_mode(regs); 80 } 81 82This is the most resilent variant as it has always a guaranteed valid 83return code. The alternative variant is: 84 85.. code-block:: c 86 87 noinstr void syscall(struct pt_regs *regs, long nr) 88 { 89 arch_syscall_enter(regs); 90 if (syscall_enter_from_user_mode_randomize_stack(regs, &nr)) { 91 instrumentation_begin(); 92 if (valid(nr) 93 result_reg(regs) = invoke_syscall(regs, nr); 94 else 95 result_reg(regs) = -ENOSYS; 96 instrumentation_end(); 97 } 98 syscall_exit_to_user_mode(regs); 99 } 100 101That works for most situations except when a probe/BPF attached to the 102syscall tracepoint sets an invalid syscall number e.g. -1 and also modifies 103the result register. So this variant will obviously overwrite the modified 104result with -ENOSYS. 105 106syscall_enter_from_user_mode_randomize_stack() first invokes 107enter_from_user_mode_randomize_stack() which establishes state in the 108following order: 109 110 * Lockdep 111 * RCU / Context tracking 112 * Tracing 113 * Apply stack randomization 114 115and then invokes the various entry work functions like ptrace, seccomp, audit, 116syscall tracing, etc. After all that is done, the instrumentable invoke_syscall 117function can be invoked. The instrumentable code section then ends, after which 118syscall_exit_to_user_mode() is invoked. 119 120syscall_exit_to_user_mode() handles all work which needs to be done before 121returning to user space like tracing, audit, signals, task work etc. After 122that it invokes exit_to_user_mode() which again handles the state 123transition in the reverse order: 124 125 * Tracing 126 * RCU / Context tracking 127 * Lockdep 128 129syscall_enter_from_user_mode_randomize_stack() and 130syscall_exit_to_user_mode() are also available as fine grained subfunctions 131in cases where the architecture code has to do extra work between the 132various steps. In such cases it has to ensure that 133enter_from_user_mode_randomize_stack() is called first on entry and 134exit_to_user_mode() is called last on exit. 135 136Do not nest syscalls. Nested syscalls will cause RCU and/or context tracking 137to print a warning. 138 139KVM 140--- 141 142Entering or exiting guest mode is very similar to syscalls. From the host 143kernel point of view the CPU goes off into user space when entering the 144guest and returns to the kernel on exit. 145 146guest_state_enter_irqoff() is a KVM-specific variant of exit_to_user_mode() 147and guest_state_exit_irqoff() is the KVM variant of enter_from_user_mode(). 148The state operations have the same ordering. 149 150Task work handling is done separately for guest at the boundary of the 151vcpu_run() loop via xfer_to_guest_mode_handle_work() which is a subset of 152the work handled on return to user space. 153 154Do not nest KVM entry/exit transitions because doing so is nonsensical. 155 156Interrupts and regular exceptions 157--------------------------------- 158 159Interrupts entry and exit handling is slightly more complex than syscalls 160and KVM transitions. 161 162If an interrupt is raised while the CPU executes in user space, the entry 163and exit handling is exactly the same as for syscalls. 164 165If the interrupt is raised while the CPU executes in kernel space the entry and 166exit handling is slightly different. RCU state is only updated when the 167interrupt is raised in the context of the CPU's idle task. Otherwise, RCU will 168already be watching. Lockdep and tracing have to be updated unconditionally. 169 170irqentry_enter() and irqentry_exit() provide the implementation for this. 171 172The architecture-specific part looks similar to syscall handling: 173 174.. code-block:: c 175 176 noinstr void interrupt(struct pt_regs *regs, int nr) 177 { 178 arch_interrupt_enter(regs); 179 state = irqentry_enter(regs); 180 181 instrumentation_begin(); 182 183 irq_enter_rcu(); 184 invoke_irq_handler(regs, nr); 185 irq_exit_rcu(); 186 187 instrumentation_end(); 188 189 irqentry_exit(regs, state); 190 } 191 192Note that the invocation of the actual interrupt handler is within a 193irq_enter_rcu() and irq_exit_rcu() pair. 194 195irq_enter_rcu() updates the preemption count which makes in_hardirq() 196return true, handles NOHZ tick state and interrupt time accounting. This 197means that up to the point where irq_enter_rcu() is invoked in_hardirq() 198returns false. 199 200irq_exit_rcu() handles interrupt time accounting, handles soft interrupts if 201possible, handles the NOHZ tick state and finally undoes the preemption count 202update. 203 204In theory, the preemption count could be updated in irqentry_enter(). In 205practice, deferring this update to irq_enter_rcu() allows the preemption-count 206code to be traced, while also maintaining symmetry with irq_exit_rcu() and 207irqentry_exit(), which are described in the next paragraph. The only downside 208is that the early entry code up to irq_enter_rcu() must be aware that the 209preemption count has not yet been updated with the HARDIRQ_OFFSET state. 210 211Note that soft interrupt handlers must run in BH context rather than in hard 212interrupt context. irq_exit_rcu() therefore replaces HARDIRQ_OFFSET with 213SOFTIRQ_OFFSET in the preemption count while it handles soft interrupts and 214puts HARDIRQ_OFFSET back afterwards, so that the remaining interrupt exit work 215is still attributed to the interrupt. HARDIRQ_OFFSET is removed before 216irq_exit_rcu() returns because irqentry_exit() might schedule, which requires 217that HARDIRQ_OFFSET has been removed from the preemption count. 218 219Even though interrupt handlers are expected to run with local interrupts 220disabled, interrupt nesting is common from an entry/exit perspective. For 221example, softirq handling happens within an irqentry_{enter,exit}() block with 222local interrupts enabled. Also, although uncommon, nothing prevents an 223interrupt handler from re-enabling interrupts. 224 225Interrupt entry/exit code doesn't strictly need to handle reentrancy, since it 226runs with local interrupts disabled. But NMIs can happen anytime, and a lot of 227the entry code is shared between the two. 228 229NMI and NMI-like exceptions 230--------------------------- 231 232NMIs and NMI-like exceptions (machine checks, double faults, debug 233interrupts, etc.) can hit any context and must be extra careful with 234the state. 235 236State changes for debug exceptions and machine-check exceptions depend on 237whether these exceptions happened in user-space (breakpoints or watchpoints) or 238in kernel mode (code patching). From user-space, they are treated like 239interrupts, while from kernel mode they are treated like NMIs. 240 241NMIs and other NMI-like exceptions handle state transitions without 242distinguishing between user-mode and kernel-mode origin. 243 244The state update on entry is handled in irqentry_nmi_enter() which updates 245state in the following order: 246 247 * Preemption counter 248 * Lockdep 249 * RCU / Context tracking 250 * Tracing 251 252The exit counterpart irqentry_nmi_exit() does the reverse operation in the 253reverse order. 254 255Note that the update of the preemption counter has to be the first 256operation on enter and the last operation on exit. The reason is that both 257lockdep and RCU rely on in_nmi() returning true in this case. The 258preemption count modification in the NMI entry/exit case must not be 259traced. 260 261Architecture-specific code looks like this: 262 263.. code-block:: c 264 265 noinstr void nmi(struct pt_regs *regs) 266 { 267 arch_nmi_enter(regs); 268 state = irqentry_nmi_enter(regs); 269 270 instrumentation_begin(); 271 nmi_handler(regs); 272 instrumentation_end(); 273 274 irqentry_nmi_exit(regs); 275 } 276 277and for e.g. a debug exception it can look like this: 278 279.. code-block:: c 280 281 noinstr void debug(struct pt_regs *regs) 282 { 283 arch_nmi_enter(regs); 284 285 debug_regs = save_debug_regs(); 286 287 if (user_mode(regs)) { 288 state = irqentry_enter(regs); 289 290 instrumentation_begin(); 291 user_mode_debug_handler(regs, debug_regs); 292 instrumentation_end(); 293 294 irqentry_exit(regs, state); 295 } else { 296 state = irqentry_nmi_enter(regs); 297 298 instrumentation_begin(); 299 kernel_mode_debug_handler(regs, debug_regs); 300 instrumentation_end(); 301 302 irqentry_nmi_exit(regs, state); 303 } 304 } 305 306There is no combined irqentry_nmi_if_kernel() function available as the 307above cannot be handled in an exception-agnostic way. 308 309NMIs can happen in any context. For example, an NMI-like exception triggered 310while handling an NMI. So NMI entry code has to be reentrant and state updates 311need to handle nesting. 312