1Real-time application monitors 2============================== 3 4- Name: rtapp 5- Type: container for multiple monitors 6- Author: Nam Cao <namcao@linutronix.de> 7 8Description 9----------- 10 11Real-time applications may have design flaws such that they experience 12unexpected latency and fail to meet their time requirements. Often, these flaws 13follow a few patterns: 14 15 - Page faults: A real-time thread may access memory that does not have a 16 mapped physical backing or must first be copied (such as for copy-on-write). 17 Thus a page fault is raised and the kernel must first perform the expensive 18 action. This causes significant delays to the real-time thread 19 - Priority inversion: A real-time thread blocks waiting for a lower-priority 20 thread. This causes the real-time thread to effectively take on the 21 scheduling priority of the lower-priority thread. For example, the real-time 22 thread needs to access a shared resource that is protected by a 23 non-pi-mutex, but the mutex is currently owned by a non-real-time thread. 24 25The `rtapp` monitor detects these patterns. It aids developers to identify 26reasons for unexpected latency with real-time applications. It is a container of 27multiple sub-monitors described in the following sections. 28 29Monitor pagefault 30+++++++++++++++++ 31 32The `pagefault` monitor reports real-time tasks raising page faults. Its 33specification is:: 34 35 RULE = always (RT imply not PAGEFAULT) 36 37To fix warnings reported by this monitor, `mlockall()` or `mlock()` can be used 38to ensure physical backing for memory. 39 40This monitor may have false negatives because the pages used by the real-time 41threads may just happen to be directly available during testing. To minimize 42this, the system can be put under memory pressure (e.g. invoking the OOM killer 43using a program that does `ptr = malloc(SIZE_OF_RAM); memset(ptr, 0, 44SIZE_OF_RAM);`) so that the kernel executes aggressive strategies to recycle as 45much physical memory as possible. 46 47Monitor sleep 48+++++++++++++ 49 50The `sleep` monitor reports real-time threads sleeping in a manner that may 51cause undesirable latency. Real-time applications should only put a real-time 52thread to sleep for one of the following reasons: 53 54 - Cyclic work: real-time thread sleeps waiting for the next 55 cycle. For this case, only the `clock_nanosleep` syscall should be 56 used with `TIMER_ABSTIME` (to avoid time drift). Additionally, 57 `CLOCK_REALTIME` should not be used (to avoid the clock being 58 changed). No other method is safe for real-time. For example, 59 threads waiting for timerfd can be woken by softirq which provides 60 no real-time guarantee. 61 - Real-time thread waiting for something to happen (e.g. another thread 62 releasing shared resources, or a completion signal from another thread). In 63 this case, only futexes (FUTEX_LOCK_PI, FUTEX_LOCK_PI2 or one of 64 FUTEX_WAIT_*) should be used. Applications usually do not use futexes 65 directly, but use PI mutexes and PI condition variables which are built on 66 top of futexes. Be aware that the C library might not implement conditional 67 variables as safe for real-time. As an alternative, the librtpi library 68 exists to provide a conditional variable implementation that is correct for 69 real-time applications in Linux. 70 - Real-time thread waiting for events using `epoll_wait`, which is a 71 real-time-safe syscall for sleeping as it uses PI-aware locking. 72 73Beside the reason for sleeping, the eventual waker should also be 74real-time-safe. Namely, one of: 75 76 - An equal-or-higher-priority thread 77 - Hard interrupt handler 78 - Non-maskable interrupt handler 79 80This monitor's warning usually means one of the following: 81 82 - Real-time thread is blocked by a non-real-time thread (e.g. due to 83 contention on a mutex without priority inheritance). This is priority 84 inversion. 85 - Time-critical work waits for something which is not safe for real-time (e.g. 86 timerfd). 87 - The work executed by the real-time thread does not need to run at real-time 88 priority at all. This is not a problem for the real-time thread itself, but 89 it is potentially taking the CPU away from other important real-time work. 90 91Application developers may purposely choose to have their real-time application 92sleep in a way that is not safe for real-time. It is debatable whether that is a 93problem. Application developers must analyze the warnings to make a proper 94assessment. 95 96The monitor's specification is:: 97 98 RULE = always ((RT and SLEEP and USER_THREAD) imply (RT_FRIENDLY_SLEEP or ALLOWLIST)) 99 100 RT_FRIENDLY_SLEEP = RT_VALID_SLEEP_REASON 101 and ((not SCHEDULE_IN) until RT_FRIENDLY_WAKE) 102 103 RT_VALID_SLEEP_REASON = FUTEX_WAIT 104 or RT_FRIENDLY_NANOSLEEP 105 or EPOLL_WAIT 106 107 RT_FRIENDLY_NANOSLEEP = CLOCK_NANOSLEEP 108 and NANOSLEEP_TIMER_ABSTIME 109 and not NANOSLEEP_CLOCK_REALTIME 110 111 RT_FRIENDLY_WAKE = WOKEN_BY_EQUAL_OR_HIGHER_PRIO 112 or WOKEN_BY_HARDIRQ 113 or WOKEN_BY_NMI 114 or ABORT_SLEEP 115 116 ALLOWLIST = BLOCK_ON_RT_MUTEX 117 or FUTEX_LOCK_PI 118 119`ABORT_SLEEP` represents a task restoring its state to `TASK_RUNNING` before 120entering the scheduler. In this case, the task does not actually block, so the 121task is back to runnable without any wakeup sequence unsafe for real-time. 122 123Beside the scenarios described above, this specification also defines an allow list 124to handle some special cases: 125 126 - `BLOCK_ON_RT_MUTEX` is included in the allowlist due to its implementation. 127 In the release path of rt_mutex, a boosted task is de-boosted before waking 128 the rt_mutex's waiter. Consequently, the monitor may see a real-time-unsafe 129 wakeup (e.g. non-real-time task waking real-time task). This is actually 130 real-time-safe because preemption is disabled for the duration. 131 - `FUTEX_LOCK_PI` is included in the allowlist for the same reason as 132 `BLOCK_ON_RT_MUTEX`. 133 134Monitor wakeup 135++++++++++++++ 136 137The `wakeup` monitor reports real-time threads being woken by lower-priority threads, 138which is a hint of priority inversion. Its specification is:: 139 140 RULE = always (((RT and USER_THREAD) imply 141 (not (WOKEN_BY_LOWER_PRIO or WOKEN_BY_SOFTIRQ)) or ALLOWLIST)) 142 143 ALLOWLIST = BLOCK_ON_RT_MUTEX 144 or FUTEX_LOCK_PI 145 146The `sleep` monitor already reports this type of problem. The difference is the 147context in which the problem is reported. While the `sleep` monitor reports the problem 148in the context of the wakee, this `wakeup` monitor reports the problem in the context of 149the waker. This monitor complement the `sleep` monitor, giving user better 150understanding of the issue. For instance, to debug a lower-priority task waking a 151higher-priority task scenario, user can enable both `wakeup` monitor and `sleep` 152monitor to get the stack traces of both tasks. 153