1Real-time application monitors 2============================== 3 4- Name: rtapp 5- Type: container for multiple monitors 6- Author: Nam Cao <namcao@linutronix.de> 7 8Description 9----------- 10 11Real-time applications may have design flaws such that they experience 12unexpected latency and fail to meet their time requirements. Often, these flaws 13follow a few patterns: 14 15 - Page faults: A real-time thread may access memory that does not have a 16 mapped physical backing or must first be copied (such as for copy-on-write). 17 Thus a page fault is raised and the kernel must first perform the expensive 18 action. This causes significant delays to the real-time thread 19 - Priority inversion: A real-time thread blocks waiting for a lower-priority 20 thread. This causes the real-time thread to effectively take on the 21 scheduling priority of the lower-priority thread. For example, the real-time 22 thread needs to access a shared resource that is protected by a 23 non-pi-mutex, but the mutex is currently owned by a non-real-time thread. 24 25The `rtapp` monitor detects these patterns. It aids developers to identify 26reasons for unexpected latency with real-time applications. It is a container of 27multiple sub-monitors described in the following sections. 28 29Monitor pagefault 30+++++++++++++++++ 31 32The `pagefault` monitor reports real-time tasks raising page faults. Its 33specification is:: 34 35 RULE = always (RT imply not PAGEFAULT) 36 37To fix warnings reported by this monitor, `mlockall()` or `mlock()` can be used 38to ensure physical backing for memory. 39 40This monitor may have false negatives because the pages used by the real-time 41threads may just happen to be directly available during testing. To minimize 42this, the system can be put under memory pressure (e.g. invoking the OOM killer 43using a program that does `ptr = malloc(SIZE_OF_RAM); memset(ptr, 0, 44SIZE_OF_RAM);`) so that the kernel executes aggressive strategies to recycle as 45much physical memory as possible. 46 47Monitor sleep 48+++++++++++++ 49 50The `sleep` monitor reports real-time threads sleeping in a manner that may 51cause undesirable latency. Real-time applications should only put a real-time 52thread to sleep for one of the following reasons: 53 54 - Cyclic work: real-time thread sleeps waiting for the next 55 cycle. For this case, only the `clock_nanosleep` syscall should be 56 used with `TIMER_ABSTIME` (to avoid time drift). Additionally, 57 `CLOCK_REALTIME` should not be used (to avoid the clock being 58 changed). No other method is safe for real-time. For example, 59 threads waiting for timerfd can be woken by softirq which provides 60 no real-time guarantee. 61 - Real-time thread waiting for something to happen (e.g. another thread 62 releasing shared resources, or a completion signal from another thread). In 63 this case, only futexes (FUTEX_LOCK_PI, FUTEX_LOCK_PI2 or one of 64 FUTEX_WAIT_*) should be used. Applications usually do not use futexes 65 directly, but use PI mutexes and PI condition variables which are built on 66 top of futexes. Be aware that the C library might not implement conditional 67 variables as safe for real-time. As an alternative, the librtpi library 68 exists to provide a conditional variable implementation that is correct for 69 real-time applications in Linux. 70 71Beside the reason for sleeping, the eventual waker should also be 72real-time-safe. Namely, one of: 73 74 - An equal-or-higher-priority thread 75 - Hard interrupt handler 76 - Non-maskable interrupt handler 77 78This monitor's warning usually means one of the following: 79 80 - Real-time thread is blocked by a non-real-time thread (e.g. due to 81 contention on a mutex without priority inheritance). This is priority 82 inversion. 83 - Time-critical work waits for something which is not safe for real-time (e.g. 84 timerfd). 85 - The work executed by the real-time thread does not need to run at real-time 86 priority at all. This is not a problem for the real-time thread itself, but 87 it is potentially taking the CPU away from other important real-time work. 88 89Application developers may purposely choose to have their real-time application 90sleep in a way that is not safe for real-time. It is debatable whether that is a 91problem. Application developers must analyze the warnings to make a proper 92assessment. 93 94The monitor's specification is:: 95 96 RULE = always ((RT and SLEEP and USER_THREAD) imply (RT_FRIENDLY_SLEEP or ALLOWLIST)) 97 98 RT_FRIENDLY_SLEEP = RT_VALID_SLEEP_REASON 99 and ((not SCHEDULE_IN) until RT_FRIENDLY_WAKE) 100 101 RT_VALID_SLEEP_REASON = FUTEX_WAIT 102 or RT_FRIENDLY_NANOSLEEP 103 or EPOLL_WAIT 104 105 RT_FRIENDLY_NANOSLEEP = CLOCK_NANOSLEEP 106 and NANOSLEEP_TIMER_ABSTIME 107 and not NANOSLEEP_CLOCK_REALTIME 108 109 RT_FRIENDLY_WAKE = WOKEN_BY_EQUAL_OR_HIGHER_PRIO 110 or WOKEN_BY_HARDIRQ 111 or WOKEN_BY_NMI 112 or ABORT_SLEEP 113 114 ALLOWLIST = BLOCK_ON_RT_MUTEX 115 or FUTEX_LOCK_PI 116 117Beside the scenarios described above, this specification also defines an allow list 118to handle some special cases: 119 120 - `BLOCK_ON_RT_MUTEX` is included in the allowlist due to its implementation. 121 In the release path of rt_mutex, a boosted task is de-boosted before waking 122 the rt_mutex's waiter. Consequently, the monitor may see a real-time-unsafe 123 wakeup (e.g. non-real-time task waking real-time task). This is actually 124 real-time-safe because preemption is disabled for the duration. 125 - `FUTEX_LOCK_PI` is included in the allowlist for the same reason as 126 `BLOCK_ON_RT_MUTEX`. 127 128Monitor wakeup 129++++++++++++++ 130 131The `wakeup` monitor reports real-time threads being woken by lower-priority threads, 132which is a hint of priority inversion. Its specification is:: 133 134 RULE = always (((RT and USER_THREAD) imply 135 (not (WOKEN_BY_LOWER_PRIO or WOKEN_BY_SOFTIRQ)) or ALLOWLIST)) 136 137 ALLOWLIST = BLOCK_ON_RT_MUTEX 138 or FUTEX_LOCK_PI 139 140The `sleep` monitor already reports this type of problem. The difference is the 141context in which the problem is reported. While the `sleep` monitor reports the problem 142in the context of the wakee, this `wakeup` monitor reports the problem in the context of 143the waker. This monitor complement the `sleep` monitor, giving user better 144understanding of the issue. For instance, to debug a lower-priority task waking a 145higher-priority task scenario, user can enable both `wakeup` monitor and `sleep` 146monitor to get the stack traces of both tasks. 147