1.. SPDX-License-Identifier: GPL-2.0 2 3==================== 4Considering hardware 5==================== 6 7:Author: Sebastian Andrzej Siewior <bigeasy@linutronix.de> 8 9The way a workload is handled can be influenced by the hardware it runs on. 10Key components include the CPU, memory, and the buses that connect them. 11These resources are shared among all applications on the system. 12As a result, heavy utilization of one resource by a single application 13can affect the deterministic handling of workloads in other applications. 14 15Below is a brief overview. 16 17System memory and cache 18----------------------- 19 20Main memory and the associated caches are the most common shared resources among 21tasks in a system. One task can dominate the available caches, forcing another 22task to wait until a cache line is written back to main memory before it can 23proceed. The impact of this contention varies based on write patterns and the 24size of the caches available. Larger caches may reduce stalls because more lines 25can be buffered before being written back. Conversely, certain write patterns 26may trigger the cache controller to flush many lines at once, causing 27applications to stall until the operation completes. 28 29This issue can be partly mitigated if applications do not share the same CPU 30cache. The kernel is aware of the cache topology and exports this information to 31user space. Tools such as **lstopo** from the Portable Hardware Locality (hwloc) 32project (https://www.open-mpi.org/projects/hwloc/) can visualize the hierarchy. 33 34Avoiding shared L2 or L3 caches is not always possible. Even when cache sharing 35is minimized, bottlenecks can still occur when accessing system memory. Memory 36is used not only by the CPU but also by peripheral devices via DMA, such as 37graphics cards or network adapters. 38 39In some cases, cache and memory bottlenecks can be controlled if the hardware 40provides the necessary support. On x86 systems, Intel offers Cache Allocation 41Technology (CAT), which enables cache partitioning among applications and 42provides control over the interconnect. AMD provides similar functionality under 43Platform Quality of Service (PQoS). On Arm64, the equivalent is Memory 44System Resource Partitioning and Monitoring (MPAM). 45 46These features can be configured through the Linux Resource Control interface. 47For details, see Documentation/filesystems/resctrl.rst. 48 49The perf tool can be used to monitor cache behavior. It can analyze 50cache misses of an application and compare how they change under 51different workloads on a neighboring CPU. Even more powerful, the perf 52c2c tool can help identify cache-to-cache issues, where multiple CPU 53cores repeatedly access and modify data on the same cache line. 54 55Hardware buses 56-------------- 57 58Real-time systems often need to access hardware directly to perform their work. 59Any latency in this process is undesirable, as it can affect the outcome of the 60task. For example, on an I/O bus, a changed output may not become immediately 61visible but instead appear with variable delay depending on the latency of the 62bus used for communication. 63 64A bus such as PCI is relatively simple because register accesses are routed 65directly to the connected device. In the worst case, a read operation stalls the 66CPU until the device responds. 67 68A bus such as USB is more complex, involving multiple layers. A register read 69or write is wrapped in a USB Request Block (URB), which is then sent by the 70USB host controller to the device. Timing and latency are influenced by the 71underlying USB bus. Requests cannot be sent immediately; they must align with 72the next frame boundary according to the endpoint type and the host controller's 73scheduling rules. This can introduce delays and additional latency. For example, 74a network device connected via USB may still deliver sufficient throughput, but 75the added latency when sending or receiving packets may fail to meet the 76requirements of certain real-time use cases. 77 78Additional restrictions on bus latency can arise from power management. For 79instance, PCIe with Active State Power Management (ASPM) enabled can suspend 80the link between the device and the host. While this behavior is beneficial for 81power savings, it delays device access and adds latency to responses. This issue 82is not limited to PCIe; internal buses within a System-on-Chip (SoC) can also be 83affected by power management mechanisms. 84 85Virtualization 86-------------- 87 88In a virtualized environment such as KVM, each guest CPU is represented as a 89thread on the host. If such a thread runs with real-time priority, the system 90should be tested to confirm it can sustain this behavior over extended periods. 91Because of its priority, the thread will not be preempted by lower-priority 92threads (such as SCHED_OTHER), which may then receive no CPU time. This can 93cause problems if a lower-priority thread is pinned to a CPU already occupied by 94a real-time task and unable to make progress. Even if a CPU has been isolated, 95the system may still (accidentally) start a per‑CPU thread on that CPU. 96Ensuring that a guest CPU goes idle is difficult, as it requires avoiding both 97task scheduling and interrupt handling. Furthermore, if the guest CPU does go 98idle but the guest system is booted with the option **idle=poll**, the guest 99CPU will never enter an idle state and will instead spin until an event 100arrives. 101 102Device handling introduces additional considerations. Emulated PCI devices or 103VirtIO devices require a counterpart on the host to complete requests. This 104adds latency because the host must intercept and either process the request 105directly or schedule a thread for its completion. These delays can be avoided if 106the required PCI device is passed directly through to the guest. Some devices, 107such as networking or storage controllers, support the PCIe SR-IOV feature. 108SR-IOV allows a single PCIe device to be divided into multiple virtual functions, 109which can then be assigned to different guests. 110 111Networking 112---------- 113 114For low-latency networking, the full networking stack may be undesirable, as it 115can introduce additional sources of delay. In this context, XDP can be used 116as a shortcut to bypass much of the stack while still relying on the kernel's 117network driver. 118 119The requirements are that the network driver must support XDP- preferably using 120an "skb pool" and that the application must use an XDP socket. Additional 121configuration may involve BPF filters, tuning networking queues, or configuring 122qdiscs for time-based transmission. These techniques are often 123applied in Time-Sensitive Networking (TSN) environments. 124 125Documenting all required steps exceeds the scope of this text. For detailed 126guidance, see the TSN documentation at https://tsn.readthedocs.io. 127 128Another useful resource is the Linux Real-Time Communication Testbench 129https://github.com/Linutronix/RTC-Testbench. 130The goal of this project is to validate real-time network communication. It can 131be thought of as a "cyclictest" for networking and also serves as a starting 132point for application development. 133 134Firmware 135-------- 136 137The firmware often plays a significant role in system operation because it can 138perform tasks that the kernel cannot directly access, and in some cases it can 139even preempt or intercept the kernel. 140 141A common example of firmware assisting the kernel is when it provides a generic 142interface to a resource. Instead of accessing an RTC chip through an I2C host 143controller, the kernel may query the firmware for the current time, and the 144firmware then accesses the RTC behind the scenes. 145 146Firmware can also intercept kernel execution by providing services that 147temporarily take control of the system. One example is memory scrubbing, where 148the firmware periodically pauses the kernel, reads back portions of system 149memory, and then returns control. During this time, the kernel is effectively 150interrupted. 151In contrast, some systems provide hardware-based memory scrubbing, which 152operates independently of firmware or software. See 153Documentation/edac/scrub.rst for details. 154 155If the kernel is intercepted for longer periods then these periods can be made 156visible with the hardware latency detector. See 157Documentation/trace/hwlat_detector.rst. 158 159The kernel can also be intercepted in response to specific events, such as 160overheating. In this case, the firmware may throttle the CPU or shut it down 161immediately to prevent hardware damage. 162 163Unless the firmware is well documented, it should be thoroughly tested to 164uncover any unexpected behaviour. 165 166EFI 167~~~~ 168 169EFI provides runtime services that act as a communication interface between the 170firmware and the operating system. One such service is reading and writing EFI 171variables, which are used, for example, to determine the boot source. 172 173Invoking a runtime service may require the architecture to disable kernel 174preemption or interrupts during the call. This means the duration of a service 175invocation directly affects the system’s observable latency. There is also 176nothing that prevents a service call from disabling interrupts internally while 177it runs. 178 179For these reasons, EFI runtime services are disabled by default on a PREEMPT_RT 180kernel. They can still be enabled at boot time or via a Kconfig option if 181required. 182The native EFI runtime service implementation (where both the EFI service and 183the kernel are either 32-bit or 64-bit executables) uses a wrapper mechanism 184that invokes the service through a dedicated workqueue. This workqueue is named 185efi_runtime, and it can be restricted to a housekeeping CPU using the 186``/sys/devices/virtual/workqueue/efi_runtime/cpumask`` sysfs file. Assigning it 187to a housekeeping CPU ensures that potentially long service invocations do not 188impact the real-time workload which is restricted to other CPUs. 189 190It must also be verified that the runtime services behave as expected. Some 191implementations on the x86 architecture pause all other CPUs while one CPU 192performs the service call. In such cases, the interruption affects all CPUs, 193and restricting the workqueue to a single CPU provides no benefit. 194 195OP-TEE (ARM) 196~~~~~~~~~~~~ 197 198Execution flows from the normal world (Linux) into the secure world (OP-TEE) 199through the secure monitor at EL3. The transition is initiated by the `smc` 200(Secure Monitor Call) opcode or the `hvc` (Hypervisor Call) opcode together 201with a function identifier. The calling convention defines two types of calls: 202**yielding calls** and **fast calls**: 203 204- A **yielding call** unmasks interrupts before handling the requested service, 205 allowing normal world interrupts to occur. 206- A **fast call** handles the requested service atomically, without allowing 207 interrupts from either the normal world or the secure world. 208 209In addition, the secure world (EL3 and OP-TEE) can receive interrupts routed to 210the secure world. While a secure world interrupt is being serviced, 211normal world interrupts are masked and cannot preempt the operation. 212 213The transition from normal world to secure monitor to OP-TEE and back introduces 214additional latency due to world switching and context save/restore. This 215overhead is typically a few microseconds and usually remains within the noise 216floor. 217 218It is worth noting that the normal world cannot mask secure interrupts, while 219the secure world can mask normal-world interrupts during execution. How OP-TEE 220affects real-time workloads depends on whether secure interrupts are enabled 221and which OP-TEE services are invoked. 222 223A practical concern is any fast call that runs longer than expected, for 224example a function that occasionally performs a long-running cryptographic 225computation. Another example that may block in an unexpected way are OP-TEE 226drivers that issue RPC requests. An OP-TEE service in the secure world (RPMB 227for instance) may need to issue a request back to the normal world (the Linux 228driver) in order to complete the operation. While Linux remains preemptible, 229the thread that issued the request stays blocked until the RPC completes and 230the secure function call returns. 231 232The TF-A project provides documentation on interrupt management: 233https://trustedfirmware-a.readthedocs.io/en/latest/design/interrupt-framework-design.html#interrupt-management-framework 234 235The OP-TEE project provides documentation on how interrupts are handled: 236https://optee.readthedocs.io/en/latest/architecture/core.html#interrupt-handling 237