1.. SPDX-License-Identifier: GPL-2.0 2 3================== 4Kernel Page Tables 5================== 6 7Introduction 8============ 9 10The kernel page tables are created early during boot and, unlike the page 11tables of user processes, most of them remain static throughout the system 12lifetime. 13 14Every architecture has a direct map (also called linear map) that maps the 15physical memory at a fixed offset, so that a physical address can be 16translated to a kernel virtual address with simple arithmetic. On most 17architectures the direct map is a part of the kernel page tables, with a few 18exceptions described in the `Direct map`_ section below. 19 20On 32-bit systems with high memory the direct map covers only a part of the 21physical memory, see Documentation/mm/highmem.rst. 22 23The vmalloc area, present on every architecture with an MMU, is used for 24allocations of virtually contiguous memory whose backing pages are not 25necessarily physically contiguous, and for mapping of the device memory. Its 26page tables are created and torn down at runtime, see 27Documentation/mm/vmalloc.rst. 28 29Architectures that use the `SPARSEMEM_VMEMMAP` memory model reserve a range of 30kernel address space for the memory map, so that `struct page` objects appear 31as a virtually contiguous array indexed by the page frame number, see 32Documentation/mm/memory-model.rst. 33 34Besides these, the kernel image may be mapped in a dedicated part of the 35kernel address space rather than accessed through the direct map. In that 36case its mapping is an alias of the direct map of the physical memory the 37image occupies. 38 39The rest of the kernel address space is architecture specific. For instance, 40x86 has a region for the EFI runtime services and s390 has a region for the 41code that has to run in the 31-bit addressing mode. 42 43Direct map 44========== 45 46On most architectures the direct map is an ordinary part of the kernel page 47tables. It is created early during boot with the largest pages the hardware 48and the kernel configuration allow. 49 50Several architectures are different. 51 52MIPS 53---- 54 55MIPS does not map the physical memory with page tables at all. Instead, a 56part of the kernel virtual address space is a window into the physical 57address space: the hardware translates the addresses that fall into that 58window by a fixed transformation of the address bits, without walking the 59page tables and without using the TLB. The memory attributes, such as 60cacheability and the privilege level required to access the memory, are a 61property of the window rather than of an individual page. 62 63There are no page table entries describing the direct map, so its properties 64cannot be changed for an individual page. On 32-bit systems the window covers 65only 512 MiB of the physical memory, so everything above that is high memory. 66 67LoongArch 68--------- 69 70Like MIPS, LoongArch maps the physical memory with a hardware window. The 71window covers 256 TiB of the physical address space on 64-bit and 512 MiB on 7232-bit systems, and everything above that is high memory. 73 74The window occupies the lower part of the kernel address space. The upper 75part, which includes the vmalloc area, is mapped with kernel page tables. 76 77PowerPC with the hash MMU 78------------------------- 79 80On 64-bit PowerPC systems with the hash MMU the direct map does not exist in 81the Linux page tables. It is installed into the hardware hash page table early 82during boot. 83 84Modifying such mappings requires updating the hash page table directly, and 85the hash MMU code implements this only for the kernel image permissions, 86`debug_pagealloc` and KFENCE. 87 88With the radix MMU the direct map is a part of the ordinary kernel page 89tables. 90 91Modifying the kernel page tables 92================================ 93 94Except for the vmalloc area, the kernel page tables are mostly static. Still, 95there are cases when the permissions of existing kernel mappings have to be 96updated, for instance when a module is loaded and its text becomes read-only 97and executable, or when a page is temporarily removed from the direct map to 98reduce its exposure. 99 100There are two families of functions for this, both declared in 101`include/linux/set_memory.h`: 102 103* `set_memory_*()` change permissions of an arbitrary kernel mapping. They 104 take a kernel virtual address and the number of pages. 105 106* `set_direct_map_*()` change permissions of the direct mapping for the range 107 of page frames starting at the page represented by a `struct page`. They take 108 a `struct page` pointer and the number of pages. 109 110Architectures that implement `set_memory()` select `CONFIG_ARCH_HAS_SET_MEMORY` 111 112Architectures that implement `set_direct_map()` select 113`CONFIG_ARCH_HAS_SET_DIRECT_MAP`. 114 115Common semantics 116---------------- 117 118Ranges 119~~~~~~ 120 121The `set_memory()` functions expect a range described by a start address and a 122number of pages. The address must be page aligned and the entire range must be 123covered by page table entries the architecture knows how to update. 124 125The entries may be marked as not present, set_memory_p() and set_memory_valid() 126exist exactly to bring such a mapping back. 127 128When a range does not qualify, an architecture will usually say so by returning 129an error and sometimes by a WARN()ing as well, unless it prefers to keep it to 130itself and return success, see `Architecture specific differences`_. 131 132The `set_direct_map()` functions expect a range described by the first 133`struct page` and a number of pages, and they update the direct map starting 134at that page. 135 136The pages that follow the first one are updated regardless of what they are, so 137the caller has to make sure that the range does not extend beyond the memory it 138owns. 139 140Some architectures cannot split a large mapping, and they reject a range that 141is a part of one, see `Architecture specific differences`_. 142 143Calling `set_memory()` with the number of pages set to zero is a no-op that 144returns success, except on arm64, where doing nothing to the wrong address is 145still an error. 146 147Aliases 148~~~~~~~ 149 150A physical page may be mapped several times, for instance in the direct map 151and in the vmalloc area, and the permissions of these mappings may differ. 152 153Whether the direct map alias is updated by a `set_memory()` call, and 154which permission bits make it there, is entirely up to the architecture, and 155there is not much agreement between them, see 156`Architecture specific differences`_. 157 158Relying on that is a gamble; code that needs the direct map alias to change 159should say so with the `set_direct_map()` APIs. 160 161Failures and partial updates 162~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 163 164Both families of functions return 0 on success and a negative error code on 165failure. The most common failures are `-EINVAL` for a range that cannot be 166handled and `-ENOMEM` when splitting a large mapping fails to allocate a page 167table. 168 169Some of the range checks happen upfront, so their failure leaves the page 170tables unchanged. 171 172**The update is not atomic and there is no rollback.** 173 174The architecture implementations walk the range and update the page tables as 175they go, and they stop at the first entry that cannot be updated. When an error 176is returned, an arbitrary prefix of the range may have been updated already, 177and the same is true for the direct map alias when the architecture updates it. 178 179For example, when a range spans two large mappings and splitting the second 180one fails because there is no memory for a page table, the first one is 181already split and updated by the time the error is returned. 182 183None of the APIs inform the caller where in the range they failed, so reverting 184such a partial update is possible in principle but unreliable in practice. 185 186The caller may try to restore the original permissions over the entire range, 187but that revert goes through the very code that has just failed, which does 188not inspire much confidence. 189 190The callers should therefore be prepared to give up on the memory in question: 191leak it or panic, but never return it to the allocator before the permissions 192are restored and never assume that the requested permissions are in effect. 193Neither option is appealing, but both beat handing out a page whose 194permissions nobody knows. 195 196TLB flushing 197~~~~~~~~~~~~ 198 199The `set_memory()` functions flush the TLB for the affected range before 200they return, so that the new permissions are in effect for every CPU. 201 202The `set_direct_map()` functions have `_noflush` in their names because when 203they were first introduced on x86, the intention was that the TLB flushing 204could be optimized by letting the caller handle it. 205 206For example, vfree() batches the TLB flushes for the areas allocated with 207`VM_FLUSH_RESET_PERMS`, folding the flush of the direct map into the flush it 208has to do for the vmalloc mapping anyway. 209 210Some architectures flush the TLB in the `_noflush` functions anyway, so the 211name is best read as a suggestion. It does not make the flush by the caller 212unnecessary, it only makes it more expensive. 213 214A caller that changes the permissions to more restrictive ones must flush the 215TLB itself. 216 217Context 218~~~~~~~ 219 220Architectures use different locking mechanisms to synchronize kernel page table 221updates, and both families of functions may sleep, for instance when they 222allocate memory to split a large mapping. 223 224The caller cannot presume it is safe to call these APIs from an atomic context. 225 226The `set_direct_map()` functions must not be called for high memory pages, 227which have no direct map alias to update. 228 229Unimplemented APIs silently succeed 230~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 231 232When an architecture does not implement these APIs, the generic stubs in 233`include/linux/set_memory.h` return 0, that is, they report success for work 234they have no intention of doing. 235 236The same happens inside several architecture implementations. The arm64 237`set_direct_map()` functions return 0 when can_set_direct_map() is false, and 238the LoongArch `set_memory()` functions return 0 for the addresses in its 239windowed direct mapping, which is not backed by page tables at all. 240 241Returning success without actually updating the page tables is a deliberate 242trade-off that keeps the callers free of `#ifdef`\ s, but they have to realize: 243 244**a return value of 0 does not imply that the permissions were actually 245changed.** 246 247For best-effort hardening that is good enough. When correctness or security 248depends on the permissions, the caller has to make sure the architecture 249really implements what it needs. For instance, secretmem depends on 250`CONFIG_ARCH_HAS_SET_DIRECT_MAP` and calls can_set_direct_map() at runtime. 251 252Architecture specific differences 253================================= 254 255The APIs are implemented by seven architectures and, beyond the common 256semantics described above, their behaviour differs in several respects. 257 258Which of the APIs are implemented: 259 260========= ===================== ========================= 261Arch `ARCH_HAS_SET_MEMORY` `ARCH_HAS_SET_DIRECT_MAP` 262========= ===================== ========================= 263arm yes no 264arm64 yes yes 265loongarch yes yes 266powerpc yes no 267riscv yes (MMU only) yes (MMU only) 268s390 yes yes 269x86 yes yes 270========= ===================== ========================= 271 272Only set_memory_ro(), set_memory_rw(), set_memory_x() and set_memory_nx() are 273available everywhere, and even these are not universal: some architectures 274restrict the ranges that can be modified, for instance arm64 rejects direct map 275addresses and accepts only addresses in vmalloc space. The architectures that 276may run on hardware without an execute permission bit, like x86 and s390, 277silently skip the update of the executable bit. 278 279set_memory_rox() has a generic implementation that calls set_memory_ro() and 280set_memory_x() in turn; PowerPC, s390 and x86 override it with a single-pass 281version. 282 283Several architectures define additional APIs. Some of those may have identical 284semantics but different names. For example, making a mapping present or not 285present is spelled differently: set_memory_p() and set_memory_np() on x86 and 286PowerPC, set_memory_valid() on arm and arm64. 287 288The direct map and the kernel image are normally mapped with the largest 289possible pages, and changing the permissions of a single page inside such a 290mapping requires splitting it, which not every architecture can do. 291 292arm 293--- 294 295* Does not implement `set_direct_map()`. 296* Provides set_memory_valid(). 297* set_memory_ro(), set_memory_rw(), set_memory_x() and set_memory_nx() accept 298 only vmalloc and module addresses. 299* set_memory_valid() accepts any address. 300* Does not update mapping aliases. 301 302arm64 303----- 304 305* Provides set_memory_valid(). 306* Provides the memory encryption helpers, which are effective only when the 307 kernel runs as a confidential guest. 308* set_memory_ro(), set_memory_rw(), set_memory_x() and set_memory_nx() accept 309 only vmalloc and module addresses: 310 311 - the range must fit in the VM area that contains its start 312 - the VM area must have `VM_ALLOC` set and `VM_ALLOW_HUGE_VMAP` clear 313 314* set_memory_valid() accepts any address. 315* The encryption helpers accept only direct map addresses. 316* The `set_memory()` functions propagate the read-only and the read-write 317 changes to the direct map alias when `rodata=full` is in effect. 318* Splits leaf mappings before the update on the hardware that supports it. 319 The split itself may be partial when it fails midway, but the permissions are 320 left untouched in that case. 321 Without support for splitting large mappings, a range that covers a leaf 322 entry only partially fails with a WARN()ing and `-EINVAL`. 323 If a range spans one or more full leaf entries and a partial leaf entry, the 324 permissions of the full leaf entries are updated before the failure. 325* The `set_direct_map()` functions return 0 without doing anything when the 326 direct map cannot be modified, see can_set_direct_map(). 327* Skips the TLB flush in `set_memory()` when the update only turns an invalid 328 mapping into a valid one. 329* Does not flush TLB in `set_direct_map()`. 330 331LoongArch 332--------- 333 334* Accepts only the addresses above the hardware window and silently returns 335 success for the rest, see `Direct map`_. 336* Does not update mapping aliases. 337* Does not split anything: a leaf entry is updated as a whole, which changes 338 the permissions of the entire large mapping. 339* Flushes the TLB in `set_direct_map()`. 340 341PowerPC 342------- 343 344* Does not implement `set_direct_map()`. 345* Provides set_memory_np() and set_memory_p(). 346* Rejects huge vmalloc mappings. 347* With the hash MMU on 64-bit systems accepts nothing but the vmalloc and the 348 I/O regions. 349* With the radix MMU accepts direct map addresses, but still cannot split a 350 large mapping. 351* Does not update mapping aliases. 352 353riscv 354----- 355 356* Implements both APIs only when the MMU is enabled. 357* Provides set_memory_rw_nx(). 358* The `set_memory()` functions accept any mapped kernel address, including the 359 direct map, but a vmalloc range must have the `pages` array of its VM area 360 populated, which rules out vmap() and ioremap() mappings. 361* On 64-bit systems the `set_memory()` functions update the direct map alias of 362 a vmalloc range, including the executable bit. 363* Does not split vmalloc ranges: a leaf entry is updated as a whole, which 364 changes the permissions of the entire large mapping. 365* Splits the direct map on 64-bit systems. 366* Flushes the TLB in `set_direct_map()`. 367 368s390 369---- 370 371* Provides set_memory_4k(), set_memory_rwnx() and the 372 `__set_memory_*(start, end)` variants that take a range rather than a page 373 count. 374* The `set_memory()` functions accept any mapped kernel address, including the 375 direct map. 376* Skips the update of the executable bit when the hardware has no support for 377 it. 378* The `set_memory()` functions propagate only the read-only and read-write 379 changes to the direct map alias of a `VM_ALLOC` area, and deliberately not 380 the executable bit. 381* Splits leaf PUD and PMD entries when the range is not aligned to them or when 382 set_memory_4k() is requested. 383* Updates the page table entries with instructions that invalidate the 384 corresponding TLB entries, so no separate flush is needed anywhere. 385 386x86 387--- 388 389* Provides the largest set of operations on top of the common ones: 390 391 - the cache attribute helpers: set_memory_uc(), set_memory_wc(), 392 set_memory_wb() 393 - presence control: set_memory_np() and set_memory_p() 394 - set_memory_4k() 395 - set_memory_global() and set_memory_nonglobal() 396 - the array variants that operate on `struct page` arrays or arrays of 397 virtual addresses 398 - memory encryption: set_memory_encrypted() and set_memory_decrypted() 399 400* The `set_memory()` functions accept any mapped kernel address, including the 401 direct map, and silently succeed for the unmapped holes inside it. 402* Does nothing in set_memory_x() and set_memory_nx() when the CPU has no 403 execute permission bit. 404* The `set_memory()` functions apply the change to the direct map alias and, 405 for the kernel image, to the high kernel mapping. The NX bit is never 406 propagated, so that the direct map stays non-executable. 407* Splits large mappings on demand and can collapse them back when the 408 permissions become uniform again. 409* Does not flush the TLB in `set_direct_map()`, but splitting a large 410 mapping flushes it anyway. 411