1.. _admin_guide_transhuge: 2 3============================ 4Transparent Hugepage Support 5============================ 6 7Objective 8========= 9 10Performance critical computing applications dealing with large memory 11working sets are already running on top of libhugetlbfs and in turn 12hugetlbfs. Transparent HugePage Support (THP) is an alternative mean of 13using huge pages for the backing of virtual memory with huge pages 14that supports the automatic promotion and demotion of page sizes and 15without the shortcomings of hugetlbfs. 16 17Currently THP only works for anonymous memory mappings and tmpfs/shmem. 18But in the future it can expand to other filesystems. 19 20.. note:: 21 in the examples below we presume that the basic page size is 4K and 22 the huge page size is 2M, although the actual numbers may vary 23 depending on the CPU architecture. 24 25The reason applications are running faster is because of two 26factors. The first factor is almost completely irrelevant and it's not 27of significant interest because it'll also have the downside of 28requiring larger clear-page copy-page in page faults which is a 29potentially negative effect. The first factor consists in taking a 30single page fault for each 2M virtual region touched by userland (so 31reducing the enter/exit kernel frequency by a 512 times factor). This 32only matters the first time the memory is accessed for the lifetime of 33a memory mapping. The second long lasting and much more important 34factor will affect all subsequent accesses to the memory for the whole 35runtime of the application. The second factor consist of two 36components: 37 381) the TLB miss will run faster (especially with virtualization using 39 nested pagetables but almost always also on bare metal without 40 virtualization) 41 422) a single TLB entry will be mapping a much larger amount of virtual 43 memory in turn reducing the number of TLB misses. With 44 virtualization and nested pagetables the TLB can be mapped of 45 larger size only if both KVM and the Linux guest are using 46 hugepages but a significant speedup already happens if only one of 47 the two is using hugepages just because of the fact the TLB miss is 48 going to run faster. 49 50THP can be enabled system wide or restricted to certain tasks or even 51memory ranges inside task's address space. Unless THP is completely 52disabled, there is ``khugepaged`` daemon that scans memory and 53collapses sequences of basic pages into huge pages. 54 55The THP behaviour is controlled via :ref:`sysfs <thp_sysfs>` 56interface and using madvise(2) and prctl(2) system calls. 57 58Transparent Hugepage Support maximizes the usefulness of free memory 59if compared to the reservation approach of hugetlbfs by allowing all 60unused memory to be used as cache or other movable (or even unmovable 61entities). It doesn't require reservation to prevent hugepage 62allocation failures to be noticeable from userland. It allows paging 63and all other advanced VM features to be available on the 64hugepages. It requires no modifications for applications to take 65advantage of it. 66 67Applications however can be further optimized to take advantage of 68this feature, like for example they've been optimized before to avoid 69a flood of mmap system calls for every malloc(4k). Optimizing userland 70is by far not mandatory and khugepaged already can take care of long 71lived page allocations even for hugepage unaware applications that 72deals with large amounts of memory. 73 74In certain cases when hugepages are enabled system wide, application 75may end up allocating more memory resources. An application may mmap a 76large region but only touch 1 byte of it, in that case a 2M page might 77be allocated instead of a 4k page for no good. This is why it's 78possible to disable hugepages system-wide and to only have them inside 79MADV_HUGEPAGE madvise regions. 80 81Embedded systems should enable hugepages only inside madvise regions 82to eliminate any risk of wasting any precious byte of memory and to 83only run faster. 84 85Applications that gets a lot of benefit from hugepages and that don't 86risk to lose memory by using hugepages, should use 87madvise(MADV_HUGEPAGE) on their critical mmapped regions. 88 89.. _thp_sysfs: 90 91sysfs 92===== 93 94Global THP controls 95------------------- 96 97Transparent Hugepage Support for anonymous memory can be entirely disabled 98(mostly for debugging purposes) or only enabled inside MADV_HUGEPAGE 99regions (to avoid the risk of consuming more memory resources) or enabled 100system wide. This can be achieved with one of:: 101 102 echo always >/sys/kernel/mm/transparent_hugepage/enabled 103 echo madvise >/sys/kernel/mm/transparent_hugepage/enabled 104 echo never >/sys/kernel/mm/transparent_hugepage/enabled 105 106It's also possible to limit defrag efforts in the VM to generate 107anonymous hugepages in case they're not immediately free to madvise 108regions or to never try to defrag memory and simply fallback to regular 109pages unless hugepages are immediately available. Clearly if we spend CPU 110time to defrag memory, we would expect to gain even more by the fact we 111use hugepages later instead of regular pages. This isn't always 112guaranteed, but it may be more likely in case the allocation is for a 113MADV_HUGEPAGE region. 114 115:: 116 117 echo always >/sys/kernel/mm/transparent_hugepage/defrag 118 echo defer >/sys/kernel/mm/transparent_hugepage/defrag 119 echo defer+madvise >/sys/kernel/mm/transparent_hugepage/defrag 120 echo madvise >/sys/kernel/mm/transparent_hugepage/defrag 121 echo never >/sys/kernel/mm/transparent_hugepage/defrag 122 123always 124 means that an application requesting THP will stall on 125 allocation failure and directly reclaim pages and compact 126 memory in an effort to allocate a THP immediately. This may be 127 desirable for virtual machines that benefit heavily from THP 128 use and are willing to delay the VM start to utilise them. 129 130defer 131 means that an application will wake kswapd in the background 132 to reclaim pages and wake kcompactd to compact memory so that 133 THP is available in the near future. It's the responsibility 134 of khugepaged to then install the THP pages later. 135 136defer+madvise 137 will enter direct reclaim and compaction like ``always``, but 138 only for regions that have used madvise(MADV_HUGEPAGE); all 139 other regions will wake kswapd in the background to reclaim 140 pages and wake kcompactd to compact memory so that THP is 141 available in the near future. 142 143madvise 144 will enter direct reclaim like ``always`` but only for regions 145 that are have used madvise(MADV_HUGEPAGE). This is the default 146 behaviour. 147 148never 149 should be self-explanatory. 150 151By default kernel tries to use huge zero page on read page fault to 152anonymous mapping. It's possible to disable huge zero page by writing 0 153or enable it back by writing 1:: 154 155 echo 0 >/sys/kernel/mm/transparent_hugepage/use_zero_page 156 echo 1 >/sys/kernel/mm/transparent_hugepage/use_zero_page 157 158Some userspace (such as a test program, or an optimized memory allocation 159library) may want to know the size (in bytes) of a transparent hugepage:: 160 161 cat /sys/kernel/mm/transparent_hugepage/hpage_pmd_size 162 163khugepaged will be automatically started when 164transparent_hugepage/enabled is set to "always" or "madvise, and it'll 165be automatically shutdown if it's set to "never". 166 167Khugepaged controls 168------------------- 169 170khugepaged runs usually at low frequency so while one may not want to 171invoke defrag algorithms synchronously during the page faults, it 172should be worth invoking defrag at least in khugepaged. However it's 173also possible to disable defrag in khugepaged by writing 0 or enable 174defrag in khugepaged by writing 1:: 175 176 echo 0 >/sys/kernel/mm/transparent_hugepage/khugepaged/defrag 177 echo 1 >/sys/kernel/mm/transparent_hugepage/khugepaged/defrag 178 179You can also control how many pages khugepaged should scan at each 180pass:: 181 182 /sys/kernel/mm/transparent_hugepage/khugepaged/pages_to_scan 183 184and how many milliseconds to wait in khugepaged between each pass (you 185can set this to 0 to run khugepaged at 100% utilization of one core):: 186 187 /sys/kernel/mm/transparent_hugepage/khugepaged/scan_sleep_millisecs 188 189and how many milliseconds to wait in khugepaged if there's an hugepage 190allocation failure to throttle the next allocation attempt:: 191 192 /sys/kernel/mm/transparent_hugepage/khugepaged/alloc_sleep_millisecs 193 194The khugepaged progress can be seen in the number of pages collapsed:: 195 196 /sys/kernel/mm/transparent_hugepage/khugepaged/pages_collapsed 197 198for each pass:: 199 200 /sys/kernel/mm/transparent_hugepage/khugepaged/full_scans 201 202``max_ptes_none`` specifies how many extra small pages (that are 203not already mapped) can be allocated when collapsing a group 204of small pages into one large page:: 205 206 /sys/kernel/mm/transparent_hugepage/khugepaged/max_ptes_none 207 208A higher value leads to use additional memory for programs. 209A lower value leads to gain less thp performance. Value of 210max_ptes_none can waste cpu time very little, you can 211ignore it. 212 213``max_ptes_swap`` specifies how many pages can be brought in from 214swap when collapsing a group of pages into a transparent huge page:: 215 216 /sys/kernel/mm/transparent_hugepage/khugepaged/max_ptes_swap 217 218A higher value can cause excessive swap IO and waste 219memory. A lower value can prevent THPs from being 220collapsed, resulting fewer pages being collapsed into 221THPs, and lower memory access performance. 222 223``max_ptes_shared`` specifies how many pages can be shared across multiple 224processes. Exceeding the number would block the collapse:: 225 226 /sys/kernel/mm/transparent_hugepage/khugepaged/max_ptes_shared 227 228A higher value may increase memory footprint for some workloads. 229 230Boot parameter 231============== 232 233You can change the sysfs boot time defaults of Transparent Hugepage 234Support by passing the parameter ``transparent_hugepage=always`` or 235``transparent_hugepage=madvise`` or ``transparent_hugepage=never`` 236to the kernel command line. 237 238Hugepages in tmpfs/shmem 239======================== 240 241You can control hugepage allocation policy in tmpfs with mount option 242``huge=``. It can have following values: 243 244always 245 Attempt to allocate huge pages every time we need a new page; 246 247never 248 Do not allocate huge pages; 249 250within_size 251 Only allocate huge page if it will be fully within i_size. 252 Also respect fadvise()/madvise() hints; 253 254advise 255 Only allocate huge pages if requested with fadvise()/madvise(); 256 257The default policy is ``never``. 258 259``mount -o remount,huge= /mountpoint`` works fine after mount: remounting 260``huge=never`` will not attempt to break up huge pages at all, just stop more 261from being allocated. 262 263There's also sysfs knob to control hugepage allocation policy for internal 264shmem mount: /sys/kernel/mm/transparent_hugepage/shmem_enabled. The mount 265is used for SysV SHM, memfds, shared anonymous mmaps (of /dev/zero or 266MAP_ANONYMOUS), GPU drivers' DRM objects, Ashmem. 267 268In addition to policies listed above, shmem_enabled allows two further 269values: 270 271deny 272 For use in emergencies, to force the huge option off from 273 all mounts; 274force 275 Force the huge option on for all - very useful for testing; 276 277Need of application restart 278=========================== 279 280The transparent_hugepage/enabled values and tmpfs mount option only affect 281future behavior. So to make them effective you need to restart any 282application that could have been using hugepages. This also applies to the 283regions registered in khugepaged. 284 285Monitoring usage 286================ 287 288The number of anonymous transparent huge pages currently used by the 289system is available by reading the AnonHugePages field in ``/proc/meminfo``. 290To identify what applications are using anonymous transparent huge pages, 291it is necessary to read ``/proc/PID/smaps`` and count the AnonHugePages fields 292for each mapping. 293 294The number of file transparent huge pages mapped to userspace is available 295by reading ShmemPmdMapped and ShmemHugePages fields in ``/proc/meminfo``. 296To identify what applications are mapping file transparent huge pages, it 297is necessary to read ``/proc/PID/smaps`` and count the FileHugeMapped fields 298for each mapping. 299 300Note that reading the smaps file is expensive and reading it 301frequently will incur overhead. 302 303There are a number of counters in ``/proc/vmstat`` that may be used to 304monitor how successfully the system is providing huge pages for use. 305 306thp_fault_alloc 307 is incremented every time a huge page is successfully 308 allocated to handle a page fault. 309 310thp_collapse_alloc 311 is incremented by khugepaged when it has found 312 a range of pages to collapse into one huge page and has 313 successfully allocated a new huge page to store the data. 314 315thp_fault_fallback 316 is incremented if a page fault fails to allocate 317 a huge page and instead falls back to using small pages. 318 319thp_fault_fallback_charge 320 is incremented if a page fault fails to charge a huge page and 321 instead falls back to using small pages even though the 322 allocation was successful. 323 324thp_collapse_alloc_failed 325 is incremented if khugepaged found a range 326 of pages that should be collapsed into one huge page but failed 327 the allocation. 328 329thp_file_alloc 330 is incremented every time a file huge page is successfully 331 allocated. 332 333thp_file_fallback 334 is incremented if a file huge page is attempted to be allocated 335 but fails and instead falls back to using small pages. 336 337thp_file_fallback_charge 338 is incremented if a file huge page cannot be charged and instead 339 falls back to using small pages even though the allocation was 340 successful. 341 342thp_file_mapped 343 is incremented every time a file huge page is mapped into 344 user address space. 345 346thp_split_page 347 is incremented every time a huge page is split into base 348 pages. This can happen for a variety of reasons but a common 349 reason is that a huge page is old and is being reclaimed. 350 This action implies splitting all PMD the page mapped with. 351 352thp_split_page_failed 353 is incremented if kernel fails to split huge 354 page. This can happen if the page was pinned by somebody. 355 356thp_deferred_split_page 357 is incremented when a huge page is put onto split 358 queue. This happens when a huge page is partially unmapped and 359 splitting it would free up some memory. Pages on split queue are 360 going to be split under memory pressure. 361 362thp_split_pmd 363 is incremented every time a PMD split into table of PTEs. 364 This can happen, for instance, when application calls mprotect() or 365 munmap() on part of huge page. It doesn't split huge page, only 366 page table entry. 367 368thp_zero_page_alloc 369 is incremented every time a huge zero page is 370 successfully allocated. It includes allocations which where 371 dropped due race with other allocation. Note, it doesn't count 372 every map of the huge zero page, only its allocation. 373 374thp_zero_page_alloc_failed 375 is incremented if kernel fails to allocate 376 huge zero page and falls back to using small pages. 377 378thp_swpout 379 is incremented every time a huge page is swapout in one 380 piece without splitting. 381 382thp_swpout_fallback 383 is incremented if a huge page has to be split before swapout. 384 Usually because failed to allocate some continuous swap space 385 for the huge page. 386 387As the system ages, allocating huge pages may be expensive as the 388system uses memory compaction to copy data around memory to free a 389huge page for use. There are some counters in ``/proc/vmstat`` to help 390monitor this overhead. 391 392compact_stall 393 is incremented every time a process stalls to run 394 memory compaction so that a huge page is free for use. 395 396compact_success 397 is incremented if the system compacted memory and 398 freed a huge page for use. 399 400compact_fail 401 is incremented if the system tries to compact memory 402 but failed. 403 404It is possible to establish how long the stalls were using the function 405tracer to record how long was spent in __alloc_pages() and 406using the mm_page_alloc tracepoint to identify which allocations were 407for huge pages. 408 409Optimizing the applications 410=========================== 411 412To be guaranteed that the kernel will map a 2M page immediately in any 413memory region, the mmap region has to be hugepage naturally 414aligned. posix_memalign() can provide that guarantee. 415 416Hugetlbfs 417========= 418 419You can use hugetlbfs on a kernel that has transparent hugepage 420support enabled just fine as always. No difference can be noted in 421hugetlbfs other than there will be less overall fragmentation. All 422usual features belonging to hugetlbfs are preserved and 423unaffected. libhugetlbfs will also work fine as usual. 424