1================== 2Memory Hot(Un)Plug 3================== 4 5This document describes generic Linux support for memory hot(un)plug with 6a focus on System RAM, including ZONE_MOVABLE support. 7 8.. contents:: :local: 9 10Introduction 11============ 12 13Memory hot(un)plug allows for increasing and decreasing the size of physical 14memory available to a machine at runtime. In the simplest case, it consists of 15physically plugging or unplugging a DIMM at runtime, coordinated with the 16operating system. 17 18Memory hot(un)plug is used for various purposes: 19 20- The physical memory available to a machine can be adjusted at runtime, up- or 21 downgrading the memory capacity. This dynamic memory resizing, sometimes 22 referred to as "capacity on demand", is frequently used with virtual machines 23 and logical partitions. 24 25- Replacing hardware, such as DIMMs or whole NUMA nodes, without downtime. One 26 example is replacing failing memory modules. 27 28- Reducing energy consumption either by physically unplugging memory modules or 29 by logically unplugging (parts of) memory modules from Linux. 30 31Further, the basic memory hot(un)plug infrastructure in Linux is nowadays also 32used to expose persistent memory, other performance-differentiated memory and 33reserved memory regions as ordinary system RAM to Linux. 34 35Linux only supports memory hot(un)plug on selected 64 bit architectures, such as 36x86_64, arm64, ppc64 and s390x. 37 38Memory Hot(Un)Plug Granularity 39------------------------------ 40 41Memory hot(un)plug in Linux uses the SPARSEMEM memory model, which divides the 42physical memory address space into chunks of the same size: memory sections. The 43size of a memory section is architecture dependent. For example, x86_64 uses 44128 MiB and ppc64 uses 16 MiB. 45 46Memory sections are combined into chunks referred to as "memory blocks". The 47size of a memory block is architecture dependent and corresponds to the smallest 48granularity that can be hot(un)plugged. The default size of a memory block is 49the same as memory section size, unless an architecture specifies otherwise. 50 51All memory blocks have the same size. 52 53Phases of Memory Hotplug 54------------------------ 55 56Memory hotplug consists of two phases: 57 58(1) Adding the memory to Linux 59(2) Onlining memory blocks 60 61In the first phase, metadata, such as the memory map ("memmap") and page tables 62for the direct mapping, is allocated and initialized, and memory blocks are 63created; the latter also creates sysfs files for managing newly created memory 64blocks. 65 66In the second phase, added memory is exposed to the page allocator. After this 67phase, the memory is visible in memory statistics, such as free and total 68memory, of the system. 69 70Phases of Memory Hotunplug 71-------------------------- 72 73Memory hotunplug consists of two phases: 74 75(1) Offlining memory blocks 76(2) Removing the memory from Linux 77 78In the first phase, memory is "hidden" from the page allocator again, for 79example, by migrating busy memory to other memory locations and removing all 80relevant free pages from the page allocator After this phase, the memory is no 81longer visible in memory statistics of the system. 82 83In the second phase, the memory blocks are removed and metadata is freed. 84 85Memory Hotplug Notifications 86============================ 87 88There are various ways how Linux is notified about memory hotplug events such 89that it can start adding hotplugged memory. This description is limited to 90systems that support ACPI; mechanisms specific to other firmware interfaces or 91virtual machines are not described. 92 93ACPI Notifications 94------------------ 95 96Platforms that support ACPI, such as x86_64, can support memory hotplug 97notifications via ACPI. 98 99In general, a firmware supporting memory hotplug defines a memory class object 100HID "PNP0C80". When notified about hotplug of a new memory device, the ACPI 101driver will hotplug the memory to Linux. 102 103If the firmware supports hotplug of NUMA nodes, it defines an object _HID 104"ACPI0004", "PNP0A05", or "PNP0A06". When notified about an hotplug event, all 105assigned memory devices are added to Linux by the ACPI driver. 106 107Similarly, Linux can be notified about requests to hotunplug a memory device or 108a NUMA node via ACPI. The ACPI driver will try offlining all relevant memory 109blocks, and, if successful, hotunplug the memory from Linux. 110 111Manual Probing 112-------------- 113 114On some architectures, the firmware may not be able to notify the operating 115system about a memory hotplug event. Instead, the memory has to be manually 116probed from user space. 117 118The probe interface is located at:: 119 120 /sys/devices/system/memory/probe 121 122Only complete memory blocks can be probed. Individual memory blocks are probed 123by providing the physical start address of the memory block:: 124 125 % echo addr > /sys/devices/system/memory/probe 126 127Which results in a memory block for the range [addr, addr + memory_block_size) 128being created. 129 130.. note:: 131 132 Using the probe interface is discouraged as it is easy to crash the kernel, 133 because Linux cannot validate user input; this interface might be removed in 134 the future. 135 136Onlining and Offlining Memory Blocks 137==================================== 138 139After a memory block has been created, Linux has to be instructed to actually 140make use of that memory: the memory block has to be "online". 141 142Before a memory block can be removed, Linux has to stop using any memory part of 143the memory block: the memory block has to be "offlined". 144 145The Linux kernel can be configured to automatically online added memory blocks 146and drivers automatically trigger offlining of memory blocks when trying 147hotunplug of memory. Memory blocks can only be removed once offlining succeeded 148and drivers may trigger offlining of memory blocks when attempting hotunplug of 149memory. 150 151Onlining Memory Blocks Manually 152------------------------------- 153 154If auto-onlining of memory blocks isn't enabled, user-space has to manually 155trigger onlining of memory blocks. Often, udev rules are used to automate this 156task in user space. 157 158Onlining of a memory block can be triggered via:: 159 160 % echo online > /sys/devices/system/memory/memoryXXX/state 161 162Or alternatively:: 163 164 % echo 1 > /sys/devices/system/memory/memoryXXX/online 165 166The kernel will select the target zone automatically, depending on the 167configured ``online_policy``. 168 169One can explicitly request to associate an offline memory block with 170ZONE_MOVABLE by:: 171 172 % echo online_movable > /sys/devices/system/memory/memoryXXX/state 173 174Or one can explicitly request a kernel zone (usually ZONE_NORMAL) by:: 175 176 % echo online_kernel > /sys/devices/system/memory/memoryXXX/state 177 178In any case, if onlining succeeds, the state of the memory block is changed to 179be "online". If it fails, the state of the memory block will remain unchanged 180and the above commands will fail. 181 182Onlining Memory Blocks Automatically 183------------------------------------ 184 185The kernel can be configured to try auto-onlining of newly added memory blocks. 186If this feature is disabled, the memory blocks will stay offline until 187explicitly onlined from user space. 188 189The configured auto-online behavior can be observed via:: 190 191 % cat /sys/devices/system/memory/auto_online_blocks 192 193Auto-onlining can be enabled by writing ``online``, ``online_kernel`` or 194``online_movable`` to that file, like:: 195 196 % echo online > /sys/devices/system/memory/auto_online_blocks 197 198Similarly to manual onlining, with ``online`` the kernel will select the 199target zone automatically, depending on the configured ``online_policy``. 200 201Modifying the auto-online behavior will only affect all subsequently added 202memory blocks only. 203 204.. note:: 205 206 In corner cases, auto-onlining can fail. The kernel won't retry. Note that 207 auto-onlining is not expected to fail in default configurations. 208 209.. note:: 210 211 DLPAR on ppc64 ignores the ``offline`` setting and will still online added 212 memory blocks; if onlining fails, memory blocks are removed again. 213 214Offlining Memory Blocks 215----------------------- 216 217In the current implementation, Linux's memory offlining will try migrating all 218movable pages off the affected memory block. As most kernel allocations, such as 219page tables, are unmovable, page migration can fail and, therefore, inhibit 220memory offlining from succeeding. 221 222Having the memory provided by memory block managed by ZONE_MOVABLE significantly 223increases memory offlining reliability; still, memory offlining can fail in 224some corner cases. 225 226Further, memory offlining might retry for a long time (or even forever), until 227aborted by the user. 228 229Offlining of a memory block can be triggered via:: 230 231 % echo offline > /sys/devices/system/memory/memoryXXX/state 232 233Or alternatively:: 234 235 % echo 0 > /sys/devices/system/memory/memoryXXX/online 236 237If offlining succeeds, the state of the memory block is changed to be "offline". 238If it fails, the state of the memory block will remain unchanged and the above 239commands will fail, for example, via:: 240 241 bash: echo: write error: Device or resource busy 242 243or via:: 244 245 bash: echo: write error: Invalid argument 246 247Observing the State of Memory Blocks 248------------------------------------ 249 250The state (online/offline/going-offline) of a memory block can be observed 251either via:: 252 253 % cat /sys/devices/system/memory/memoryXXX/state 254 255Or alternatively (1/0) via:: 256 257 % cat /sys/devices/system/memory/memoryXXX/online 258 259For an online memory block, the managing zone can be observed via:: 260 261 % cat /sys/devices/system/memory/memoryXXX/valid_zones 262 263Configuring Memory Hot(Un)Plug 264============================== 265 266There are various ways how system administrators can configure memory 267hot(un)plug and interact with memory blocks, especially, to online them. 268 269Memory Hot(Un)Plug Configuration via Sysfs 270------------------------------------------ 271 272Some memory hot(un)plug properties can be configured or inspected via sysfs in:: 273 274 /sys/devices/system/memory/ 275 276The following files are currently defined: 277 278====================== ========================================================= 279``auto_online_blocks`` read-write: set or get the default state of new memory 280 blocks; configure auto-onlining. 281 282 The default value depends on the 283 CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE kernel configuration 284 option. 285 286 See the ``state`` property of memory blocks for details. 287``block_size_bytes`` read-only: the size in bytes of a memory block. 288``probe`` write-only: add (probe) selected memory blocks manually 289 from user space by supplying the physical start address. 290 291 Availability depends on the CONFIG_ARCH_MEMORY_PROBE 292 kernel configuration option. 293``uevent`` read-write: generic udev file for device subsystems. 294``crash_hotplug`` read-only: when changes to the system memory map 295 occur due to hot un/plug of memory, this file contains 296 '1' if the kernel updates the kdump capture kernel memory 297 map itself (via elfcorehdr), or '0' if userspace must update 298 the kdump capture kernel memory map. 299 300 Availability depends on the CONFIG_MEMORY_HOTPLUG kernel 301 configuration option. 302====================== ========================================================= 303 304.. note:: 305 306 When the CONFIG_MEMORY_FAILURE kernel configuration option is enabled, two 307 additional files ``hard_offline_page`` and ``soft_offline_page`` are available 308 to trigger hwpoisoning of pages, for example, for testing purposes. Note that 309 this functionality is not really related to memory hot(un)plug or actual 310 offlining of memory blocks. 311 312Memory Block Configuration via Sysfs 313------------------------------------ 314 315Each memory block is represented as a memory block device that can be 316onlined or offlined. All memory blocks have their device information located in 317sysfs. Each present memory block is listed under 318``/sys/devices/system/memory`` as:: 319 320 /sys/devices/system/memory/memoryXXX 321 322where XXX is the memory block id; the number of digits is variable. 323 324A present memory block indicates that some memory in the range is present; 325however, a memory block might span memory holes. A memory block spanning memory 326holes cannot be offlined. 327 328For example, assume 1 GiB memory block size. A device for a memory starting at 3290x100000000 is ``/sys/devices/system/memory/memory4``:: 330 331 (0x100000000 / 1Gib = 4) 332 333This device covers address range [0x100000000 ... 0x140000000) 334 335The following files are currently defined: 336 337=================== ============================================================ 338``online`` read-write: simplified interface to trigger onlining / 339 offlining and to observe the state of a memory block. 340 When onlining, the zone is selected automatically. 341``phys_device`` read-only: legacy interface only ever used on s390x to 342 expose the covered storage increment. 343``phys_index`` read-only: the memory block id (XXX). 344``removable`` read-only: legacy interface that indicated whether a memory 345 block was likely to be offlineable or not. Nowadays, the 346 kernel return ``1`` if and only if it supports memory 347 offlining. 348``state`` read-write: advanced interface to trigger onlining / 349 offlining and to observe the state of a memory block. 350 351 When writing, ``online``, ``offline``, ``online_kernel`` and 352 ``online_movable`` are supported. 353 354 ``online_movable`` specifies onlining to ZONE_MOVABLE. 355 ``online_kernel`` specifies onlining to the default kernel 356 zone for the memory block, such as ZONE_NORMAL. 357 ``online`` let's the kernel select the zone automatically. 358 359 When reading, ``online``, ``offline`` and ``going-offline`` 360 may be returned. 361``uevent`` read-write: generic uevent file for devices. 362``valid_zones`` read-only: when a block is online, shows the zone it 363 belongs to; when a block is offline, shows what zone will 364 manage it when the block will be onlined. 365 366 For online memory blocks, ``DMA``, ``DMA32``, ``Normal``, 367 ``Movable`` and ``none`` may be returned. ``none`` indicates 368 that memory provided by a memory block is managed by 369 multiple zones or spans multiple nodes; such memory blocks 370 cannot be offlined. ``Movable`` indicates ZONE_MOVABLE. 371 Other values indicate a kernel zone. 372 373 For offline memory blocks, the first column shows the 374 zone the kernel would select when onlining the memory block 375 right now without further specifying a zone. 376 377 Availability depends on the CONFIG_MEMORY_HOTREMOVE 378 kernel configuration option. 379=================== ============================================================ 380 381.. note:: 382 383 If the CONFIG_NUMA kernel configuration option is enabled, the memoryXXX/ 384 directories can also be accessed via symbolic links located in the 385 ``/sys/devices/system/node/node*`` directories. 386 387 For example:: 388 389 /sys/devices/system/node/node0/memory9 -> ../../memory/memory9 390 391 A backlink will also be created:: 392 393 /sys/devices/system/memory/memory9/node0 -> ../../node/node0 394 395Command Line Parameters 396----------------------- 397 398Some command line parameters affect memory hot(un)plug handling. The following 399command line parameters are relevant: 400 401======================== ======================================================= 402``memhp_default_state`` configure auto-onlining by essentially setting 403 ``/sys/devices/system/memory/auto_online_blocks``. 404``movable_node`` configure automatic zone selection in the kernel when 405 using the ``contig-zones`` online policy. When 406 set, the kernel will default to ZONE_MOVABLE when 407 onlining a memory block, unless other zones can be kept 408 contiguous. 409======================== ======================================================= 410 411See Documentation/admin-guide/kernel-parameters.txt for a more generic 412description of these command line parameters. 413 414Module Parameters 415------------------ 416 417Instead of additional command line parameters or sysfs files, the 418``memory_hotplug`` subsystem now provides a dedicated namespace for module 419parameters. Module parameters can be set via the command line by predicating 420them with ``memory_hotplug.`` such as:: 421 422 memory_hotplug.memmap_on_memory=1 423 424and they can be observed (and some even modified at runtime) via:: 425 426 /sys/module/memory_hotplug/parameters/ 427 428The following module parameters are currently defined: 429 430================================ =============================================== 431``memmap_on_memory`` read-write: Allocate memory for the memmap from 432 the added memory block itself. Even if enabled, 433 actual support depends on various other system 434 properties and should only be regarded as a 435 hint whether the behavior would be desired. 436 437 While allocating the memmap from the memory 438 block itself makes memory hotplug less likely 439 to fail and keeps the memmap on the same NUMA 440 node in any case, it can fragment physical 441 memory in a way that huge pages in bigger 442 granularity cannot be formed on hotplugged 443 memory. 444 445 With value "force" it could result in memory 446 wastage due to memmap size limitations. For 447 example, if the memmap for a memory block 448 requires 1 MiB, but the pageblock size is 2 449 MiB, 1 MiB of hotplugged memory will be wasted. 450 Note that there are still cases where the 451 feature cannot be enforced: for example, if the 452 memmap is smaller than a single page, or if the 453 architecture does not support the forced mode 454 in all configurations. 455 456``online_policy`` read-write: Set the basic policy used for 457 automatic zone selection when onlining memory 458 blocks without specifying a target zone. 459 ``contig-zones`` has been the kernel default 460 before this parameter was added. After an 461 online policy was configured and memory was 462 online, the policy should not be changed 463 anymore. 464 465 When set to ``contig-zones``, the kernel will 466 try keeping zones contiguous. If a memory block 467 intersects multiple zones or no zone, the 468 behavior depends on the ``movable_node`` kernel 469 command line parameter: default to ZONE_MOVABLE 470 if set, default to the applicable kernel zone 471 (usually ZONE_NORMAL) if not set. 472 473 When set to ``auto-movable``, the kernel will 474 try onlining memory blocks to ZONE_MOVABLE if 475 possible according to the configuration and 476 memory device details. With this policy, one 477 can avoid zone imbalances when eventually 478 hotplugging a lot of memory later and still 479 wanting to be able to hotunplug as much as 480 possible reliably, very desirable in 481 virtualized environments. This policy ignores 482 the ``movable_node`` kernel command line 483 parameter and isn't really applicable in 484 environments that require it (e.g., bare metal 485 with hotunpluggable nodes) where hotplugged 486 memory might be exposed via the 487 firmware-provided memory map early during boot 488 to the system instead of getting detected, 489 added and onlined later during boot (such as 490 done by virtio-mem or by some hypervisors 491 implementing emulated DIMMs). As one example, a 492 hotplugged DIMM will be onlined either 493 completely to ZONE_MOVABLE or completely to 494 ZONE_NORMAL, not a mixture. 495 As another example, as many memory blocks 496 belonging to a virtio-mem device will be 497 onlined to ZONE_MOVABLE as possible, 498 special-casing units of memory blocks that can 499 only get hotunplugged together. *This policy 500 does not protect from setups that are 501 problematic with ZONE_MOVABLE and does not 502 change the zone of memory blocks dynamically 503 after they were onlined.* 504``auto_movable_ratio`` read-write: Set the maximum MOVABLE:KERNEL 505 memory ratio in % for the ``auto-movable`` 506 online policy. Whether the ratio applies only 507 for the system across all NUMA nodes or also 508 per NUMA nodes depends on the 509 ``auto_movable_numa_aware`` configuration. 510 511 All accounting is based on present memory pages 512 in the zones combined with accounting per 513 memory device. Memory dedicated to the CMA 514 allocator is accounted as MOVABLE, although 515 residing on one of the kernel zones. The 516 possible ratio depends on the actual workload. 517 The kernel default is "301" %, for example, 518 allowing for hotplugging 24 GiB to a 8 GiB VM 519 and automatically onlining all hotplugged 520 memory to ZONE_MOVABLE in many setups. The 521 additional 1% deals with some pages being not 522 present, for example, because of some firmware 523 allocations. 524 525 Note that ZONE_NORMAL memory provided by one 526 memory device does not allow for more 527 ZONE_MOVABLE memory for a different memory 528 device. As one example, onlining memory of a 529 hotplugged DIMM to ZONE_NORMAL will not allow 530 for another hotplugged DIMM to get onlined to 531 ZONE_MOVABLE automatically. In contrast, memory 532 hotplugged by a virtio-mem device that got 533 onlined to ZONE_NORMAL will allow for more 534 ZONE_MOVABLE memory within *the same* 535 virtio-mem device. 536``auto_movable_numa_aware`` read-write: Configure whether the 537 ``auto_movable_ratio`` in the ``auto-movable`` 538 online policy also applies per NUMA 539 node in addition to the whole system across all 540 NUMA nodes. The kernel default is "Y". 541 542 Disabling NUMA awareness can be helpful when 543 dealing with NUMA nodes that should be 544 completely hotunpluggable, onlining the memory 545 completely to ZONE_MOVABLE automatically if 546 possible. 547 548 Parameter availability depends on CONFIG_NUMA. 549================================ =============================================== 550 551ZONE_MOVABLE 552============ 553 554ZONE_MOVABLE is an important mechanism for more reliable memory offlining. 555Further, having system RAM managed by ZONE_MOVABLE instead of one of the 556kernel zones can increase the number of possible transparent huge pages and 557dynamically allocated huge pages. 558 559Most kernel allocations are unmovable. Important examples include the memory 560map (usually 1/64ths of memory), page tables, and kmalloc(). Such allocations 561can only be served from the kernel zones. 562 563Most user space pages, such as anonymous memory, and page cache pages are 564movable. Such allocations can be served from ZONE_MOVABLE and the kernel zones. 565 566Only movable allocations are served from ZONE_MOVABLE, resulting in unmovable 567allocations being limited to the kernel zones. Without ZONE_MOVABLE, there is 568absolutely no guarantee whether a memory block can be offlined successfully. 569 570Zone Imbalances 571--------------- 572 573Having too much system RAM managed by ZONE_MOVABLE is called a zone imbalance, 574which can harm the system or degrade performance. As one example, the kernel 575might crash because it runs out of free memory for unmovable allocations, 576although there is still plenty of free memory left in ZONE_MOVABLE. 577 578Usually, MOVABLE:KERNEL ratios of up to 3:1 or even 4:1 are fine. Ratios of 63:1 579are definitely impossible due to the overhead for the memory map. 580 581Actual safe zone ratios depend on the workload. Extreme cases, like excessive 582long-term pinning of pages, might not be able to deal with ZONE_MOVABLE at all. 583 584.. note:: 585 586 CMA memory part of a kernel zone essentially behaves like memory in 587 ZONE_MOVABLE and similar considerations apply, especially when combining 588 CMA with ZONE_MOVABLE. 589 590ZONE_MOVABLE Sizing Considerations 591---------------------------------- 592 593We usually expect that a large portion of available system RAM will actually 594be consumed by user space, either directly or indirectly via the page cache. In 595the normal case, ZONE_MOVABLE can be used when allocating such pages just fine. 596 597With that in mind, it makes sense that we can have a big portion of system RAM 598managed by ZONE_MOVABLE. However, there are some things to consider when using 599ZONE_MOVABLE, especially when fine-tuning zone ratios: 600 601- Having a lot of offline memory blocks. Even offline memory blocks consume 602 memory for metadata and page tables in the direct map; having a lot of offline 603 memory blocks is not a typical case, though. 604 605- Memory ballooning without balloon compaction is incompatible with 606 ZONE_MOVABLE. Only some implementations, such as virtio-balloon and 607 pseries CMM, fully support balloon compaction. 608 609 Further, the CONFIG_BALLOON_COMPACTION kernel configuration option might be 610 disabled. In that case, balloon inflation will only perform unmovable 611 allocations and silently create a zone imbalance, usually triggered by 612 inflation requests from the hypervisor. 613 614- Gigantic pages are unmovable, resulting in user space consuming a 615 lot of unmovable memory. 616 617- Huge pages are unmovable when an architectures does not support huge 618 page migration, resulting in a similar issue as with gigantic pages. 619 620- Page tables are unmovable. Excessive swapping, mapping extremely large 621 files or ZONE_DEVICE memory can be problematic, although only really relevant 622 in corner cases. When we manage a lot of user space memory that has been 623 swapped out or is served from a file/persistent memory/... we still need a lot 624 of page tables to manage that memory once user space accessed that memory. 625 626- In certain DAX configurations the memory map for the device memory will be 627 allocated from the kernel zones. 628 629- KASAN can have a significant memory overhead, for example, consuming 1/8th of 630 the total system memory size as (unmovable) tracking metadata. 631 632- Long-term pinning of pages. Techniques that rely on long-term pinnings 633 (especially, RDMA and vfio/mdev) are fundamentally problematic with 634 ZONE_MOVABLE, and therefore, memory offlining. Pinned pages cannot reside 635 on ZONE_MOVABLE as that would turn these pages unmovable. Therefore, they 636 have to be migrated off that zone while pinning. Pinning a page can fail 637 even if there is plenty of free memory in ZONE_MOVABLE. 638 639 In addition, using ZONE_MOVABLE might make page pinning more expensive, 640 because of the page migration overhead. 641 642By default, all the memory configured at boot time is managed by the kernel 643zones and ZONE_MOVABLE is not used. 644 645To enable ZONE_MOVABLE to include the memory present at boot and to control the 646ratio between movable and kernel zones there are two command line options: 647``kernelcore=`` and ``movablecore=``. See 648Documentation/admin-guide/kernel-parameters.rst for their description. 649 650Memory Offlining and ZONE_MOVABLE 651--------------------------------- 652 653Even with ZONE_MOVABLE, there are some corner cases where offlining a memory 654block might fail: 655 656- Memory blocks with memory holes; this applies to memory blocks present during 657 boot and can apply to memory blocks hotplugged via the XEN balloon and the 658 Hyper-V balloon. 659 660- Mixed NUMA nodes and mixed zones within a single memory block prevent memory 661 offlining; this applies to memory blocks present during boot only. 662 663- Special memory blocks prevented by the system from getting offlined. Examples 664 include any memory available during boot on arm64 or memory blocks spanning 665 the crashkernel area on s390x; this usually applies to memory blocks present 666 during boot only. 667 668- Memory blocks overlapping with CMA areas cannot be offlined, this applies to 669 memory blocks present during boot only. 670 671- Concurrent activity that operates on the same physical memory area, such as 672 allocating gigantic pages, can result in temporary offlining failures. 673 674- Out of memory when dissolving huge pages, especially when HugeTLB Vmemmap 675 Optimization (HVO) is enabled. 676 677 Offlining code may be able to migrate huge page contents, but may not be able 678 to dissolve the source huge page because it fails allocating (unmovable) pages 679 for the vmemmap, because the system might not have free memory in the kernel 680 zones left. 681 682 Users that depend on memory offlining to succeed for movable zones should 683 carefully consider whether the memory savings gained from this feature are 684 worth the risk of possibly not being able to offline memory in certain 685 situations. 686 687Further, when running into out of memory situations while migrating pages, or 688when still encountering permanently unmovable pages within ZONE_MOVABLE 689(-> BUG), memory offlining will keep retrying until it eventually succeeds. 690 691When offlining is triggered from user space, the offlining context can be 692terminated by sending a signal. A timeout based offlining can easily be 693implemented via:: 694 695 % timeout $TIMEOUT offline_block | failure_handling 696