1.. _admin_guide_memory_hotplug: 2 3============== 4Memory Hotplug 5============== 6 7:Created: Jul 28 2007 8:Updated: Add description of notifier of memory hotplug: Oct 11 2007 9 10This document is about memory hotplug including how-to-use and current status. 11Because Memory Hotplug is still under development, contents of this text will 12be changed often. 13 14.. contents:: :local: 15 16.. CONTENTS 17 18 1. Introduction 19 1.1 Purpose of memory hotplug 20 1.2. Phases of memory hotplug 21 1.3. Unit of Memory online/offline operation 22 2. Kernel Configuration 23 3. sysfs files for memory hotplug 24 4. Physical memory hot-add phase 25 4.1 Hardware(Firmware) Support 26 4.2 Notify memory hot-add event by hand 27 5. Logical Memory hot-add phase 28 5.1. State of memory 29 5.2. How to online memory 30 6. Logical memory remove 31 6.1 Memory offline and ZONE_MOVABLE 32 6.2. How to offline memory 33 7. Physical memory remove 34 9. Future Work List 35 36 37.. note:: 38 39 (1) x86_64's has special implementation for memory hotplug. 40 This text does not describe it. 41 (2) This text assumes that sysfs is mounted at ``/sys``. 42 43 44Introduction 45============ 46 47Purpose of memory hotplug 48------------------------- 49 50Memory Hotplug allows users to increase/decrease the amount of memory. 51Generally, there are two purposes. 52 53(A) For changing the amount of memory. 54 This is to allow a feature like capacity on demand. 55(B) For installing/removing DIMMs or NUMA-nodes physically. 56 This is to exchange DIMMs/NUMA-nodes, reduce power consumption, etc. 57 58(A) is required by highly virtualized environments and (B) is required by 59hardware which supports memory power management. 60 61Linux memory hotplug is designed for both purpose. 62 63Phases of memory hotplug 64------------------------ 65 66There are 2 phases in Memory Hotplug: 67 68 1) Physical Memory Hotplug phase 69 2) Logical Memory Hotplug phase. 70 71The First phase is to communicate hardware/firmware and make/erase 72environment for hotplugged memory. Basically, this phase is necessary 73for the purpose (B), but this is good phase for communication between 74highly virtualized environments too. 75 76When memory is hotplugged, the kernel recognizes new memory, makes new memory 77management tables, and makes sysfs files for new memory's operation. 78 79If firmware supports notification of connection of new memory to OS, 80this phase is triggered automatically. ACPI can notify this event. If not, 81"probe" operation by system administration is used instead. 82(see :ref:`memory_hotplug_physical_mem`). 83 84Logical Memory Hotplug phase is to change memory state into 85available/unavailable for users. Amount of memory from user's view is 86changed by this phase. The kernel makes all memory in it as free pages 87when a memory range is available. 88 89In this document, this phase is described as online/offline. 90 91Logical Memory Hotplug phase is triggered by write of sysfs file by system 92administrator. For the hot-add case, it must be executed after Physical Hotplug 93phase by hand. 94(However, if you writes udev's hotplug scripts for memory hotplug, these 95phases can be execute in seamless way.) 96 97Unit of Memory online/offline operation 98--------------------------------------- 99 100Memory hotplug uses SPARSEMEM memory model which allows memory to be divided 101into chunks of the same size. These chunks are called "sections". The size of 102a memory section is architecture dependent. For example, power uses 16MiB, ia64 103uses 1GiB. 104 105Memory sections are combined into chunks referred to as "memory blocks". The 106size of a memory block is architecture dependent and represents the logical 107unit upon which memory online/offline operations are to be performed. The 108default size of a memory block is the same as memory section size unless an 109architecture specifies otherwise. (see :ref:`memory_hotplug_sysfs_files`.) 110 111To determine the size (in bytes) of a memory block please read this file:: 112 113 /sys/devices/system/memory/block_size_bytes 114 115Kernel Configuration 116==================== 117 118To use memory hotplug feature, kernel must be compiled with following 119config options. 120 121- For all memory hotplug: 122 - Memory model -> Sparse Memory (``CONFIG_SPARSEMEM``) 123 - Allow for memory hot-add (``CONFIG_MEMORY_HOTPLUG``) 124 125- To enable memory removal, the following are also necessary: 126 - Allow for memory hot remove (``CONFIG_MEMORY_HOTREMOVE``) 127 - Page Migration (``CONFIG_MIGRATION``) 128 129- For ACPI memory hotplug, the following are also necessary: 130 - Memory hotplug (under ACPI Support menu) (``CONFIG_ACPI_HOTPLUG_MEMORY``) 131 - This option can be kernel module. 132 133- As a related configuration, if your box has a feature of NUMA-node hotplug 134 via ACPI, then this option is necessary too. 135 136 - ACPI0004,PNP0A05 and PNP0A06 Container Driver (under ACPI Support menu) 137 (``CONFIG_ACPI_CONTAINER``). 138 139 This option can be kernel module too. 140 141 142.. _memory_hotplug_sysfs_files: 143 144sysfs files for memory hotplug 145============================== 146 147All memory blocks have their device information in sysfs. Each memory block 148is described under ``/sys/devices/system/memory`` as:: 149 150 /sys/devices/system/memory/memoryXXX 151 152where XXX is the memory block id. 153 154For the memory block covered by the sysfs directory. It is expected that all 155memory sections in this range are present and no memory holes exist in the 156range. Currently there is no way to determine if there is a memory hole, but 157the existence of one should not affect the hotplug capabilities of the memory 158block. 159 160For example, assume 1GiB memory block size. A device for a memory starting at 1610x100000000 is ``/sys/device/system/memory/memory4``:: 162 163 (0x100000000 / 1Gib = 4) 164 165This device covers address range [0x100000000 ... 0x140000000) 166 167Under each memory block, you can see 5 files: 168 169- ``/sys/devices/system/memory/memoryXXX/phys_index`` 170- ``/sys/devices/system/memory/memoryXXX/phys_device`` 171- ``/sys/devices/system/memory/memoryXXX/state`` 172- ``/sys/devices/system/memory/memoryXXX/removable`` 173- ``/sys/devices/system/memory/memoryXXX/valid_zones`` 174 175=================== ============================================================ 176``phys_index`` read-only and contains memory block id, same as XXX. 177``state`` read-write 178 179 - at read: contains online/offline state of memory. 180 - at write: user can specify "online_kernel", 181 182 "online_movable", "online", "offline" command 183 which will be performed on all sections in the block. 184``phys_device`` read-only: designed to show the name of physical memory 185 device. This is not well implemented now. 186``removable`` read-only: contains an integer value indicating 187 whether the memory block is removable or not 188 removable. A value of 1 indicates that the memory 189 block is removable and a value of 0 indicates that 190 it is not removable. A memory block is removable only if 191 every section in the block is removable. 192``valid_zones`` read-only: designed to show which zones this memory block 193 can be onlined to. 194 195 The first column shows it`s default zone. 196 197 "memory6/valid_zones: Normal Movable" shows this memoryblock 198 can be onlined to ZONE_NORMAL by default and to ZONE_MOVABLE 199 by online_movable. 200 201 "memory7/valid_zones: Movable Normal" shows this memoryblock 202 can be onlined to ZONE_MOVABLE by default and to ZONE_NORMAL 203 by online_kernel. 204=================== ============================================================ 205 206.. note:: 207 208 These directories/files appear after physical memory hotplug phase. 209 210If CONFIG_NUMA is enabled the memoryXXX/ directories can also be accessed 211via symbolic links located in the ``/sys/devices/system/node/node*`` directories. 212 213For example:: 214 215 /sys/devices/system/node/node0/memory9 -> ../../memory/memory9 216 217A backlink will also be created:: 218 219 /sys/devices/system/memory/memory9/node0 -> ../../node/node0 220 221.. _memory_hotplug_physical_mem: 222 223Physical memory hot-add phase 224============================= 225 226Hardware(Firmware) Support 227-------------------------- 228 229On x86_64/ia64 platform, memory hotplug by ACPI is supported. 230 231In general, the firmware (ACPI) which supports memory hotplug defines 232memory class object of _HID "PNP0C80". When a notify is asserted to PNP0C80, 233Linux's ACPI handler does hot-add memory to the system and calls a hotplug udev 234script. This will be done automatically. 235 236But scripts for memory hotplug are not contained in generic udev package(now). 237You may have to write it by yourself or online/offline memory by hand. 238Please see :ref:`memory_hotplug_how_to_online_memory` and 239:ref:`memory_hotplug_how_to_offline_memory`. 240 241If firmware supports NUMA-node hotplug, and defines an object _HID "ACPI0004", 242"PNP0A05", or "PNP0A06", notification is asserted to it, and ACPI handler 243calls hotplug code for all of objects which are defined in it. 244If memory device is found, memory hotplug code will be called. 245 246Notify memory hot-add event by hand 247----------------------------------- 248 249On some architectures, the firmware may not notify the kernel of a memory 250hotplug event. Therefore, the memory "probe" interface is supported to 251explicitly notify the kernel. This interface depends on 252CONFIG_ARCH_MEMORY_PROBE and can be configured on powerpc, sh, and x86 253if hotplug is supported, although for x86 this should be handled by ACPI 254notification. 255 256Probe interface is located at:: 257 258 /sys/devices/system/memory/probe 259 260You can tell the physical address of new memory to the kernel by:: 261 262 % echo start_address_of_new_memory > /sys/devices/system/memory/probe 263 264Then, [start_address_of_new_memory, start_address_of_new_memory + 265memory_block_size] memory range is hot-added. In this case, hotplug script is 266not called (in current implementation). You'll have to online memory by 267yourself. Please see :ref:`memory_hotplug_how_to_online_memory`. 268 269Logical Memory hot-add phase 270============================ 271 272State of memory 273--------------- 274 275To see (online/offline) state of a memory block, read 'state' file:: 276 277 % cat /sys/device/system/memory/memoryXXX/state 278 279 280- If the memory block is online, you'll read "online". 281- If the memory block is offline, you'll read "offline". 282 283 284.. _memory_hotplug_how_to_online_memory: 285 286How to online memory 287-------------------- 288 289When the memory is hot-added, the kernel decides whether or not to "online" 290it according to the policy which can be read from "auto_online_blocks" file:: 291 292 % cat /sys/devices/system/memory/auto_online_blocks 293 294The default depends on the CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE kernel config 295option. If it is disabled the default is "offline" which means the newly added 296memory is not in a ready-to-use state and you have to "online" the newly added 297memory blocks manually. Automatic onlining can be requested by writing "online" 298to "auto_online_blocks" file:: 299 300 % echo online > /sys/devices/system/memory/auto_online_blocks 301 302This sets a global policy and impacts all memory blocks that will subsequently 303be hotplugged. Currently offline blocks keep their state. It is possible, under 304certain circumstances, that some memory blocks will be added but will fail to 305online. User space tools can check their "state" files 306(``/sys/devices/system/memory/memoryXXX/state``) and try to online them manually. 307 308If the automatic onlining wasn't requested, failed, or some memory block was 309offlined it is possible to change the individual block's state by writing to the 310"state" file:: 311 312 % echo online > /sys/devices/system/memory/memoryXXX/state 313 314This onlining will not change the ZONE type of the target memory block, 315If the memory block doesn't belong to any zone an appropriate kernel zone 316(usually ZONE_NORMAL) will be used unless movable_node kernel command line 317option is specified when ZONE_MOVABLE will be used. 318 319You can explicitly request to associate it with ZONE_MOVABLE by:: 320 321 % echo online_movable > /sys/devices/system/memory/memoryXXX/state 322 323.. note:: current limit: this memory block must be adjacent to ZONE_MOVABLE 324 325Or you can explicitly request a kernel zone (usually ZONE_NORMAL) by:: 326 327 % echo online_kernel > /sys/devices/system/memory/memoryXXX/state 328 329.. note:: current limit: this memory block must be adjacent to ZONE_NORMAL 330 331An explicit zone onlining can fail (e.g. when the range is already within 332and existing and incompatible zone already). 333 334After this, memory block XXX's state will be 'online' and the amount of 335available memory will be increased. 336 337This may be changed in future. 338 339Logical memory remove 340===================== 341 342Memory offline and ZONE_MOVABLE 343------------------------------- 344 345Memory offlining is more complicated than memory online. Because memory offline 346has to make the whole memory block be unused, memory offline can fail if 347the memory block includes memory which cannot be freed. 348 349In general, memory offline can use 2 techniques. 350 351(1) reclaim and free all memory in the memory block. 352(2) migrate all pages in the memory block. 353 354In the current implementation, Linux's memory offline uses method (2), freeing 355all pages in the memory block by page migration. But not all pages are 356migratable. Under current Linux, migratable pages are anonymous pages and 357page caches. For offlining a memory block by migration, the kernel has to 358guarantee that the memory block contains only migratable pages. 359 360Now, a boot option for making a memory block which consists of migratable pages 361is supported. By specifying "kernelcore=" or "movablecore=" boot option, you can 362create ZONE_MOVABLE...a zone which is just used for movable pages. 363(See also Documentation/admin-guide/kernel-parameters.rst) 364 365Assume the system has "TOTAL" amount of memory at boot time, this boot option 366creates ZONE_MOVABLE as following. 367 3681) When kernelcore=YYYY boot option is used, 369 Size of memory not for movable pages (not for offline) is YYYY. 370 Size of memory for movable pages (for offline) is TOTAL-YYYY. 371 3722) When movablecore=ZZZZ boot option is used, 373 Size of memory not for movable pages (not for offline) is TOTAL - ZZZZ. 374 Size of memory for movable pages (for offline) is ZZZZ. 375 376.. note:: 377 378 Unfortunately, there is no information to show which memory block belongs 379 to ZONE_MOVABLE. This is TBD. 380 381.. _memory_hotplug_how_to_offline_memory: 382 383How to offline memory 384--------------------- 385 386You can offline a memory block by using the same sysfs interface that was used 387in memory onlining:: 388 389 % echo offline > /sys/devices/system/memory/memoryXXX/state 390 391If offline succeeds, the state of the memory block is changed to be "offline". 392If it fails, some error core (like -EBUSY) will be returned by the kernel. 393Even if a memory block does not belong to ZONE_MOVABLE, you can try to offline 394it. If it doesn't contain 'unmovable' memory, you'll get success. 395 396A memory block under ZONE_MOVABLE is considered to be able to be offlined 397easily. But under some busy state, it may return -EBUSY. Even if a memory 398block cannot be offlined due to -EBUSY, you can retry offlining it and may be 399able to offline it (or not). (For example, a page is referred to by some kernel 400internal call and released soon.) 401 402Consideration: 403 Memory hotplug's design direction is to make the possibility of memory 404 offlining higher and to guarantee unplugging memory under any situation. But 405 it needs more work. Returning -EBUSY under some situation may be good because 406 the user can decide to retry more or not by himself. Currently, memory 407 offlining code does some amount of retry with 120 seconds timeout. 408 409Physical memory remove 410====================== 411 412Need more implementation yet.... 413 - Notification completion of remove works by OS to firmware. 414 - Guard from remove if not yet. 415 416Future Work 417=========== 418 419 - allowing memory hot-add to ZONE_MOVABLE. maybe we need some switch like 420 sysctl or new control file. 421 - showing memory block and physical device relationship. 422 - test and make it better memory offlining. 423 - support HugeTLB page migration and offlining. 424 - memmap removing at memory offline. 425 - physical remove memory. 426