1.. SPDX-License-Identifier: GPL-2.0 2 3================== 4CXL Device Hotplug 5================== 6 7Device hotplug refers to *physical* hotplug of a device (addition or removal 8of a physical device from the machine). 9 10BIOS/EFI software is expected to configure sufficient resources **at boot 11time** to allow hotplugged devices to be configured by software (such as 12proximity domains, HPA regions, and host-bridge configurations). 13 14BIOS/EFI is not expected (**nor suggested**) to configure hotplugged 15devices at hotplug time (i.e. HDM decoders should be left unprogrammed). 16 17This document covers some examples of those resources, but should not 18be considered exhaustive. 19 20Hot-Remove 21========== 22Hot removal of a device typically requires careful removal of software 23constructs (memory regions, associated drivers) which manage these devices. 24 25Hard-removing a CXL.mem device without carefully tearing down driver stacks 26is likely to cause the system to machine-check (or at least SIGBUS if memory 27access is limited to user space). 28 29Memory Device Hot-Add 30===================== 31A device present at boot may be associated with a CXL Fixed Memory Window 32reported in :doc:`CEDT<acpi/cedt>`. That CFMWS may match the size of the 33device, but the construction of the CEDT CFMWS is platform-defined. 34 35Hot-adding a memory device requires this pre-defined, **static** CFMWS to 36have sufficient HPA space to describe that device. 37 38There are a few common scenarios to consider. 39 40Single-Endpoint Memory Device Present at Boot 41--------------------------------------------- 42A device present at boot likely had its capacity reported in the 43:doc:`CEDT<acpi/cedt>`. If a device is removed and a new device hotplugged, 44the capacity of the new device will be limited to the original CFMWS capacity. 45 46Adding capacity larger than the original device will cause memory region 47creation to fail if the region size is greater than the CFMWS size. 48 49The CFMWS is **static** and cannot be adjusted. Platforms which may expect 50different sized devices to be hotplugged must allocate sufficient CFMWS space 51**at boot time** to cover all future expected devices. 52 53Multi-Endpoint Memory Device Present at Boot 54-------------------------------------------- 55Non-switch-based Multi-Endpoint devices are outside the scope of what the 56CXL specification describes, but they are technically possible. We describe 57them here for instructive reasons only - this does not imply Linux support. 58 59A hot-plug capable CXL memory device, such as one which presents multiple 60expanders as a single large-capacity device, should report the **maximum 61possible capacity** for the device at boot. :: 62 63 HB0 64 RP0 65 | 66 [Multi-Endpoint Memory Device] 67 _____|_____ 68 | | 69 [Endpoint0] [Empty] 70 71 72Limiting the size to the capacity preset at boot will limit hot-add support 73to replacing capacity that was present at boot. 74 75No CXL Device Present at Boot 76----------------------------- 77When no CXL memory device is present on boot, some platforms omit the CFMWS 78in the :doc:`CEDT<acpi/cedt>`. When this occurs, hot-add is not possible. 79 80This describes the base case for any given device not being present at boot. 81If a future possible device is not described in the CEDT at boot, hot-add 82of that device is either limited or not possible. 83 84For a platform to support hot-add of a full memory device, it must allocate 85a CEDT CFMWS region with sufficient memory capacity to cover all future 86potentially added capacity (along with any relevant CEDT CHBS entry). 87 88To support memory hotplug directly on the host bridge/root port, or on a switch 89downstream of the host bridge, a platform must construct a CEDT CFMWS at boot 90with sufficient resources to support the max possible (or expected) hotplug 91memory capacity. :: 92 93 HB0 HB1 94 RP0 RP1 RP2 95 | | | 96 Empty Empty USP 97 ________|________ 98 | | | | 99 DSP DSP DSP DSP 100 | | | | 101 All Empty 102 103For example, a BIOS/EFI may expose an option to configure a CEDT CFMWS with 104a pre-configured amount of memory capacity (per host bridge, or host bridge 105interleave set), even if no device is attached to Root Ports or Downstream 106Ports at boot (as depicted in the figure above). 107 108 109Interleave Sets 110=============== 111 112Host Bridge Interleave 113---------------------- 114Host-bridge interleaved memory regions are defined **statically** in the 115:doc:`CEDT<acpi/cedt>`. To apply cross-host-bridge interleave, a CFMWS entry 116describing that interleave must have been provided **at boot**. Hotplugged 117devices cannot add host-bridge interleave capabilities at hotplug time. 118 119See the :doc:`Flexible CEDT Configuration<example-configurations/flexible>` 120example to see how a platform can provide this kind of flexibility regarding 121hotplugged memory devices. BIOS/EFI software should consider options to 122present flexible CEDT configurations with hotplug support. 123 124HDM Interleave 125-------------- 126Decoder-applied interleave can flexibly handle hotplugged devices, as decoders 127can be re-programmed after hotplug. 128 129To add or remove a device to/from an existing HDM-applied interleaved region, 130that region must be torn down an re-created. 131