1.. SPDX-License-Identifier: GPL-2.0 2 3======================= 4DAMON-based Reclamation 5======================= 6 7DAMON-based Reclamation (DAMON_RECLAIM) is a static kernel module that aimed to 8be used for proactive and lightweight reclamation under light memory pressure. 9It doesn't aim to replace the LRU-list based page_granularity reclamation, but 10to be selectively used for different level of memory pressure and requirements. 11 12Where Proactive Reclamation is Required? 13======================================== 14 15On general memory over-committed systems, proactively reclaiming cold pages 16helps saving memory and reducing latency spikes that incurred by the direct 17reclaim of the process or CPU consumption of kswapd, while incurring only 18minimal performance degradation [1]_ [2]_ . 19 20Free Pages Reporting [3]_ based memory over-commit virtualization systems are 21good example of the cases. In such systems, the guest VMs reports their free 22memory to host, and the host reallocates the reported memory to other guests. 23As a result, the memory of the systems are fully utilized. However, the 24guests could be not so memory-frugal, mainly because some kernel subsystems and 25user-space applications are designed to use as much memory as available. Then, 26guests could report only small amount of memory as free to host, results in 27memory utilization drop of the systems. Running the proactive reclamation in 28guests could mitigate this problem. 29 30How It Works? 31============= 32 33DAMON_RECLAIM finds memory regions that didn't accessed for specific time 34duration and page out. To avoid it consuming too much CPU for the paging out 35operation, a speed limit can be configured. Under the speed limit, it pages 36out memory regions that didn't accessed longer time first. System 37administrators can also configure under what situation this scheme should 38automatically activated and deactivated with three memory pressure watermarks. 39 40Interface: Module Parameters 41============================ 42 43To use this feature, you should first ensure your system is running on a kernel 44that is built with ``CONFIG_DAMON_RECLAIM=y``. 45 46To let sysadmins enable or disable it and tune for the given system, 47DAMON_RECLAIM utilizes module parameters. That is, you can put 48``damon_reclaim.<parameter>=<value>`` on the kernel boot command line or write 49proper values to ``/sys/module/damon_reclaim/parameters/<parameter>`` files. 50 51Below are the description of each parameter. 52 53enabled 54------- 55 56Enable or disable DAMON_RECLAIM. 57 58You can enable DAMON_RCLAIM by setting the value of this parameter as ``Y``. 59Setting it as ``N`` disables DAMON_RECLAIM. Note that DAMON_RECLAIM could do 60no real monitoring and reclamation due to the watermarks-based activation 61condition. Refer to below descriptions for the watermarks parameter for this. 62 63commit_inputs 64------------- 65 66Make DAMON_RECLAIM reads the input parameters again, except ``enabled``. 67 68Input parameters that updated while DAMON_RECLAIM is running are not applied 69by default. Once this parameter is set as ``Y``, DAMON_RECLAIM reads values 70of parametrs except ``enabled`` again. Once the re-reading is done, this 71parameter is set as ``N``. If invalid parameters are found while the 72re-reading, DAMON_RECLAIM will be disabled. 73 74Once ``Y`` is written to this parameter, the user must not write to any 75parameters until reading ``commit_inputs`` again returns ``N``. If users 76violate this rule, the kernel may exhibit undefined behavior. 77 78min_age 79------- 80 81Time threshold for cold memory regions identification in microseconds. 82 83If a memory region is not accessed for this or longer time, DAMON_RECLAIM 84identifies the region as cold, and reclaims it. 85 86120 seconds by default. 87 88quota_ms 89-------- 90 91Limit of time for the reclamation in milliseconds. 92 93DAMON_RECLAIM tries to use only up to this time within a time window 94(quota_reset_interval_ms) for trying reclamation of cold pages. This can be 95used for limiting CPU consumption of DAMON_RECLAIM. If the value is zero, the 96limit is disabled. 97 9810 ms by default. 99 100quota_sz 101-------- 102 103Limit of size of memory for the reclamation in bytes. 104 105DAMON_RECLAIM charges amount of memory which it tried to reclaim within a time 106window (quota_reset_interval_ms) and makes no more than this limit is tried. 107This can be used for limiting consumption of CPU and IO. If this value is 108zero, the limit is disabled. 109 110128 MiB by default. 111 112quota_reset_interval_ms 113----------------------- 114 115The time/size quota charge reset interval in milliseconds. 116 117The charget reset interval for the quota of time (quota_ms) and size 118(quota_sz). That is, DAMON_RECLAIM does not try reclamation for more than 119quota_ms milliseconds or quota_sz bytes within quota_reset_interval_ms 120milliseconds. 121 1221 second by default. 123 124quota_mem_pressure_us 125--------------------- 126 127Desired level of memory pressure-stall time in microseconds. 128 129While keeping the caps that set by other quotas, DAMON_RECLAIM automatically 130increases and decreases the effective level of the quota aiming this level of 131memory pressure is incurred. System-wide ``some`` memory PSI in microseconds 132per quota reset interval (``quota_reset_interval_ms``) is collected and 133compared to this value to see if the aim is satisfied. Value zero means 134disabling this auto-tuning feature. 135 136Disabled by default. 137 138quota_autotune_feedback 139----------------------- 140 141User-specifiable feedback for auto-tuning of the effective quota. 142 143While keeping the caps that set by other quotas, DAMON_RECLAIM automatically 144increases and decreases the effective level of the quota aiming receiving this 145feedback of value ``10,000`` from the user. DAMON_RECLAIM assumes the feedback 146value and the quota are positively proportional. Value zero means disabling 147this auto-tuning feature. 148 149Disabled by default. 150 151wmarks_interval 152--------------- 153 154Minimal time to wait before checking the watermarks, when DAMON_RECLAIM is 155enabled but inactive due to its watermarks rule. 156 157wmarks_high 158----------- 159 160Free memory rate (per thousand) for the high watermark. 161 162If free memory of the system in bytes per thousand bytes is higher than this, 163DAMON_RECLAIM becomes inactive, so it does nothing but only periodically checks 164the watermarks. 165 166wmarks_mid 167---------- 168 169Free memory rate (per thousand) for the middle watermark. 170 171If free memory of the system in bytes per thousand bytes is between this and 172the low watermark, DAMON_RECLAIM becomes active, so starts the monitoring and 173the reclaiming. 174 175wmarks_low 176---------- 177 178Free memory rate (per thousand) for the low watermark. 179 180If free memory of the system in bytes per thousand bytes is lower than this, 181DAMON_RECLAIM becomes inactive, so it does nothing but periodically checks the 182watermarks. In the case, the system falls back to the LRU-list based page 183granularity reclamation logic. 184 185sample_interval 186--------------- 187 188Sampling interval for the monitoring in microseconds. 189 190The sampling interval of DAMON for the cold memory monitoring. Please refer to 191the DAMON documentation (:doc:`usage`) for more detail. 192 193aggr_interval 194------------- 195 196Aggregation interval for the monitoring in microseconds. 197 198The aggregation interval of DAMON for the cold memory monitoring. Please 199refer to the DAMON documentation (:doc:`usage`) for more detail. 200 201min_nr_regions 202-------------- 203 204Minimum number of monitoring regions. 205 206The minimal number of monitoring regions of DAMON for the cold memory 207monitoring. This can be used to set lower-bound of the monitoring quality. 208But, setting this too high could result in increased monitoring overhead. 209Please refer to the DAMON documentation (:doc:`usage`) for more detail. 210 211Note that this must be 3 or higher. Please refer to the :ref:`Monitoring 212<damon_design_monitoring>` section of the design document for the rationale 213behind this lower bound. 214 215max_nr_regions 216-------------- 217 218Maximum number of monitoring regions. 219 220The maximum number of monitoring regions of DAMON for the cold memory 221monitoring. This can be used to set upper-bound of the monitoring overhead. 222However, setting this too low could result in bad monitoring quality. Please 223refer to the DAMON documentation (:doc:`usage`) for more detail. 224 225monitor_region_start 226-------------------- 227 228Start of target memory region in physical address. 229 230The start physical address of memory region that DAMON_RECLAIM will do work 231against. That is, DAMON_RECLAIM will find cold memory regions in this region 232and reclaims. By default, biggest System RAM is used as the region. 233 234monitor_region_end 235------------------ 236 237End of target memory region in physical address. 238 239The end physical address of memory region that DAMON_RECLAIM will do work 240against. That is, DAMON_RECLAIM will find cold memory regions in this region 241and reclaims. By default, biggest System RAM is used as the region. 242 243addr_unit 244--------- 245 246A scale factor for memory addresses and bytes. 247 248This parameter is for setting and getting the :ref:`address unit 249<damon_design_addr_unit>` parameter of the DAMON instance for DAMON_RECLAIM. 250 251``monitor_region_start`` and ``monitor_region_end`` should be provided in this 252unit. For example, let's suppose ``addr_unit``, ``monitor_region_start`` and 253``monitor_region_end`` are set as ``1024``, ``0`` and ``10``, respectively. 254Then DAMON_RECLAIM will work for 10 KiB length of physical address range that 255starts from address zero (``[0 * 1024, 10 * 1024)`` in bytes). 256 257``bytes_reclaim_tried_regions`` and ``bytes_reclaimed_regions`` are also in 258this unit. For example, let's suppose values of ``addr_unit``, 259``bytes_reclaim_tried_regions`` and ``bytes_reclaimed_regions`` are ``1024``, 260``42``, and ``32``, respectively. Then it means DAMON_RECLAIM tried to reclaim 26142 KiB memory and successfully reclaimed 32 KiB memory in total. 262 263If unsure, use only the default value (``1``) and forget about this. 264 265skip_anon 266--------- 267 268Skip anonymous pages reclamation. 269 270If this parameter is set as ``Y``, DAMON_RECLAIM does not reclaim anonymous 271pages. By default, ``N``. 272 273 274kdamond_pid 275----------- 276 277PID of the DAMON thread. 278 279If DAMON_RECLAIM is enabled, this becomes the PID of the worker thread. Else, 280-1. 281 282nr_reclaim_tried_regions 283------------------------ 284 285Number of memory regions that tried to be reclaimed by DAMON_RECLAIM. 286 287bytes_reclaim_tried_regions 288--------------------------- 289 290Total bytes of memory regions that tried to be reclaimed by DAMON_RECLAIM. 291 292nr_reclaimed_regions 293-------------------- 294 295Number of memory regions that successfully be reclaimed by DAMON_RECLAIM. 296 297bytes_reclaimed_regions 298----------------------- 299 300Total bytes of memory regions that successfully be reclaimed by DAMON_RECLAIM. 301 302nr_quota_exceeds 303---------------- 304 305Number of times that the time/space quota limits have exceeded. 306 307Example 308======= 309 310Below runtime example commands make DAMON_RECLAIM to find memory regions that 311not accessed for 30 seconds or more and pages out. The reclamation is limited 312to be done only up to 1 GiB per second to avoid DAMON_RECLAIM consuming too 313much CPU time for the paging out operation. It also asks DAMON_RECLAIM to do 314nothing if the system's free memory rate is more than 50%, but start the real 315works if it becomes lower than 40%. If DAMON_RECLAIM doesn't make progress and 316therefore the free memory rate becomes lower than 20%, it asks DAMON_RECLAIM to 317do nothing again, so that we can fall back to the LRU-list based page 318granularity reclamation. :: 319 320 # cd /sys/module/damon_reclaim/parameters 321 # echo 30000000 > min_age 322 # echo $((1 * 1024 * 1024 * 1024)) > quota_sz 323 # echo 1000 > quota_reset_interval_ms 324 # echo 500 > wmarks_high 325 # echo 400 > wmarks_mid 326 # echo 200 > wmarks_low 327 # echo Y > enabled 328 329Note that this module (damon_reclaim) cannot run simultaneously with other 330DAMON-based special-purpose modules. Refer to :ref:`DAMON design special 331purpose modules exclusivity <damon_design_special_purpose_modules_exclusivity>` 332for more details. 333 334.. [1] https://research.google/pubs/pub48551/ 335.. [2] https://lwn.net/Articles/787611/ 336.. [3] https://www.kernel.org/doc/html/latest/mm/free_page_reporting.html 337