1.\" Copyright (c) 2015 Adrian Chadd <adrian@FreeBSD.org> 2.\" All rights reserved. 3.\" 4.\" Redistribution and use in source and binary forms, with or without 5.\" modification, are permitted provided that the following conditions 6.\" are met: 7.\" 1. Redistributions of source code must retain the above copyright 8.\" notice, this list of conditions and the following disclaimer. 9.\" 2. Redistributions in binary form must reproduce the above copyright 10.\" notice, this list of conditions and the following disclaimer in the 11.\" documentation and/or other materials provided with the distribution. 12.\" 13.\" THIS SOFTWARE IS PROVIDED BY THE AUTHORS AND CONTRIBUTORS ``AS IS'' AND 14.\" ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 15.\" IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE 16.\" ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHORS OR CONTRIBUTORS BE LIABLE 17.\" FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL 18.\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS 19.\" OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) 20.\" HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT 21.\" LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY 22.\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF 23.\" SUCH DAMAGE. 24.\" 25.\" $FreeBSD$ 26.\" 27.Dd May 10, 2015 28.Dt NUMA 4 29.Os 30.Sh NAME 31.Nm NUMA 32.Nd Non-Uniform Memory Access 33.Sh SYNOPSIS 34.Cd options SMP 35.Cd options MAXMEMDOM=16 36.Pp 37.In sys/numa.h 38.In sys/cpuset.h 39.In sys/bus.h 40.Sh DESCRIPTION 41Non-Uniform Memory Access is a computer architecture design which 42involves unequal costs between processors, memory and IO devices 43in a given system. 44.Pp 45In a 46.Nm 47architecture, the latency to access specific memory or IO devices 48depends upon which processor the memory or device is attached to. 49Accessing memory local to a processor is faster than accessing memory 50that is connected to one of the other processors. 51.Pp 52.Nm 53is enabled when the 54.Cd MAXMEMDOM 55option is used in a kernel configuration 56file and is set to a value greater than 1. 57.Pp 58Thread and process 59.Nm 60policies are controlled with the 61.Xr numa_setaffinity 2 62and 63.Xr numa_getaffinity 2 64syscalls. 65.Pp 66The 67.Xr numactl 1 68tool is available for starting processes with a non-default 69policy, or to change the policy of an existing thread or process. 70.Pp 71Systems with non-uniform access to I/O devices may mark those devices 72with the local VM domain identifier. 73Drivers can find out their local domain information by calling 74.Xr bus_get_domain 9 . 75.Ss MIB Variables 76The operation of 77.Nm 78is controlled and exposes information with these 79.Xr sysctl 8 80MIB variables: 81.Pp 82.Bl -tag -width indent -compact 83.It Va vm.ndomains 84The number of VM domains which have been detected. 85.Pp 86.It Va vm.default_policy 87The default VM domain allocation policy. 88Defaults to "first-touch-rr". 89The valid values are "first-touch", "first-touch-rr", 90"rr", where "rr" is a short-hand for "round-robin." 91See 92.Xr numa_setaffinity 2 93for more information about the available policies. 94.Pp 95.It Va vm.phys_locality 96A table indicating the relative cost of each VM domain to each other. 97A value of 10 indicates equal cost. 98A value of -1 means the locality map is not available or no 99locality information is available. 100.Pp 101.It Va vm.phys_segs 102The map of physical memory, grouped by VM domain. 103.El 104.Sh IMPLEMENTATION NOTES 105The current 106.Nm 107implementation is VM-focused. 108The hardware 109.Nm 110domains are mapped into a contiguous, non-sparse 111VM domain space, starting from 0. 112Thus, VM domain information (for example, the domain identifier) is not 113necessarily the same as is found in the hardware specific information. 114.Pp 115The 116.Nm 117allocation policies are implemented as a policy and iterator in 118.Pa sys/vm/vm_domain.c 119and 120.Pa sys/vm/vm_domain.h . 121Policy information is available in both struct thread and struct proc. 122Processes inherit 123.Nm 124policy from parent processes and threads inherit 125.Nm 126policy from parent threads. 127Note that threads do not explicitly inherit their 128.Nm 129policy from processes. 130Instead, if no thread policy is set, the system 131will fall back to the process policy. 132.Pp 133For now, 134.Nm 135domain policies only influence physical page allocation in 136.Pa sys/vm/vm_phys.c . 137This is useful for userland memory allocation, but not for kernel 138and driver memory allocation. 139These features will be implemented in future work. 140.Sh SEE ALSO 141.Xr numactl 1 , 142.Xr numa_getaffinity 2 , 143.Xr numa_setaffinity 2 , 144.Xr bus_get_domain 9 145.Sh HISTORY 146.Nm 147first appeared in 148.Fx 9.0 149as a first-touch allocation policy with a fail-over to round-robin allocation 150and was not configurable. 151It was then modified in 152.Fx 10.0 153to implement a round-robin allocation policy and was also not configurable. 154.Pp 155The 156.Xr numa_getaffinity 2 157and 158.Xr numa_setaffinity 2 159syscalls first appeared in 160.Fx 11.0 . 161.Pp 162The 163.Xr numactl 1 164tool first appeared in 165.Fx 11.0 . 166.Sh AUTHORS 167This manual page written by 168.An Adrian Chadd Aq Mt adrian@FreeBSD.org . 169.Sh NOTES 170No statistics are kept to indicate how often 171.Nm 172allocation policies succeed or fail. 173