1.\" Copyright (c) 2018 Jeffrey Roberson <jeff@FreeBSD.org> 2.\" All rights reserved. 3.\" 4.\" Redistribution and use in source and binary forms, with or without 5.\" modification, are permitted provided that the following conditions 6.\" are met: 7.\" 1. Redistributions of source code must retain the above copyright 8.\" notice, this list of conditions and the following disclaimer. 9.\" 2. Redistributions in binary form must reproduce the above copyright 10.\" notice, this list of conditions and the following disclaimer in the 11.\" documentation and/or other materials provided with the distribution. 12.\" 13.\" THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' 14.\" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED 15.\" TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR 16.\" PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE 17.\" LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR 18.\" CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF 19.\" SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS 20.\" INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN 21.\" CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) 22.\" ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE 23.\" POSSIBILITY OF SUCH DAMAGE. 24.\" 25.\" $FreeBSD$ 26.\" 27.Dd October 20, 2018 28.Dt DOMAINSET 9 29.Os 30.Sh NAME 31.Nm domainset(9) 32.Nd domainset functions and operation 33.Sh SYNOPSIS 34.In sys/_domainset.h 35.In sys/domainset.h 36.\" 37.Bd -literal -offset indent 38struct domainset { 39 domainset_t ds_mask; 40 uint16_t ds_policy; 41 domainid_t ds_prefer; 42 ... 43}; 44.Ed 45.Pp 46.Fn DOMAINSET_RR 47.Fn DOMAINSET_PREF domain 48.Ft struct domainset * 49.Fn domainset_create "const struct domainset *key" 50.Ft int 51.Fn sysctl_handle_domainset "SYSCTL_HANDLER_ARGS" 52.Sh DESCRIPTION 53The 54.Nm 55API provides memory domain allocation policy for NUMA machines. 56Each 57.Vt domainset 58contains a bitmask of allowed domains, an integer policy, and an optional 59preferred domain. 60Together, these specify a search order for memory allocations as well as 61the ability to restrict threads and objects to a subset of available 62memory domains for system partitioning and resource management. 63.Pp 64Every thread in the system and optionally every 65.Vt vm_object_t , 66which is used to represent files and other memory sources, has 67a reference to a 68.Vt struct domainset . 69The domainset associated with the object is consulted first and the system 70falls back to the thread policy if none exists. 71.Pp 72The allocation policy has the following possible values: 73.Bl -tag -width "foo" 74.It Dv DOMAINSET_POLICY_ROUNDROBIN 75Memory is allocated from each domain in the mask in a round-robin fashion. 76This distributes bandwidth evenly among available domains. 77This policy can specify a single domain for a fixed allocation. 78.It Dv DOMAINSET_POLICY_FIRSTTOUCH 79Memory is allocated from the node that it is first accessed on. 80Allocation falls back to round-robin if the current domain is not in the 81allowed set or is out of memory. 82This policy optimizes for locality but may give pessimal results if the 83memory is accessed from many CPUs that are not in the local domain. 84.It Dv DOMAINSET_POLICY_PREFER 85Memory is allocated from the node in the 86.Vt prefer 87member. The preferred node must be set in the allowed mask. 88If the preferred node is out of memory the allocation falls back to 89round-robin among allowed sets. 90.It Dv DOMAINSET_POLICY_INTERLEAVE 91Memory is allocated in a striped fashion with multiple pages 92allocated to each domain in the set according to the offset within 93the object. 94The strip width is object dependent and may be as large as a 95super-page (2MB on amd64). 96This gives good distribution among memory domains while keeping system 97efficiency higher and is preferential to round-robin for general use. 98.El 99.Pp 100The 101.Fn DOMAINSET_RR 102and 103.Fn DOMAINSET_PREF 104provide pointers to global pre-defined policies for use when the 105desired policy is known at compile time. 106The 107.Fn domainset_create 108function takes a partially filled in domainset as a key and returns a 109valid domainset or NULL. 110It is critical that consumers not use domainsets that have not been 111returned by this function. 112.Vt 113domainset 114is an immutable type that is shared among all matching keys and must 115not be modified after return. 116.Pp 117The 118.Fn sysctl_handle_domainset 119function is provided as a convenience for modifying or viewing domainsets 120that are not accessible via 121.Xr cpuset 2 . 122It is intended for use with 123.Xr sysctl 9 . 124.Pp 125.Sh SEE ALSO 126.Xr cpuset 1 , 127.Xr cpuset 2 , 128.Xr cpuset_setdomain 2 , 129.Xr bitset 9 130.Sh HISTORY 131.In sys/domainset.h 132first appeared in 133.Fx 12.0 . 134