1 e5818a53SJeff Roberson.\" Copyright (c) 2018 Jeffrey Roberson <jeff@FreeBSD.org> 2 e5818a53SJeff Roberson.\" All rights reserved. 3 e5818a53SJeff Roberson.\" 4 e5818a53SJeff Roberson.\" Redistribution and use in source and binary forms, with or without 5 e5818a53SJeff Roberson.\" modification, are permitted provided that the following conditions 6 e5818a53SJeff Roberson.\" are met: 7 e5818a53SJeff Roberson.\" 1. Redistributions of source code must retain the above copyright 8 e5818a53SJeff Roberson.\" notice, this list of conditions and the following disclaimer. 9 e5818a53SJeff Roberson.\" 2. Redistributions in binary form must reproduce the above copyright 10 e5818a53SJeff Roberson.\" notice, this list of conditions and the following disclaimer in the 11 e5818a53SJeff Roberson.\" documentation and/or other materials provided with the distribution. 12 e5818a53SJeff Roberson.\" 13 e5818a53SJeff Roberson.\" THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' 14 e5818a53SJeff Roberson.\" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED 15 e5818a53SJeff Roberson.\" TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR 16 e5818a53SJeff Roberson.\" PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE 17 e5818a53SJeff Roberson.\" LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR 18 e5818a53SJeff Roberson.\" CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF 19 e5818a53SJeff Roberson.\" SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS 20 e5818a53SJeff Roberson.\" INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN 21 e5818a53SJeff Roberson.\" CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) 22 e5818a53SJeff Roberson.\" ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE 23 e5818a53SJeff Roberson.\" POSSIBILITY OF SUCH DAMAGE. 24 e5818a53SJeff Roberson.\" 25 *29bb6c19SMark Johnston.Dd April 14, 2021 26 e5818a53SJeff Roberson.Dt DOMAINSET 9 27 e5818a53SJeff Roberson.Os 28 e5818a53SJeff Roberson.Sh NAME 29 e5818a53SJeff Roberson.Nm domainset(9) 30 e5818a53SJeff Roberson.Nd domainset functions and operation 31 e5818a53SJeff Roberson.Sh SYNOPSIS 32 e5818a53SJeff Roberson.In sys/_domainset.h 33 e5818a53SJeff Roberson.In sys/domainset.h 34 e5818a53SJeff Roberson.\" 35 e5818a53SJeff Roberson.Bd -literal -offset indent 36 e5818a53SJeff Robersonstruct domainset { 37 e5818a53SJeff Roberson domainset_t ds_mask; 38 e5818a53SJeff Roberson uint16_t ds_policy; 39 e5818a53SJeff Roberson domainid_t ds_prefer; 40 e5818a53SJeff Roberson ... 41 e5818a53SJeff Roberson}; 42 e5818a53SJeff Roberson.Ed 43 e5818a53SJeff Roberson.Pp 44 9978bd99SMark Johnston.Ft struct domainset * 45 9978bd99SMark Johnston.Fn DOMAINSET_FIXED domain 46 9978bd99SMark Johnston.Ft struct domainset * 47 *29bb6c19SMark Johnston.Fn DOMAINSET_FT 48 *29bb6c19SMark Johnston.Ft struct domainset * 49 *29bb6c19SMark Johnston.Fn DOMAINSET_IL 50 *29bb6c19SMark Johnston.Ft struct domainset * 51 662e7fa8SMark Johnston.Fn DOMAINSET_RR 52 9978bd99SMark Johnston.Ft struct domainset * 53 662e7fa8SMark Johnston.Fn DOMAINSET_PREF domain 54 e5818a53SJeff Roberson.Ft struct domainset * 55 e5818a53SJeff Roberson.Fn domainset_create "const struct domainset *key" 56 e5818a53SJeff Roberson.Ft int 57 e5818a53SJeff Roberson.Fn sysctl_handle_domainset "SYSCTL_HANDLER_ARGS" 58 e5818a53SJeff Roberson.Sh DESCRIPTION 59 e5818a53SJeff RobersonThe 60 e5818a53SJeff Roberson.Nm 61 e5818a53SJeff RobersonAPI provides memory domain allocation policy for NUMA machines. 62 e5818a53SJeff RobersonEach 63 e5818a53SJeff Roberson.Vt domainset 64 e5818a53SJeff Robersoncontains a bitmask of allowed domains, an integer policy, and an optional 65 e5818a53SJeff Robersonpreferred domain. 66 e5818a53SJeff RobersonTogether, these specify a search order for memory allocations as well as 67 e5818a53SJeff Robersonthe ability to restrict threads and objects to a subset of available 68 e5818a53SJeff Robersonmemory domains for system partitioning and resource management. 69 e5818a53SJeff Roberson.Pp 70 e5818a53SJeff RobersonEvery thread in the system and optionally every 71 e5818a53SJeff Roberson.Vt vm_object_t , 72 e5818a53SJeff Robersonwhich is used to represent files and other memory sources, has 73 e5818a53SJeff Robersona reference to a 74 e5818a53SJeff Roberson.Vt struct domainset . 75 e5818a53SJeff RobersonThe domainset associated with the object is consulted first and the system 76 e5818a53SJeff Robersonfalls back to the thread policy if none exists. 77 e5818a53SJeff Roberson.Pp 78 e5818a53SJeff RobersonThe allocation policy has the following possible values: 79 e5818a53SJeff Roberson.Bl -tag -width "foo" 80 e5818a53SJeff Roberson.It Dv DOMAINSET_POLICY_ROUNDROBIN 81 e5818a53SJeff RobersonMemory is allocated from each domain in the mask in a round-robin fashion. 82 e5818a53SJeff RobersonThis distributes bandwidth evenly among available domains. 83 e5818a53SJeff RobersonThis policy can specify a single domain for a fixed allocation. 84 e5818a53SJeff Roberson.It Dv DOMAINSET_POLICY_FIRSTTOUCH 85 e5818a53SJeff RobersonMemory is allocated from the node that it is first accessed on. 86 e5818a53SJeff RobersonAllocation falls back to round-robin if the current domain is not in the 87 e5818a53SJeff Robersonallowed set or is out of memory. 88 e5818a53SJeff RobersonThis policy optimizes for locality but may give pessimal results if the 89 e5818a53SJeff Robersonmemory is accessed from many CPUs that are not in the local domain. 90 e5818a53SJeff Roberson.It Dv DOMAINSET_POLICY_PREFER 91 e5818a53SJeff RobersonMemory is allocated from the node in the 92 e5818a53SJeff Roberson.Vt prefer 93 6f563a6fSGordon Berglingmember. 94 6f563a6fSGordon BerglingThe preferred node must be set in the allowed mask. 95 e5818a53SJeff RobersonIf the preferred node is out of memory the allocation falls back to 96 e5818a53SJeff Robersonround-robin among allowed sets. 97 e5818a53SJeff Roberson.It Dv DOMAINSET_POLICY_INTERLEAVE 98 e5818a53SJeff RobersonMemory is allocated in a striped fashion with multiple pages 99 e5818a53SJeff Robersonallocated to each domain in the set according to the offset within 100 e5818a53SJeff Robersonthe object. 101 e5818a53SJeff RobersonThe strip width is object dependent and may be as large as a 102 e5818a53SJeff Robersonsuper-page (2MB on amd64). 103 e5818a53SJeff RobersonThis gives good distribution among memory domains while keeping system 104 e5818a53SJeff Robersonefficiency higher and is preferential to round-robin for general use. 105 e5818a53SJeff Roberson.El 106 e5818a53SJeff Roberson.Pp 107 e5818a53SJeff RobersonThe 108 9978bd99SMark Johnston.Fn DOMAINSET_FIXED , 109 *29bb6c19SMark Johnston.Fn DOMAINSET_FT , 110 *29bb6c19SMark Johnston.Fn DOMAINSET_IL , 111 662e7fa8SMark Johnston.Fn DOMAINSET_RR 112 662e7fa8SMark Johnstonand 113 662e7fa8SMark Johnston.Fn DOMAINSET_PREF 114 9978bd99SMark Johnstonmacros provide pointers to global pre-defined policies for use when the 115 662e7fa8SMark Johnstondesired policy is known at compile time. 116 9978bd99SMark Johnston.Fn DOMAINSET_FIXED 117 9978bd99SMark Johnstonis a policy which only permits allocations from the specified domain. 118 *29bb6c19SMark Johnston.Fn DOMAINSET_FT 119 *29bb6c19SMark Johnstonis a policy which attempts to allocate memory local to the current CPU, 120 *29bb6c19SMark Johnstonfalling back to a round-robin policy if the initial allocation fails. 121 *29bb6c19SMark Johnston.Fn DOMAINSET_IL 122 *29bb6c19SMark Johnstonand 123 9978bd99SMark Johnston.Fn DOMAINSET_RR 124 *29bb6c19SMark Johnstonprovide round-robin selection among all domains in the system, corresponding 125 *29bb6c19SMark Johnstonto the 126 *29bb6c19SMark Johnston.Dv DOMAINSET_POLICY_INTERLEAVE 127 *29bb6c19SMark Johnstonand 128 *29bb6c19SMark Johnston.Dv DOMAINSET_POLICY_ROUNDROBIN 129 *29bb6c19SMark Johnstonpolicies, respectively. 130 9978bd99SMark JohnstonThe 131 9978bd99SMark Johnston.Fn DOMAINSET_PREF 132 9978bd99SMark Johnstonpolicies attempt allocation from the specified domain, but unlike 133 9978bd99SMark Johnston.Fn DOMAINSET_FIXED 134 9978bd99SMark Johnstonwill fall back to other domains to satisfy the request. 135 9978bd99SMark JohnstonThese policies should be used in preference to 136 9978bd99SMark Johnston.Fn DOMAINSET_FIXED 137 9978bd99SMark Johnstonto avoid blocking indefinitely on a 138 9978bd99SMark Johnston.Dv M_WAITOK 139 9978bd99SMark Johnstonrequest. 140 662e7fa8SMark JohnstonThe 141 e5818a53SJeff Roberson.Fn domainset_create 142 e5818a53SJeff Robersonfunction takes a partially filled in domainset as a key and returns a 143 e5818a53SJeff Robersonvalid domainset or NULL. 144 e5818a53SJeff RobersonIt is critical that consumers not use domainsets that have not been 145 e5818a53SJeff Robersonreturned by this function. 146 6f563a6fSGordon Bergling.Vt domainset 147 e5818a53SJeff Robersonis an immutable type that is shared among all matching keys and must 148 e5818a53SJeff Robersonnot be modified after return. 149 e5818a53SJeff Roberson.Pp 150 e5818a53SJeff RobersonThe 151 e5818a53SJeff Roberson.Fn sysctl_handle_domainset 152 e5818a53SJeff Robersonfunction is provided as a convenience for modifying or viewing domainsets 153 e5818a53SJeff Robersonthat are not accessible via 154 e5818a53SJeff Roberson.Xr cpuset 2 . 155 e5818a53SJeff RobersonIt is intended for use with 156 e5818a53SJeff Roberson.Xr sysctl 9 . 157 e5818a53SJeff Roberson.Sh SEE ALSO 158 e5818a53SJeff Roberson.Xr cpuset 1 , 159 e5818a53SJeff Roberson.Xr cpuset 2 , 160 e5818a53SJeff Roberson.Xr cpuset_setdomain 2 , 161 e5818a53SJeff Roberson.Xr bitset 9 162 e5818a53SJeff Roberson.Sh HISTORY 163 e5818a53SJeff Roberson.In sys/domainset.h 164 e5818a53SJeff Robersonfirst appeared in 165 e5818a53SJeff Roberson.Fx 12.0 . 166