1.\" Copyright (c) 2018 Jeffrey Roberson <jeff@FreeBSD.org> 2.\" All rights reserved. 3.\" 4.\" Redistribution and use in source and binary forms, with or without 5.\" modification, are permitted provided that the following conditions 6.\" are met: 7.\" 1. Redistributions of source code must retain the above copyright 8.\" notice, this list of conditions and the following disclaimer. 9.\" 2. Redistributions in binary form must reproduce the above copyright 10.\" notice, this list of conditions and the following disclaimer in the 11.\" documentation and/or other materials provided with the distribution. 12.\" 13.\" THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' 14.\" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED 15.\" TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR 16.\" PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE 17.\" LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR 18.\" CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF 19.\" SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS 20.\" INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN 21.\" CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) 22.\" ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE 23.\" POSSIBILITY OF SUCH DAMAGE. 24.\" 25.\" $FreeBSD$ 26.\" 27.Dd April 14, 2021 28.Dt DOMAINSET 9 29.Os 30.Sh NAME 31.Nm domainset(9) 32.Nd domainset functions and operation 33.Sh SYNOPSIS 34.In sys/_domainset.h 35.In sys/domainset.h 36.\" 37.Bd -literal -offset indent 38struct domainset { 39 domainset_t ds_mask; 40 uint16_t ds_policy; 41 domainid_t ds_prefer; 42 ... 43}; 44.Ed 45.Pp 46.Ft struct domainset * 47.Fn DOMAINSET_FIXED domain 48.Ft struct domainset * 49.Fn DOMAINSET_FT 50.Ft struct domainset * 51.Fn DOMAINSET_IL 52.Ft struct domainset * 53.Fn DOMAINSET_RR 54.Ft struct domainset * 55.Fn DOMAINSET_PREF domain 56.Ft struct domainset * 57.Fn domainset_create "const struct domainset *key" 58.Ft int 59.Fn sysctl_handle_domainset "SYSCTL_HANDLER_ARGS" 60.Sh DESCRIPTION 61The 62.Nm 63API provides memory domain allocation policy for NUMA machines. 64Each 65.Vt domainset 66contains a bitmask of allowed domains, an integer policy, and an optional 67preferred domain. 68Together, these specify a search order for memory allocations as well as 69the ability to restrict threads and objects to a subset of available 70memory domains for system partitioning and resource management. 71.Pp 72Every thread in the system and optionally every 73.Vt vm_object_t , 74which is used to represent files and other memory sources, has 75a reference to a 76.Vt struct domainset . 77The domainset associated with the object is consulted first and the system 78falls back to the thread policy if none exists. 79.Pp 80The allocation policy has the following possible values: 81.Bl -tag -width "foo" 82.It Dv DOMAINSET_POLICY_ROUNDROBIN 83Memory is allocated from each domain in the mask in a round-robin fashion. 84This distributes bandwidth evenly among available domains. 85This policy can specify a single domain for a fixed allocation. 86.It Dv DOMAINSET_POLICY_FIRSTTOUCH 87Memory is allocated from the node that it is first accessed on. 88Allocation falls back to round-robin if the current domain is not in the 89allowed set or is out of memory. 90This policy optimizes for locality but may give pessimal results if the 91memory is accessed from many CPUs that are not in the local domain. 92.It Dv DOMAINSET_POLICY_PREFER 93Memory is allocated from the node in the 94.Vt prefer 95member. 96The preferred node must be set in the allowed mask. 97If the preferred node is out of memory the allocation falls back to 98round-robin among allowed sets. 99.It Dv DOMAINSET_POLICY_INTERLEAVE 100Memory is allocated in a striped fashion with multiple pages 101allocated to each domain in the set according to the offset within 102the object. 103The strip width is object dependent and may be as large as a 104super-page (2MB on amd64). 105This gives good distribution among memory domains while keeping system 106efficiency higher and is preferential to round-robin for general use. 107.El 108.Pp 109The 110.Fn DOMAINSET_FIXED , 111.Fn DOMAINSET_FT , 112.Fn DOMAINSET_IL , 113.Fn DOMAINSET_RR 114and 115.Fn DOMAINSET_PREF 116macros provide pointers to global pre-defined policies for use when the 117desired policy is known at compile time. 118.Fn DOMAINSET_FIXED 119is a policy which only permits allocations from the specified domain. 120.Fn DOMAINSET_FT 121is a policy which attempts to allocate memory local to the current CPU, 122falling back to a round-robin policy if the initial allocation fails. 123.Fn DOMAINSET_IL 124and 125.Fn DOMAINSET_RR 126provide round-robin selection among all domains in the system, corresponding 127to the 128.Dv DOMAINSET_POLICY_INTERLEAVE 129and 130.Dv DOMAINSET_POLICY_ROUNDROBIN 131policies, respectively. 132The 133.Fn DOMAINSET_PREF 134policies attempt allocation from the specified domain, but unlike 135.Fn DOMAINSET_FIXED 136will fall back to other domains to satisfy the request. 137These policies should be used in preference to 138.Fn DOMAINSET_FIXED 139to avoid blocking indefinitely on a 140.Dv M_WAITOK 141request. 142The 143.Fn domainset_create 144function takes a partially filled in domainset as a key and returns a 145valid domainset or NULL. 146It is critical that consumers not use domainsets that have not been 147returned by this function. 148.Vt domainset 149is an immutable type that is shared among all matching keys and must 150not be modified after return. 151.Pp 152The 153.Fn sysctl_handle_domainset 154function is provided as a convenience for modifying or viewing domainsets 155that are not accessible via 156.Xr cpuset 2 . 157It is intended for use with 158.Xr sysctl 9 . 159.Sh SEE ALSO 160.Xr cpuset 1 , 161.Xr cpuset 2 , 162.Xr cpuset_setdomain 2 , 163.Xr bitset 9 164.Sh HISTORY 165.In sys/domainset.h 166first appeared in 167.Fx 12.0 . 168