1*e5818a53SJeff Roberson.\" Copyright (c) 2018 Jeffrey Roberson <jeff@FreeBSD.org> 2*e5818a53SJeff Roberson.\" All rights reserved. 3*e5818a53SJeff Roberson.\" 4*e5818a53SJeff Roberson.\" Redistribution and use in source and binary forms, with or without 5*e5818a53SJeff Roberson.\" modification, are permitted provided that the following conditions 6*e5818a53SJeff Roberson.\" are met: 7*e5818a53SJeff Roberson.\" 1. Redistributions of source code must retain the above copyright 8*e5818a53SJeff Roberson.\" notice, this list of conditions and the following disclaimer. 9*e5818a53SJeff Roberson.\" 2. Redistributions in binary form must reproduce the above copyright 10*e5818a53SJeff Roberson.\" notice, this list of conditions and the following disclaimer in the 11*e5818a53SJeff Roberson.\" documentation and/or other materials provided with the distribution. 12*e5818a53SJeff Roberson.\" 13*e5818a53SJeff Roberson.\" THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' 14*e5818a53SJeff Roberson.\" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED 15*e5818a53SJeff Roberson.\" TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR 16*e5818a53SJeff Roberson.\" PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE 17*e5818a53SJeff Roberson.\" LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR 18*e5818a53SJeff Roberson.\" CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF 19*e5818a53SJeff Roberson.\" SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS 20*e5818a53SJeff Roberson.\" INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN 21*e5818a53SJeff Roberson.\" CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) 22*e5818a53SJeff Roberson.\" ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE 23*e5818a53SJeff Roberson.\" POSSIBILITY OF SUCH DAMAGE. 24*e5818a53SJeff Roberson.\" 25*e5818a53SJeff Roberson.\" $FreeBSD$ 26*e5818a53SJeff Roberson.\" 27*e5818a53SJeff Roberson.Dd March 24, 2018 28*e5818a53SJeff Roberson.Dt DOMAINSET 9 29*e5818a53SJeff Roberson.Os 30*e5818a53SJeff Roberson.Sh NAME 31*e5818a53SJeff Roberson.Nm domainset(9) 32*e5818a53SJeff Roberson\(em 33*e5818a53SJeff Roberson.Nm domainset_create , 34*e5818a53SJeff Roberson.Nm sysctl_handle_domainset . 35*e5818a53SJeff Roberson.Nd domainset functions and operation 36*e5818a53SJeff Roberson.Sh SYNOPSIS 37*e5818a53SJeff Roberson.In sys/_domainset.h 38*e5818a53SJeff Roberson.In sys/domainset.h 39*e5818a53SJeff Roberson.\" 40*e5818a53SJeff Roberson.Bd -literal -offset indent 41*e5818a53SJeff Robersonstruct domainset { 42*e5818a53SJeff Roberson domainset_t ds_mask; 43*e5818a53SJeff Roberson uint16_t ds_policy; 44*e5818a53SJeff Roberson domainid_t ds_prefer; 45*e5818a53SJeff Roberson ... 46*e5818a53SJeff Roberson}; 47*e5818a53SJeff Roberson.Ed 48*e5818a53SJeff Roberson.Pp 49*e5818a53SJeff Roberson.Ft struct domainset * 50*e5818a53SJeff Roberson.Fn domainset_create "const struct domainset *key" 51*e5818a53SJeff Roberson.Ft int 52*e5818a53SJeff Roberson.Fn sysctl_handle_domainset "SYSCTL_HANDLER_ARGS" 53*e5818a53SJeff Roberson.Sh DESCRIPTION 54*e5818a53SJeff RobersonThe 55*e5818a53SJeff Roberson.Nm 56*e5818a53SJeff RobersonAPI provides memory domain allocation policy for NUMA machines. 57*e5818a53SJeff RobersonEach 58*e5818a53SJeff Roberson.Vt domainset 59*e5818a53SJeff Robersoncontains a bitmask of allowed domains, an integer policy, and an optional 60*e5818a53SJeff Robersonpreferred domain. 61*e5818a53SJeff RobersonTogether, these specify a search order for memory allocations as well as 62*e5818a53SJeff Robersonthe ability to restrict threads and objects to a subset of available 63*e5818a53SJeff Robersonmemory domains for system partitioning and resource management. 64*e5818a53SJeff Roberson.Pp 65*e5818a53SJeff RobersonEvery thread in the system and optionally every 66*e5818a53SJeff Roberson.Vt vm_object_t , 67*e5818a53SJeff Robersonwhich is used to represent files and other memory sources, has 68*e5818a53SJeff Robersona reference to a 69*e5818a53SJeff Roberson.Vt struct domainset . 70*e5818a53SJeff RobersonThe domainset associated with the object is consulted first and the system 71*e5818a53SJeff Robersonfalls back to the thread policy if none exists. 72*e5818a53SJeff Roberson.Pp 73*e5818a53SJeff RobersonThe allocation policy has the following possible values: 74*e5818a53SJeff Roberson.Bl -tag -width "foo" 75*e5818a53SJeff Roberson.It Dv DOMAINSET_POLICY_ROUNDROBIN 76*e5818a53SJeff RobersonMemory is allocated from each domain in the mask in a round-robin fashion. 77*e5818a53SJeff RobersonThis distributes bandwidth evenly among available domains. 78*e5818a53SJeff RobersonThis policy can specify a single domain for a fixed allocation. 79*e5818a53SJeff Roberson.It Dv DOMAINSET_POLICY_FIRSTTOUCH 80*e5818a53SJeff RobersonMemory is allocated from the node that it is first accessed on. 81*e5818a53SJeff RobersonAllocation falls back to round-robin if the current domain is not in the 82*e5818a53SJeff Robersonallowed set or is out of memory. 83*e5818a53SJeff RobersonThis policy optimizes for locality but may give pessimal results if the 84*e5818a53SJeff Robersonmemory is accessed from many CPUs that are not in the local domain. 85*e5818a53SJeff Roberson.It Dv DOMAINSET_POLICY_PREFER 86*e5818a53SJeff RobersonMemory is allocated from the node in the 87*e5818a53SJeff Roberson.Vt prefer 88*e5818a53SJeff Robersonmember. The preferred node must be set in the allowed mask. 89*e5818a53SJeff RobersonIf the preferred node is out of memory the allocation falls back to 90*e5818a53SJeff Robersonround-robin among allowed sets. 91*e5818a53SJeff Roberson.It Dv DOMAINSET_POLICY_INTERLEAVE 92*e5818a53SJeff RobersonMemory is allocated in a striped fashion with multiple pages 93*e5818a53SJeff Robersonallocated to each domain in the set according to the offset within 94*e5818a53SJeff Robersonthe object. 95*e5818a53SJeff RobersonThe strip width is object dependent and may be as large as a 96*e5818a53SJeff Robersonsuper-page (2MB on amd64). 97*e5818a53SJeff RobersonThis gives good distribution among memory domains while keeping system 98*e5818a53SJeff Robersonefficiency higher and is preferential to round-robin for general use. 99*e5818a53SJeff Roberson.El 100*e5818a53SJeff Roberson.Pp 101*e5818a53SJeff RobersonThe 102*e5818a53SJeff Roberson.Fn domainset_create 103*e5818a53SJeff Robersonfunction takes a partially filled in domainset as a key and returns a 104*e5818a53SJeff Robersonvalid domainset or NULL. 105*e5818a53SJeff RobersonIt is critical that consumers not use domainsets that have not been 106*e5818a53SJeff Robersonreturned by this function. 107*e5818a53SJeff Roberson.Vt 108*e5818a53SJeff Robersondomainset 109*e5818a53SJeff Robersonis an immutable type that is shared among all matching keys and must 110*e5818a53SJeff Robersonnot be modified after return. 111*e5818a53SJeff Roberson.Pp 112*e5818a53SJeff RobersonThe 113*e5818a53SJeff Roberson.Fn sysctl_handle_domainset 114*e5818a53SJeff Robersonfunction is provided as a convenience for modifying or viewing domainsets 115*e5818a53SJeff Robersonthat are not accessible via 116*e5818a53SJeff Roberson.Xr cpuset 2 . 117*e5818a53SJeff RobersonIt is intended for use with 118*e5818a53SJeff Roberson.Xr sysctl 9 . 119*e5818a53SJeff Roberson.Pp 120*e5818a53SJeff Roberson.Sh SEE ALSO 121*e5818a53SJeff Roberson.Xr cpuset 1 , 122*e5818a53SJeff Roberson.Xr cpuset 2 , 123*e5818a53SJeff Roberson.Xr cpuset_setdomain 2 , 124*e5818a53SJeff Roberson.Xr bitset 9 125*e5818a53SJeff Roberson.Sh HISTORY 126*e5818a53SJeff Roberson.In sys/domainset.h 127*e5818a53SJeff Robersonfirst appeared in 128*e5818a53SJeff Roberson.Fx 12.0 . 129