1.\" $FreeBSD$ 2.\" 3.Dd October 23, 2016 4.Dt LIBRSS 3 5.Os 6.Sh NAME 7.Nm librss 8.Nd Provide Receive-side scaling awareness to userland applications 9.Sh LIBRARY 10.Lb librss 11.Sh SYNOPSIS 12.In sys/param.h 13.In sys/cpuset.h 14.In librss.h 15.Ft struct rss_config * 16.Fn rss_config_get "void" 17.Ft void 18.Fn rss_config_free "struct rss_config *cfg" 19.Ft int 20.Fn rss_config_get_bucket_count "struct rss_config *cfg" 21.Ft int 22.Fn rss_get_bucket_cpuset "struct rss_config *rc" "rss_bucket_type_t btype" "int bucket" "cpuset_t *cs" 23.Ft int 24.Fn rss_set_bucket_rebalance_cb "rss_bucket_rebalance_cb_t *cb" "void *cbdata" 25.Ft int 26.Fn rss_sock_set_bindmulti "int fd" "int af" "int val" 27.Ft int 28.Fn rss_sock_set_rss_bucket "int fd" "int af" "int rss_bucket" 29.Ft int 30.Fn rss_sock_set_recvrss "int fd" "int af" "int val" 31.Sh DESCRIPTION 32The 33.Nm 34library and the functions it provides are used for both fetching 35the system RSS configuration and interacting with RSS aware 36sockets. 37.Pp 38Applications will typically call 39.Fn rss_config_get 40to fetch the current RSS configuration from the system and perform 41initial setup. 42This typically involves spawning worker threads, one per RSS bucket, 43and optionally binding them to the per-bucket CPU set. 44.Pp 45The 46.Vt rss_config 47struct is defined as: 48.Bd -literal 49struct rss_config { 50 int rss_ncpus; 51 int rss_nbuckets; 52 int rss_basecpu; 53 int *rss_bucket_map; 54}; 55.Ed 56.Pp 57Applications will typically use the 58.Fn rss_config_get_bucket_count 59function to fetch the number of RSS buckets, create one thread 60per RSS bucket for RSS aware work, then one RSS aware socket to receive 61UDP datagrams or TCP connections 62in each particular RSS bucket / thread. 63.Pp 64The 65.Fn rss_get_bucket_cpuset 66function sets the given cpuset up for the given 67RSS bucket and behaviour. 68Typically applications will wish to just query for 69.Vt RSS_BUCKET_TYPE_KERNEL_ALL 70unless they wish to potentially setup different 71worker threads for transmit and receive. 72.Pp 73The 74.Vt rss_bucket_type_t 75enum is defined as: 76.Bd -literal 77typedef enum { 78 RSS_BUCKET_TYPE_NONE = 0, 79 RSS_BUCKET_TYPE_KERNEL_ALL = 1, 80 RSS_BUCKET_TYPE_KERNEL_TX = 2, 81 RSS_BUCKET_TYPE_KERNEL_RX = 3, 82 RSS_BUCKET_TYPE_MAX = 3, 83} rss_bucket_type_t; 84.Ed 85.Pp 86The rebalance callback 87.Vt rss_bucket_rebalance_cb_t 88is defined as: 89.Bd -literal 90typedef void rss_bucket_rebalance_cb_t(void *arg); 91.Ed 92.Pp 93The 94.Fn rss_set_bucket_rebalance_cb 95function sets an optional callback that will be called if the kernel 96rebalances RSS buckets. 97This is intended as a future expansion to rebalance buckets rather than 98reprogram the RSS key, so typically the only work to be performed 99is to rebind worker threads to an updated cpuset. 100.Pp 101Once RSS setup is completed, 102.Fn rss_config_free 103is called to free the RSS configuration structure. 104.Pp 105To make a 106.Vt bind 107socket RSS aware, the 108.Fn rss_sock_set_bindmulti 109function is used to enable or disable per-RSS bucket 110behaviour. 111The socket filedescriptor, address family and enable flag 112.Vt val 113are passed in. 114.Pp 115If 116.Vt val 117is set to 1, the socket can be placed in an RSS bucket and will only accept 118datagrams (for UDP) or connections (for TCP) that are received for that 119RSS bucket. 120If set to 0, the socket is placed in the default PCB and will see 121datagrams/connections that are not initially consumed by a PCB aware 122socket. 123.Pp 124The 125.Fn rss_sock_set_rss_bucket 126function configures the RSS bucket which a socket belongs in. 127Note that TCP sockets created by 128.Xr accept 2 129will automatically be assigned to the RSS bucket. 130.Pp 131The 132.Fn rss_sock_set_recvrss 133function enables or disables receiving RSS related information 134as socket options in. 135.2 recvmsg 136calls. 137.Pp 138When enabled, UDP datagrams will have a message with the 139.Vt IP_RECVFLOWID 140option indicating the 32-bit receive flowid as a uint32_t, 141and the 142.Vt IP_RECVRSSBUCKETID 143option indicating the 32 bit RSS bucket id as a uint32_t. 144.Sh ERRORS 145The functions return either <0 or NULL as appropriate upon error. 146.Sh SEE ALSO 147.Xr PCBGROUP 9 148.Sh HISTORY 149The 150.Xr librss.3 151library first appeared in 152.Fx 11.0 . 153.Sh AUTHORS 154.An Adrian Chadd Aq Mt adrian@FreeBSD.org 155.Sh BUGS 156There is currently no kernel mechanism to rebalance the RSS bucket to CPU 157mapping, and so the callback mechanism is a no-op. 158