1.\"- 2.\" Copyright (c) 2013 Gleb Smirnoff <glebius@FreeBSD.org> 3.\" All rights reserved. 4.\" 5.\" Redistribution and use in source and binary forms, with or without 6.\" modification, are permitted provided that the following conditions 7.\" are met: 8.\" 1. Redistributions of source code must retain the above copyright 9.\" notice, this list of conditions and the following disclaimer. 10.\" 2. Redistributions in binary form must reproduce the above copyright 11.\" notice, this list of conditions and the following disclaimer in the 12.\" documentation and/or other materials provided with the distribution. 13.\" 14.\" THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND 15.\" ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 16.\" IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE 17.\" ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE 18.\" FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL 19.\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS 20.\" OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) 21.\" HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT 22.\" LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY 23.\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF 24.\" SUCH DAMAGE. 25.\" 26.Dd June 19, 2025 27.Dt COUNTER 9 28.Os 29.Sh NAME 30.Nm counter 31.Nd "SMP-friendly kernel counter implementation" 32.Sh SYNOPSIS 33.In sys/types.h 34.In sys/systm.h 35.In sys/counter.h 36.Ft counter_u64_t 37.Fn counter_u64_alloc "int wait" 38.Ft void 39.Fn counter_u64_free "counter_u64_t c" 40.Ft void 41.Fn counter_u64_add "counter_u64_t c" "int64_t v" 42.Ft void 43.Fn counter_enter 44.Ft void 45.Fn counter_exit 46.Ft void 47.Fn counter_u64_add_protected "counter_u64_t c" "int64_t v" 48.Ft uint64_t 49.Fn counter_u64_fetch "counter_u64_t c" 50.Ft void 51.Fn counter_u64_zero "counter_u64_t c" 52.Ft struct counter_rate * 53.Fn counter_rate_alloc "int flags" "int period" 54.Ft int64_t 55.Fn counter_ratecheck "struct counter_rate *cr" "int64_t limit" 56.Ft uint64_t 57.Fn counter_rate_get "struct counter_rate *cr" 58.Ft void 59.Fn counter_rate_free "struct counter_rate *cr" 60.Fn COUNTER_U64_SYSINIT "counter_u64_t c" 61.Fn COUNTER_U64_DEFINE_EARLY "counter_u64_t c" 62.In sys/sysctl.h 63.Fn SYSCTL_COUNTER_U64 parent nbr name access ptr descr 64.Fn SYSCTL_ADD_COUNTER_U64 ctx parent nbr name access ptr descr 65.Fn SYSCTL_COUNTER_U64_ARRAY parent nbr name access ptr len descr 66.Fn SYSCTL_ADD_COUNTER_U64_ARRAY ctx parent nbr name access ptr len descr 67.Sh DESCRIPTION 68.Nm 69is a generic facility to create counters 70that can be utilized for any purpose (such as collecting statistical 71data). 72A 73.Nm 74is guaranteed to be lossless when several kernel threads do simultaneous 75updates. 76However, 77.Nm 78does not block the calling thread, 79also no 80.Xr atomic 9 81operations are used for the update, therefore the counters 82can be used in any non-interrupt context. 83Moreover, 84.Nm 85has special optimisations for SMP environments, making 86.Nm 87update faster than simple arithmetic on the global variable. 88Thus 89.Nm 90is considered suitable for accounting in the performance-critical 91code paths. 92.Bl -tag -width indent 93.It Fn counter_u64_alloc wait 94Allocate a new 64-bit unsigned counter. 95The 96.Fa wait 97argument is the 98.Xr malloc 9 99wait flag, should be either 100.Va M_NOWAIT 101or 102.Va M_WAITOK . 103If 104.Va M_NOWAIT 105is specified the operation may fail and return 106.Dv NULL . 107.It Fn counter_u64_free c 108Free the previously allocated counter 109.Fa c . 110It is safe to pass 111.Dv NULL . 112.It Fn counter_u64_add c v 113Add 114.Fa v 115to 116.Fa c . 117The KPI does not guarantee any protection from wraparound. 118.It Fn counter_enter 119Enter mode that would allow the safe update of several counters via 120.Fn counter_u64_add_protected . 121On some machines this expands to 122.Xr critical 9 123section, while on other is a nop. 124See 125.Sx IMPLEMENTATION DETAILS . 126.It Fn counter_exit 127Exit mode for updating several counters. 128.It Fn counter_u64_add_protected c v 129Same as 130.Fn counter_u64_add , 131but should be preceded by 132.Fn counter_enter . 133.It Fn counter_u64_fetch c 134Take a snapshot of counter 135.Fa c . 136The data obtained is not guaranteed to reflect the real cumulative 137value for any moment. 138.It Fn counter_u64_zero c 139Clear the counter 140.Fa c 141and set it to zero. 142.It Fn counter_rate_alloc flags period 143Allocate a new struct counter_rate. 144.Fa flags 145is passed to 146.Xr malloc 9 . 147.Fa period 148is the time over which the rate is checked. 149.It Fn counter_ratecheck cr limit 150The function is a multiprocessor-friendly version of 151.Fn ppsratecheck 152which uses 153.Nm 154internally. 155Returns non-negative value if the rate is not yet reached during the current 156period, and a negative value otherwise. 157If the limit was reached during the previous period, but was just reset back 158to zero, then 159.Fn counter_ratecheck 160returns number of events since previous reset. 161.It Fn counter_rate_get cr 162The number of hits to this check within the current period. 163.It Fn counter_rate_free cr 164Free the 165.Fa cr 166counter. 167.It Fn COUNTER_U64_SYSINIT c 168Define a 169.Xr SYSINIT 9 170initializer for the global counter 171.Fa c . 172.It Fn COUNTER_U64_DEFINE_EARLY c 173Define and initialize a global counter 174.Fa c . 175It is always safe to increment 176.Fa c , 177though updates prior to the 178.Dv SI_SUB_COUNTER 179.Xr SYSINIT 9 180event are lost. 181.It Fn SYSCTL_COUNTER_U64 parent nbr name access ptr descr 182Declare a static 183.Xr sysctl 9 184oid that would represent a 185.Nm . 186The 187.Fa ptr 188argument should be a pointer to allocated 189.Vt counter_u64_t . 190A read of the oid returns value obtained through 191.Fn counter_u64_fetch . 192Any write to the oid zeroes it. 193.It Fn SYSCTL_ADD_COUNTER_U64 ctx parent nbr name access ptr descr 194Create a 195.Xr sysctl 9 196oid that would represent a 197.Nm . 198The 199.Fa ptr 200argument should be a pointer to allocated 201.Vt counter_u64_t . 202A read of the oid returns value obtained through 203.Fn counter_u64_fetch . 204Any write to the oid zeroes it. 205.It Fn SYSCTL_COUNTER_U64_ARRAY parent nbr name access ptr len descr 206Declare a static 207.Xr sysctl 9 208oid that would represent an array of 209.Nm . 210The 211.Fa ptr 212argument should be a pointer to allocated array of 213.Vt counter_u64_t's . 214The 215.Fa len 216argument should specify number of elements in the array. 217A read of the oid returns len-sized array of 218.Vt uint64_t 219values obtained through 220.Fn counter_u64_fetch . 221Any write to the oid zeroes all array elements. 222.It Fn SYSCTL_ADD_COUNTER_U64_ARRAY ctx parent nbr name access ptr len descr 223Create a 224.Xr sysctl 9 225oid that would represent an array of 226.Nm . 227The 228.Fa ptr 229argument should be a pointer to allocated array of 230.Vt counter_u64_t's . 231The 232.Fa len 233argument should specify number of elements in the array. 234A read of the oid returns len-sized array of 235.Vt uint64_t 236values obtained through 237.Fn counter_u64_fetch . 238Any write to the oid zeroes all array elements. 239.El 240.Sh IMPLEMENTATION DETAILS 241On all architectures 242.Nm 243is implemented using per-CPU data fields that are specially aligned 244in memory, to avoid inter-CPU bus traffic due to shared use 245of the variables between CPUs. 246These are allocated using 247.Va UMA_ZONE_PCPU 248.Xr uma 9 249zone. 250The update operation only touches the field that is private to current CPU. 251Fetch operation loops through all per-CPU fields and obtains a snapshot 252sum of all fields. 253.Pp 254On amd64 a 255.Nm counter 256update is implemented as a single instruction without lock semantics, 257operating on the private data for the current CPU, 258which is safe against preemption and interrupts. 259.Pp 260On i386 architecture, when machine supports the cmpxchg8 instruction, 261this instruction is used. 262The multi-instruction sequence provides the same guarantees as the 263amd64 single-instruction implementation. 264.Pp 265On some architectures updating a counter require a 266.Xr critical 9 267section. 268.Sh EXAMPLES 269The following example creates a static counter array exported to 270userspace through a sysctl: 271.Bd -literal -offset indent 272#define MY_SIZE 8 273static counter_u64_t array[MY_SIZE]; 274SYSCTL_COUNTER_U64_ARRAY(_debug, OID_AUTO, counter_array, CTLFLAG_RW, 275 &array[0], MY_SIZE, "Test counter array"); 276.Ed 277.Sh SEE ALSO 278.Xr atomic 9 , 279.Xr critical 9 , 280.Xr locking 9 , 281.Xr malloc 9 , 282.Xr ratecheck 9 , 283.Xr sysctl 9 , 284.Xr SYSINIT 9 , 285.Xr uma 9 286.Sh HISTORY 287The 288.Nm 289facility first appeared in 290.Fx 10.0 . 291.Sh AUTHORS 292.An -nosplit 293The 294.Nm 295facility was written by 296.An Gleb Smirnoff 297and 298.An Konstantin Belousov . 299