1.\" Copyright (c) 2000-2001 John H. Baldwin <jhb@FreeBSD.org> 2.\" All rights reserved. 3.\" 4.\" Redistribution and use in source and binary forms, with or without 5.\" modification, are permitted provided that the following conditions 6.\" are met: 7.\" 1. Redistributions of source code must retain the above copyright 8.\" notice, this list of conditions and the following disclaimer. 9.\" 2. Redistributions in binary form must reproduce the above copyright 10.\" notice, this list of conditions and the following disclaimer in the 11.\" documentation and/or other materials provided with the distribution. 12.\" 13.\" THIS SOFTWARE IS PROVIDED BY THE DEVELOPERS ``AS IS'' AND ANY EXPRESS OR 14.\" IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES 15.\" OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. 16.\" IN NO EVENT SHALL THE DEVELOPERS BE LIABLE FOR ANY DIRECT, INDIRECT, 17.\" INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT 18.\" NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, 19.\" DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY 20.\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT 21.\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF 22.\" THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 23.\" 24.\" $FreeBSD$ 25.\" 26.Dd August 14, 2015 27.Dt ATOMIC 9 28.Os 29.Sh NAME 30.Nm atomic_add , 31.Nm atomic_clear , 32.Nm atomic_cmpset , 33.Nm atomic_fetchadd , 34.Nm atomic_load , 35.Nm atomic_readandclear , 36.Nm atomic_set , 37.Nm atomic_subtract , 38.Nm atomic_store 39.Nd atomic operations 40.Sh SYNOPSIS 41.In sys/types.h 42.In machine/atomic.h 43.Ft void 44.Fn atomic_add_[acq_|rel_]<type> "volatile <type> *p" "<type> v" 45.Ft void 46.Fn atomic_clear_[acq_|rel_]<type> "volatile <type> *p" "<type> v" 47.Ft int 48.Fo atomic_cmpset_[acq_|rel_]<type> 49.Fa "volatile <type> *dst" 50.Fa "<type> old" 51.Fa "<type> new" 52.Fc 53.Ft <type> 54.Fn atomic_fetchadd_<type> "volatile <type> *p" "<type> v" 55.Ft <type> 56.Fn atomic_load_acq_<type> "volatile <type> *p" 57.Ft <type> 58.Fn atomic_readandclear_<type> "volatile <type> *p" 59.Ft void 60.Fn atomic_set_[acq_|rel_]<type> "volatile <type> *p" "<type> v" 61.Ft void 62.Fn atomic_subtract_[acq_|rel_]<type> "volatile <type> *p" "<type> v" 63.Ft void 64.Fn atomic_store_rel_<type> "volatile <type> *p" "<type> v" 65.Ft <type> 66.Fn atomic_swap_<type> "volatile <type> *p" "<type> v" 67.Ft int 68.Fn atomic_testandset_<type> "volatile <type> *p" "u_int v" 69.Sh DESCRIPTION 70Each of the atomic operations is guaranteed to be atomic across multiple 71threads and in the presence of interrupts. 72They can be used to implement reference counts or as building blocks for more 73advanced synchronization primitives such as mutexes. 74.Ss Types 75Each atomic operation operates on a specific 76.Fa type . 77The type to use is indicated in the function name. 78The available types that can be used are: 79.Pp 80.Bl -tag -offset indent -width short -compact 81.It Li int 82unsigned integer 83.It Li long 84unsigned long integer 85.It Li ptr 86unsigned integer the size of a pointer 87.It Li 32 88unsigned 32-bit integer 89.It Li 64 90unsigned 64-bit integer 91.El 92.Pp 93For example, the function to atomically add two integers is called 94.Fn atomic_add_int . 95.Pp 96Certain architectures also provide operations for types smaller than 97.Dq Li int . 98.Pp 99.Bl -tag -offset indent -width short -compact 100.It Li char 101unsigned character 102.It Li short 103unsigned short integer 104.It Li 8 105unsigned 8-bit integer 106.It Li 16 107unsigned 16-bit integer 108.El 109.Pp 110These must not be used in MI code because the instructions to implement them 111efficiently might not be available. 112.Ss Acquire and Release Operations 113By default, a thread's accesses to different memory locations might not be 114performed in 115.Em program order , 116that is, the order in which the accesses appear in the source code. 117To optimize the program's execution, both the compiler and processor might 118reorder the thread's accesses. 119However, both ensure that their reordering of the accesses is not visible to 120the thread. 121Otherwise, the traditional memory model that is expected by single-threaded 122programs would be violated. 123Nonetheless, other threads in a multithreaded program, such as the 124.Fx 125kernel, might observe the reordering. 126Moreover, in some cases, such as the implementation of synchronization between 127threads, arbitrary reordering might result in the incorrect execution of the 128program. 129To constrain the reordering that both the compiler and processor might perform 130on a thread's accesses, the thread should use atomic operations with 131.Em acquire 132and 133.Em release 134semantics. 135.Pp 136Most of the atomic operations on memory have three variants. 137The first variant performs the operation without imposing any ordering 138constraints on memory accesses to other locations. 139The second variant has acquire semantics, and the third variant has release 140semantics. 141In effect, operations with acquire and release semantics establish one-way 142barriers to reordering. 143.Pp 144When an atomic operation has acquire semantics, the effects of the operation 145must have completed before any subsequent load or store (by program order) is 146performed. 147Conversely, acquire semantics do not require that prior loads or stores have 148completed before the atomic operation is performed. 149To denote acquire semantics, the suffix 150.Dq Li _acq 151is inserted into the function name immediately prior to the 152.Dq Li _ Ns Aq Fa type 153suffix. 154For example, to subtract two integers ensuring that subsequent loads and 155stores happen after the subtraction is performed, use 156.Fn atomic_subtract_acq_int . 157.Pp 158When an atomic operation has release semantics, the effects of all prior 159loads or stores (by program order) must have completed before the operation 160is performed. 161Conversely, release semantics do not require that the effects of the 162atomic operation must have completed before any subsequent load or store is 163performed. 164To denote release semantics, the suffix 165.Dq Li _rel 166is inserted into the function name immediately prior to the 167.Dq Li _ Ns Aq Fa type 168suffix. 169For example, to add two long integers ensuring that all prior loads and 170stores happen before the addition, use 171.Fn atomic_add_rel_long . 172.Pp 173The one-way barriers provided by acquire and release operations allow the 174implementations of common synchronization primitives to express their 175ordering requirements without also imposing unnecessary ordering. 176For example, for a critical section guarded by a mutex, an acquire operation 177when the mutex is locked and a release operation when the mutex is unlocked 178will prevent any loads or stores from moving outside of the critical 179section. 180However, they will not prevent the compiler or processor from moving loads 181or stores into the critical section, which does not violate the semantics of 182a mutex. 183.Ss Multiple Processors 184In multiprocessor systems, the atomicity of the atomic operations on memory 185depends on support for cache coherence in the underlying architecture. 186In general, cache coherence on the default memory type, 187.Dv VM_MEMATTR_DEFAULT , 188is guaranteed by all architectures that are supported by 189.Fx . 190For example, cache coherence is guaranteed on write-back memory by the 191.Tn amd64 192and 193.Tn i386 194architectures. 195However, on some architectures, cache coherence might not be enabled on all 196memory types. 197To determine if cache coherence is enabled for a non-default memory type, 198consult the architecture's documentation. 199.Ss Semantics 200This section describes the semantics of each operation using a C like notation. 201.Bl -hang 202.It Fn atomic_add p v 203.Bd -literal -compact 204*p += v; 205.Ed 206.It Fn atomic_clear p v 207.Bd -literal -compact 208*p &= ~v; 209.Ed 210.It Fn atomic_cmpset dst old new 211.Bd -literal -compact 212if (*dst == old) { 213 *dst = new; 214 return (1); 215} else 216 return (0); 217.Ed 218.El 219.Pp 220The 221.Fn atomic_cmpset 222functions are not implemented for the types 223.Dq Li char , 224.Dq Li short , 225.Dq Li 8 , 226and 227.Dq Li 16 . 228.Bl -hang 229.It Fn atomic_fetchadd p v 230.Bd -literal -compact 231tmp = *p; 232*p += v; 233return (tmp); 234.Ed 235.El 236.Pp 237The 238.Fn atomic_fetchadd 239functions are only implemented for the types 240.Dq Li int , 241.Dq Li long 242and 243.Dq Li 32 244and do not have any variants with memory barriers at this time. 245.Bl -hang 246.It Fn atomic_load p 247.Bd -literal -compact 248return (*p); 249.Ed 250.El 251.Pp 252The 253.Fn atomic_load 254functions are only provided with acquire memory barriers. 255.Bl -hang 256.It Fn atomic_readandclear p 257.Bd -literal -compact 258tmp = *p; 259*p = 0; 260return (tmp); 261.Ed 262.El 263.Pp 264The 265.Fn atomic_readandclear 266functions are not implemented for the types 267.Dq Li char , 268.Dq Li short , 269.Dq Li ptr , 270.Dq Li 8 , 271and 272.Dq Li 16 273and do not have any variants with memory barriers at this time. 274.Bl -hang 275.It Fn atomic_set p v 276.Bd -literal -compact 277*p |= v; 278.Ed 279.It Fn atomic_subtract p v 280.Bd -literal -compact 281*p -= v; 282.Ed 283.It Fn atomic_store p v 284.Bd -literal -compact 285*p = v; 286.Ed 287.El 288.Pp 289The 290.Fn atomic_store 291functions are only provided with release memory barriers. 292.Bl -hang 293.It Fn atomic_swap p v 294.Bd -literal -compact 295tmp = *p; 296*p = v; 297return (tmp); 298.Ed 299.El 300.Pp 301The 302.Fn atomic_swap 303functions are not implemented for the types 304.Dq Li char , 305.Dq Li short , 306.Dq Li ptr , 307.Dq Li 8 , 308and 309.Dq Li 16 310and do not have any variants with memory barriers at this time. 311.Bl -hang 312.It Fn atomic_testandset p v 313.Bd -literal -compact 314bit = 1 << (v % (sizeof(*p) * NBBY)); 315tmp = (*p & bit) != 0; 316*p |= bit; 317return (tmp); 318.Ed 319.El 320.Pp 321The 322.Fn atomic_testandset 323functions are only implemented for the types 324.Dq Li int , 325.Dq Li long 326and 327.Dq Li 32 328and do not have any variants with memory barriers at this time. 329.Pp 330The type 331.Dq Li 64 332is currently not implemented for any of the atomic operations on the 333.Tn arm , 334.Tn i386 , 335and 336.Tn powerpc 337architectures. 338.Sh RETURN VALUES 339The 340.Fn atomic_cmpset 341function returns the result of the compare operation. 342The 343.Fn atomic_fetchadd , 344.Fn atomic_load , 345.Fn atomic_readandclear , 346and 347.Fn atomic_swap 348functions return the value at the specified address. 349The 350.Fn atomic_testandset 351function returns the result of the test operation. 352.Sh EXAMPLES 353This example uses the 354.Fn atomic_cmpset_acq_ptr 355and 356.Fn atomic_set_ptr 357functions to obtain a sleep mutex and handle recursion. 358Since the 359.Va mtx_lock 360member of a 361.Vt "struct mtx" 362is a pointer, the 363.Dq Li ptr 364type is used. 365.Bd -literal 366/* Try to obtain mtx_lock once. */ 367#define _obtain_lock(mp, tid) \\ 368 atomic_cmpset_acq_ptr(&(mp)->mtx_lock, MTX_UNOWNED, (tid)) 369 370/* Get a sleep lock, deal with recursion inline. */ 371#define _get_sleep_lock(mp, tid, opts, file, line) do { \\ 372 uintptr_t _tid = (uintptr_t)(tid); \\ 373 \\ 374 if (!_obtain_lock(mp, tid)) { \\ 375 if (((mp)->mtx_lock & MTX_FLAGMASK) != _tid) \\ 376 _mtx_lock_sleep((mp), _tid, (opts), (file), (line));\\ 377 else { \\ 378 atomic_set_ptr(&(mp)->mtx_lock, MTX_RECURSE); \\ 379 (mp)->mtx_recurse++; \\ 380 } \\ 381 } \\ 382} while (0) 383.Ed 384.Sh HISTORY 385The 386.Fn atomic_add , 387.Fn atomic_clear , 388.Fn atomic_set , 389and 390.Fn atomic_subtract 391operations were first introduced in 392.Fx 3.0 . 393This first set only supported the types 394.Dq Li char , 395.Dq Li short , 396.Dq Li int , 397and 398.Dq Li long . 399The 400.Fn atomic_cmpset , 401.Fn atomic_load , 402.Fn atomic_readandclear , 403and 404.Fn atomic_store 405operations were added in 406.Fx 5.0 . 407The types 408.Dq Li 8 , 409.Dq Li 16 , 410.Dq Li 32 , 411.Dq Li 64 , 412and 413.Dq Li ptr 414and all of the acquire and release variants 415were added in 416.Fx 5.0 417as well. 418The 419.Fn atomic_fetchadd 420operations were added in 421.Fx 6.0 . 422The 423.Fn atomic_swap 424and 425.Fn atomic_testandset 426operations were added in 427.Fx 10.0 . 428