1.\" Copyright (c) 2000-2001 John H. Baldwin <jhb@FreeBSD.org> 2.\" All rights reserved. 3.\" 4.\" Redistribution and use in source and binary forms, with or without 5.\" modification, are permitted provided that the following conditions 6.\" are met: 7.\" 1. Redistributions of source code must retain the above copyright 8.\" notice, this list of conditions and the following disclaimer. 9.\" 2. Redistributions in binary form must reproduce the above copyright 10.\" notice, this list of conditions and the following disclaimer in the 11.\" documentation and/or other materials provided with the distribution. 12.\" 13.\" THIS SOFTWARE IS PROVIDED BY THE DEVELOPERS ``AS IS'' AND ANY EXPRESS OR 14.\" IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES 15.\" OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. 16.\" IN NO EVENT SHALL THE DEVELOPERS BE LIABLE FOR ANY DIRECT, INDIRECT, 17.\" INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT 18.\" NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, 19.\" DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY 20.\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT 21.\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF 22.\" THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 23.\" 24.\" $FreeBSD$ 25.\" 26.Dd March 22, 2017 27.Dt ATOMIC 9 28.Os 29.Sh NAME 30.Nm atomic_add , 31.Nm atomic_clear , 32.Nm atomic_cmpset , 33.Nm atomic_fcmpset , 34.Nm atomic_fetchadd , 35.Nm atomic_load , 36.Nm atomic_readandclear , 37.Nm atomic_set , 38.Nm atomic_subtract , 39.Nm atomic_store 40.Nd atomic operations 41.Sh SYNOPSIS 42.In sys/types.h 43.In machine/atomic.h 44.Ft void 45.Fn atomic_add_[acq_|rel_]<type> "volatile <type> *p" "<type> v" 46.Ft void 47.Fn atomic_clear_[acq_|rel_]<type> "volatile <type> *p" "<type> v" 48.Ft int 49.Fo atomic_cmpset_[acq_|rel_]<type> 50.Fa "volatile <type> *dst" 51.Fa "<type> old" 52.Fa "<type> new" 53.Fc 54.Ft int 55.Fo atomic_fcmpset_[acq_|rel_]<type> 56.Fa "volatile <type> *dst" 57.Fa "<type> *old" 58.Fa "<type> new" 59.Fc 60.Ft <type> 61.Fn atomic_fetchadd_<type> "volatile <type> *p" "<type> v" 62.Ft <type> 63.Fn atomic_load_acq_<type> "volatile <type> *p" 64.Ft <type> 65.Fn atomic_readandclear_<type> "volatile <type> *p" 66.Ft void 67.Fn atomic_set_[acq_|rel_]<type> "volatile <type> *p" "<type> v" 68.Ft void 69.Fn atomic_subtract_[acq_|rel_]<type> "volatile <type> *p" "<type> v" 70.Ft void 71.Fn atomic_store_rel_<type> "volatile <type> *p" "<type> v" 72.Ft <type> 73.Fn atomic_swap_<type> "volatile <type> *p" "<type> v" 74.Ft int 75.Fn atomic_testandclear_<type> "volatile <type> *p" "u_int v" 76.Ft int 77.Fn atomic_testandset_<type> "volatile <type> *p" "u_int v" 78.Sh DESCRIPTION 79Each of the atomic operations is guaranteed to be atomic across multiple 80threads and in the presence of interrupts. 81They can be used to implement reference counts or as building blocks for more 82advanced synchronization primitives such as mutexes. 83.Ss Types 84Each atomic operation operates on a specific 85.Fa type . 86The type to use is indicated in the function name. 87The available types that can be used are: 88.Pp 89.Bl -tag -offset indent -width short -compact 90.It Li int 91unsigned integer 92.It Li long 93unsigned long integer 94.It Li ptr 95unsigned integer the size of a pointer 96.It Li 32 97unsigned 32-bit integer 98.It Li 64 99unsigned 64-bit integer 100.El 101.Pp 102For example, the function to atomically add two integers is called 103.Fn atomic_add_int . 104.Pp 105Certain architectures also provide operations for types smaller than 106.Dq Li int . 107.Pp 108.Bl -tag -offset indent -width short -compact 109.It Li char 110unsigned character 111.It Li short 112unsigned short integer 113.It Li 8 114unsigned 8-bit integer 115.It Li 16 116unsigned 16-bit integer 117.El 118.Pp 119These must not be used in MI code because the instructions to implement them 120efficiently might not be available. 121.Ss Acquire and Release Operations 122By default, a thread's accesses to different memory locations might not be 123performed in 124.Em program order , 125that is, the order in which the accesses appear in the source code. 126To optimize the program's execution, both the compiler and processor might 127reorder the thread's accesses. 128However, both ensure that their reordering of the accesses is not visible to 129the thread. 130Otherwise, the traditional memory model that is expected by single-threaded 131programs would be violated. 132Nonetheless, other threads in a multithreaded program, such as the 133.Fx 134kernel, might observe the reordering. 135Moreover, in some cases, such as the implementation of synchronization between 136threads, arbitrary reordering might result in the incorrect execution of the 137program. 138To constrain the reordering that both the compiler and processor might perform 139on a thread's accesses, the thread should use atomic operations with 140.Em acquire 141and 142.Em release 143semantics. 144.Pp 145Most of the atomic operations on memory have three variants. 146The first variant performs the operation without imposing any ordering 147constraints on memory accesses to other locations. 148The second variant has acquire semantics, and the third variant has release 149semantics. 150In effect, operations with acquire and release semantics establish one-way 151barriers to reordering. 152.Pp 153When an atomic operation has acquire semantics, the effects of the operation 154must have completed before any subsequent load or store (by program order) is 155performed. 156Conversely, acquire semantics do not require that prior loads or stores have 157completed before the atomic operation is performed. 158To denote acquire semantics, the suffix 159.Dq Li _acq 160is inserted into the function name immediately prior to the 161.Dq Li _ Ns Aq Fa type 162suffix. 163For example, to subtract two integers ensuring that subsequent loads and 164stores happen after the subtraction is performed, use 165.Fn atomic_subtract_acq_int . 166.Pp 167When an atomic operation has release semantics, the effects of all prior 168loads or stores (by program order) must have completed before the operation 169is performed. 170Conversely, release semantics do not require that the effects of the 171atomic operation must have completed before any subsequent load or store is 172performed. 173To denote release semantics, the suffix 174.Dq Li _rel 175is inserted into the function name immediately prior to the 176.Dq Li _ Ns Aq Fa type 177suffix. 178For example, to add two long integers ensuring that all prior loads and 179stores happen before the addition, use 180.Fn atomic_add_rel_long . 181.Pp 182The one-way barriers provided by acquire and release operations allow the 183implementations of common synchronization primitives to express their 184ordering requirements without also imposing unnecessary ordering. 185For example, for a critical section guarded by a mutex, an acquire operation 186when the mutex is locked and a release operation when the mutex is unlocked 187will prevent any loads or stores from moving outside of the critical 188section. 189However, they will not prevent the compiler or processor from moving loads 190or stores into the critical section, which does not violate the semantics of 191a mutex. 192.Ss Multiple Processors 193In multiprocessor systems, the atomicity of the atomic operations on memory 194depends on support for cache coherence in the underlying architecture. 195In general, cache coherence on the default memory type, 196.Dv VM_MEMATTR_DEFAULT , 197is guaranteed by all architectures that are supported by 198.Fx . 199For example, cache coherence is guaranteed on write-back memory by the 200.Tn amd64 201and 202.Tn i386 203architectures. 204However, on some architectures, cache coherence might not be enabled on all 205memory types. 206To determine if cache coherence is enabled for a non-default memory type, 207consult the architecture's documentation. 208.Ss Semantics 209This section describes the semantics of each operation using a C like notation. 210.Bl -hang 211.It Fn atomic_add p v 212.Bd -literal -compact 213*p += v; 214.Ed 215.It Fn atomic_clear p v 216.Bd -literal -compact 217*p &= ~v; 218.Ed 219.It Fn atomic_cmpset dst old new 220.Bd -literal -compact 221if (*dst == old) { 222 *dst = new; 223 return (1); 224} else 225 return (0); 226.Ed 227.El 228.Pp 229Some architectures do not implement the 230.Fn atomic_cmpset 231functions for the types 232.Dq Li char , 233.Dq Li short , 234.Dq Li 8 , 235and 236.Dq Li 16 . 237.Bl -hang 238.It Fn atomic_fcmpset dst *old new 239.El 240.Pp 241On architectures implementing 242.Em Compare And Swap 243operation in hardware, the functionality can be described as 244.Bd -literal -offset indent -compact 245if (*dst == *old) { 246 *dst = new; 247 return (1); 248} else { 249 *old = *dst; 250 return (0); 251} 252.Ed 253On architectures which provide 254.Em Load Linked/Store Conditional 255primitive, the write to 256.Dv *dst 257might also fail for several reasons, most important of which 258is a parallel write to 259.Dv *dst 260cache line by other CPU. 261In this case 262.Fn atomic_fcmpset 263function also returns 264.Dv false , 265despite 266.Dl *old == *dst . 267.Pp 268Some architectures do not implement the 269.Fn atomic_fcmpset 270functions for the types 271.Dq Li char , 272.Dq Li short , 273.Dq Li 8 , 274and 275.Dq Li 16 . 276.Bl -hang 277.It Fn atomic_fetchadd p v 278.Bd -literal -compact 279tmp = *p; 280*p += v; 281return (tmp); 282.Ed 283.El 284.Pp 285The 286.Fn atomic_fetchadd 287functions are only implemented for the types 288.Dq Li int , 289.Dq Li long 290and 291.Dq Li 32 292and do not have any variants with memory barriers at this time. 293.Bl -hang 294.It Fn atomic_load p 295.Bd -literal -compact 296return (*p); 297.Ed 298.El 299.Pp 300The 301.Fn atomic_load 302functions are only provided with acquire memory barriers. 303.Bl -hang 304.It Fn atomic_readandclear p 305.Bd -literal -compact 306tmp = *p; 307*p = 0; 308return (tmp); 309.Ed 310.El 311.Pp 312The 313.Fn atomic_readandclear 314functions are not implemented for the types 315.Dq Li char , 316.Dq Li short , 317.Dq Li ptr , 318.Dq Li 8 , 319and 320.Dq Li 16 321and do not have any variants with memory barriers at this time. 322.Bl -hang 323.It Fn atomic_set p v 324.Bd -literal -compact 325*p |= v; 326.Ed 327.It Fn atomic_subtract p v 328.Bd -literal -compact 329*p -= v; 330.Ed 331.It Fn atomic_store p v 332.Bd -literal -compact 333*p = v; 334.Ed 335.El 336.Pp 337The 338.Fn atomic_store 339functions are only provided with release memory barriers. 340.Bl -hang 341.It Fn atomic_swap p v 342.Bd -literal -compact 343tmp = *p; 344*p = v; 345return (tmp); 346.Ed 347.El 348.Pp 349The 350.Fn atomic_swap 351functions are not implemented for the types 352.Dq Li char , 353.Dq Li short , 354.Dq Li ptr , 355.Dq Li 8 , 356and 357.Dq Li 16 358and do not have any variants with memory barriers at this time. 359.Bl -hang 360.It Fn atomic_testandclear p v 361.Bd -literal -compact 362bit = 1 << (v % (sizeof(*p) * NBBY)); 363tmp = (*p & bit) != 0; 364*p &= ~bit; 365return (tmp); 366.Ed 367.El 368.Bl -hang 369.It Fn atomic_testandset p v 370.Bd -literal -compact 371bit = 1 << (v % (sizeof(*p) * NBBY)); 372tmp = (*p & bit) != 0; 373*p |= bit; 374return (tmp); 375.Ed 376.El 377.Pp 378The 379.Fn atomic_testandset 380and 381.Fn atomic_testandclear 382functions are only implemented for the types 383.Dq Li int , 384.Dq Li long 385and 386.Dq Li 32 387and do not have any variants with memory barriers at this time. 388.Pp 389The type 390.Dq Li 64 391is currently not implemented for any of the atomic operations on the 392.Tn arm , 393.Tn i386 , 394and 395.Tn powerpc 396architectures. 397.Sh RETURN VALUES 398The 399.Fn atomic_cmpset 400function returns the result of the compare operation. 401The 402.Fn atomic_fcmpset 403function returns 404.Dv true 405if the operation succeeded. 406Otherwise it returns 407.Dv false 408and sets 409.Va *old 410to the found value. 411The 412.Fn atomic_fetchadd , 413.Fn atomic_load , 414.Fn atomic_readandclear , 415and 416.Fn atomic_swap 417functions return the value at the specified address. 418The 419.Fn atomic_testandset 420and 421.Fn atomic_testandclear 422function returns the result of the test operation. 423.Sh EXAMPLES 424This example uses the 425.Fn atomic_cmpset_acq_ptr 426and 427.Fn atomic_set_ptr 428functions to obtain a sleep mutex and handle recursion. 429Since the 430.Va mtx_lock 431member of a 432.Vt "struct mtx" 433is a pointer, the 434.Dq Li ptr 435type is used. 436.Bd -literal 437/* Try to obtain mtx_lock once. */ 438#define _obtain_lock(mp, tid) \\ 439 atomic_cmpset_acq_ptr(&(mp)->mtx_lock, MTX_UNOWNED, (tid)) 440 441/* Get a sleep lock, deal with recursion inline. */ 442#define _get_sleep_lock(mp, tid, opts, file, line) do { \\ 443 uintptr_t _tid = (uintptr_t)(tid); \\ 444 \\ 445 if (!_obtain_lock(mp, tid)) { \\ 446 if (((mp)->mtx_lock & MTX_FLAGMASK) != _tid) \\ 447 _mtx_lock_sleep((mp), _tid, (opts), (file), (line));\\ 448 else { \\ 449 atomic_set_ptr(&(mp)->mtx_lock, MTX_RECURSE); \\ 450 (mp)->mtx_recurse++; \\ 451 } \\ 452 } \\ 453} while (0) 454.Ed 455.Sh HISTORY 456The 457.Fn atomic_add , 458.Fn atomic_clear , 459.Fn atomic_set , 460and 461.Fn atomic_subtract 462operations were first introduced in 463.Fx 3.0 . 464This first set only supported the types 465.Dq Li char , 466.Dq Li short , 467.Dq Li int , 468and 469.Dq Li long . 470The 471.Fn atomic_cmpset , 472.Fn atomic_load , 473.Fn atomic_readandclear , 474and 475.Fn atomic_store 476operations were added in 477.Fx 5.0 . 478The types 479.Dq Li 8 , 480.Dq Li 16 , 481.Dq Li 32 , 482.Dq Li 64 , 483and 484.Dq Li ptr 485and all of the acquire and release variants 486were added in 487.Fx 5.0 488as well. 489The 490.Fn atomic_fetchadd 491operations were added in 492.Fx 6.0 . 493The 494.Fn atomic_swap 495and 496.Fn atomic_testandset 497operations were added in 498.Fx 10.0 . 499.Fn atomic_testandclear 500operation was added in 501.Fx 11.0 . 502