1.\" Copyright (c) 2000-2001 John H. Baldwin <jhb@FreeBSD.org> 2.\" 3.\" Redistribution and use in source and binary forms, with or without 4.\" modification, are permitted provided that the following conditions 5.\" are met: 6.\" 1. Redistributions of source code must retain the above copyright 7.\" notice, this list of conditions and the following disclaimer. 8.\" 2. Redistributions in binary form must reproduce the above copyright 9.\" notice, this list of conditions and the following disclaimer in the 10.\" documentation and/or other materials provided with the distribution. 11.\" 12.\" THIS SOFTWARE IS PROVIDED BY THE DEVELOPERS ``AS IS'' AND ANY EXPRESS OR 13.\" IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES 14.\" OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. 15.\" IN NO EVENT SHALL THE DEVELOPERS BE LIABLE FOR ANY DIRECT, INDIRECT, 16.\" INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT 17.\" NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, 18.\" DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY 19.\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT 20.\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF 21.\" THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 22.\" 23.\" $FreeBSD$ 24.\" 25.Dd March 6, 2021 26.Dt ATOMIC 9 27.Os 28.Sh NAME 29.Nm atomic_add , 30.Nm atomic_clear , 31.Nm atomic_cmpset , 32.Nm atomic_fcmpset , 33.Nm atomic_fetchadd , 34.Nm atomic_interrupt_fence , 35.Nm atomic_load , 36.Nm atomic_readandclear , 37.Nm atomic_set , 38.Nm atomic_subtract , 39.Nm atomic_store , 40.Nm atomic_thread_fence 41.Nd atomic operations 42.Sh SYNOPSIS 43.In sys/types.h 44.In machine/atomic.h 45.Ft void 46.Fn atomic_add_[acq_|rel_]<type> "volatile <type> *p" "<type> v" 47.Ft void 48.Fn atomic_clear_[acq_|rel_]<type> "volatile <type> *p" "<type> v" 49.Ft int 50.Fo atomic_cmpset_[acq_|rel_]<type> 51.Fa "volatile <type> *dst" 52.Fa "<type> old" 53.Fa "<type> new" 54.Fc 55.Ft int 56.Fo atomic_fcmpset_[acq_|rel_]<type> 57.Fa "volatile <type> *dst" 58.Fa "<type> *old" 59.Fa "<type> new" 60.Fc 61.Ft <type> 62.Fn atomic_fetchadd_<type> "volatile <type> *p" "<type> v" 63.Ft void 64.Fn atomic_interrupt_fence "void" 65.Ft <type> 66.Fn atomic_load_[acq_]<type> "volatile <type> *p" 67.Ft <type> 68.Fn atomic_readandclear_<type> "volatile <type> *p" 69.Ft void 70.Fn atomic_set_[acq_|rel_]<type> "volatile <type> *p" "<type> v" 71.Ft void 72.Fn atomic_subtract_[acq_|rel_]<type> "volatile <type> *p" "<type> v" 73.Ft void 74.Fn atomic_store_[rel_]<type> "volatile <type> *p" "<type> v" 75.Ft <type> 76.Fn atomic_swap_<type> "volatile <type> *p" "<type> v" 77.Ft int 78.Fn atomic_testandclear_<type> "volatile <type> *p" "u_int v" 79.Ft int 80.Fn atomic_testandset_<type> "volatile <type> *p" "u_int v" 81.Ft void 82.Fn atomic_thread_fence_[acq|acq_rel|rel|seq_cst] "void" 83.Sh DESCRIPTION 84Atomic operations are commonly used to implement reference counts and as 85building blocks for synchronization primitives, such as mutexes. 86.Pp 87All of these operations are performed 88.Em atomically 89across multiple threads and in the presence of interrupts, meaning that they 90are performed in an indivisible manner from the perspective of concurrently 91running threads and interrupt handlers. 92.Pp 93On all architectures supported by 94.Fx , 95ordinary loads and stores of integers in cache-coherent memory are 96inherently atomic if the integer is naturally aligned and its size does not 97exceed the processor's word size. 98However, such loads and stores may be elided from the program by 99the compiler, whereas atomic operations are always performed. 100.Pp 101When atomic operations are performed on cache-coherent memory, all 102operations on the same location are totally ordered. 103.Pp 104When an atomic load is performed on a location in cache-coherent memory, 105it reads the entire value that was defined by the last atomic store to 106each byte of the location. 107An atomic load will never return a value out of thin air. 108When an atomic store is performed on a location, no other thread or 109interrupt handler will observe a 110.Em torn write , 111or partial modification of the location. 112.Pp 113Except as noted below, the semantics of these operations are almost 114identical to the semantics of similarly named C11 atomic operations. 115.Ss Types 116Most atomic operations act upon a specific 117.Fa type . 118That type is indicated in the function name. 119In contrast to C11 atomic operations, 120.Fx Ns 's 121atomic operations are performed on ordinary integer types. 122The available types are: 123.Pp 124.Bl -tag -offset indent -width short -compact 125.It Li int 126unsigned integer 127.It Li long 128unsigned long integer 129.It Li ptr 130unsigned integer the size of a pointer 131.It Li 32 132unsigned 32-bit integer 133.It Li 64 134unsigned 64-bit integer 135.El 136.Pp 137For example, the function to atomically add two integers is called 138.Fn atomic_add_int . 139.Pp 140Certain architectures also provide operations for types smaller than 141.Dq Li int . 142.Pp 143.Bl -tag -offset indent -width short -compact 144.It Li char 145unsigned character 146.It Li short 147unsigned short integer 148.It Li 8 149unsigned 8-bit integer 150.It Li 16 151unsigned 16-bit integer 152.El 153.Pp 154These types must not be used in machine-independent code. 155.Ss Acquire and Release Operations 156By default, a thread's accesses to different memory locations might not be 157performed in 158.Em program order , 159that is, the order in which the accesses appear in the source code. 160To optimize the program's execution, both the compiler and processor might 161reorder the thread's accesses. 162However, both ensure that their reordering of the accesses is not visible to 163the thread. 164Otherwise, the traditional memory model that is expected by single-threaded 165programs would be violated. 166Nonetheless, other threads in a multithreaded program, such as the 167.Fx 168kernel, might observe the reordering. 169Moreover, in some cases, such as the implementation of synchronization between 170threads, arbitrary reordering might result in the incorrect execution of the 171program. 172To constrain the reordering that both the compiler and processor might perform 173on a thread's accesses, a programmer can use atomic operations with 174.Em acquire 175and 176.Em release 177semantics. 178.Pp 179Atomic operations on memory have up to three variants. 180The first, or 181.Em relaxed 182variant, performs the operation without imposing any ordering constraints on 183accesses to other memory locations. 184This variant is the default. 185The second variant has acquire semantics, and the third variant has release 186semantics. 187.Pp 188When an atomic operation has acquire semantics, the operation must have 189completed before any subsequent load or store (by program order) is 190performed. 191Conversely, acquire semantics do not require that prior loads or stores have 192completed before the atomic operation is performed. 193An atomic operation can only have acquire semantics if it performs a load 194from memory. 195To denote acquire semantics, the suffix 196.Dq Li _acq 197is inserted into the function name immediately prior to the 198.Dq Li _ Ns Aq Fa type 199suffix. 200For example, to subtract two integers ensuring that the subtraction is 201completed before any subsequent loads and stores are performed, use 202.Fn atomic_subtract_acq_int . 203.Pp 204When an atomic operation has release semantics, all prior loads or stores 205(by program order) must have completed before the operation is performed. 206Conversely, release semantics do not require that the atomic operation must 207have completed before any subsequent load or store is performed. 208An atomic operation can only have release semantics if it performs a store 209to memory. 210To denote release semantics, the suffix 211.Dq Li _rel 212is inserted into the function name immediately prior to the 213.Dq Li _ Ns Aq Fa type 214suffix. 215For example, to add two long integers ensuring that all prior loads and 216stores are completed before the addition is performed, use 217.Fn atomic_add_rel_long . 218.Pp 219When a release operation by one thread 220.Em synchronizes with 221an acquire operation by another thread, usually meaning that the acquire 222operation reads the value written by the release operation, then the effects 223of all prior stores by the releasing thread must become visible to 224subsequent loads by the acquiring thread. 225Moreover, the effects of all stores (by other threads) that were visible to 226the releasing thread must also become visible to the acquiring thread. 227These rules only apply to the synchronizing threads. 228Other threads might observe these stores in a different order. 229.Pp 230In effect, atomic operations with acquire and release semantics establish 231one-way barriers to reordering that enable the implementations of 232synchronization primitives to express their ordering requirements without 233also imposing unnecessary ordering. 234For example, for a critical section guarded by a mutex, an acquire operation 235when the mutex is locked and a release operation when the mutex is unlocked 236will prevent any loads or stores from moving outside of the critical 237section. 238However, they will not prevent the compiler or processor from moving loads 239or stores into the critical section, which does not violate the semantics of 240a mutex. 241.Ss Thread Fence Operations 242Alternatively, a programmer can use atomic thread fence operations to 243constrain the reordering of accesses. 244In contrast to other atomic operations, fences do not, themselves, access 245memory. 246.Pp 247When a fence has acquire semantics, all prior loads (by program order) must 248have completed before any subsequent load or store is performed. 249Thus, an acquire fence is a two-way barrier for load operations. 250To denote acquire semantics, the suffix 251.Dq Li _acq 252is appended to the function name, for example, 253.Fn atomic_thread_fence_acq . 254.Pp 255When a fence has release semantics, all prior loads or stores (by program 256order) must have completed before any subsequent store operation is 257performed. 258Thus, a release fence is a two-way barrier for store operations. 259To denote release semantics, the suffix 260.Dq Li _rel 261is appended to the function name, for example, 262.Fn atomic_thread_fence_rel . 263.Pp 264Although 265.Fn atomic_thread_fence_acq_rel 266implements both acquire and release semantics, it is not a full barrier. 267For example, a store prior to the fence (in program order) may be completed 268after a load subsequent to the fence. 269In contrast, 270.Fn atomic_thread_fence_seq_cst 271implements a full barrier. 272Neither loads nor stores may cross this barrier in either direction. 273.Pp 274In C11, a release fence by one thread synchronizes with an acquire fence by 275another thread when an atomic load that is prior to the acquire fence (by 276program order) reads the value written by an atomic store that is subsequent 277to the release fence. 278In constrast, in 279.Fx , 280because of the atomicity of ordinary, naturally 281aligned loads and stores, fences can also be synchronized by ordinary loads 282and stores. 283This simplifies the implementation and use of some synchronization 284primitives in 285.Fx . 286.Pp 287Since neither a compiler nor a processor can foresee which (atomic) load 288will read the value written by an (atomic) store, the ordering constraints 289imposed by fences must be more restrictive than acquire loads and release 290stores. 291Essentially, this is why fences are two-way barriers. 292.Pp 293Although fences impose more restrictive ordering than acquire loads and 294release stores, by separating access from ordering, they can sometimes 295facilitate more efficient implementations of synchronization primitives. 296For example, they can be used to avoid executing a memory barrier until a 297memory access shows that some condition is satisfied. 298.Ss Interrupt Fence Operations 299The 300.Fn atomic_interrupt_fence() 301function establishes ordering between its call location and any interrupt 302handler executing on the same CPU. 303It is modeled after the similar C11 function 304.Fn atomic_signal_fence() , 305and adapted for the kernel environment. 306.Ss Multiple Processors 307In multiprocessor systems, the atomicity of the atomic operations on memory 308depends on support for cache coherence in the underlying architecture. 309In general, cache coherence on the default memory type, 310.Dv VM_MEMATTR_DEFAULT , 311is guaranteed by all architectures that are supported by 312.Fx . 313For example, cache coherence is guaranteed on write-back memory by the 314.Tn amd64 315and 316.Tn i386 317architectures. 318However, on some architectures, cache coherence might not be enabled on all 319memory types. 320To determine if cache coherence is enabled for a non-default memory type, 321consult the architecture's documentation. 322.Ss Semantics 323This section describes the semantics of each operation using a C like notation. 324.Bl -hang 325.It Fn atomic_add p v 326.Bd -literal -compact 327*p += v; 328.Ed 329.It Fn atomic_clear p v 330.Bd -literal -compact 331*p &= ~v; 332.Ed 333.It Fn atomic_cmpset dst old new 334.Bd -literal -compact 335if (*dst == old) { 336 *dst = new; 337 return (1); 338} else 339 return (0); 340.Ed 341.El 342.Pp 343Some architectures do not implement the 344.Fn atomic_cmpset 345functions for the types 346.Dq Li char , 347.Dq Li short , 348.Dq Li 8 , 349and 350.Dq Li 16 . 351.Bl -hang 352.It Fn atomic_fcmpset dst *old new 353.El 354.Pp 355On architectures implementing 356.Em Compare And Swap 357operation in hardware, the functionality can be described as 358.Bd -literal -offset indent -compact 359if (*dst == *old) { 360 *dst = new; 361 return (1); 362} else { 363 *old = *dst; 364 return (0); 365} 366.Ed 367On architectures which provide 368.Em Load Linked/Store Conditional 369primitive, the write to 370.Dv *dst 371might also fail for several reasons, most important of which 372is a parallel write to 373.Dv *dst 374cache line by other CPU. 375In this case 376.Fn atomic_fcmpset 377function also returns 378.Dv false , 379despite 380.Dl *old == *dst . 381.Pp 382Some architectures do not implement the 383.Fn atomic_fcmpset 384functions for the types 385.Dq Li char , 386.Dq Li short , 387.Dq Li 8 , 388and 389.Dq Li 16 . 390.Bl -hang 391.It Fn atomic_fetchadd p v 392.Bd -literal -compact 393tmp = *p; 394*p += v; 395return (tmp); 396.Ed 397.El 398.Pp 399The 400.Fn atomic_fetchadd 401functions are only implemented for the types 402.Dq Li int , 403.Dq Li long 404and 405.Dq Li 32 406and do not have any variants with memory barriers at this time. 407.Bl -hang 408.It Fn atomic_load p 409.Bd -literal -compact 410return (*p); 411.Ed 412.It Fn atomic_readandclear p 413.Bd -literal -compact 414tmp = *p; 415*p = 0; 416return (tmp); 417.Ed 418.El 419.Pp 420The 421.Fn atomic_readandclear 422functions are not implemented for the types 423.Dq Li char , 424.Dq Li short , 425.Dq Li ptr , 426.Dq Li 8 , 427and 428.Dq Li 16 429and do not have any variants with memory barriers at this time. 430.Bl -hang 431.It Fn atomic_set p v 432.Bd -literal -compact 433*p |= v; 434.Ed 435.It Fn atomic_subtract p v 436.Bd -literal -compact 437*p -= v; 438.Ed 439.It Fn atomic_store p v 440.Bd -literal -compact 441*p = v; 442.Ed 443.It Fn atomic_swap p v 444.Bd -literal -compact 445tmp = *p; 446*p = v; 447return (tmp); 448.Ed 449.El 450.Pp 451The 452.Fn atomic_swap 453functions are not implemented for the types 454.Dq Li char , 455.Dq Li short , 456.Dq Li ptr , 457.Dq Li 8 , 458and 459.Dq Li 16 460and do not have any variants with memory barriers at this time. 461.Bl -hang 462.It Fn atomic_testandclear p v 463.Bd -literal -compact 464bit = 1 << (v % (sizeof(*p) * NBBY)); 465tmp = (*p & bit) != 0; 466*p &= ~bit; 467return (tmp); 468.Ed 469.El 470.Bl -hang 471.It Fn atomic_testandset p v 472.Bd -literal -compact 473bit = 1 << (v % (sizeof(*p) * NBBY)); 474tmp = (*p & bit) != 0; 475*p |= bit; 476return (tmp); 477.Ed 478.El 479.Pp 480The 481.Fn atomic_testandset 482and 483.Fn atomic_testandclear 484functions are only implemented for the types 485.Dq Li int , 486.Dq Li long 487and 488.Dq Li 32 489and do not have any variants with memory barriers at this time. 490.Pp 491The type 492.Dq Li 64 493is currently not implemented for some of the atomic operations on the 494.Tn arm , 495.Tn i386 , 496and 497.Tn powerpc 498architectures. 499.Sh RETURN VALUES 500The 501.Fn atomic_cmpset 502function returns the result of the compare operation. 503The 504.Fn atomic_fcmpset 505function returns 506.Dv true 507if the operation succeeded. 508Otherwise it returns 509.Dv false 510and sets 511.Va *old 512to the found value. 513The 514.Fn atomic_fetchadd , 515.Fn atomic_load , 516.Fn atomic_readandclear , 517and 518.Fn atomic_swap 519functions return the value at the specified address. 520The 521.Fn atomic_testandset 522and 523.Fn atomic_testandclear 524function returns the result of the test operation. 525.Sh EXAMPLES 526This example uses the 527.Fn atomic_cmpset_acq_ptr 528and 529.Fn atomic_set_ptr 530functions to obtain a sleep mutex and handle recursion. 531Since the 532.Va mtx_lock 533member of a 534.Vt "struct mtx" 535is a pointer, the 536.Dq Li ptr 537type is used. 538.Bd -literal 539/* Try to obtain mtx_lock once. */ 540#define _obtain_lock(mp, tid) \\ 541 atomic_cmpset_acq_ptr(&(mp)->mtx_lock, MTX_UNOWNED, (tid)) 542 543/* Get a sleep lock, deal with recursion inline. */ 544#define _get_sleep_lock(mp, tid, opts, file, line) do { \\ 545 uintptr_t _tid = (uintptr_t)(tid); \\ 546 \\ 547 if (!_obtain_lock(mp, tid)) { \\ 548 if (((mp)->mtx_lock & MTX_FLAGMASK) != _tid) \\ 549 _mtx_lock_sleep((mp), _tid, (opts), (file), (line));\\ 550 else { \\ 551 atomic_set_ptr(&(mp)->mtx_lock, MTX_RECURSE); \\ 552 (mp)->mtx_recurse++; \\ 553 } \\ 554 } \\ 555} while (0) 556.Ed 557.Sh HISTORY 558The 559.Fn atomic_add , 560.Fn atomic_clear , 561.Fn atomic_set , 562and 563.Fn atomic_subtract 564operations were introduced in 565.Fx 3.0 . 566Initially, these operations were defined on the types 567.Dq Li char , 568.Dq Li short , 569.Dq Li int , 570and 571.Dq Li long . 572.Pp 573The 574.Fn atomic_cmpset , 575.Fn atomic_load_acq , 576.Fn atomic_readandclear , 577and 578.Fn atomic_store_rel 579operations were added in 580.Fx 5.0 . 581Simultaneously, the acquire and release variants were introduced, and 582support was added for operation on the types 583.Dq Li 8 , 584.Dq Li 16 , 585.Dq Li 32 , 586.Dq Li 64 , 587and 588.Dq Li ptr . 589.Pp 590The 591.Fn atomic_fetchadd 592operation was added in 593.Fx 6.0 . 594.Pp 595The 596.Fn atomic_swap 597and 598.Fn atomic_testandset 599operations were added in 600.Fx 10.0 . 601.Pp 602The 603.Fn atomic_testandclear 604and 605.Fn atomic_thread_fence 606operations were added in 607.Fx 11.0 . 608.Pp 609The relaxed variants of 610.Fn atomic_load 611and 612.Fn atomic_store 613were added in 614.Fx 12.0 . 615.Pp 616The 617.Fn atomic_interrupt_fence 618operation was added in 619.Fx 13.0 . 620