atomics: Constify loads

In order to match reality, allow using these functions with pointers on
const objects, and bring us closer to C11.

Remove the '+' modifier in the atomic_load_acq_64_i586()'s

atomics: Constify loads

In order to match reality, allow using these functions with pointers on
const objects, and bring us closer to C11.

Remove the '+' modifier in the atomic_load_acq_64_i586()'s inline asm
statement's constraint for '*p' (the value to load). CMPXCHG8B always
writes back some value, even when the value exchange does not happen in
which case what was read is written back. atomic_load_acq_64_i586()
further takes care of the operation atomically writing back the same
value that was read in any case. All in all, this makes the inline
asm's write back undetectable by any other code, whether executing on
other CPUs or code on the same CPU before and after the call to
atomic_load_acq_64_i586(), except for the fact that CMPXCHG8B will
trigger a #GP(0) if the memory address is part of a read-only mapping.
This unfortunate property is however out of scope of the C abstract
machine, and in particular independent of whether the 'uint64_t' pointed
to is declared 'const' or not.

Approved by: markj (mentor)
MFC after: 5 days
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D46887

show more ...

atomic(9): Implement atomic_testand(clear|set)_ptr

For current architectures, these are just aliases for the existing
operation on the relevant scalar integer.

Reviewed by: imp, kib
Obtained from:

atomic(9): Implement atomic_testand(clear|set)_ptr

For current architectures, these are just aliases for the existing
operation on the relevant scalar integer.

Reviewed by: imp, kib
Obtained from: CheriBSD
Sponsored by: AFRL, DARPA
Differential Revision: https://reviews.freebsd.org/D47631

show more ...

atomic(9): Remove fcmpset-based fallback for atomic_testand(clear|set)

All architectures implement a MD version

Reviewed by: kib
Sponsored by: AFRL, DARPA
Differential Revision: https://reviews.fre

atomic(9): Remove fcmpset-based fallback for atomic_testand(clear|set)

All architectures implement a MD version

Reviewed by: kib
Sponsored by: AFRL, DARPA
Differential Revision: https://reviews.freebsd.org/D47629

show more ...

sys: Remove $FreeBSD$: two-line .h pattern

Remove /^\s*\*\n \*\s+\$FreeBSD\$$\n/

spdx: The BSD-2-Clause-FreeBSD identifier is obsolete, drop -FreeBSD

The SPDX folks have obsoleted the BSD-2-Clause-FreeBSD identifier. Catch
up to that fact and revert to their recommended match of

spdx: The BSD-2-Clause-FreeBSD identifier is obsolete, drop -FreeBSD

The SPDX folks have obsoleted the BSD-2-Clause-FreeBSD identifier. Catch
up to that fact and revert to their recommended match of BSD-2-Clause.

Discussed with: pfg
MFC After: 3 days
Sponsored by: Netflix

show more ...

Remove checks for <sys/cdefs.h> being included.

These files no longer depend on the macros required when these checks
were added.

PR: 263102 (exp-run)
Reviewed by: brooks, imp, emaste
Differential

Remove checks for <sys/cdefs.h> being included.

These files no longer depend on the macros required when these checks
were added.

PR: 263102 (exp-run)
Reviewed by: brooks, imp, emaste
Differential Revision: https://reviews.freebsd.org/D34804

show more ...

powerpc/atomic: Fix atomic_testand_*_long on powerpc64

After b5d227b0 FreeBSD was panicking on boot with "Duplicate free" in
UMA. Analyzing the asm, the '1' mask was treated as an integer, rather
t

powerpc/atomic: Fix atomic_testand_*_long on powerpc64

After b5d227b0 FreeBSD was panicking on boot with "Duplicate free" in
UMA. Analyzing the asm, the '1' mask was treated as an integer, rather
than a long, causing 'slw' (shift left word) to be used for the shifting
instruction, not 'sld' (shift left double). This means the upper bits
of the bitfield were not getting used, resulting in corruption of the
bitfield.

While fixing this, the 'and' check of the mask does not need to be
recorded, so don't record (drop the '.').

show more ...

powerpc: Fix syntax atomic.h

Fixes: b5d227b0b27 powerpc: Add atomic_testand_{set,clear}_{int,long}

powerpc: Add atomic_testand_{set,clear}_{int,long}

Add machine-optimized implementations for the following:
* atomic_testandset_int
* atomic_testandclear_int
* atomic_testandset_long
* atomic_testan

powerpc: Add atomic_testand_{set,clear}_{int,long}

Add machine-optimized implementations for the following:
* atomic_testandset_int
* atomic_testandclear_int
* atomic_testandset_long
* atomic_testandclear_long

This fixes the build with ISA_206_ATOMICS enabled.
Add the associated atomic_testandset_32, atomic_testandclear_32, so
that ice(4) can potentially build.

show more ...

[PowerPC] Fix ISA_206 subword atomics

The POWER7 subword atomics were not using the correct instructions for
byte and halfword stores in the atomic_fcmpset code.

This only affects builds with custo

[PowerPC] Fix ISA_206 subword atomics

The POWER7 subword atomics were not using the correct instructions for
byte and halfword stores in the atomic_fcmpset code.

This only affects builds with custom CFLAGS that have explicitly enabled
ISA_206_ATOMICS.

show more ...

powerpc: clean up empty lines in .c and .h files

[PowerPC] Fix atomic_cmpset_masked().

A recent kernel change caused the previously unused atomic_cmpset_masked() to
be used.

It had a typo in it.

Instead of reading the old value from an uninitial

[PowerPC] Fix atomic_cmpset_masked().

A recent kernel change caused the previously unused atomic_cmpset_masked() to
be used.

It had a typo in it.

Instead of reading the old value from an uninitialized variable, read it
from the passed-in pointer as intended.

This fixes crashes on 64 bit Book-E.

Obtained from: jhibbits

show more ...

Expand generic subword atomic primitives

The goal of this change is to make the atomic_load_acq_{8,16},
atomic_testandset{,_acq}_long, and atomic_testandclear_long primitives
available in MI-namespa

Expand generic subword atomic primitives

The goal of this change is to make the atomic_load_acq_{8,16},
atomic_testandset{,_acq}_long, and atomic_testandclear_long primitives
available in MI-namespace.

The second goal is to get this draft out of my local tree, as anything that
requires a full tinderbox is a big burden out of tree. MD specifics can be
refined individually afterwards.

The generic implementations may not be ideal for your architecture; feel
free to implement better versions. If no subword_atomic definitions are
needed, the include can be removed from your arch's machine/atomic.h.
Generic definitions are guarded by defined macros of the same name. To
avoid picking up conflicting generic definitions, some macro defines are
added to various MD machine/atomic.h to register an existing implementation.

Include _atomic_subword.h in arm and arm64 machine/atomic.h.

For some odd reason, KCSAN only generates some versions of primitives.
Generate the _acq variants of atomic_load.*_8, atomic_load.*_16, and
atomic_testandset.*_long. There are other questionably disabled primitives,
but I didn't run into them, so I left them alone. KCSAN is only built for
amd64 in tinderbox for now.

Add atomic_subword implementations of atomic_load_acq_{8,16} implemented
using masking and atomic_load_acq_32.

Add generic atomic_subword implementations of atomic_testandset_long(),
atomic_testandclear_long(), and atomic_testandset_acq_long(), using
atomic_fcmpset_long() and atomic_fcmpset_acq_long().

On x86, add atomic_testandset_acq_long as an alias for
atomic_testandset_long.

Reviewed by: kevans, rlibby (previous versions both)
Differential Revision: https://reviews.freebsd.org/D22963

show more ...

[PowerPC] [MIPS] Implement 32-bit kernel emulation of atomic64 operations

This is a lock-based emulation of 64-bit atomics for kernel use, split off
from an earlier patch by jhibbits.

This is neede

[PowerPC] [MIPS] Implement 32-bit kernel emulation of atomic64 operations

This is a lock-based emulation of 64-bit atomics for kernel use, split off
from an earlier patch by jhibbits.

This is needed to unblock future improvements that reduce the need for
locking on 64-bit platforms by using atomic updates.

The implementation allows for future integration with userland atomic64,
but as that implies going through sysarch for every use, the current
status quo of userland doing its own locking may be for the best.

Submitted by: jhibbits (original patch), kevans (mips bits)
Reviewed by: jhibbits, jeff, kevans
Differential Revision: https://reviews.freebsd.org/D22976

show more ...

atomic: Add atomic_cmpset_masked to powerpc and use it

Summary:
This is a more optimal way of doing atomic_compset_masked() than the
fallback in sys/_atomic_subword.h. There's also an override for

atomic: Add atomic_cmpset_masked to powerpc and use it

Summary:
This is a more optimal way of doing atomic_compset_masked() than the
fallback in sys/_atomic_subword.h. There's also an override for
_atomic_fcmpset_masked_word(), which may or may not be necessary, and is
unused for powerpc.

Reviewed by: kevans, kib
Differential Revision: https://reviews.freebsd.org/D22359

show more ...

powerpc/atomic: Fix atomic_cmpset_rel()

Need a release barrier, not an acquire barrier, else bad things happen.

Merge ^/head r352764 through r353315.

powerpc: Implement atomic_(f)cmpset_ for short and char
|
This adds two implementations for each atomic_fcmpset_ and atomic_cmpset_
short and char functions, selectable at compile time for the target

powerpc: Implement atomic_(f)cmpset_ for short and char
|
This adds two implementations for each atomic_fcmpset_ and atomic_cmpset_
short and char functions, selectable at compile time for the target
architecture. By default, it uses a generic shift-and-mask to perform atomic
updates to sub-components of 32-bit words from <sys/_atomic_subword.h>.
However, if ISA_206_ATOMICS is defined it uses the ll/sc instructions for
halfword and bytes, introduced in PowerISA 2.06. These instructions are
supported by all IBM processors from POWER7 on, as well as the Freescale/NXP
e6500 core. Although the e5500 and e500mc both implement PowerISA 2.06 they
do not implement these instructions.

As part of this, clean up the atomic_(f)cmpset_acq and _rel wrappers, by
using macros to reduce code duplication.

ISA_206_ATOMICS requires clang or newer binutils (2.20 or later).

Differential Revision: https://reviews.freebsd.org/D21682

show more ...

Merge ^/head r352587 through r352763.

powerpc/atomic: Follow recommendations on atomic primitive comparisons

Both IBM and Freescale programming examples presume the cmpset operands will
favor equal, and pessimize the non-equal case inst

powerpc/atomic: Follow recommendations on atomic primitive comparisons

Both IBM and Freescale programming examples presume the cmpset operands will
favor equal, and pessimize the non-equal case instead. Do the same for
atomic_cmpset_* and atomic_fcmpset_*. This slightly pessimizes the failure
case, in favor of the success case.

MFC after: 3 weeks

show more ...

Merge ^/head r341764 through r341812.

Implement atomic_swap_xxx() for all platforms.

Differential Revision: https://reviews.freebsd.org/D18450
Reviewed by: kib@
MFC after: 3 days
Sponsored by: Mellanox Technologies

Merge ^/head r340126 through r340212.

powerpc/atomic: Loosen the memory barrier on atomic_load_acq_*()

'sync' is pretty heavy-handed, and is unnecessary for this use case. It's a
full barrier, which is applicable for all storage types.

powerpc/atomic: Loosen the memory barrier on atomic_load_acq_*()

'sync' is pretty heavy-handed, and is unnecessary for this use case. It's a
full barrier, which is applicable for all storage types. However,
atomic_load_acq_*() is only expected to operate on physical memory, not
device memory, so lwsync is sufficient (lwsync provides access ordering on
memory that is marked as Coherency Required and is not Write Through nor
Cache Inhibited). On 32-bit systems, this is a nop, since powerpc_lwsync()
is defined to use sync, as a workaround for a silicon bug in the Freescale
e500 core.

show more ...

Merge ^/head r326936 through r327149.