atomic: Update interceptor function signatures after commit 5e9a82e898d5

Fixes: 5e9a82e898d5 ("atomics: Constify loads")

atomic(9): Implement atomic_testand(clear|set)_ptr

For current architectures, these are just aliases for the existing
operation on the relevant scalar integer.

Reviewed by: imp, kib
Obtained from:

atomic(9): Implement atomic_testand(clear|set)_ptr

For current architectures, these are just aliases for the existing
operation on the relevant scalar integer.

Reviewed by: imp, kib
Obtained from: CheriBSD
Sponsored by: AFRL, DARPA
Differential Revision: https://reviews.freebsd.org/D47631

show more ...

sys: Automated cleanup of cdefs and other formatting

Apply the following automated changes to try to eliminate
no-longer-needed sys/cdefs.h includes as well as now-empty
blank lines in a row.

Remov

sys: Automated cleanup of cdefs and other formatting

Apply the following automated changes to try to eliminate
no-longer-needed sys/cdefs.h includes as well as now-empty
blank lines in a row.

Remove /^#if.*\n#endif.*\n#include\s+<sys/cdefs.h>.*\n/
Remove /\n+#include\s+<sys/cdefs.h>.*\n+#if.*\n#endif.*\n+/
Remove /\n+#if.*\n#endif.*\n+/
Remove /^#if.*\n#endif.*\n/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/types.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/param.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/capsicum.h>/

Sponsored by: Netflix

show more ...

sys: Remove $FreeBSD$: one-line .c pattern

Remove /^[\s*]*__FBSDID\("\$FreeBSD\$"\);?\s*\n/

kcsan: add __tsan_mem(cpy|move|set) aliases for clang >= 16

Summary:
After https://github.com/llvm/llvm-project/commit/b4257d3bf58c ("[tsan]
Replace mem intrinsics with calls to interceptors") intri

kcsan: add __tsan_mem(cpy|move|set) aliases for clang >= 16

Summary:
After https://github.com/llvm/llvm-project/commit/b4257d3bf58c ("[tsan]
Replace mem intrinsics with calls to interceptors") intrinsic calls to
memcpy, memmove or memset will directly call sanitizer interceptors,
e.g. __tsan_memcpy, __tsan_memmove or __tsan_memset.

Building GENERIC-KCSAN with clang >= 16 would thus result in link errors
similar to:

ld: error: undefined symbol: __tsan_memcpy
>>> referenced by cam_compat.c:150 (/usr/src/sys/cam/cam_compat.c:150)
>>> cam_compat.o:(cam_compat_handle_0x17)
>>> referenced by cam_compat.c:151 (/usr/src/sys/cam/cam_compat.c:151)
>>> cam_compat.o:(cam_compat_handle_0x17)
>>> referenced by cam_compat.c:152 (/usr/src/sys/cam/cam_compat.c:152)
>>> cam_compat.o:(cam_compat_handle_0x17)
>>> referenced 1692 more times

Similar to subr_msan.c, add aliases from the existing kcsan_* versions
of these functions to __tsan_* names.

Reviewed by: markj
MFC after: 3 days
Differential Revision: https://reviews.freebsd.org/D39772

show more ...

atomic: Intercept atomic_(load|store)_bool for kernel sanitizers

Fixes: 2bed73739aac ("atomic: Add plain atomic_load/store_bool()")

Built all KCSAN atomic interceptors on arm64

These atomic functions are now supported. Add them to KCSAN.

Sponsored by: The FreeBSD Foundation

Simplify kernel sanitizer interceptors

KASAN and KCSAN implement interceptors for various primitive operations
that are not instrumented by the compiler. KMSAN requires them as well.
Rather than ad

Simplify kernel sanitizer interceptors

KASAN and KCSAN implement interceptors for various primitive operations
that are not instrumented by the compiler. KMSAN requires them as well.
Rather than adding new cases for each sanitizer which requires
interceptors, implement the following protocol:
- When interceptor definitions are required, define
SAN_NEEDS_INTERCEPTORS and SANITIZER_INTERCEPTOR_PREFIX.
- In headers that declare functions which need to be intercepted by a
sanitizer runtime, use SANITIZER_INTERCEPTOR_PREFIX to provide
declarations.
- When SAN_RUNTIME is defined, do not redefine the names of intercepted
functions. This is typically the case in files which implement
sanitizer runtimes but is also needed in, for example, files which
define ifunc selectors for intercepted operations.

MFC after: 2 weeks
Sponsored by: The FreeBSD Foundation

show more ...

Generalize sanitizer interceptors for memory and string routines

Similar to commit 3ead60236f ("Generalize bus_space(9) and atomic(9)
sanitizer interceptors"), use a more generic scheme for interpos

Generalize sanitizer interceptors for memory and string routines

Similar to commit 3ead60236f ("Generalize bus_space(9) and atomic(9)
sanitizer interceptors"), use a more generic scheme for interposing
sanitizer implementations of routines like memcpy().

No functional change intended.

MFC after: 1 month
Sponsored by: The FreeBSD Foundation

show more ...

Generalize bus_space(9) and atomic(9) sanitizer interceptors

Make it easy to define interceptors for new sanitizer runtimes, rather
than assuming KCSAN. Lay a bit of groundwork for KASAN and KMSAN.

Generalize bus_space(9) and atomic(9) sanitizer interceptors

Make it easy to define interceptors for new sanitizer runtimes, rather
than assuming KCSAN. Lay a bit of groundwork for KASAN and KMSAN.

When a sanitizer is compiled in, atomic(9) and bus_space(9) definitions
in atomic_san.h are used by default instead of the inline
implementations in the platform's atomic.h. These definitions are
implemented in the sanitizer runtime, which includes
machine/{atomic,bus}.h with SAN_RUNTIME defined to pull in the actual
implementations.

No functional change intended.

MFC after: 1 month
Sponsored by: The FreeBSD Foundation

show more ...

Rename _cscan_atomic.h and _cscan_bus.h to atomic_san.h and bus_san.h

Other kernel sanitizers (KMSAN, KASAN) require interceptors as well, so
put these in a more generic place as a step towards impo

Rename _cscan_atomic.h and _cscan_bus.h to atomic_san.h and bus_san.h

Other kernel sanitizers (KMSAN, KASAN) require interceptors as well, so
put these in a more generic place as a step towards importing the other
sanitizers.

No functional change intended.

MFC after: 1 week
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D29103

show more ...

Add NetBSD compatible bus_space_peek_N() and bus_space_poke_N() functions.
One problem with the bus_space_read_N() and bus_space_write_N() family of
functions is that they provide no protection again

Add NetBSD compatible bus_space_peek_N() and bus_space_poke_N() functions.
One problem with the bus_space_read_N() and bus_space_write_N() family of
functions is that they provide no protection against exceptions which can
occur when no physical hardware or device responds to the read or write
cycles. In such a situation, the system typically would panic due to a
kernel-mode bus error. The bus_space_peek_N() and bus_space_poke_N() family
of functions provide a mechanism to handle these exceptions gracefully
without the risk of crashing the system.

Typical example is access to PCI(e) configuration space in bus enumeration
function on badly implemented PCI(e) root complexes (RK3399 or Neoverse
N1 N1SDP and/or access to PCI(e) register when device is in deep sleep state.

This commit adds a real implementation for arm64 only. The remaining
architectures have bus_space_peek()/bus_space_poke() emulated by using
bus_space_read()/bus_space_write() (without exception handling).

MFC after: 1 month
Reviewed by: kib
Differential Revision: https://reviews.freebsd.org/D25371

show more ...

kern: clean up empty lines in .c and .h files

Add char and short types to kcsan

copystr(9): Move to deprecate (attempt #2)

This reapplies logical r360944 and r360946 (reverting r360955), with fixed
copystr() stand-in replacement macro. Eventually the goal is to convert
consume

copystr(9): Move to deprecate (attempt #2)

This reapplies logical r360944 and r360946 (reverting r360955), with fixed
copystr() stand-in replacement macro. Eventually the goal is to convert
consumers and kill the macro, but for a first step it helps if the macro is
correct.

Prior commit message:

Unlike the other copy*() functions, it does not serve to copy from one
address space to another or protect against potential faults. It's just
an older incarnation of the now-more-common strlcpy().

Add a coccinelle script to tools/ which can be used to mechanically
convert existing instances where replacement with strlcpy is trivial.
In the two cases which matched, fuse_vfsops.c and union_vfsops.c, the
code was further refactored manually to simplify.

Replace the declaration of copystr() in systm.h with a small macro
wrapper around strlcpy (with correction from brooks@ -- thanks).

Remove N redundant MI implementations of copystr. For MIPS, this
entailed inlining the assembler copystr into the only consumer,
copyinstr, and making the latter a leaf function.

Reviewed by: jhb (earlier version)
Discussed with: brooks (thanks!)
Differential Revision: https://reviews.freebsd.org/D24672

show more ...

Revert r360944 and r360946 until reported issues can be resolved

Reported by: cy

copystr(9): Move to deprecate [2/2]

Unlike the other copy*() functions, it does not serve to copy from one
address space to another or protect against potential faults. It's just
an older incarnati

copystr(9): Move to deprecate [2/2]

Unlike the other copy*() functions, it does not serve to copy from one
address space to another or protect against potential faults. It's just
an older incarnation of the now-more-common strlcpy().

Add a coccinelle script to tools/ which can be used to mechanically
convert existing instances where replacement with strlcpy is trivial.
In the two cases which matched, fuse_vfsops.c and union_vfsops.c, the
code was further refactored manually to simplify.

Replace the declaration of copystr() in systm.h with a small macro
wrapper around strlcpy.

Remove N redundant MI implementations of copystr. For MIPS, this
entailed inlining the assembler copystr into the only consumer,
copyinstr, and making the latter a leaf function.

Reviewed by: jhb
Differential Revision: https://reviews.freebsd.org/D24672

show more ...

Expand generic subword atomic primitives

The goal of this change is to make the atomic_load_acq_{8,16},
atomic_testandset{,_acq}_long, and atomic_testandclear_long primitives
available in MI-namespa

Expand generic subword atomic primitives

The goal of this change is to make the atomic_load_acq_{8,16},
atomic_testandset{,_acq}_long, and atomic_testandclear_long primitives
available in MI-namespace.

The second goal is to get this draft out of my local tree, as anything that
requires a full tinderbox is a big burden out of tree. MD specifics can be
refined individually afterwards.

The generic implementations may not be ideal for your architecture; feel
free to implement better versions. If no subword_atomic definitions are
needed, the include can be removed from your arch's machine/atomic.h.
Generic definitions are guarded by defined macros of the same name. To
avoid picking up conflicting generic definitions, some macro defines are
added to various MD machine/atomic.h to register an existing implementation.

Include _atomic_subword.h in arm and arm64 machine/atomic.h.

For some odd reason, KCSAN only generates some versions of primitives.
Generate the _acq variants of atomic_load.*_8, atomic_load.*_16, and
atomic_testandset.*_long. There are other questionably disabled primitives,
but I didn't run into them, so I left them alone. KCSAN is only built for
amd64 in tinderbox for now.

Add atomic_subword implementations of atomic_load_acq_{8,16} implemented
using masking and atomic_load_acq_32.

Add generic atomic_subword implementations of atomic_testandset_long(),
atomic_testandclear_long(), and atomic_testandset_acq_long(), using
atomic_fcmpset_long() and atomic_fcmpset_acq_long().

On x86, add atomic_testandset_acq_long as an alias for
atomic_testandset_long.

Reviewed by: kevans, rlibby (previous versions both)
Differential Revision: https://reviews.freebsd.org/D22963

show more ...

Merge ^/head r357270 through r357349.

Remove duplicated empty lines from kern/*.c

No functional changes.

Add KERNEL_PANICKED macro for use in place of direct panicstr tests

Disable KCSAN within a panic.

The kernel is single threaded at this point and the panic is more
important.

Sponsored by: DARPA, AFRL

Add kcsan_md_unsupported from NetBSD.

It's used to ignore virtual addresses that may have a different physical
address depending on the CPU.

Sponsored by: DARPA, AFRL

Fix the bus_space functions with KCSAN on arm64.

Arm64 doesn't define the bus_space_set_multi_stream and
bus_space_set_region_stream functions. Don't try to define them there.

Sponsored by: DARPA,

Fix the bus_space functions with KCSAN on arm64.

Arm64 doesn't define the bus_space_set_multi_stream and
bus_space_set_region_stream functions. Don't try to define them there.

Sponsored by: DARPA, AFRL

show more ...

Port the NetBSD KCSAN runtime to FreeBSD.

Update the NetBSD Kernel Concurrency Sanitizer (KCSAN) runtime to work in
the FreeBSD kernel. It is a useful tool for finding data races between
threads exe

Port the NetBSD KCSAN runtime to FreeBSD.

Update the NetBSD Kernel Concurrency Sanitizer (KCSAN) runtime to work in
the FreeBSD kernel. It is a useful tool for finding data races between
threads executing on different CPUs.

This can be enabled by enabling KCSAN in the kernel config, or by using the
GENERIC-KCSAN amd64 kernel. It works on amd64 and arm64, however the later
needs a compiler change to allow -fsanitize=thread that KCSAN uses.

Sponsored by: DARPA, AFRL
Differential Revision: https://reviews.freebsd.org/D22315

show more ...