rangelocks: stop caching per-thread rl_q_entry

This should reduce the frequency of smr_synchronize() calls, that
otherwise occur on almost each rangelock unlock.

Reviewed by: markj
Sponsored by: Th

rangelocks: stop caching per-thread rl_q_entry

This should reduce the frequency of smr_synchronize() calls, that
otherwise occur on almost each rangelock unlock.

Reviewed by: markj
Sponsored by: The FreeBSD Foundation
Differential revision: https://reviews.freebsd.org/D46482

show more ...

rangelocks: restore caching of the single rl entry in the struct thread

Reviewed by: markj, Olivier Certner <olce.freebsd@certner.fr>
Tested by: pho
Sponsored by: The FreeBSD Foundation
Differential

rangelocks: restore caching of the single rl entry in the struct thread

Reviewed by: markj, Olivier Certner <olce.freebsd@certner.fr>
Tested by: pho
Sponsored by: The FreeBSD Foundation
Differential revision: https://reviews.freebsd.org/D41787

show more ...

Re-implement rangelocks part 1

Using the algorithms from https://doi.org/10.1145/3342195.3387533.

For the first part, consider all range lock requests as exclusive.

Reviewed by: markj, Olivier Cer

Re-implement rangelocks part 1

Using the algorithms from https://doi.org/10.1145/3342195.3387533.

For the first part, consider all range lock requests as exclusive.

Reviewed by: markj, Olivier Certner <olce.freebsd@certner.fr>
Tested by: pho
Sponsored by: The FreeBSD Foundation
Differential revision: https://reviews.freebsd.org/D41787

show more ...

proc: Remove kernel stack swapping support, part 9

setrunnable() no longer needs to return a value. Make its return type
void and fix up callers. Now a number of other interfaces no longer
need to

proc: Remove kernel stack swapping support, part 9

setrunnable() no longer needs to return a value. Make its return type
void and fix up callers. Now a number of other interfaces no longer
need to return a value, they will be fixed up subsequently.

Tested by: pho
Reviewed by: kib
Differential Revision: https://reviews.freebsd.org/D46127

show more ...

kern_thread.c: remove unneeded include of sys/param.h

Handled by sys/systm.h already.

Reviewed by: markj
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://review

kern_thread.c: remove unneeded include of sys/param.h

Handled by sys/systm.h already.

Reviewed by: markj
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D44867

show more ...

thread: Add a missing include of asan.h

I didn't notice this during testing because invariants-enabled kernels
implicitly include asan.h via kassert.h.

Reported by: Lexi Winter <lexi@le-Fay.org>
Fi

thread: Add a missing include of asan.h

I didn't notice this during testing because invariants-enabled kernels
implicitly include asan.h via kassert.h.

Reported by: Lexi Winter <lexi@le-Fay.org>
Fixes: 800da341bc4a ("thread: Simplify sanitizer integration with thread creation")

show more ...

thread: Simplify sanitizer integration with thread creation

fork() may allocate a new thread in one of two ways: from UMA, or cached
in a freed proc that was just allocated from UMA. In either case

thread: Simplify sanitizer integration with thread creation

fork() may allocate a new thread in one of two ways: from UMA, or cached
in a freed proc that was just allocated from UMA. In either case, KASAN
and KMSAN need to initialize some state; in particular they need to
initialize the shadow mapping of the new thread's stack.

This is done differently between KASAN and KMSAN, which is confusing.
This patch improves things a bit:
- Add a new thread_recycle() function, which moves all kernel stack
handling out of kern_fork.c, since it doesn't really belong there.
- Then, thread_alloc_stack() has only one local caller, so just inline
it.
- Avoid redundant shadow stack initialization: thread_alloc()
initializes the KMSAN shadow stack (via kmsan_thread_alloc()) even
through vm_thread_new() already did that.
- Add kasan_thread_alloc(), for consistency with kmsan_thread_alloc().

No functional change intended.

Reviewed by: khng
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D44891

show more ...

thread_single(9): decline external requests for traced or debugger-stopped procs

Debugger has the powers to cause unbound delay in single-threading,
which then blocks the threaded taskqueue. The re

thread_single(9): decline external requests for traced or debugger-stopped procs

Debugger has the powers to cause unbound delay in single-threading,
which then blocks the threaded taskqueue. The reproducer is
`truss -f timeout 2 sleep 10`.

Reported by: mjg
Reviewed by: markj
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D44523

show more ...

sys: Automated cleanup of cdefs and other formatting

Apply the following automated changes to try to eliminate
no-longer-needed sys/cdefs.h includes as well as now-empty
blank lines in a row.

Remov

sys: Automated cleanup of cdefs and other formatting

Apply the following automated changes to try to eliminate
no-longer-needed sys/cdefs.h includes as well as now-empty
blank lines in a row.

Remove /^#if.*\n#endif.*\n#include\s+<sys/cdefs.h>.*\n/
Remove /\n+#include\s+<sys/cdefs.h>.*\n+#if.*\n#endif.*\n+/
Remove /\n+#if.*\n#endif.*\n+/
Remove /^#if.*\n#endif.*\n/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/types.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/param.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/capsicum.h>/

Sponsored by: Netflix

show more ...

Ensure 'struct thread' is aligned to a cache line

Using the new UMA_ALIGN_CACHE_AND_MASK() facility, which allows to
simultaneously guarantee a minimum of 32 bytes of alignment (the 5 lower
bits are

Ensure 'struct thread' is aligned to a cache line

Using the new UMA_ALIGN_CACHE_AND_MASK() facility, which allows to
simultaneously guarantee a minimum of 32 bytes of alignment (the 5 lower
bits are always 0).

For the record, to this day, here's a (possibly non-exhaustive) list of
synchronization primitives using lower bits to store flags in pointers
to thread structures:
- lockmgr, rwlock and sx all use the 5 bits directly.
- rmlock indirectly relies on sx, so can use the 5 bits.
- mtx (non-spin) relies on the 3 lower bits.

Reviewed by: markj, kib
MFC after: 2 week
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D42266

show more ...

thread: add td_wantedlock

This enables obtaining lock information threads are actively waiting for
while sampling. Without the change one would only see a bunch of calls
to lock_delay(), where the s

thread: add td_wantedlock

This enables obtaining lock information threads are actively waiting for
while sampling. Without the change one would only see a bunch of calls
to lock_delay(), where the stacktrace often does not reveal what the
lock might be.

Note this is not the same as lock profiling, which only produces data
for cases which wait for locks.

struct thread already has a td_lockname field, but I did not use it
because it has different semantics -- denotes when the thread is off
cpu. At the same time it could not be converted to hold a lock_object
pointer because non-curthread access would no longer be guaranteed to be
safe -- by the time it reads the pointer the lock might have been taken,
released and the object containing it freed.

Sample usage with dtrace:
rm /tmp/out.kern_stacks ; dtrace -x stackframes=100 -n 'profile-997 { @[curthread->td_wantedlock != NULL ? stringof(curthread->td_wantedlock->lo_name) : stringof("\n"), stack()] = count(); }' -o /tmp/out.kern_stacks

This also facilitates addition of lock information to traces produced by
hwpmc.

Note: spinlocks are not supported at the moment.

Sponsored by: Rubicon Communications, LLC ("Netgate")

show more ...

thread: reorder td_no_sleeping to plug two 4-byte holes on LP64

Sponsored by: Rubicon Communications, LLC ("Netgate")

sys: Remove $FreeBSD$: one-line .c pattern

Remove /^[\s*]*__FBSDID\("\$FreeBSD\$"\);?\s*\n/

Revert "killpg(): close a race with fork(), part 2"

This reverts commits 81a37995c757b4e3ad8a5c699864197fd1ebdcf5 and
565a343ae3a30bc2973182ff8dfd2fa37d7f615f.

There is still a leakage of the p_kil

Revert "killpg(): close a race with fork(), part 2"

This reverts commits 81a37995c757b4e3ad8a5c699864197fd1ebdcf5 and
565a343ae3a30bc2973182ff8dfd2fa37d7f615f.

There is still a leakage of the p_killpg_cnt, some but not all sources
of which were identified.

Second, and more important, is that there is a fundamental issue with
blocked signals having KSI_KILLPG flag set. Queueing of such signal
increments p_killpg_cnt, but it cannot be decremented until the signal
is delivered. If, for instance, a single-threaded process with blocked
signal receives killpg-kill and executes fork(2), the fork enter check
returns with ERESTART. And since signal is blocked, the condition
cannot be cleared.

Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D41128

show more ...

killpg(): close a race with fork(), part 2

When we are sending terminating signal to the group, killpg() needs to
guarantee that all group members are to be terminated (it does not need
to ensure th

killpg(): close a race with fork(), part 2

When we are sending terminating signal to the group, killpg() needs to
guarantee that all group members are to be terminated (it does not need
to ensure that they are terminated on return from killpg()). The
pg_killsx change eliminates the largest window there, but still, if a
multithreaded process is signalled, the following could happen:
- thread 1 is selected for the signal delivery and gets descheduled
- thread 2 waits for pg_killsx lock, obtains it and forks
- thread 1 continue executing and terminates the process
This scenario allows the child to escape still.

To fix it, count the number of signals sent to the process with
killpg(2), in p_killpg_cnt variable, which is incremented in killpg()
and decremented after signal handler frame is created or in exit1()
after single-threading. This way we avoid forking if the termination is
due.

Noted and reviewed by: markj (previous version)
Tested by: pho
Sponsored by: The FreeBSD Foundation
MFC after: 2 weeks
Differential revision: https://reviews.freebsd.org/D40493

show more ...

td: line up with other clean up in thread_reap_domain

NFC

rlimit: line up with other clean up in thread_reap_domain

NFC

proc: s/short/int lock-related counters

short is small enough that it can overflow in practice for certain cases
and then trigger wrong asserts.

One known example is vfs namecache resize which grab

proc: s/short/int lock-related counters

short is small enough that it can overflow in practice for certain cases
and then trigger wrong asserts.

One known example is vfs namecache resize which grabs all locks at once.

Reported by: gallatin
Sponsored by: Rubicon Communications, LLC ("Netgate")

show more ...

spdx: The BSD-2-Clause-FreeBSD identifier is obsolete, drop -FreeBSD

The SPDX folks have obsoleted the BSD-2-Clause-FreeBSD identifier. Catch
up to that fact and revert to their recommended match of

spdx: The BSD-2-Clause-FreeBSD identifier is obsolete, drop -FreeBSD

The SPDX folks have obsoleted the BSD-2-Clause-FreeBSD identifier. Catch
up to that fact and revert to their recommended match of BSD-2-Clause.

Discussed with: pfg
MFC After: 3 days
Sponsored by: Netflix

show more ...

umtx: allow to configure minimal timeout (in nanoseconds)

PR: 270785
Reviewed by: markj, mav
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org

umtx: allow to configure minimal timeout (in nanoseconds)

PR: 270785
Reviewed by: markj, mav
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D39584

show more ...

whitespace: rewrap to match case directly above

It's easier to visually diff the two case blocks if there aren't
gratutious whitespace differences.

Sponsored by: DARPA

arm64: Handle translation faults for thread structures

The break-before-make requirement poses a problem when promoting or
demoting mappings containing thread structures: a CPU may raise a
translati

arm64: Handle translation faults for thread structures

The break-before-make requirement poses a problem when promoting or
demoting mappings containing thread structures: a CPU may raise a
translation fault while accessing curthread, and data_abort() accesses
the thread again before pmap_fault() can translate the address and
return.

Normally this isn't a problem because we have a hack to ensure that
slabs used by the thread zone are always accessed via the direct map,
where promotions and demotions are rare. However, this hack doesn't
work properly with UMA_MD_SMALL_ALLOC disabled, as is the case with
KASAN configured (since our KASAN implementation does not shadow the
direct map and so tries to force the use of the kernel map wherever
possible).

Fix the problem by modifying data_abort() to handle translation faults
in the kernel map without dereferencing "td", i.e., curthread, and
without enabling interrupts. pmap_klookup() has special handling for
translation faults which makes it safe to call in this context. Then,
revert the aforementioned hack.

Reviewed by: kevans, alc, kib, andrew
MFC after: 1 month
Sponsored by: Juniper Networks, Inc.
Sponsored by: Klara, Inc.
Differential Revision: https://reviews.freebsd.org/D37231

show more ...

Remove TDF_DOING_SA

We cannot see a thread with the flag set in unsuspend, after we stopped
doing SINGLE_ALLPROC from user processes.

Reviewed by: markj
Tested by: pho
Sponsored by: The FreeBSD Fou

Remove TDF_DOING_SA

We cannot see a thread with the flag set in unsuspend, after we stopped
doing SINGLE_ALLPROC from user processes.

Reviewed by: markj
Tested by: pho
Sponsored by: The FreeBSD Foundation
MFC after: 2 weeks
Differential revision: https://reviews.freebsd.org/D36207

show more ...

Remove struct proc p_singlethr member

It does not serve any purpose after we stopped doing
thread_single(SINGLE_ALLPROC) from stoppable user processes.

Reviewed by: markj
Tested by: pho
Sponsored b

Remove struct proc p_singlethr member

It does not serve any purpose after we stopped doing
thread_single(SINGLE_ALLPROC) from stoppable user processes.

Reviewed by: markj
Tested by: pho
Sponsored by: The FreeBSD Foundation
MFC after: 2 weeks
Differential revision: https://reviews.freebsd.org/D36207

show more ...

ksiginfo_alloc(): change to directly take M_WAITOK/NOWAIT flags

Also style, and remove unneeded cast.

Reviewed by: markj
Tested by: pho
Sponsored by: The FreeBSD Foundation
MFC after: 2 weeks
Diffe

ksiginfo_alloc(): change to directly take M_WAITOK/NOWAIT flags

Also style, and remove unneeded cast.

Reviewed by: markj
Tested by: pho
Sponsored by: The FreeBSD Foundation
MFC after: 2 weeks
Differential revision: https://reviews.freebsd.org/D36207

show more ...