umtx: handle allocation failire in umtx_pi_alloc()

Don't assume that this allocation will succeed. We may have been passed
M_NOWAIT.

The calling code already handles allocation failures, but the fu

umtx: handle allocation failire in umtx_pi_alloc()

Don't assume that this allocation will succeed. We may have been passed
M_NOWAIT.

The calling code already handles allocation failures, but the function
itself did not.

PR: 283807
MFC after: 1 week

show more ...

shm: Handle swap pager allocation failures

shm_alloc() can fail if swap reservation fails (i.e., vm.overcommit is
non-zero) or racct is imposing some limits on swap usage.

PR: 282994
MFC after: 2

shm: Handle swap pager allocation failures

shm_alloc() can fail if swap reservation fails (i.e., vm.overcommit is
non-zero) or racct is imposing some limits on swap usage.

PR: 282994
MFC after: 2 weeks
Reviewed by: olce, kib
Differential Revision: https://reviews.freebsd.org/D47839

show more ...

_utmx_op: don't recurse on chain busy

In handling a PP mutex, we'll busy it as soon as we enter the loop and
unbusy it either prior to sleeping or at exit time. In this particular
case, if we fail

_utmx_op: don't recurse on chain busy

In handling a PP mutex, we'll busy it as soon as we enter the loop and
unbusy it either prior to sleeping or at exit time. In this particular
case, if we fail to transition the mutex from OWNERDEAD -> owned because
of casueword(9) failure and the suspend check fails, we'll start over
and attempt to busy an already-busied chain and irrecoverably lock up
both this thread and anything else that tries to busy the chain.

Unbusy the chain prior to restarting because I couldn't decide if that
was a better or worse idea than just keeping track of whether we dirtied
it in do_lock_pp() and avoiding re-dirty. This is marginally easier to
reason about as it returns us to expected state on entry to the loop.

While we're here, simplify the code a bit as `error` will be clobbered
right after the branch anyways.

Reviewed by: kib, olce (both earlier version)
Differential Revision: https://reviews.freebsd.org/D47493

show more ...

umtx: shm: 'ushm_refcnt > 0' => 'ushm_refcnt != 0'

'ushm_refcnt' is unsigned. Don't leave the impression it isn't.

No functional change (intended).

Reviewed by: kib
Approved by: emaste (men

umtx: shm: 'ushm_refcnt > 0' => 'ushm_refcnt != 0'

'ushm_refcnt' is unsigned. Don't leave the impression it isn't.

No functional change (intended).

Reviewed by: kib
Approved by: emaste (mentor)
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D46126

show more ...

umtx: shm: Prevent reference counting overflow

This hardens against provoked use-after-free occurences should there be
reference counting leaks in the future (which is currently not the
case).

At t

umtx: shm: Prevent reference counting overflow

This hardens against provoked use-after-free occurences should there be
reference counting leaks in the future (which is currently not the
case).

At the deepest level, umtx_shm_find_reg_unlocked() now returns EOVERFLOW
when it cannot grant an additional reference to the registry object, and
so will umtx_shm_find_reg(). umtx_shm_create_reg() will fail if calling
umtx_shm_find_reg() returns EOVERFLOW (meaning a SHM object for the
passed key already exists, but we can't acquire another reference on
it), avoiding the creation of a duplicate registry entry for a given key
(this wouldn't pose problem for the rest of the code in its current
form, but is expressly avoided for intelligibility and hardening
purposes).

Since umtx_shm_find_reg*(), and consequently the whole _umtx_op() system
call, can only return EOVERFLOW on such a bug manifesting, we don't
document that return value.

Reviewed by: kib, emaste
Approved by: emaste (mentor)
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D46126

show more ...

umtx: shm: Fix use-after-free due to multiple drops of the registry reference

umtx_shm_unref_reg_locked() would unconditionally drop the "registry"
reference, tied to USHMF_LINKED.

This is not a pr

umtx: shm: Fix use-after-free due to multiple drops of the registry reference

umtx_shm_unref_reg_locked() would unconditionally drop the "registry"
reference, tied to USHMF_LINKED.

This is not a problem for caller umtx_shm_object_terminated(), which
operates under the 'umtx_shm_lock' lock end-to-end, but it is for
indirect caller umtx_shm(), which drops the lock between
umtx_shm_find_reg() and the call to umtx_shm_unref_reg(true) that
deregisters the umtx shared region (from 'umtx_shm_registry';
umtx_shm_find_reg() only finds registered shared mutexes).

Thus, two concurrent user-space callers of _umtx_op() with UMTX_OP_SHM
and flags UMTX_SHM_DESTROY, both progressing past umtx_shm_find_reg()
but before umtx_shm_unref_reg(true), would then decrease twice the
reference count for the single reference standing for the shared mutex's
registration.

Reported by: Synacktiv
Reviewed by: kib
Approved by: emaste (mentor)
Security: FreeBSD-SA-24:14.umtx
Security: CVE-2024-43102
Security: CAP-01
Sponsored by: The Alpha-Omega Project
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D46126

show more ...

umtx: shm: Collapse USHMF_REG_LINKED and USHMF_OBJ_LINKED flags

...into the only USHMF_LINKED, as they are always set or unset together.

This is both to stop giving the impression that they can be

umtx: shm: Collapse USHMF_REG_LINKED and USHMF_OBJ_LINKED flags

...into the only USHMF_LINKED, as they are always set or unset together.

This is both to stop giving the impression that they can be set/unset
independently, which they can't with the current code, and to make it
clearer that an upcoming reference counting fix is correct.

Reviewed by: kib
Approved by: emaste (mentor)
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D46126

show more ...

Reduce reliance on sys/sysproto.h pollution

Add sys/errno.h, sys/malloc.h, sys/queue.h, and vm/uma.h as needed.

sys/sysproto.h currently includes sys/acl.h which currently includes
sys/param.h, sys

Reduce reliance on sys/sysproto.h pollution

Add sys/errno.h, sys/malloc.h, sys/queue.h, and vm/uma.h as needed.

sys/sysproto.h currently includes sys/acl.h which currently includes
sys/param.h, sys/queue.h, and vm/uma.h which in turn bring in
sys/errno.h sys/malloc.h.

Reviewed by: kib
Differential Revision: https://reviews.freebsd.org/D44465

show more ...

PP mutexes: lock: Reduce 'umtx_lock' holding before taking the user lock

There is no need to have it for the priority check (that the thread
doesn't have a higher priority than the mutex's ceiling),

PP mutexes: lock: Reduce 'umtx_lock' holding before taking the user lock

There is no need to have it for the priority check (that the thread
doesn't have a higher priority than the mutex's ceiling), and there's
also no need to take it if the thread doesn't have privileges to set its
priority to the mutex's ceiling.

While here, turn 'su' into a 'bool' and compute the internal priority
corresponding to the mutex's ceiling once and for all, putting it in new
'new_pri'.

Reviewed by: kib
Approved by: emaste (mentor)
MFC after: 2 weeks
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D44045

show more ...

PP mutexes: lock: Check if priority is too high against base one

Doing this instead of using the current (user) priority, which includes
current lendings, prevents gratuitous failures for threads in

PP mutexes: lock: Check if priority is too high against base one

Doing this instead of using the current (user) priority, which includes
current lendings, prevents gratuitous failures for threads involved in
multiple locking groups, where each group is defined as the threads that
can lock a particular PP or PI mutex. No deadlock can occur in this
case. Indeed, if a thread holds such a lock A giving it a higher
priority than the ceiling of some other lock B that is PP, and B is
acquired by another thread, effectively the latter may not be able to
run but this situation can only last until the first thread releases A,
which it will do eventually.

Reviewed by: kib
Approved by: emaste (mentor)
MFC after: 2 weeks
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D44044

show more ...

PP mutexes: unlock: Reset inherited prio regardless of privileges

'uq_inherited_pri' contains the current priority inherited from Priority
Protection mutexes. If -1 is passed through 'm_ceilings[1]

PP mutexes: unlock: Reset inherited prio regardless of privileges

'uq_inherited_pri' contains the current priority inherited from Priority
Protection mutexes. If -1 is passed through 'm_ceilings[1]', meaning
that there are no such mutexes held anymore, this must be reflected into
it by setting it to PRI_MAX, regardless of whether the thread has
privilege to set realtime priorities (PRI_MAX is also obviously not
a realtime priority level). By contrast, it shall not be updated and
the computed 'new_inherited_pri' shall stay unused if the thread doesn't
have the ability to set a realtime priority, possibly keeping an older
such priority acquired previously.

Reviewed by: kib
Approved by: emaste (mentor)
MFC after: 2 weeks
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D43984

show more ...

umtx: Check for errors from suword32()

This is in preparation for annotating copyin() and related functions
with __result_use_check.

Reviewed by: kib
MFC after: 2 weeks
Differential Revision: https

umtx: Check for errors from suword32()

This is in preparation for annotating copyin() and related functions
with __result_use_check.

Reviewed by: kib
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D43144

show more ...

sys: Remove $FreeBSD$: one-line .c pattern

Remove /^[\s*]*__FBSDID\("\$FreeBSD\$"\);?\s*\n/

spdx: The BSD-2-Clause-FreeBSD identifier is obsolete, drop -FreeBSD

The SPDX folks have obsoleted the BSD-2-Clause-FreeBSD identifier. Catch
up to that fact and revert to their recommended match of

spdx: The BSD-2-Clause-FreeBSD identifier is obsolete, drop -FreeBSD

The SPDX folks have obsoleted the BSD-2-Clause-FreeBSD identifier. Catch
up to that fact and revert to their recommended match of BSD-2-Clause.

Discussed with: pfg
MFC After: 3 days
Sponsored by: Netflix

show more ...

umtx: allow to configure minimal timeout (in nanoseconds)

PR: 270785
Reviewed by: markj, mav
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org

umtx: allow to configure minimal timeout (in nanoseconds)

PR: 270785
Reviewed by: markj, mav
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D39584

show more ...

kern_umtx.c do_wait(): correct confusing indent

Sponsored by: The FreeBSD Foundation
MFC after: 3 days

time(3): Align fast clock times to avoid firing multiple timers.

In non-periodic mode absolute timers fire at exactly the time given.
When specifying a fast clock, align the firing time so that less

time(3): Align fast clock times to avoid firing multiple timers.

In non-periodic mode absolute timers fire at exactly the time given.
When specifying a fast clock, align the firing time so that less
timer interrupt events are needed.

Reviewed by: rrs @
Differential Revision: https://reviews.freebsd.org/D36858
MFC after: 1 week
Sponsored by: NVIDIA Networking

show more ...

Fix compat10 semaphore interface race

Wrong has-waiters and missing unconditional _count==0 check may cause
infinite waiting with already non-zero count.
1) properly clear _has_waiters flag when wai

Fix compat10 semaphore interface race

Wrong has-waiters and missing unconditional _count==0 check may cause
infinite waiting with already non-zero count.
1) properly clear _has_waiters flag when waiting failed to start
2) always check _count before start waiting

PR: 265997
Reviewed by: kib
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D36272

show more ...

vm_object: Use the vm_object_(set|clear)_flag() helpers

... rather than setting and clearing flags inline. No functional change
intended.

Reviewed by: alc, kib
MFC after: 1 week
Sponsored by: The

vm_object: Use the vm_object_(set|clear)_flag() helpers

... rather than setting and clearing flags inline. No functional change
intended.

Reviewed by: alc, kib
MFC after: 1 week
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D35469

show more ...

sysent: Get rid of bogus sys/sysent.h include.

Where appropriate hide sysent.h under proper condition.

MFC after: 2 weeks

Handle cas failure when the compare succeeds

When locking a priority inherit mutex we perform a compare and swap
operation to try and acquire the mutex. This may fail even when the
compare succeeds.

Handle cas failure when the compare succeeds

When locking a priority inherit mutex we perform a compare and swap
operation to try and acquire the mutex. This may fail even when the
compare succeeds.

Check and handle this case.

PR: 263825
Reviewed by: kib, markj
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D35150

show more ...

Add timespecvalid_interval macro and use it.

Reviewed by: jhb, imp (early rev)
Differential revision: https://reviews.freebsd.org/D34848
MFC after: 2 weeks

Fix umtxq_sleep() regression caused by 56070dd2e4d.

umtxq_requeue() moves the queue to a different hash chain and different
lock, so we can't rely on msleep_sbt() reacquiring the same old lock.
We h

Fix umtxq_sleep() regression caused by 56070dd2e4d.

umtxq_requeue() moves the queue to a different hash chain and different
lock, so we can't rely on msleep_sbt() reacquiring the same old lock.
We have to use PDROP and update the queue chain and so lock pointer.

PR: 262587
MFC after: 2 weeks

show more ...

Improve timeout precision of pthread_cond_timedwait().

This code was not touched when all other user-space sleep functions were
switched to sbintime_t and decoupled from hardclock. When it is possi

Improve timeout precision of pthread_cond_timedwait().

This code was not touched when all other user-space sleep functions were
switched to sbintime_t and decoupled from hardclock. When it is possible,
convert supplied times into sbinuptime to supply directly to msleep_sbt()
with C_ABSOLUTE. This provides the timeout resolution of few microseconds
instead of 2 milliseconds, plus avoids few clock reads and conversions.

Reviewed by: vangyzen
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D34163

show more ...

freebsd32: add _'s to _umtx_(un)lock

This aligns with the default ABI's configuration.

Reviewed by: kib