callout: provide CALLOUT_TRYLOCK flag

If a callout was initialized with the flag, then the callout(9) system
will not drop the callwheel lock in softclock_call_cc() to obtain the
callout lock. Inst

callout: provide CALLOUT_TRYLOCK flag

If a callout was initialized with the flag, then the callout(9) system
will not drop the callwheel lock in softclock_call_cc() to obtain the
callout lock. Instead it will use try-lock semantic to obtain the
callout's lock. In case of a failure the callout will be rescheduled to
the 50% of the precision value. The main benefit of such behavior is not
the avoidance of the lock contention in the callout thread, but the fact
that callout with such flag can be actually stopped in a safe manner,
because the race window in the beginning of softclock_call_cc() is closed.

Call of callout_stop() on such a callout would guarantee that nothing will
be executed after callout_stop() returns, neither callout lock will be
dereferenced. A callout marked as CALLOUT_TRYLOCK |
CALLOUT_RETURNUNLOCKED can call callout_stop() from the callout function
itself (0, a failure to stop, will be returned), then unlock the lock and
then free the memory containing the callout structure.

Caveat: when calling callout_stop() from outside the callout function, the
return value from callout_stop() is still inconsistent. A race window at
the end of softclock_call_cc() still exists, so callout_stop() may report
failure to stop, which would not be true.

Reviewed by: jtl, kib
Differential Revision: https://reviews.freebsd.org/D45746

show more ...

callout: assert that callout_init_*lock* functions are called with a lock

Quick grep around kernel confirms they all do.

callout: tidy up _callout_init_lock()

Separate function into assertive part and into assigning part.
Consistently use __func__ in the assertions. Write the assigning code in
a declarative style.

T

callout: tidy up _callout_init_lock()

Separate function into assertive part and into assigning part.
Consistently use __func__ in the assertions. Write the assigning code in
a declarative style.

The functional change is that we no longer validate flags in the
non-INVARIANT kernel. The assertion that checks flags has been there for
17 years, so all code that calls with invalid flags must have been
filtered and fixed.

show more ...

callout: retire callout_async_drain()

This function was used only in TCP before 446ccdd08e2a. It was born in
pain in 2016 to plug different complex panics in TCP timers. It wasn't
warmly accepted

callout: retire callout_async_drain()

This function was used only in TCP before 446ccdd08e2a. It was born in
pain in 2016 to plug different complex panics in TCP timers. It wasn't
warmly accepted in phabricator by all of the reviewers and my recollection
of overall agreement was that "if you need this KPI, then you'd better fix
your code to not need it". However, the function served its duty well all
the way to FreeBSD 14. But now that TCP doesn't need it anymore, let's
retire it to reduce complexity of callout code and also to avoid its
further use.

Reviewed by: jhb, markj, kib, rrs
Differential Revision: https://reviews.freebsd.org/D43546

show more ...

sys: Remove ancient SCCS tags.

Remove ancient SCCS tags from the tree, automated scripting, with two
minor fixup to keep things compiling. All the common forms in the tree
were removed with a perl s

sys: Remove ancient SCCS tags.

Remove ancient SCCS tags from the tree, automated scripting, with two
minor fixup to keep things compiling. All the common forms in the tree
were removed with a perl script.

Sponsored by: Netflix

show more ...

sys: Remove $FreeBSD$: one-line .c pattern

Remove /^[\s*]*__FBSDID\("\$FreeBSD\$"\);?\s*\n/

callout: Remove an unneeded MTX_NEW

Reported by: hselasky
Fixes: 78cfa762ebf2 ("callout: Move per-CPU callout state into the dpcpu region")

callout: Move per-CPU callout state into the dpcpu region

This eliminates some static bloat in amd64 kernels and reduces the
penalty of increasing MAXCPU. The structures now also maintain NUMA
affi

callout: Move per-CPU callout state into the dpcpu region

This eliminates some static bloat in amd64 kernels and reduces the
penalty of increasing MAXCPU. The structures now also maintain NUMA
affinity. No functional change intended.

PR: 269572
Reviewed by: mjg, kib
MFC after: 2 weeks
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D39807

show more ...

callout(9): adopt old references to timeout(9)

timeout(9) was removed a couple of years ago; all consumers now use the
callout(9) interface.

Explicitly do not bump .Dd anywhere, as this is not a co

callout(9): adopt old references to timeout(9)

timeout(9) was removed a couple of years ago; all consumers now use the
callout(9) interface.

Explicitly do not bump .Dd anywhere, as this is not a content or
semantic change.

Reviewed by: markj, jhb, Pau Amma <pauamma@gundo.com>
MFC after: 1 week
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D39136

show more ...

ddb: annotate some commands with DB_CMD_MEMSAFE

This is not completely exhaustive, but covers a large majority of
commands in the tree.

Reviewed by: markj
Sponsored by: Juniper Networks, Inc.
Spons

ddb: annotate some commands with DB_CMD_MEMSAFE

This is not completely exhaustive, but covers a large majority of
commands in the tree.

Reviewed by: markj
Sponsored by: Juniper Networks, Inc.
Sponsored by: Klara, Inc.
Differential Revision: https://reviews.freebsd.org/D35583

show more ...

callout: Simplify the inner loop in callout_process() a bit

- Use LIST_FOREACH_SAFE.
- Simplify control flow.

No functional change intended.

MFC after: 1 week
Sponsored by: The FreeBSD Foundation

callout: Remove a redundant parameter to callout_cc_add()

The passed cpuid is always equal to the one stored in the callout
structure. No functional change intended.

MFC after: 1 week
Sponsored by

callout: Remove a redundant parameter to callout_cc_add()

The passed cpuid is always equal to the one stored in the callout
structure. No functional change intended.

MFC after: 1 week
Sponsored by: The FreeBSD Foundation

show more ...

ithreads: Support priority adjustment by schedulers.

Use sched_wakeup instead of sched_add when marking an ithread
runnable. This allows schedulers to reset their internal time slice
tracking state

ithreads: Support priority adjustment by schedulers.

Use sched_wakeup instead of sched_add when marking an ithread
runnable. This allows schedulers to reset their internal time slice
tracking state and restore the base ithread priority when an ithread
resumes from idle.

Reviewed by: markj
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D35643

show more ...

Add sched_ithread_prio to set the base priority of an interrupt thread.

Use it instead of sched_prio when setting the priority of an interrupt
thread.

Reviewed by: kib, markj
Sponsored by: Netflix

Add sched_ithread_prio to set the base priority of an interrupt thread.

Use it instead of sched_prio when setting the priority of an interrupt
thread.

Reviewed by: kib, markj
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D35642

show more ...

callout: Simplify cpuid validation in callout_reset_sbt_on()

- Remove a flag variable.
- Convert a runtime check of the passed cpuid to a KASSERT.
- Remove the cc_inited flag. An attempt to schedul

callout: Simplify cpuid validation in callout_reset_sbt_on()

- Remove a flag variable.
- Convert a runtime check of the passed cpuid to a KASSERT.
- Remove the cc_inited flag. An attempt to schedule a callout before
SI_SUB_CPU will crash anyway since the per-CPU mutexes won't have been
initialized, and that flag was only checked in the case where a cpuid
was explicitly specified by the caller.

No functional change intended.

MFC after: 2 weeks
Sponsored by: The FreeBSD Foundation

show more ...

Run softclock threads at a hardware ithread priority.

Add a new PI_SOFTCLOCK for use by softclock threads. Currently this
maps to PI_AV which is the second-highest ithread priority.

Reviewed by: m

Run softclock threads at a hardware ithread priority.

Add a new PI_SOFTCLOCK for use by softclock threads. Currently this
maps to PI_AV which is the second-highest ithread priority.

Reviewed by: mav, kib
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D33693

show more ...

callout: fix using shared rmlocks

15b1eb142c changed the callout code to store the CALLOUT_SHAREDLOCK flag
in c_iflags (where it used to be c_flags), but failed to update the
check in softclock_call

callout: fix using shared rmlocks

15b1eb142c changed the callout code to store the CALLOUT_SHAREDLOCK flag
in c_iflags (where it used to be c_flags), but failed to update the
check in softclock_call_cc(). This resulted in the callout code always
taking the write lock, even if a read lock had been requested (with
the CALLOUT_SHAREDLOCK flag in callout_init_rm()).

Reviewed by: markj
MFC after: 1 week
Sponsored by: Rubicon Communications, LLC ("Netgate")
Differential Revision: https://reviews.freebsd.org/D34959

show more ...

callout: Remove the CS_EXECUTING flag

It is now unused.

MFC after: 1 month
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D34626

kern: harvest entropy from callouts

74cf7cae4d22 ("softclock: Use dedicated ithreads for running callouts.")
switched callouts away from the swi infrastructure. It turns out that
this was a major s

kern: harvest entropy from callouts

74cf7cae4d22 ("softclock: Use dedicated ithreads for running callouts.")
switched callouts away from the swi infrastructure. It turns out that
this was a major source of entropy in early boot, which we've now lost.

As a result, first boot on hardware without a 'fast' entropy source
would block waiting for fortuna to be seeded with little hope of
progressing without manual intervention.

Let's resolve it by explicitly harvesting entropy in callout_process()
if we've handled any callouts. cc/curthread/now seem to be reasonable
sources of entropy, so use those.

Discussed with: jhb (also proposed initial patch)
Reported by: many
Reviewed by: cem, markm (both csprng)
Differential Revision: https://reviews.freebsd.org/D34150

show more ...

Reduce minimum idle hardclock rate from 2Hz to 1Hz.

On idle 80-thread system it allows to improve package-level idle state
residency and so power consumption by several percent.

MFC after: 2 weeks

callout: Wait for the softclock thread to switch before rescheduling

When a softclock thread prepares to go off-CPU, the following happens in
the context of the thread:

1. callout state is locked
2

callout: Wait for the softclock thread to switch before rescheduling

When a softclock thread prepares to go off-CPU, the following happens in
the context of the thread:

1. callout state is locked
2. thread state is set to IWAIT
3. thread lock is switched from the tdq lock to the callout lock
4. tdq lock is released
5. sched_switch() sets td_lock to &blocked_lock
6. sched_switch() releases old td_lock (callout lock)
7. sched_switch() removes td from its runqueue
8. cpu_switch() sets td_lock back to the callout lock

Suppose a timer interrupt fires while the softclock thread is switching
off, and callout_process() schedules the softclock thread. Then there
is a window between steps 5 and 8 where callout_process() can call
sched_add() while td_lock is &blocked_lock, but this is not correct
since the thread is not logically locked.

callout_process() thus needs to spin waiting for the softclock thread to
finish switching off (i.e., after step 8 completes) before rescheduling
it, since callout_process() does not acquire the thread lock directly.

Reported by: syzbot+fb44dbf6734ff492c337@syzkaller.appspotmail.com
Fixes: 74cf7cae4d22 ("softclock: Use dedicated ithreads for running callouts.")
Reviewed by: mav, kib, jhb
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D33709

show more ...

softclock: Use dedicated ithreads for running callouts.

Rather than using the swi infrastructure, rewrite softclock() as a
thread loop (softclock_thread()) and use it as the main routine of the
soft

softclock: Use dedicated ithreads for running callouts.

Rather than using the swi infrastructure, rewrite softclock() as a
thread loop (softclock_thread()) and use it as the main routine of the
softclock threads. The threads use the CC_LOCK as the thread lock
when idle.

Reviewed by: mav, imp, kib
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D33683

show more ...

callout(9): Allow spin locks use with callout_init_mtx().

Implement lock_spin()/unlock_spin() lock class methods, moving the
assertion to _sleep() instead. Change assertions in callout(9) to
allow

callout(9): Allow spin locks use with callout_init_mtx().

Implement lock_spin()/unlock_spin() lock class methods, moving the
assertion to _sleep() instead. Change assertions in callout(9) to
allow spin locks for both regular and C_DIRECT_EXEC cases. In case of
C_DIRECT_EXEC callouts spin locks are the only locks allowed actually.

As the first use case allow taskqueue_enqueue_timeout() use on fast
task queues. It actually becomes more efficient due to avoided extra
context switches in callout(9) thanks to C_DIRECT_EXEC.

MFC after: 2 weeks
Reviewed by: hselasky
Differential Revision: https://reviews.freebsd.org/D31778

show more ...

callout: Make cc_cpu local to kern_timeout.c

No functional change intended.

MFC after: 1 week
Sponsored by: The FreeBSD Foundation

Balance parentheses in sysctl descriptions