sx: fix compilation on UP kernels after r313855

sx primitives use inlines as opposed to macros. Change the tested condition
to LOCK_DEBUG which covers the case, but is slightly overzelaous.

Reporte

sx: fix compilation on UP kernels after r313855

sx primitives use inlines as opposed to macros. Change the tested condition
to LOCK_DEBUG which covers the case, but is slightly overzelaous.

Reported by: kib

show more ...

locks: let primitives for modules unlock without always goging to the slsow path

It is only needed if the LOCK_PROFILING is enabled. It has to always check if
the lock is about to be released which

locks: let primitives for modules unlock without always goging to the slsow path

It is only needed if the LOCK_PROFILING is enabled. It has to always check if
the lock is about to be released which requires an avoidable read if the option
is not specified..

show more ...

locks: remove SCHEDULER_STOPPED checks from primitives for modules

They all fallback to the slow path if necessary and the check is there.

This means a panicked kernel executing code from modules w

locks: remove SCHEDULER_STOPPED checks from primitives for modules

They all fallback to the slow path if necessary and the check is there.

This means a panicked kernel executing code from modules will be able to
succeed doing actual lock/unlock, but this was already the case for core code
which has said primitives inlined.

show more ...

Merge ^/head r313301 through r313643.

locks: tidy up unlock fallback paths

Update comments to note these functions are reachable if lockstat is
enabled.

Check if the lock has any bits set before attempting unlock, which saves
an unnece

locks: tidy up unlock fallback paths

Update comments to note these functions are reachable if lockstat is
enabled.

Check if the lock has any bits set before attempting unlock, which saves
an unnecessary atomic operation.

show more ...

sx: implement slock/sunlock fast path

See r313454.

MFhead@r313398

locks: change backoff to exponential

Previous implementation would use a random factor to spread readers and
reduce chances of starvation. This visibly reduces effectiveness of the
mechanism.

Switc

locks: change backoff to exponential

Previous implementation would use a random factor to spread readers and
reduce chances of starvation. This visibly reduces effectiveness of the
mechanism.

Switch to the more traditional exponential variant. Try to limit starvation
by imposing an upper limit of spins after which spinning is half of what
other threads get. Note the mechanism is turned off by default.

Reviewed by: kib (previous version)

show more ...

MFhead@r313360

locks: fix recursion support after recent changes

When a relevant lockstat probe is enabled the fallback primitive is called with
a constant signifying a free lock. This works fine for typical cases

locks: fix recursion support after recent changes

When a relevant lockstat probe is enabled the fallback primitive is called with
a constant signifying a free lock. This works fine for typical cases but breaks
with recursion, since it checks if the passed value is that of the executing
thread.

Read the value if necessary.

show more ...

Merge ^/head r313055 through r313300.

sx: move lockstat handling out of inline primitives

See r313275 for details.

sx: add witness support missed in r313272

sx: uninline slock/sunlock

Shared locking routines explicitly read the value and test it. If the
change attempt fails, they fall back to a regular function which would
retry in a loop.

The problem

sx: uninline slock/sunlock

Shared locking routines explicitly read the value and test it. If the
change attempt fails, they fall back to a regular function which would
retry in a loop.

The problem is that with many concurrent readers the risk of failure is pretty
high and even the value returned by fcmpset is very likely going to be stale
by the time the loop in the fallback routine is reached.

Uninline said primitives. It gives a throughput increase when doing concurrent
slocks/sunlocks with 80 hardware threads from ~50 mln/s to ~56 mln/s.

Interestingly, rwlock primitives are already not inlined.

show more ...

sx: switch to fcmpset

Discussed with: jhb
Tested by: pho (previous version)

MFhead@r313243

Merge ^/head r312720 through r312893.

Sprinkle __read_mostly on backoff and lock profiling code.

MFC after: 1 month

Merge ^/head r312309 through r312623.

sx: reduce lock accesses similarly to r311172

Discussed with: jhb
Tested by: pho (previous version)

Merge ^/head r309758 through r309803.

Return a non-NULL owner only if the lock is exclusively held in owner_sx().

Fix some whitespace bugs while here.

MFC after: 2 weeks

Merge ^/head r303250 through r304235.

locks: fix sx compilation on mips after r303643

The kernel.h header is required for the SYSINIT macro, which apparently
was present on amd64 by accident.

Reported by: kib

locks: fix compilation for KDTRACE_HOOKS && !ADAPTIVE_* case

Reported by: Michael Butler <imb protected-networks.net>