Merge ^/head r325999 through r326131.

sx: unbreak debug after r326107

An assertion was modified to use the found value, but it was not updated to
handle a race where blocked threads appear after the entrance to the func.

Move the asser

sx: unbreak debug after r326107

An assertion was modified to use the found value, but it was not updated to
handle a race where blocked threads appear after the entrance to the func.

Move the assertion down to the area protected with sleepq lock where the
lock is read anyway. This does not affect coverage of the assertion and
is consistent with what rw locks are doing.

Reported by: Shawn Webb

show more ...

locks: pass the found lock value to unlock slow path

This avoids an explicit read later.

While here whack the cheaply obtainable 'tid' argument.

locks: remove the file + line argument from internal primitives when not used

The pair is of use only in debug or LOCKPROF kernels, but was passed (zeroed)
for many locks even in production kernels.

locks: remove the file + line argument from internal primitives when not used

The pair is of use only in debug or LOCKPROF kernels, but was passed (zeroed)
for many locks even in production kernels.

While here whack the tid argument from wlock hard and xlock hard.

There is no kbi change of any sort - "external" primitives still accept the
pair.

show more ...

Merge ^/head r325842 through r325998.

locks: fix compilation issues without SMP or KDTRACE_HOOKS

sx: perform a minor cleanup of the unlock slowpath

No functional changes.

locks: pull up PMC_SOFT_CALLs out of slow path loops

sx: avoid branches if in the slow path if lockstat is disabled

Merge ^/head r323559 through r325504.

MFhead@r324482

locks: take the number of readers into account when waiting

Previous code would always spin once before checking the lock. But a lock
with e.g. 6 readers is not going to become free in the duration

locks: take the number of readers into account when waiting

Previous code would always spin once before checking the lock. But a lock
with e.g. 6 readers is not going to become free in the duration of once spin
even if they start draining immediately.

Conservatively perform one for each reader.

Note that the total number of allowed spins is still extremely small and is
subject to change later.

MFC after: 1 week

show more ...

locks: partially tidy up waiting on readers

spin first instant of instantly re-readoing and don't re-read after
spinning is finished - the state is already known.

Note the code is subject to signif

locks: partially tidy up waiting on readers

spin first instant of instantly re-readoing and don't re-read after
spinning is finished - the state is already known.

Note the code is subject to significant changes later.

MFC after: 1 week

show more ...

MFH @ r323558.

MFhead@r323343

Sprinkle __read_frequently on few obvious places.

Note that some of annotated variables should probably change their types
to something smaller, preferably bit-sized.

Merge ^/head r319973 through 321382.

Merge ^/head r320042 through r320397.

Fix the !TD_IS_IDLETHREAD(curthread) locking assertions.

Most of the lock slowpaths assert that the calling thread isn't an idle
thread. However, this may not be true if the system has panicked, and

Fix the !TD_IS_IDLETHREAD(curthread) locking assertions.

Most of the lock slowpaths assert that the calling thread isn't an idle
thread. However, this may not be true if the system has panicked, and in
some cases the assertion appears before a SCHEDULER_STOPPED() check.

MFC after: 3 days
Sponsored by: Dell EMC Isilon

show more ...

Merge ^/head r314420 through r314481.

locks: ensure proper barriers are used with atomic ops when necessary

Unclear how, but the locking routine for mutexes was using the *release*
barrier instead of acquire. This must have been either

locks: ensure proper barriers are used with atomic ops when necessary

Unclear how, but the locking routine for mutexes was using the *release*
barrier instead of acquire. This must have been either a copy-pasto or bad
completion.

Going through other uses of atomics shows no barriers in:
- upgrade routines (addressed in this patch)
- sections protected with turnstile locks - this should be fine as necessary
barriers are in the worst case provided by turnstile unlock

I would like to thank Mark Millard and andreast@ for reporting the problem and
testing previous patches before the issue got identified.

ps.
.-'---`-.
,' `.
| \
| \
\ _ \
,\ _ ,'-,/-)\
( * \ \,' ,' ,'-)
`._,) -',-')
\/ ''/
) / /
/ ,'-'

Hardware provided by: IBM LTC

show more ...

Merge ^/head r313896 through r314128.

locks: make trylock routines check for 'unowned' value

Since fcmpset can fail without lock contention e.g. on arm, it was possible
to get spurious failures when the caller was expecting the primitiv

locks: make trylock routines check for 'unowned' value

Since fcmpset can fail without lock contention e.g. on arm, it was possible
to get spurious failures when the caller was expecting the primitive to succeed.

Reported by: mmel

show more ...

locks: clean up trylock primitives

In particular thius reduces accesses of the lock itself.

Merge ^/head r313644 through r313895.