MFH @ r337607, in preparation for boarding

Merge ^/head r337286 through r337585.

epoch_block_wait: don't check TD_RUNNING

struct epoch_thread is not type safe (stack allocated) and thus cannot be dereferenced from another CPU

Reported by: novel@

epoch(9): simplify initialization

replace manual NUMA aware allocation with a pcpu zone

epoch(9): make nesting assert in epoch_wait_preempt more specific

Reported by: markj

epoch(9): allow preemptible epochs to compose

- Add tracker argument to preemptible epochs
- Inline epoch read path in kernel and tied modules
- Change in_epoch to take an epoch as argument
- Simpli

epoch(9): allow preemptible epochs to compose

- Add tracker argument to preemptible epochs
- Inline epoch read path in kernel and tied modules
- Change in_epoch to take an epoch as argument
- Simplify tfb_tcp_do_segment to not take a ti_locked argument,
there's no longer any benefit to dropping the pcbinfo lock
and trying to do so just adds an error prone branchfest to
these functions
- Remove cases of same function recursion on the epoch as
recursing is no longer free.
- Remove the the TAILQ_ENTRY and epoch_section from struct
thread as the tracker field is now stack or heap allocated
as appropriate.

Tested by: pho and Limelight Networks
Reviewed by: kbowling at llnw dot com
Sponsored by: Limelight Networks
Differential Revision: https://reviews.freebsd.org/D16066

show more ...

fix assert and conditionally allow mutexes to be held across epoch_wait_preempt

epoch(9): Don't trigger taskq enqueue before the grouptaskqs are setup

If EARLY_AP_STARTUP is not defined it is possible for an epoch to be
allocated prior to it being possible to call epoch_call wi

epoch(9): Don't trigger taskq enqueue before the grouptaskqs are setup

If EARLY_AP_STARTUP is not defined it is possible for an epoch to be
allocated prior to it being possible to call epoch_call without
issue.

Based on patch by andrew@

PR: 229014
Reported by: andrew

show more ...

epoch(9): make non-preemptible variant work early boot

epoch(9): make epoch closer to style(9)

Don't pass a section cookie to CK for non-preemptible epoch sections.

They're only useful when multiple threads may share an epoch record,
and that can't happen with non-preemptible sections.

Revie

Don't pass a section cookie to CK for non-preemptible epoch sections.

They're only useful when multiple threads may share an epoch record,
and that can't happen with non-preemptible sections.

Reviewed by: mmacy
Differential Revision: https://reviews.freebsd.org/D15507

show more ...

subr_epoch.c fix unused variable warnings

epoch(9): assert that epoch is allocated post-configure

epoch(9): Make epochs non-preemptible by default

There are risks associated with waiting on a preemptible epoch section.
Change the name to make them not be the default and document the issue
under

epoch(9): Make epochs non-preemptible by default

There are risks associated with waiting on a preemptible epoch section.
Change the name to make them not be the default and document the issue
under CAVEATS.

Reported by: markj

show more ...

epoch: actually allocate the counters we've assigned sysctls too

Approved by: sbruno

epoch: add non-preemptible "critical" variant

adds:
- epoch_enter_critical() - can be called inside a different epoch,
starts a section that will acquire any MTX_DEF mutexes or do
anything that

epoch: add non-preemptible "critical" variant

adds:
- epoch_enter_critical() - can be called inside a different epoch,
starts a section that will acquire any MTX_DEF mutexes or do
anything that might sleep.
- epoch_exit_critical() - corresponding exit call
- epoch_wait_critical() - wait variant that is guaranteed that any
threads in a section are running.
- epoch_global_critical - an epoch_wait_critical safe epoch instance

Requested by: markj
Approved by: sbruno

show more ...

epoch: skip poll function call in hardclock unless there are callbacks pending

Reported by: mjg
Approved by: sbruno

epoch(9): schedule pcpu callback task in hardclock if there are callbacks pending

Approved by: sbruno

epoch(9): eliminate the need to wait when polling for callbacks to run

by using ck's own callback handling mechanism we can simply check which
callbacks have had a grace period elapse

Approved by:

epoch(9): eliminate the need to wait when polling for callbacks to run

by using ck's own callback handling mechanism we can simply check which
callbacks have had a grace period elapse

Approved by: sbruno

show more ...

epoch(9): fix potential deadlock

Don't acquire a waiting thread's lock while holding our own

Approved by: sbruno

epoch(9): restore thread priority on exit if it was changed by a waiter

Reported by: markj
Approved by: sbruno

epoch(9): make recursion lighter weight

There isn't any real work to do except bump td_epochnest when recursing.
Skip the additional work in this case.

Approved by: sbruno

epoch(9): Guarantee forward progress on busy sections

Add epoch section to struct thread. We can use this to
ennable epoch counter to advance even if a section is
perpetually occupied by a thread.

epoch(9): Guarantee forward progress on busy sections

Add epoch section to struct thread. We can use this to
ennable epoch counter to advance even if a section is
perpetually occupied by a thread.

Approved by: sbruno

show more ...

epoch(9): allow sx locks to be held across epoch_wait()

The INVARIANTS checks in epoch_wait() were intended to
prevent the block handler from returning with locks held.
What it in fact did was preve

epoch(9): allow sx locks to be held across epoch_wait()

The INVARIANTS checks in epoch_wait() were intended to
prevent the block handler from returning with locks held.
What it in fact did was preventing anything except Giant
from being held across it. Check that the number of locks
held has not changed instead.

Approved by: sbruno@

show more ...

epoch(9): cleanups, additional debug checks, and add global_epoch

- GC the _nopreempt routines
- to really benefit we'd need a separate routine
- they're not currently in use
- they comp

epoch(9): cleanups, additional debug checks, and add global_epoch

- GC the _nopreempt routines
- to really benefit we'd need a separate routine
- they're not currently in use
- they complicate the API for no benefit at this time

- check that we're actually in a epoch section at exit

- handle epoch_call() early in boot

- Fix copyright declaration language

Approved by: sbruno@

show more ...