sys: Remove $FreeBSD$: two-line .h pattern

Remove /^\s*\*\n \*\s+\$FreeBSD\$$\n/

LinuxKPI: Constantly use _LINUXKPI_ prefix in include guards

MFC after: 1 week
Reviewed by: bz, emaste, hselasky, manu
Differential Revision: https://reviews.freebsd.org/D33562

Implement synchronize_srcu_expedited() in the LinuxKPI.

Differential Revision: https://reviews.freebsd.org/D24798
MFC after: 1 week
Sponsored by: Mellanox Technologies

Some fixes for SRCU in the LinuxKPI.

- Make sure to use READ_ONCE() when deferring variables.
- Remove superfluous zero initializer.

MFC after: 1 week
Sponsored by: Mellanox Technologies

MFHead@r345275

Implement DEFINE_STATIC_SRCU() function macro in the LinuxKPI.

Submitted by: Johannes Lundberg <johalun0@gmail.com>
MFC after: 1 week
Sponsored by: Limelight Networks
Sponsored by: Mellanox Tech

Implement DEFINE_STATIC_SRCU() function macro in the LinuxKPI.

Submitted by: Johannes Lundberg <johalun0@gmail.com>
MFC after: 1 week
Sponsored by: Limelight Networks
Sponsored by: Mellanox Technologies

show more ...

Merge ^/head r316992 through r317215.

Fix problem regarding priority inversion when using the concurrency
kit, CK, in the LinuxKPI.

When threads are pinned to a CPU core or when there is only one CPU,
it can happen that a higher priorit

Fix problem regarding priority inversion when using the concurrency
kit, CK, in the LinuxKPI.

When threads are pinned to a CPU core or when there is only one CPU,
it can happen that a higher priority thread can call the CK
synchronize function while a lower priority thread holds the read
lock. Because the CK's synchronize is a simple wait loop this can lead
to a deadlock situation. To solve this problem use the recently
introduced CK's wait callback function.

When detecting a CK blocking condition figure out the lowest priority
among the blockers and update the calling thread's priority and
yield. If another CPU core is holding the read lock, pin the thread to
the blocked CPU core and update the priority. The calling threads
priority and CPU bindings are restored before return.

If a thread holding a CK read lock is detected to be sleeping, pause()
will be used instead of yield().

MFC after: 1 week
Sponsored by: Mellanox Technologies

show more ...

Remove duplicate prototype in the LinuxKPI to fix compilation warning.

Reported by: emaste @
MFC after: 1 week
Sponsored by: Mellanox Technologies

Update the LinuxKPI RCU and SRCU wrappers for the concurrency kit, CK.

- Optimise the RCU implementation to not allocate and free
ck_epoch_records during runtime. Instead allocate two sets of
ck_epo

Update the LinuxKPI RCU and SRCU wrappers for the concurrency kit, CK.

- Optimise the RCU implementation to not allocate and free
ck_epoch_records during runtime. Instead allocate two sets of
ck_epoch_records per CPU for general purpose use. The first set is
only used for reader locks and the second set is only used for
synchronization and barriers and is protected with a regular mutex to
prevent simultaneous issues.

- Move the task structure away from the rcu_head structure and into
the per-CPU structures. This allows the size of the rcu_head structure
to be reduced down to the size of two pointers.

- Fix a bug where the linux_rcu_barrier() function only waited for one
per-CPU epoch record to be completed instead of all.

- Use a critical section or a mutex to protect ck_epoch_begin() and
ck_epoch_end() depending on RCU or SRCU type. All the ck_epoch_xxx()
functions, except ck_epoch_register(), ck_epoch_unregister() and
ck_epoch_recycle() are not re-entrant and needs a critical section or
a mutex to operate in the LinuxKPI, after inspecting the CK
implementation of the above mentioned functions. The simultaneous
issues arise from per-CPU epoch records being shared between multiple
threads depending on the amount of taskswitching and how many threads
are involved with the RCU and SRCU operations.

- Properly free all epoch records by using safe list traversal at
LinuxKPI module unload. It turns out the ck_epoch_recycle() always
have the records on an internal list and use a flag in the epoch
record to track allocated and free entries. This would lead to use
after free during module unload.

- Remove redundant synchronize_rcu() call from the
linux_compat_uninit() function. Let the linux_rcu_runtime_uninit()
function do the final rcu_barrier() instead.

MFC after: 1 week
Sponsored by: Mellanox Technologies

show more ...

Merge ^/head r314178 through r314269.

Implement srcu_dereference() macro in the LinuxKPI.

MFC after: 1 week
Sponsored by: Mellanox Technologies

Merge ^/head r313896 through r314128.

Replace dummy implementation of RCU in the LinuxKPI with one based on
the in-kernel concurrency kit's ck_epoch API. Factor RCU hlist_xxx()
functions into own rculist.h header file.

Obtained from: k

Replace dummy implementation of RCU in the LinuxKPI with one based on
the in-kernel concurrency kit's ck_epoch API. Factor RCU hlist_xxx()
functions into own rculist.h header file.

Obtained from: kmacy @
MFC after: 1 week
Sponsored by: Mellanox Technologies

show more ...

MFH r289384-r293170

Sponsored by: The FreeBSD Foundation

MFH @r292599

This includes the pluggable TCP framework and other chnages to the
netstack to track for VNET stability.

Security: The FreeBSD Foundation

Implement sleepable RCU mechanism using shared exclusive locks.

MFC after: 1 week
Sponsored by: Mellanox Technologies