ipsec offload: ipsec_accel_fill_xh() should indirect through fn pointer

The config with IPSEC_SUPPORT + IPSEC_OFFLOAD is the valid one.

Fixes: 5be5a0bde5f990dbc680272eee74132bcde815f2
Sponsored by:

ipsec offload: ipsec_accel_fill_xh() should indirect through fn pointer

The config with IPSEC_SUPPORT + IPSEC_OFFLOAD is the valid one.

Fixes: 5be5a0bde5f990dbc680272eee74132bcde815f2
Sponsored by: Nvidia networking

show more ...

ipsec_offload: pre-calculate xform_history for recorded accelerated SA

Also provide the helper to fill the xform_history into mtag, provided by
a driver.

Reviewed by: Ariel Ehrenberg <aehrenberg@nv

ipsec_offload: pre-calculate xform_history for recorded accelerated SA

Also provide the helper to fill the xform_history into mtag, provided by
a driver.

Reviewed by: Ariel Ehrenberg <aehrenberg@nvidia.com>, slavash
Sponsored by: Nvidia networking

show more ...

machine/stdarg.h -> sys/stdarg.h

Switch to using sys/stdarg.h for va_list type and va_* builtins.

Make an attempt to insert the include in a sensible place. Where
style(9) was followed this is eas

machine/stdarg.h -> sys/stdarg.h

Switch to using sys/stdarg.h for va_list type and va_* builtins.

Make an attempt to insert the include in a sensible place. Where
style(9) was followed this is easy, where it was ignored, aim for the
first block of sys/*.h headers and don't get too fussy or try to fix
other style bugs.

Reviewed by: imp
Exp-run by: antoine (PR 286274)
Pull Request: https://github.com/freebsd/freebsd-src/pull/1595

show more ...

net: Include private header in more needed places

sys/netinet and sys/netipsec are both part of the 'blessed' netstack, so
can access struct ifnet directly. With this structure becoming private
ver

net: Include private header in more needed places

sys/netinet and sys/netipsec are both part of the 'blessed' netstack, so
can access struct ifnet directly. With this structure becoming private
very soon, the necessary files need to get direct access.

Sponsored by: Juniper Networks, Inc.

show more ...

ipsec: Drain async ipsec_offload work when destroying a vnet

Re-apply commit e196b12f4d4d. This was reverted by commit 28294dc92476
because it could trigger a deadlock, but the underlying problem t

ipsec: Drain async ipsec_offload work when destroying a vnet

Re-apply commit e196b12f4d4d. This was reverted by commit 28294dc92476
because it could trigger a deadlock, but the underlying problem there
was fixed in commit f76826b892de.

Reported by: KASAN
Reviewed by: kib
Fixes: ef2a572bf6bd ("ipsec_offload: kernel infrastructure")
Sponsored by: Klara, Inc.
Differential Revision: https://reviews.freebsd.org/D46483

show more ...

ipsec offload: use private taskqueue thread

Using global taskqueue_thread XXX with the vnet tasks scheduled during
VNET destruction. VNET shutdown needs to wait for all vnet-scoped
SAs/SPs to be ha

ipsec offload: use private taskqueue thread

Using global taskqueue_thread XXX with the vnet tasks scheduled during
VNET destruction. VNET shutdown needs to wait for all vnet-scoped
SAs/SPs to be handled, and doing that from taskqueue_thread task
deadlocks because the same thread proceeds the removals.

Reviewed by: markj
Sponsored by: NVidia networking
Differential revision: https://reviews.freebsd.org/D46494

show more ...

ipsec_offlad: remove not needed IFP_HS_INPUT/OUTPUT flags

Calculate the hdr_ext_size unconditionally, it is kept unused for SAs
not handling the input.

Sponsored by: NVidia networking

ipsec_accel_output(): do not process packet if interface rejected offload

Sponsored by: NVidia networking

Revert "ipsec: Drain async ipsec_offload work when destroying a vnet"

This change can cause a deadlock in some cases, since it's possible for
VNET teardown to happen in the context of taskqueue_thre

Revert "ipsec: Drain async ipsec_offload work when destroying a vnet"

This change can cause a deadlock in some cases, since it's possible for
VNET teardown to happen in the context of taskqueue_thread, and
ipsec_accel_sync() drains taskqueue_thread's work queue.

This reverts commit e196b12f4d4d13982d42cf7841b77dca1405effa.

show more ...

ipsec: Drain async ipsec_offload work when destroying a vnet

The ipsec_offload code in some cases releases object references in an
asynchronous context where it needs to set the current VNET. Make

ipsec: Drain async ipsec_offload work when destroying a vnet

The ipsec_offload code in some cases releases object references in an
asynchronous context where it needs to set the current VNET. Make sure
that all such work completes before the VNET is actually destroyed,
otherwise a use-after-free is possible.

Reported by: KASAN
Reviewed by: kib
Fixes: ef2a572bf6bd ("ipsec_offload: kernel infrastructure")
Sponsored by: Klara, Inc.
Differential Revision: https://reviews.freebsd.org/D46483

show more ...

ipsec_offload: add ipsec_accel_drv_sa_lifetime_fetch()

A function to fetch hardware counters for offloaded SA on specific
interface.

Sponsored by: NVidia networking

ipsec_offload: add handler for interface down events

Remove all offloaded SAs and SPs on ifdown.

Sponsored by: NVIDIA networking

ipsec_offload: indirect two more functions on the ipsec.ko module load

Specifically, ipsec_accel_on_ifdown() and ipsec_accel_drv_sa_lifetime_update()
should be present in kernel for future mlx5en dr

ipsec_offload: indirect two more functions on the ipsec.ko module load

Specifically, ipsec_accel_on_ifdown() and ipsec_accel_drv_sa_lifetime_update()
should be present in kernel for future mlx5en driver to be statically
linkable into the kernel built with IPSEC_HOOKS + IPSEC_OFFLOAD.

Sponsored by: NVIDIA networking

show more ...

ipsec_offload: hide SA/SP offload lifecycle prints under verbose sysctl

Reported and reviewed by: kp
Discussed with: np
Sponsored by: NVIDIA networking
Differential revision: https://reviews.freebsd

ipsec_offload: hide SA/SP offload lifecycle prints under verbose sysctl

Reported and reviewed by: kp
Discussed with: np
Sponsored by: NVIDIA networking
Differential revision: https://reviews.freebsd.org/D46045

show more ...

ipsec_offload: switch TF2_IPSEC_TSO on/off as appropriate on output

after the interface ipsec_accel method if_hwassist() is consulted.

Sponsored by: NVIDIA networking

ipsec_offload: offload inner checksums calculations for UDP/TCP/TSO

and allow the interface driver to declare such support.

Sponsored by: NVIDIA networking
Differential revision: https://reviews.fr

ipsec_offload: offload inner checksums calculations for UDP/TCP/TSO

and allow the interface driver to declare such support.

Sponsored by: NVIDIA networking
Differential revision: https://reviews.freebsd.org/D44221

show more ...

ipsec_offload: kernel infrastructure

Inline IPSEC offload moves almost whole IPSEC processing from the
CPU/MCU and possibly crypto accelerator, to the network card.

The transmitted packet content i

ipsec_offload: kernel infrastructure

Inline IPSEC offload moves almost whole IPSEC processing from the
CPU/MCU and possibly crypto accelerator, to the network card.

The transmitted packet content is not touched by CPU during TX
operations, kernel only does the required policy and security
association lookups to find out that given flow is offloaded, and then
packet is transmitted as plain text to the card. For driver convenience,
a metadata is attached to the packet identifying SA which must process
the packet. Card does encryption of the payload, padding, calculates
authentication, and does the reformat according to the policy.

Similarly, on receive, card does the decapsulation, decryption, and
authentification. Kernel receives the identifier of SA that was
used to process the packet, together with the plain-text packet.

Overall, payload octets are only read or written by card DMA engine,
removing a lot of memory subsystem overhead, and saving CPU time because
IPSEC algos calculations are avoided.

If driver declares support for inline IPSEC offload (with the
IFCAP2_IPSEC_OFFLOAD capability set and registering method table struct
if_ipsec_accel_methods), kernel offers the SPD and SAD to driver.
Driver decides which policies and SAs can be offloaded based on
hardware capacity, and acks/nacks each SA for given interface to
kernel. Kernel needs to keep this information to make a decision to
skip software processing on TX, and to assume processing already done
on RX. This shadow SPD/SAD database of offloads is rooted from
policies (struct secpolicy accel_ifps, struct ifp_handle_sp) and SAs
(struct secasvar accel_ipfs, struct ifp_handle_sav).

Some extensions to the PF_KEY socket allow to limit interfaces for
which given SP/SA could be offloaded (proposed for offload). Also,
additional statistics extensions allow to observe allocation/octet/use
counters for specific SA.

Since SPs and SAs are typically instantiated in non-sleepable context,
while offloading them into card is expected to require costly async
manipulations of the card state, calls to the driver for offload and
termination are executed in the threaded taskqueue. It also solves
the issue of allocating resources needed for the offload database.
Neither ipf_handle_sp nor ipf_handle_sav do not add reference to the
owning SP/SA, the offload must be terminated before last reference is
dropped. ipsec_accel only adds transient references to ensure safe
pointer ownership by taskqueue.

Maintaining the SA counters for hardware-accelerated packets is the
duty of the driver. The helper ipsec_accel_drv_sa_lifetime_update()
is provided to hide accel infrastructure from drivers which would use
expected callout to query hardware periodically for updates.

Reviewed by: rscheff (transport, stack integration), np
Sponsored by: NVIDIA networking
Differential revision: https://reviews.freebsd.org/D44219

show more ...