ktls: Remove the socket parameter to ktls_ocf_try()

The socket is unused, and not passing it means that there's less to
think about when considering how KTLS is synchronized with the rest of
the soc

ktls: Remove the socket parameter to ktls_ocf_try()

The socket is unused, and not passing it means that there's less to
think about when considering how KTLS is synchronized with the rest of
the socket code. No functional change intended.

Reviewed by: gallatin
MFC after: 2 weeks
Sponsored by: Klara, Inc.
Sponsored by: Stormshield
Differential Revision: https://reviews.freebsd.org/D45675

show more ...

sys: Automated cleanup of cdefs and other formatting

Apply the following automated changes to try to eliminate
no-longer-needed sys/cdefs.h includes as well as now-empty
blank lines in a row.

Remov

sys: Automated cleanup of cdefs and other formatting

Apply the following automated changes to try to eliminate
no-longer-needed sys/cdefs.h includes as well as now-empty
blank lines in a row.

Remove /^#if.*\n#endif.*\n#include\s+<sys/cdefs.h>.*\n/
Remove /\n+#include\s+<sys/cdefs.h>.*\n+#if.*\n#endif.*\n+/
Remove /\n+#if.*\n#endif.*\n+/
Remove /^#if.*\n#endif.*\n/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/types.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/param.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/capsicum.h>/

Sponsored by: Netflix

show more ...

sys: Remove $FreeBSD$: one-line .c pattern

Remove /^[\s*]*__FBSDID\("\$FreeBSD\$"\);?\s*\n/

ktls_ocf: Reject encrypted TLS records using AEAD that are too small.

If a TLS record is too small to contain the required explicit IV,
record_type (TLS 1.3), and MAC, reject attempts to decrypt it

ktls_ocf: Reject encrypted TLS records using AEAD that are too small.

If a TLS record is too small to contain the required explicit IV,
record_type (TLS 1.3), and MAC, reject attempts to decrypt it with
EMSGSIZE without submitting it to OCF. OCF drivers may not properly
detect that regions in the crypto request are outside the bounds of
the mbuf chain. The caller isn't supposed to submit such requests.

Reviewed by: markj
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D37372

show more ...

ktls: Add software support for AES-CBC decryption for TLS 1.1+.

This is mainly intended to provide a fallback for TOE TLS which may
need to use software decryption for an initial record at the start

ktls: Add software support for AES-CBC decryption for TLS 1.1+.

This is mainly intended to provide a fallback for TOE TLS which may
need to use software decryption for an initial record at the start
of a connection.

Reviewed by: markj
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D37370

show more ...

KTLS: Add a new recrypt operation to the software backend.

When using NIC TLS RX, packets that are dropped and retransmitted are
not decrypted by the NIC but are passed along as-is. As a result, a

KTLS: Add a new recrypt operation to the software backend.

When using NIC TLS RX, packets that are dropped and retransmitted are
not decrypted by the NIC but are passed along as-is. As a result, a
received TLS record might contain a mix of encrypted and decrypted
data. If this occurs, the already-decrypted data needs to be
re-encrypted so that the resulting record can then be decrypted
normally.

Add support for this for sessions using AES-GCM with TLS 1.2 or TLS
1.3. For the recrypt operation, allocate a temporary buffer and
encrypt the the payload portion of the TLS record with AES-CTR with an
initial IV constructed from the AES-GCM nonce. Then fixup the
original mbuf chain by copying the results from the temporary buffer
back into the original mbufs for any mbufs containing decrypted data.

Once it has been recrypted, the mbuf chain can then be decrypted via
the normal software decryption path.

Co-authored by: Hans Petter Selasky <hselasky@FreeBSD.org>
Reviewed by: hselasky
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D35012

show more ...

KTLS: Construct IV directly in crp.crp_iv for TLS 1.3 AEAD encryption.

Previously this used a temporary nonce[] buffer. The decrypt hook for
TLS 1.3 as well as the hooks for TLS 1.2 already constru

KTLS: Construct IV directly in crp.crp_iv for TLS 1.3 AEAD encryption.

Previously this used a temporary nonce[] buffer. The decrypt hook for
TLS 1.3 as well as the hooks for TLS 1.2 already constructed the IV
directly in crp.crp_iv.

Reviewed by: hselasky
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D35027

show more ...

KTLS: Move OCF function pointers out of ktls_session.

Instead, create a switch structure private to ktls_ocf.c and store a
pointer to the switch in the ocf_session. This will permit adding an
addit

KTLS: Move OCF function pointers out of ktls_session.

Instead, create a switch structure private to ktls_ocf.c and store a
pointer to the switch in the ocf_session. This will permit adding an
additional function pointer needed for NIC TLS RX without further
bloating ktls_session.

Reviewed by: hselasky
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D35011

show more ...

KTLS: Free the MAC session when destroying AES-CBC software sessions.

Reviewed by: hselasky
MFC after: 1 week
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D35013

ktls: Support for TLS 1.3 receive offload.

Note that support for TLS 1.3 receive offload in OpenSSL is still an
open pull request in active development. However, potential changes
to that pull requ

ktls: Support for TLS 1.3 receive offload.

Note that support for TLS 1.3 receive offload in OpenSSL is still an
open pull request in active development. However, potential changes
to that pull request should not affect the kernel interface.

Reviewed by: hselasky
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D33007

show more ...

ktls: Split encrypt vs decrypt OCF counters.

Reviewed by: gallatin, markj
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D33006

Drop "All rights reserved" from a Netflix copyright.

Reviewed by: imp
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D32778

Remove unused ocf_operation structure from stack.

Reviewed by: jhb@
MFC after: 1 week
Sponsored by: NVIDIA Networking

ktls: Fix assertion for TLS 1.0 CBC when using non-zero starting seqno.

The starting sequence number used to verify that TLS 1.0 CBC records
are encrypted in-order in the OCF layer was always set to

ktls: Fix assertion for TLS 1.0 CBC when using non-zero starting seqno.

The starting sequence number used to verify that TLS 1.0 CBC records
are encrypted in-order in the OCF layer was always set to 0 and not to
the initial sequence number from the struct tls_enable.

In practice, OpenSSL always starts TLS transmit offload with a
sequence number of zero, so this only matters for tests that use a
random starting sequence number.

Reviewed by: markj
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D32676

show more ...

ktls: Change struct ktls_session.cipher to an OCF-specific type.

As a followup to SW KTLS assuming an OCF backend, rename
struct ocf_session to struct ktls_ocf_session and forward
declare it in <sys

ktls: Change struct ktls_session.cipher to an OCF-specific type.

As a followup to SW KTLS assuming an OCF backend, rename
struct ocf_session to struct ktls_ocf_session and forward
declare it in <sys/ktls.h> to use as the type of
struct ktls_session.cipher.

Reviewed by: gallatin, hselasky
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D32565

show more ...

ktls: Support asynchronous dispatch of AEAD ciphers.

KTLS OCF support was originally targeted at software backends that
used host CPU cycles to encrypt TLS records. As a result, each KTLS
worker th

ktls: Support asynchronous dispatch of AEAD ciphers.

KTLS OCF support was originally targeted at software backends that
used host CPU cycles to encrypt TLS records. As a result, each KTLS
worker thread queued a single TLS record at a time and waited for it
to be encrypted before processing another TLS record. This works well
for software backends but limits throughput on OCF drivers for
coprocessors that support asynchronous operation such as qat(4) or
ccr(4). This change uses an alternate function (ktls_encrypt_async)
when encrypt TLS records via a coprocessor. This function queues TLS
records for encryption and returns. It defers the work done after a
TLS record has been encrypted (such as marking the mbufs ready) to a
callback invoked asynchronously by the coprocessor driver when a
record has been encrypted.

- Add a struct ktls_ocf_state that holds the per-request state stored
on the stack for synchronous requests. Asynchronous requests malloc
this structure while synchronous requests continue to allocate this
structure on the stack.

- Add a ktls_encrypt_async() variant of ktls_encrypt() which does not
perform request completion after dispatching a request to OCF.
Instead, the ktls_ocf backends invoke ktls_encrypt_cb() when a TLS
record request completes for an asynchronous request.

- Flag AEAD software TLS sessions as async if the backend driver
selected by OCF is an async driver.

- Pull code to create and dispatch an OCF request out of
ktls_encrypt() into a new ktls_encrypt_one() function used by both
ktls_encrypt() and ktls_encrypt_async().

- Pull code to "finish" the VM page shuffling for a file-backed TLS
record into a helper function ktls_finish_noanon() used by both
ktls_encrypt() and ktls_encrypt_cb().

Reviewed by: markj
Tested on: ccr(4) (jhb), qat(4) (markj)
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D31665

show more ...

Include the trailer in the original dst_iov.

This avoids creating a duplicate copy on the stack just to
append the trailer.

Reviewed by: gallatin, markj
Sponsored by: Netflix
Differential Revision:

Include the trailer in the original dst_iov.

This avoids creating a duplicate copy on the stack just to
append the trailer.

Reviewed by: gallatin, markj
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D30139

show more ...

Assume OCF is the only KTLS software backend.

This removes support for loadable software backends. The KTLS OCF
support is now always included in kernels with KERN_TLS and the
ktls_ocf.ko module ha

Assume OCF is the only KTLS software backend.

This removes support for loadable software backends. The KTLS OCF
support is now always included in kernels with KERN_TLS and the
ktls_ocf.ko module has been removed. The software encryption routines
now take an mbuf directly and use the TLS mbuf as the crypto buffer
when possible.

Bump __FreeBSD_version for software backends in ports.

Reviewed by: gallatin, markj
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D30138

show more ...

ktls_ocf: Fix a few places to not hardcode the GMAC hash length.

This is not a functional change as the Poly1305 hash is the same
length as the GMAC hash length.

Reviewed by: gallatin, markj
Sponso

ktls_ocf: Fix a few places to not hardcode the GMAC hash length.

This is not a functional change as the Poly1305 hash is the same
length as the GMAC hash length.

Reviewed by: gallatin, markj
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D30137

show more ...

ktls: Fix non-inplace TLS 1.3 encryption.

Copy the iovec for the trailer from the proper place. This is the same
fix for CBC encryption from ff6a7e4ba6bf.

Reported by: gallatin
Reviewed by: gallat

ktls: Fix non-inplace TLS 1.3 encryption.

Copy the iovec for the trailer from the proper place. This is the same
fix for CBC encryption from ff6a7e4ba6bf.

Reported by: gallatin
Reviewed by: gallatin, markj
Fixes: 49f6925ca
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D29177

show more ...

ktls: Fix CBC encryption when input and output iov sizes are different

Reported by: gallatin
Tested by: gallatin
Fixes: 49f6925ca
Differential Revision: https://reviews.freebsd.org/D29073

ktls: Cache output buffers for software encryption

Maintain a cache of physically contiguous runs of pages for use as
output buffers when software encryption is configured and in-place
encryption is

ktls: Cache output buffers for software encryption

Maintain a cache of physically contiguous runs of pages for use as
output buffers when software encryption is configured and in-place
encryption is not possible. This makes allocation and free cheaper
since in the common case we avoid touching the vm_page structures for
the buffer, and fewer calls into UMA are needed. gallatin@ reports a
~10% absolute decrease in CPU usage with sendfile/KTLS on a Xeon after
this change.

It is possible that we will not be able to allocate these buffers if
physical memory is fragmented. To avoid frequently calling into the
physical memory allocator in this scenario, rate-limit allocation
attempts after a failure. In the failure case we fall back to the old
behaviour of allocating a page at a time.

N.B.: this scheme could be simplified, either by simply using malloc()
and looking up the PAs of the pages backing the buffer, or by falling
back to page by page allocation and creating a mapping in the cache
zone. This requires some way to save a mapping of an M_EXTPG page array
in the mbuf, though. m_data is not really appropriate. The second
approach may be possible by saving the mapping in the plinks union of
the first vm_page structure of the array, but this would force a vm_page
access when freeing an mbuf.

Reviewed by: gallatin, jhb
Tested by: gallatin
Sponsored by: Ampere Computing
Submitted by: Klara, Inc.
Differential Revision: https://reviews.freebsd.org/D28556

show more ...

Add Chacha20-Poly1305 support in the OCF backend for KTLS.

This supports Chacha20-Poly1305 for both send and receive for TLS 1.2
and for send in TLS 1.3.

Reviewed by: gallatin
Sponsored by: Netflix

Add Chacha20-Poly1305 support in the OCF backend for KTLS.

This supports Chacha20-Poly1305 for both send and receive for TLS 1.2
and for send in TLS 1.3.

Reviewed by: gallatin
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D27841

show more ...

ktls: Avoid wakeups and locking for synchronous callbacks

When performing encryption in software, the KTLS crypto callback always
locks the session to deliver a wakeup. But, if we're handling the
o

ktls: Avoid wakeups and locking for synchronous callbacks

When performing encryption in software, the KTLS crypto callback always
locks the session to deliver a wakeup. But, if we're handling the
operation synchronously this is wasted effort and can result in
sleepqueue lock contention on large systems.

Use CRYPTO_SESS_SYNC() to determine whether the operation will be
completed asynchronously or not, and select a callback appropriately.
Avoid locking the session to check for completion if the session handles
requests synchronously.

Reviewed by: jhb
Sponsored by: Ampere Computing
Submitted by: Klara, Inc.
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D28195

show more ...

ktls: Use COUNTER_U64_DEFINE_EARLY

This makes it a bit more straightforward to add new counters when
debugging. No functional change intended.

Reviewed by: jhb
Sponsored by: Ampere Computing
Submi

ktls: Use COUNTER_U64_DEFINE_EARLY

This makes it a bit more straightforward to add new counters when
debugging. No functional change intended.

Reviewed by: jhb
Sponsored by: Ampere Computing
Submitted by: Klara, Inc.
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D28498

show more ...