Introduce the UMA_ZONE_NOTRIM uma zone type

The ktls buffer zone allocates 16k contiguous buffers, and often needs
to call vm_page_reclaim_contig_domain_ext() to free up contiguous
memory, which can

Introduce the UMA_ZONE_NOTRIM uma zone type

The ktls buffer zone allocates 16k contiguous buffers, and often needs
to call vm_page_reclaim_contig_domain_ext() to free up contiguous
memory, which can be expensive. Web servers which have a daily
pattern of peaks and troughs end up having UMA trim the
ktls_buffer_zone when they are in their trough, and end up re-building
it on the way to their peak.

Rather than calling vm_page_reclaim_contig_domain_ext() multiple times
on a daily basis, lets mark the ktls_buffer_zone with a new UMA flag,
UMA_ZONE_NOTRIM. This disables UMA_RECLAIM_TRIM on the zone, but
allows UMA_RECLAIM_DRAIN* operations, so that if we become extremely
short of memory (vm_page_count_severe()), the uma reclaim worker can
still free up memory.

Note that UMA_ZONE_UNMANAGED already exists, but can never be drained
or trimmed, so it may hold on to memory during times of severe memory
pressure. Using UMA_ZONE_NOTRIM rather than UMA_ZONE_UNMANAGED is an
attempt to keep this zone more reactive in the face of severe memory
pressure.

Sponsored by: Netflix
Reviewed by: jhb, kib, markj (via slack)
Differential Revision: https://reviews.freebsd.org/D48451

show more ...

ktls: Enable by default

Reviewed by: gallatin, markj
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D47735

ktls: Mark mbufs containing outbound encrypted TLS records read-only

Reviewed by: gallatin, kp
Differential Revision: https://reviews.freebsd.org/D46784

ktls: Remove the socket parameter to ktls_ocf_try()

The socket is unused, and not passing it means that there's less to
think about when considering how KTLS is synchronized with the rest of
the soc

ktls: Remove the socket parameter to ktls_ocf_try()

The socket is unused, and not passing it means that there's less to
think about when considering how KTLS is synchronized with the rest of
the socket code. No functional change intended.

Reviewed by: gallatin
MFC after: 2 weeks
Sponsored by: Klara, Inc.
Sponsored by: Stormshield
Differential Revision: https://reviews.freebsd.org/D45675

show more ...

ktls: Fix races that can lead to double initialization

ktls_enable_rx() and ktls_enable_tx() have checks to return EALREADY if
the socket already has KTLS enabled. However, these are done without
a

ktls: Fix races that can lead to double initialization

ktls_enable_rx() and ktls_enable_tx() have checks to return EALREADY if
the socket already has KTLS enabled. However, these are done without
any locks held and nothing blocks concurrent attempts to set the socket
option. I believe the worst outcome of the race is leaked memory.

Fix the problem by rechecking under the sockbuf lock. While here, unify
the locking protocol for sb_tls_info: require both the sockbuf and
socket I/O locks in order to enable KTLS. This means that either lock
is sufficient for checking whether KTLS is enabled in a given sockbuf,
which simplifies some refactoring further down the road.

Note that the SOLISTENING() check can go away because
SOCK_IO_RECV_LOCK() atomically locks the socket buffer and checks
whether the socket is a listening socket. This changes the returned
errno value, so update a test which checks it.

Reviewed by: gallatin
MFC after: 2 weeks
Sponsored by: Klara, Inc.
Sponsored by: Stormshield
Differential Revision: https://reviews.freebsd.org/D45674

show more ...

ktls: catch invalid parameters earlier

Move safety checks forward from ktls_session_create() to
ktls_copyin_tls_enable(). Prevents zero mallocs, and excessively
large kernel mallocs.

Reported-by: s

ktls: catch invalid parameters earlier

Move safety checks forward from ktls_session_create() to
ktls_copyin_tls_enable(). Prevents zero mallocs, and excessively
large kernel mallocs.

Reported-by: syzbot+72022fa9163fa958b66c@syzkaller.appspotmail.com
Reported-by: syzbot+8992893e13058ce0670a@syzkaller.appspotmail.com
Sponsored by: NetApp, Inc.
X-NetApp-PR: #79
Reviewed By: tuexen
Differential Revision: https://reviews.freebsd.org/D44364

show more ...

ktls: deep copy tls_enable struct for in-kernel tcp consumers

Doing a deep copy of the keys early allows users of the
tls_enable structure to assume kernel memory.
This enables the socket options to

ktls: deep copy tls_enable struct for in-kernel tcp consumers

Doing a deep copy of the keys early allows users of the
tls_enable structure to assume kernel memory.
This enables the socket options to be set by kernel threads.

Reviewed By: #transport, tuexen, jhb, rrs
Sponsored by: NetApp, Inc.
X-NetApp-PR: #79
Differential Revision: https://reviews.freebsd.org/D44250

show more ...

ktls: fix vnet-related panic in ktls_reset_receive_tag()

Reviewed by: gallatin, jhb
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D43400

Avoid waiting on physical allocations that can't possibly be satisfied

- Change vm_page_reclaim_contig[_domain] to return an errno instead
of a boolean. 0 indicates a successful reclaim, ENOMEM i

Avoid waiting on physical allocations that can't possibly be satisfied

- Change vm_page_reclaim_contig[_domain] to return an errno instead
of a boolean. 0 indicates a successful reclaim, ENOMEM indicates
lack of available memory to reclaim, with any other error (currently
only ERANGE) indicating that reclamation is impossible for the
specified address range. Change all callers to only follow
up with vm_page_wait* in the ENOMEM case.

- Introduce vm_domainset_iter_ignore(), which marks the specified
domain as unavailable for further use by the iterator. Use this
function to ignore domains that can't possibly satisfy a physical
allocation request. Since WAITOK allocations run the iterators
repeatedly, this avoids the possibility of infinitely spinning
in domain iteration if no available domain can satisfy the
allocation request.

PR: 274252
Reported by: kevans
Tested by: kevans
Reviewed by: markj
Differential Revision: https://reviews.freebsd.org/D42706

show more ...

ktls: Remove unneeded vm/uma_dbg.h include

It was used in original implementation, but is no longer.

MFC after: 2 weeks

sys: Remove $FreeBSD$: one-line .c pattern

Remove /^[\s*]*__FBSDID\("\$FreeBSD\$"\);?\s*\n/

ktls_alloc_rcv_tag: Fix capability checks for RXTLS4/6.

IFCAP2_* has the bit position and not the shifted value.

Reviewed by: kib@
MFC after: 1 week
Sponsored by: Chelsio Communications
Differentia

ktls_alloc_rcv_tag: Fix capability checks for RXTLS4/6.

IFCAP2_* has the bit position and not the shifted value.

Reviewed by: kib@
MFC after: 1 week
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D41100

show more ...

ktls: don't try to unlock pcb if tcp_drop() already did

Reviewed by: rrs, gallatin

ktls: re-work alloc thread

When the ktls_buffer zone needs to expand, it may fail due
to a lack of physically contiguous memory. We tried to rectify
that by introducing an alloc thread to provide a

ktls: re-work alloc thread

When the ktls_buffer zone needs to expand, it may fail due
to a lack of physically contiguous memory. We tried to rectify
that by introducing an alloc thread to provide a context where
it is harmless to sleep, and letting that thread repopulate
the ktls_buffer zone.

However, it turns out that M_WAITOK is not enough, and we
must call vm_page_reclaim_contig_domain() to reclaim contig
memory. Worse, M_WAITOK results in the allocation essentially
busy-looping around vm_domain_alloc_fail() returning EAGIN,
causing vm_page_alloc_noobj_contig_domain() to loop and resulting
in the alloc thread consuming 100% CPU.

To fix this, we change the alloc thread to call
vm_page_reclaim_contig_domain_ext()

In order to prevent the busy loop around vm_domain_alloc_fail(), we
must change the uma_zalloc flags to M_NORECLAIM | M_NOWAIT. However,
once that is done, these allocations become no different than the
allocations done in the critical path in ktls_buffer_alloc(), so its
best to just eliminate them.

Since we're no longer doing allocations but just calling
vm_page_reclaim_contig_domain_ext(), the name has changed to the ktls
reclaim thread.

Reviewed by: jhb, markj
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D39421

show more ...

ktls: Drop all the INET and INET6 compile-time guards.

Consistent with 9fd0d9b16e93ff2a3bd375a98763dca0150dcee0, KERN_TLS is
not supported on kernels without any INET support.

Reviewed by: gallatin

ktls: Drop all the INET and INET6 compile-time guards.

Consistent with 9fd0d9b16e93ff2a3bd375a98763dca0150dcee0, KERN_TLS is
not supported on kernels without any INET support.

Reviewed by: gallatin, hselasky
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D39232

show more ...

ktls: Fix interlocking between ktls_enable_rx() and listen(2)

The TCP_TXTLS_ENABLE and TCP_RXTLS_ENABLE socket option handlers check
whether the socket is listening socket and fail if so, but this c

ktls: Fix interlocking between ktls_enable_rx() and listen(2)

The TCP_TXTLS_ENABLE and TCP_RXTLS_ENABLE socket option handlers check
whether the socket is listening socket and fail if so, but this check is
racy. Since we have to lock the socket buffer later anyway, defer the
check to that point.

ktls_enable_tx() locks the send buffer's I/O lock, which will fail if
the socket is a listening socket, so no explicit checks are needed. In
ktls_enable_rx(), which does not acquire the I/O lock (see the review
for some discussion on this), use an explicit SOLISTENING() check after
locking the recv socket buffer.

Otherwise, a concurrent solisten_proto() call can trigger crashes and
memory leaks by wiping out socket buffers as ktls_enable_*() is
modifying them.

Also make sure that a KTLS-enabled socket can't be converted to a
listening socket, and use SOCK_(SEND|RECV)BUF_LOCK macros instead of the
old ones while here.

Add some simple regression tests involving listen(2).

Reported by: syzkaller
MFC after: 2 weeks
Reviewed by: gallatin, glebius, jhb
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D38504

show more ...

ktls: Use IfAPI accessors to get capabilities

Summary:
Avoid referencing the ifnet struct directly, and use the IfAPI accessors
instead.

Reviewed by: gallatin
Sponsored by: Juniper Networks, Inc.
D

ktls: Use IfAPI accessors to get capabilities

Summary:
Avoid referencing the ifnet struct directly, and use the IfAPI accessors
instead.

Reviewed by: gallatin
Sponsored by: Juniper Networks, Inc.
Differential Revision: https://reviews.freebsd.org/D38932

show more ...

ktls: Fix comments & whitespace issues with c0e4090e3d43

Address some last minute review feedback on c0e4090e3d43
by fixing spacing around comments, and clarifying that the
newly added destroy_task

ktls: Fix comments & whitespace issues with c0e4090e3d43

Address some last minute review feedback on c0e4090e3d43
by fixing spacing around comments, and clarifying that the
newly added destroy_task is not related to tls 1.0.
No functional change intended.

Pointed out by: jhb
Sponsored by: Netflix

show more ...

ktls: Accurately track if ifnet ktls is enabled

This allows us to avoid spurious calls to ktls_disable_ifnet()

When we implemented ifnet kTLSe, we set a flag in the tx socket
buffer (SB_TLS_IFNET)

ktls: Accurately track if ifnet ktls is enabled

This allows us to avoid spurious calls to ktls_disable_ifnet()

When we implemented ifnet kTLSe, we set a flag in the tx socket
buffer (SB_TLS_IFNET) to indicate ifnet kTLS. This flag meant that
now, or in the past, ifnet ktls was active on a socket. Later,
I added code to switch ifnet ktls sessions to software in the case
of lossy TCP connections that have a high retransmit rate.
Because TCP was using SB_TLS_IFNET to know if it needed to do math
to calculate the retransmit ratio and potentially call into
ktls_disable_ifnet(), it was doing unneeded work long after
a session was moved to software.

This patch carefully tracks whether or not ifnet ktls is still enabled
on a TCP connection. Because the inp is now embedded in the tcpcb, and
because TCP is the most frequent accessor of this state, it made sense to
move this from the socket buffer flags to the tcpcb. Because we now need
reliable access to the tcbcb, we take a ref on the inp when creating a tx
ktls session.

While here, I noticed that rack/bbr were incorrectly implementing
tfb_hwtls_change(), and applying the change to all pending sends,
when it should apply only to future sends.

This change reduces spurious calls to ktls_disable_ifnet() by 95% or so
in a Netflix CDN environment.

Reviewed by: markj, rrs
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D38380

show more ...

ktls_disable_ifnet_help: Set curvnet around sorele().

This is required in kernels with VIMAGE such as GENERIC.

MFC after: 1 week
Sponsored by: Chelsio Communications

ktls: Post receive errors on partially closed sockets.

If an error such as an invalid record or one whose decryption fails is
detected on a socket that has received a RST then ktls_drop() could
igno

ktls: Post receive errors on partially closed sockets.

If an error such as an invalid record or one whose decryption fails is
detected on a socket that has received a RST then ktls_drop() could
ignore the error since INP_DROPPED could already be set. In this case
soreceive_generic hangs since it does not return from a KTLS socket
with pending encrypted data unless there is an error (so_error) (this
behavior is to ensure that soreceive_generic doesn't return a
premature EOF when there is pending data still being decrypted).

Note that this was a bug prior to
69542f26820b7edb8351398b36edda5299c1db56 as tcp_usr_abort would also
have ignored the error in this case.

Reviewed by: gallatin
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D37775

show more ...

ktls: Close a race with setting so_error when dropping a connection.

pr_abort calls tcp_usr_abort which calls tcp_drop with ECONNABORTED.
After pr_abort returns, the so_error is then set to a more s

ktls: Close a race with setting so_error when dropping a connection.

pr_abort calls tcp_usr_abort which calls tcp_drop with ECONNABORTED.
After pr_abort returns, the so_error is then set to a more specific
error. However, a reader can observe and return the ECONNABORTED
error before so_error is set to the desired error value. This is
resulting in spurious test failures of recently added tests for
invalid conditions such as invalid headers.

To fix, refactor the code to abort a connection to call tcp_drop
directly with the desired error value. ktls_reset_send_tag already
calls tcp_drop directly when it aborts a connection due to an error.

Reviewed by: gallatin
Reported by: CI (jenkins), gallatin, olivier
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D37692

show more ...

ktls: Add software support for AES-CBC decryption for TLS 1.1+.

This is mainly intended to provide a fallback for TOE TLS which may
need to use software decryption for an initial record at the start

ktls: Add software support for AES-CBC decryption for TLS 1.1+.

This is mainly intended to provide a fallback for TOE TLS which may
need to use software decryption for an initial record at the start
of a connection.

Reviewed by: markj
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D37370

show more ...

ktls: Inline ktls_cleanup() into ktls_destroy().

Reviewed by: gallatin, markj
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D37353

ktls: Don't leak ktls session objects for certain errors.

ktls_cleanup() does not free ktls session objects, it merely
cleans (and frees) members of the object.

Change callers to use ktls_free() in

ktls: Don't leak ktls session objects for certain errors.

ktls_cleanup() does not free ktls session objects, it merely
cleans (and frees) members of the object.

Change callers to use ktls_free() instead.

Reviewed by: gallatin, markj
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D37352

show more ...