protosw: refactor protosw and domain static declaration and load

o Assert that every protosw has pr_attach. Now this structure is
only for socket protocols declarations and nothing else.
o Merge

protosw: refactor protosw and domain static declaration and load

o Assert that every protosw has pr_attach. Now this structure is
only for socket protocols declarations and nothing else.
o Merge struct pr_usrreqs into struct protosw. This was suggested
in 1996 by wollman@ (see 7b187005d18ef), and later reiterated
in 2006 by rwatson@ (see 6fbb9cf860dcd).
o Make struct domain hold a variable sized array of protosw pointers.
For most protocols these pointers are initialized statically.
Those domains that may have loadable protocols have spacers. IPv4
and IPv6 have 8 spacers each (andre@ dff3237ee54ea).
o For inetsw and inet6sw leave a comment noting that many protosw
entries very likely are dead code.
o Refactor pf_proto_[un]register() into protosw_[un]register().
o Isolate pr_*_notsupp() methods into uipc_domain.c

Reviewed by: melifaro
Differential revision: https://reviews.freebsd.org/D36232

show more ...

cxgbe tom: Force unsigned modulus for queue indices.

The final transmit and receive queue indices need to be positive
values. However, since txq_idx and rxq_idx are signed (to permit
using -1 to as

cxgbe tom: Force unsigned modulus for queue indices.

The final transmit and receive queue indices need to be positive
values. However, since txq_idx and rxq_idx are signed (to permit
using -1 to as a marker for uninitialized values), using %= with
another integer type (vi->nofld[tr]xq) yielded a sign-extended modulus
value. This resulted in negative queue indices and a buffer underrun
when arc4random() returned a value with the sign bit set. Use a
temporary unsigned variable to hold the "raw" queue index to force
unsigned modulus.

This worked previously because the modulus was previously applied
directly to the return value of arc4random() which is unsigned before
the result was assigned to txq_idx and rxq_idx.

Discussed with: np
Fixes: db28d4a0cd1c cxgbe/t4_tom: Support for round-robin selection of offload queues.
Sponsored by: Chelsio Communications

show more ...

cxgbe/t4_tom: Support for round-robin selection of offload queues.

A COP (Connection Offload Policy) rule can now specify that the tx
and/or rx queue for a new tid should be selected in a round-robi

cxgbe/t4_tom: Support for round-robin selection of offload queues.

A COP (Connection Offload Policy) rule can now specify that the tx
and/or rx queue for a new tid should be selected in a round-robin
manner. There is no change in default behavior.

Reviewed by: jhb@
MFC after: 1 week
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D34921

show more ...

cxgbe tom: Don't queue AIO requests on listen sockets.

This is similar to the fixes in 141fe2dceeae. One difference is that
TOE sockets do not change states (listen vs non-listen) once created,
so

cxgbe tom: Don't queue AIO requests on listen sockets.

This is similar to the fixes in 141fe2dceeae. One difference is that
TOE sockets do not change states (listen vs non-listen) once created,
so no lock is needed for SOLISTENING().

Sponsored by: Chelsio Communications

show more ...

cxgbei: Wait for the final CPL to be received in icl_cxgbei_conn_close.

A socket in the FIN_WAIT_1 state is marked disconnected by
do_close_con_rpl() even though there might still receive data pendi

cxgbei: Wait for the final CPL to be received in icl_cxgbei_conn_close.

A socket in the FIN_WAIT_1 state is marked disconnected by
do_close_con_rpl() even though there might still receive data pending.
This is because the socket at that point has set SBS_CANTRCVMORE which
causes the protocol layer to discard any data received before the FIN.
However, icl_cxgbei_conn_close needs to wait until all the data has
been discarded. Replace the wait for SS_ISDISCONNECTED with instead
waiting for final_cpl_received() to be called.

Reported by: Jithesh Arakkan @ Chelsio
Sponsored by: Chelsio Communications

show more ...

cxgbe(4): Do not configure traffic classes automatically on attach.

The driver used to configure all available classes with some default
parameters on attach and the rest of t4_sched.c was written w

cxgbe(4): Do not configure traffic classes automatically on attach.

The driver used to configure all available classes with some default
parameters on attach and the rest of t4_sched.c was written with the
assumption that all traffic classes are always valid in the hardware.
But this resulted in a lot of informational messages being logged in the
firmware's circular log, crowding out other more useful messages.

This change leaves the tx scheduler alone during attach to reduce the
spam in the devlog. The state of every class is now tracked separately
from its flags and there is support for an 'uninitialized' state.

MFC after: 2 weeks
Sponsored by: Chelsio Communications

show more ...

cxgbe(4): Get the number of usable traffic classes from the firmware.

Recent firmwares are able to utilize the traffic classes of tx channels
that were previously unused. This effectively doubles t

cxgbe(4): Get the number of usable traffic classes from the firmware.

Recent firmwares are able to utilize the traffic classes of tx channels
that were previously unused. This effectively doubles the number of
traffic classes available per port for 2 port cards. Stop using the raw
per-channel value in the driver and ask the firmware for the number of
usable traffic classes instead.

MFC after: 2 weeks
Sponsored by: Chelsio Communications

show more ...

cxgbe tom: Remove support for non-KTLS TLS offload.

TOE TLS offload was first supported via a customized OpenSSL developed
by Chelsio with proprietary socket options prior to KTLS being present
eith

cxgbe tom: Remove support for non-KTLS TLS offload.

TOE TLS offload was first supported via a customized OpenSSL developed
by Chelsio with proprietary socket options prior to KTLS being present
either in FreeBSD or upstream OpenSSL. With the addition of KTLS in
both places, cxgbe's TOE driver was extended to support TLS offload
via KTLS as well. This change removes the older interface leaving
only the KTLS bindings for TOE TLS.

Since KTLS was added to TOE TLS second, it was somehat shoe-horned
into the existing code. In addition to removing the non-KTLS TLS
offload, refactor and simplify the code to assume KTLS, e.g. not
copying keys into a helper structure that mimic'ed the non-KTLS mode,
but using the KTLS session object directly when constructing key
contexts.

This also removes some unused code to send TX keys inline in work
requests for TOE TLS. This code was never enabled, and was arguably
sending the wrong thing (it was not sending the raw key context as we
do for NIC TLS when using inline keys).

Sponsored by: Chelsio Communications

show more ...

cxgbe tom: Free pending iSCSI mbufs on connection shutdown.

If an iSCSI connection is shutdown abruptly (e.g. by a RST from the
peer), pending iSCSI PDUs and page pod work requests can be in the
ulp

cxgbe tom: Free pending iSCSI mbufs on connection shutdown.

If an iSCSI connection is shutdown abruptly (e.g. by a RST from the
peer), pending iSCSI PDUs and page pod work requests can be in the
ulp_pduq when the final CPL is received indicating the death of the
connection.

Reported by: Jithesh Arakkan @ Chelsio

show more ...

cxgbe(4): Overhaul CLIP (Compressed Local IPv6) table management.

- Process the list of local IPs once instead of once per adapter. Add
addresses from all VNETs to the driver's list but leave har

cxgbe(4): Overhaul CLIP (Compressed Local IPv6) table management.

- Process the list of local IPs once instead of once per adapter. Add
addresses from all VNETs to the driver's list but leave hardware
updates for later when the global VNET/IFADDR list locks have been
released.

- Add address to the hardware table synchronously when a CLIP entry is
requested for an address that's not already in there.

- Provide ioctls that allow userspace tools to manage addresses in the
CLIP table.

- Add a knob (hw.cxgbe.clip_db_auto) that controls whether local IPs are
automatically added to the CLIP table or not.

MFC after: 2 weeks
Sponsored by: Chelsio Communications

show more ...

cxgbe/t4_tom: Implement tod_pmtu_update.

tod_pmtu_update was added to the kernel in 01d74fe1ffc.

Sponsored by: Chelsio Communications

cxgbe/t4_tom: restore socket's protosw before entering TIME_WAIT.

This fixes a panic due to stale so->so_proto if t4_tom is unloaded and
one or more connections that were previously offloaded are st

cxgbe/t4_tom: restore socket's protosw before entering TIME_WAIT.

This fixes a panic due to stale so->so_proto if t4_tom is unloaded and
one or more connections that were previously offloaded are still around
in TIME_WAIT state.

Reviewed by: jhb@
MFC after: 1 week
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D29503

show more ...

cxgbe(4): Use firmware commands to get/set filter configuration.

1. Query the firmware for filter mode, mask, and related ingress config
instead of trying to figure them out from hardware registe

cxgbe(4): Use firmware commands to get/set filter configuration.

1. Query the firmware for filter mode, mask, and related ingress config
instead of trying to figure them out from hardware registers. Read
configuration from the registers only when the firmware does not
support this query.

2. Use the firmware to set the filter mode. This is the correct way to
do it and is more flexible as well. The filter mode (and associated
ingress config) can now be changed any time it is safe to do so.

The user can specify a subset of a valid mode and the driver will
enable enough bits to make sure that the mode is maxed out -- that
is, it is not possible to set another bit without exceeding the
total width for optional filter fields. This is a hardware
requirement that was not enforced by the driver previously.

MFC after: 2 weeks
Sponsored by: Chelsio Communications

show more ...

Clear TLS offload mode if a TLS socket hangs without receiving data.

By default, if a TOE TLS socket stops receiving data for more than 5
seconds, revert the connection back to plain TOE mode. This

Clear TLS offload mode if a TLS socket hangs without receiving data.

By default, if a TOE TLS socket stops receiving data for more than 5
seconds, revert the connection back to plain TOE mode. This provides
a fallback if the userland SSL library does not support KTLS. In
addition, for client TLS 1.3 sockets using connect(), the TOE socket
blocks before the handshake has completed since the socket option is
only invoked for the final handshake.

The timeout defaults to 5 seconds, but can be changed at boot via the
hw.cxgbe.toe.tls_rx_timeout tunable or for an individual interface via
the dev.<nexus>.toe.tls_rx_timeout sysctl.

Reviewed by: np
MFC after: 2 weeks
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D27470

show more ...

Clear tp->tod in t4_pcb_detach().

Otherwise, a socket can have a non-NULL tp->tod while TF_TOE is clear.
In particular, if a newly accepted socket falls back to non-TOE due to
an active open failure

Clear tp->tod in t4_pcb_detach().

Otherwise, a socket can have a non-NULL tp->tod while TF_TOE is clear.
In particular, if a newly accepted socket falls back to non-TOE due to
an active open failure, the non-TOE socket will still have tp->tod set
even though TF_TOE is clear.

Reviewed by: np
MFC after: 2 weeks
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D27028

show more ...

Re-enable receive flow control for TOE TLS sockets.

Flow control was disabled during initial TOE TLS development to
workaround a hang (and to match the Linux TOE TLS support for T6).
The rest of the

Re-enable receive flow control for TOE TLS sockets.

Flow control was disabled during initial TOE TLS development to
workaround a hang (and to match the Linux TOE TLS support for T6).
The rest of the TOE TLS code maintained credits as if flow control was
enabled which was inherited from before the workaround was added with
the exception that the receive window was allowed to go negative.
This negative receive window handling (rcv_over) was because I hadn't
realized the full implications of disabling flow control.

To clean this up, re-enable flow control on TOE TLS sockets. The
existing TPF_FORCE_CREDITS workaround is sufficient for the original
hang. Now that flow control is enabled, remove the rcv_over
workaround and instead assert that the receive window never goes
negative matching plain TCP TOE sockets.

Reviewed by: np
MFC after: 2 weeks
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D26799

show more ...

cxgbe(4): Add a pointer to the adapter softc in vi_info.

There were quite a few places where port_info was being accessed only to
get to the adapter.

Reviewed by: jhb@
MFC after: 1 week
Sponsored b

cxgbe(4): Add a pointer to the adapter softc in vi_info.

There were quite a few places where port_info was being accessed only to
get to the adapter.

Reviewed by: jhb@
MFC after: 1 week
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D25432

show more ...

Add support for KTLS RX over TOE to T6.

This largely reuses the TLS TOE support added in r330884. However,
this uses the KTLS framework in upstream OpenSSL rather than requiring
Chelsio-specific pa

Add support for KTLS RX over TOE to T6.

This largely reuses the TLS TOE support added in r330884. However,
this uses the KTLS framework in upstream OpenSSL rather than requiring
Chelsio-specific patches to OpenSSL. As with the existing TLS TOE
support, use of RX offload requires setting the tls_rx_ports sysctl.

Reviewed by: np
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D24453

show more ...

Initial support for kernel offload of TLS receive.

- Add a new TCP_RXTLS_ENABLE socket option to set the encryption and
authentication algorithms and keys as well as the initial sequence
number.

Initial support for kernel offload of TLS receive.

- Add a new TCP_RXTLS_ENABLE socket option to set the encryption and
authentication algorithms and keys as well as the initial sequence
number.

- When reading from a socket using KTLS receive, applications must use
recvmsg(). Each successful call to recvmsg() will return a single
TLS record. A new TCP control message, TLS_GET_RECORD, will contain
the TLS record header of the decrypted record. The regular message
buffer passed to recvmsg() will receive the decrypted payload. This
is similar to the interface used by Linux's KTLS RX except that
Linux does not return the full TLS header in the control message.

- Add plumbing to the TOE KTLS interface to request either transmit
or receive KTLS sessions.

- When a socket is using receive KTLS, redirect reads from
soreceive_stream() into soreceive_generic().

- Note that this interface is currently only defined for TLS 1.1 and
1.2, though I believe we will be able to reuse the same interface
and structures for 1.3.

show more ...

Clear CPL_GET_TCB_RPL handler on module unload.

This fixes a panic when unloading and reloading t4_tom.ko since the
old pointer is still stored when t4_tom_load tries to set it.

Reviewed by: np
Spo

Clear CPL_GET_TCB_RPL handler on module unload.

This fixes a panic when unloading and reloading t4_tom.ko since the
old pointer is still stored when t4_tom_load tries to set it.

Reviewed by: np
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D24358

show more ...

Merge ^/head r358712 through r358730.

cxgbe/t4_tom: Do not uninitialize a toepcb that has not been initialized.

This fixes the following panic:
--- trap 0xc, rip = 0xffffffff80c00411, rsp = 0xfffffe0025192840, rbp = 0xfffffe0025192860 -

cxgbe/t4_tom: Do not uninitialize a toepcb that has not been initialized.

This fixes the following panic:
--- trap 0xc, rip = 0xffffffff80c00411, rsp = 0xfffffe0025192840, rbp = 0xfffffe0025192860 ---
vmem_xfree() at vmem_xfree+0xd1/frame 0xfffffe0025192860
tls_uninit_toep() at tls_uninit_toep+0x78/frame 0xfffffe0025192880
free_toepcb() at free_toepcb+0x32/frame 0xfffffe00251928a0
t4_connect() at t4_connect+0x3be/frame 0xfffffe0025192950
tcp_offload_connect() at tcp_offload_connect+0xa4/frame 0xfffffe0025192990
tcp_usr_connect() at tcp_usr_connect+0xec/frame 0xfffffe00251929f0
soconnect() at soconnect+0xae/frame 0xfffffe0025192a30
kern_connectat() at kern_connectat+0xe2/frame 0xfffffe0025192a90
sys_connect() at sys_connect+0x75/frame 0xfffffe0025192ad0
amd64_syscall() at amd64_syscall+0x137/frame 0xfffffe0025192bf0
fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe0025192bf0
--- syscall (98, FreeBSD ELF64, sys_connect), rip = 0x8008e9d8a, rsp = 0x7fffffffc0f8, rbp = 0x7fffffffc130 ---

Reviewed by: jhb@
MFC after: 3 days
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D23989

show more ...

vnet: virtualise more network stack sysctls.

Virtualise tcp_always_keepalive, TCP and UDP log_in_vain. All three are
set in the netoptions startup script, which we would love to run for VNETs
as we

vnet: virtualise more network stack sysctls.

Virtualise tcp_always_keepalive, TCP and UDP log_in_vain. All three are
set in the netoptions startup script, which we would love to run for VNETs
as well [1].

While virtualising the log_in_vain sysctls seems pointles at first for as
long as the kernel message buffer is not virtualised, it at least allows
an administrator to debug the base system or an individual jail if needed
without turning the logging on for all jails running on a system.

PR: 243193 [1]
MFC after: 2 weeks

show more ...

Always allocate the atid table during attach.

Previously the table was allocated on first use by TOE and the
ratelimit code. The forthcoming NIC KTLS code also uses this table.
Allocate it uncondit

Always allocate the atid table during attach.

Previously the table was allocated on first use by TOE and the
ratelimit code. The forthcoming NIC KTLS code also uses this table.
Allocate it unconditionally during attach to simplify consumers.

Reviewed by: np
Differential Revision: https://reviews.freebsd.org/D22028

show more ...

Merge ^/head r353316 through r353350.