cxgbe tom: Restore support for zerocopy TCP receive for aio_read()

The commit to introduce TCP_USE_DDP support had a couple of bugs that
broke support for zerocopy receive via aio_read(). First, th

cxgbe tom: Restore support for zerocopy TCP receive for aio_read()

The commit to introduce TCP_USE_DDP support had a couple of bugs that
broke support for zerocopy receive via aio_read(). First, the length
and offset arguments to mk_update_tcb_for_ddp() were reversed which
prevented DDP from working. Second, the AIO state in the toep was
initialized too late when the first aio_read() request was queued.

Reported by: Harshavardhan Tanneru @ Chelsio
Fixes: eba13bbc37ab cxgbe: Support TCP_USE_DDP on offloaded TOE connections
MFC after: 1 week
Sponsored by: Chelsio Communications

show more ...

Deprecate contigfree(9) in favour of free(9)

As of 9e6544dd6e02c46b805d11ab925c4f3b18ad7a4b contigfree(9) is no longer
needed and should not be used anymore. We leave a wrapper for 3rd party
code i

Deprecate contigfree(9) in favour of free(9)

As of 9e6544dd6e02c46b805d11ab925c4f3b18ad7a4b contigfree(9) is no longer
needed and should not be used anymore. We leave a wrapper for 3rd party
code in at least 15.x but remove (almost) all other cases from the tree.

This leaves one use of contigfree(9) untouched; that was the original
trigger for 9e6544dd6e02 and is handled in D45813 (to be committed
seperately later).

Sponsored by: The FreeBSD Foundation
Reviewed by: markj, kib
Tested by: pho (10h stress test run)
Differential Revision: https://reviews.freebsd.org/D46099

show more ...

cxgbe(4): Consolidate all mk_set_tcb_field_ulp in one place.

MFC after: 1 week
Sponsored by: Chelsio Communications

cxgbe: Support TCP_USE_DDP on offloaded TOE connections

When this socket option is enabled, relatively large contiguous
buffers are allocated and used to receive data from the remote
connection. Wh

cxgbe: Support TCP_USE_DDP on offloaded TOE connections

When this socket option is enabled, relatively large contiguous
buffers are allocated and used to receive data from the remote
connection. When data is received a wrapper M_EXT mbuf is queued to
the socket's receive buffer. This reduces the length of the linked
list of received mbufs and allows consumers to consume receive data in
larger chunks.

To minimize reprogramming the page pods in the adapter, receive
buffers for a given connection are recycled. When a buffer has been
fully consumed by the receiver and freed, the buffer is placed on a
per-connection free buffers list.

The size of the receive buffers defaults to 256k and can be set via
the hw.cxgbe.toe.ddp_rcvbuf_len sysctl. The
hw.cxgbe.toe.ddp_rcvbuf_cache sysctl (defaults to 4) determines the
maximum number of free buffers cached per connection. Note that this
limit does not apply to "in-flight" receive buffers that are
associated with mbufs in the socket's receive buffer.

Co-authored-by: Navdeep Parhar <np@FreeBSD.org>
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D44001

show more ...

ddp: Clear active DDP buffer members to NULL to pacify an assertion

Reviewed by: np
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D43999

cxgbe tom: Enable ULP_MODE_TCPDDP on demand

Most ULP modes in cxgbe's TOE are enabled on the fly when a protocol
is needed (e.g. ULP_MODE_ISCSI is enabled by cxgbei when offloading a
connection usin

cxgbe tom: Enable ULP_MODE_TCPDDP on demand

Most ULP modes in cxgbe's TOE are enabled on the fly when a protocol
is needed (e.g. ULP_MODE_ISCSI is enabled by cxgbei when offloading a
connection using iSCSI, and ULP_MODE_TLS is enabled when RX TLS keys
are programmed for a TOE connection). The one exception to this is
ULP_MODE_TCPDDP.

Currently the cxgbe driver enables ULP_MODE_TCPDDP when a TOE
connection is first created. However, since DDP connections cannot be
converted to other connection types, this requires some special
handling in the driver. For example, iSCSI daemons use the SO_NO_DDP
socket option to ensure TOE connections use ULP_MODE_NONE so they can
be converted to ULP_MODE_ISCSI. Similarly, using TLS receive offload
(ULP_MODE_TLS) requires disabling TCP DDP for new connections by
default.

This commit changes cxgbe to instead switch a connection from
ULP_MODE_NONE to ULP_MODE_TCPDDP when a connection first attempts to
use TCP DDP via aio_read(2). This permits connections to always start
as ULP_MODE_NONE and switch to a protocol-specific mode as needed.

Reviewed by: np
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D43670

show more ...

cxgbe: Add counters for POSIX async I/O requests handled by the driver

Reviewed by: np
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D43668

sys: Remove $FreeBSD$: one-line .c pattern

Remove /^[\s*]*__FBSDID\("\$FreeBSD\$"\);?\s*\n/

spdx: The BSD-2-Clause-FreeBSD identifier is obsolete, drop -FreeBSD

The SPDX folks have obsoleted the BSD-2-Clause-FreeBSD identifier. Catch
up to that fact and revert to their recommended match of

spdx: The BSD-2-Clause-FreeBSD identifier is obsolete, drop -FreeBSD

The SPDX folks have obsoleted the BSD-2-Clause-FreeBSD identifier. Catch
up to that fact and revert to their recommended match of BSD-2-Clause.

Discussed with: pfg
MFC After: 3 days
Sponsored by: Netflix

show more ...

cxgbe: use standard sototcpcb() accessor macro to get socket's tcpcb

Reviewed by: np
Differential revision: https://reviews.freebsd.org/D37041

tcp: remove INP_TIMEWAIT flag

Mechanically cleanup INP_TIMEWAIT from the kernel sources. After
0d7445193ab, this commit shall not cause any functional changes.

Note: this flag was very often check

tcp: remove INP_TIMEWAIT flag

Mechanically cleanup INP_TIMEWAIT from the kernel sources. After
0d7445193ab, this commit shall not cause any functional changes.

Note: this flag was very often checked together with INP_DROPPED.
If we modify in_pcblookup*() not to return INP_DROPPED pcbs, we
will be able to remove most of this checks and turn them to
assertions. Some of them can be turned into assertions right now,
but that should be carefully done on a case by case basis.

Differential revision: https://reviews.freebsd.org/D36400

show more ...

sockets: use socket buffer mutexes in struct socket directly

Since c67f3b8b78e the sockbuf mutexes belong to the containing socket,
and socket buffers just point to it. In 74a68313b50 macros that a

sockets: use socket buffer mutexes in struct socket directly

Since c67f3b8b78e the sockbuf mutexes belong to the containing socket,
and socket buffers just point to it. In 74a68313b50 macros that access
this mutex directly were added. Go over the core socket code and
eliminate code that reaches the mutex by dereferencing the sockbuf
compatibility pointer.

This change requires a KPI change, as some functions were given the
sockbuf pointer only without any hint if it is a receive or send buffer.

This change doesn't cover the whole kernel, many protocols still use
compatibility pointers internally. However, it allows operation of a
protocol that doesn't use them.

Reviewed by: markj
Differential revision: https://reviews.freebsd.org/D35152

show more ...

cxgbei: Support unmapped I/O requests.

- Add icl_pdu_append_bio and icl_pdu_get_bio methods.

- Add new page pod routines for allocating and writing page pods for
unmapped bio requests. Use these

cxgbei: Support unmapped I/O requests.

- Add icl_pdu_append_bio and icl_pdu_get_bio methods.

- Add new page pod routines for allocating and writing page pods for
unmapped bio requests. Use these new routines for setting up DDP
for iSCSI tasks with a SCSI I/O CCB which uses CAM_DATA_BIO.

- When ICL_NOCOPY is used to append data from an unmapped I/O request
to a PDU, construct unmapped mbufs from the relevant pages backing
the struct bio. This also requires changes in the t4_push_pdus path
to support unmapped mbufs.

Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D34383

show more ...

cxgbe tom: Compile fix for disabled KTR trace.

Sponsored by: Chelsio Communications

cxgbe tom: Use VM_PAGE_TO_PHYS().

Sponsored by: Chelsio Communications

cxgbe tom: Use be64toh instead of htobe64 to convert to host order.

This is a no-op but more accurately conveys intent.

Sponsored by: Chelsio Communications

cxgbe tom: Use vm_paddr_t for physical addresses in page pod routines.

Sponsored by: Chelsio Communications

cxgbe: Move page pods KTR traces under VERBOSE_TRACES.

cxgbe(4): Fix "set but not used [-Wunused-but-set-variable]" warnings.

MFC after: 1 week
Sponsored by: Chelsio Communications

cxgbe tom: Update rcv_nxt for a FIN after handle_ddp_close().

For TCP DDP, handle_ddp_close() needs to see the pre-FIN rcv_nxt to
determine how much data was placed in the local buffer before the FI

cxgbe tom: Update rcv_nxt for a FIN after handle_ddp_close().

For TCP DDP, handle_ddp_close() needs to see the pre-FIN rcv_nxt to
determine how much data was placed in the local buffer before the FIN
was received. The changes in d59f1c49e26b broke this by updating
rcv_nxt before calling handle_ddp_close().

Fixes: d59f1c49e26b cxgbe tom: Permit rcv_nxt mismatches on FIN for iSCSI connections on T6.
Sponsored by: Chelsio Communications

show more ...

cxgbei: Fix a race between transfer setup and a peer reset.

In 4427ac3675f9, the TOM driver stopped sending work requests to
program iSCSI page pods directly and instead queued them to be written
as

cxgbei: Fix a race between transfer setup and a peer reset.

In 4427ac3675f9, the TOM driver stopped sending work requests to
program iSCSI page pods directly and instead queued them to be written
asynchronously with iSCSI PDUs. The queue of mbufs to send is
protected by the inp lock. However, the inp cannot be safely obtained
from the toep since a RST from the remote peer might have cleared
toep->inp asynchronously in an ithread. To fix, obtain the inp from
the socket as is already done in icl_cxgbei_conn_pdu_queue_cb() and
fail the new transfer setup with ECONNRESET if the connection has been
reset.

To avoid passing sockets or inps into the page pod routines, pull the
mbufq out of the two relevant page pod routines such that the routines
queue new work request mbufs to a caller-supplied mbufq.

Reported by: Jithesh Arakkan @ Chelsio
Fixes: 4427ac3675f91df039d54a23518132e0e0fede86

show more ...

cxgbe: Cast pointer arguments to trunc_page() to vm_offset_t.

Reported by: mjg, jenkins, rmacklem
Fixes: 46bee8043ee2bd352d420cd573e0364ca45f813e
Sponsored by: Chelsio Communications

cxgbei: Support DDP for target I/O S/G lists with more than one entry.

A CAM target layer I/O CCB can use a S/G list of virtual address ranges
to describe its data buffer. This change adds zero-cop

cxgbei: Support DDP for target I/O S/G lists with more than one entry.

A CAM target layer I/O CCB can use a S/G list of virtual address ranges
to describe its data buffer. This change adds zero-copy receive support
for such requests.

Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D29908

show more ...

cxgbe tom: Set the tid in the work requests to program page pods for iSCSI.

As a result, CPL_FW4_ACK now returns credits for these work requests.
To support this, page pod work requests are now cons

cxgbe tom: Set the tid in the work requests to program page pods for iSCSI.

As a result, CPL_FW4_ACK now returns credits for these work requests.
To support this, page pod work requests are now constructed in special
mbufs similar to "raw" mbufs used for NIC TLS in plain TX queues.
These special mbufs are stored in the ulp_pduq and dispatched in order
with PDU work requests.

Sponsored by: Chelsio Communications
Discussed with: np
Differential Revision: https://reviews.freebsd.org/D29904

show more ...

cxgbe ddp: Use CPL_COOKIE_DDP* instead of DDP_BUF*_INVALIDATED.

This avoids mixing the use of two different enums which modern C
compilers warn about.

Reviewed by: np
MFC after: 2 weeks
Sponsored b

cxgbe ddp: Use CPL_COOKIE_DDP* instead of DDP_BUF*_INVALIDATED.

This avoids mixing the use of two different enums which modern C
compilers warn about.

Reviewed by: np
MFC after: 2 weeks
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D29301

show more ...