mlx5en ipsec offload: copy xform_history to the ipsec_accel_in_tag

Reviewed by: Ariel Ehrenberg <aehrenberg@nvidia.com>, slavash
Sponsored by: Nvidia networking

mlx5en: sync channel close with the rq completion processing

Without the wait, mlx5e_destroy_rq() might free mbuf that is passed up
to the network stack on receive in mlx5e_poll_rx_cq().

Sponsored

mlx5en: sync channel close with the rq completion processing

Without the wait, mlx5e_destroy_rq() might free mbuf that is passed up
to the network stack on receive in mlx5e_poll_rx_cq().

Sponsored by: NVidia networking
MFC after: 1 week

show more ...

mlx5en: fix rq->wqe_sz usage

Define it as the size of the single data segment in wqe.

Reviewed by: Ariel Ehrenberg <aehrenberg@nvidia.com>, Slava Shwartsman <slavash@nvidia.com>
Sponsored by: NVidi

mlx5en: fix rq->wqe_sz usage

Define it as the size of the single data segment in wqe.

Reviewed by: Ariel Ehrenberg <aehrenberg@nvidia.com>, Slava Shwartsman <slavash@nvidia.com>
Sponsored by: NVidia networking
MFC after: 1 week

show more ...

mlx5: overwrite only the echo reply timestamp from the last packet in LRO

Reviewed by: Ariel Ehrenberg <aehrenberg@nvidia.com>, Slava Shwartsman <slavash@nvidia.com>
Sponsored by: NVidia networking

mlx5: overwrite only the echo reply timestamp from the last packet in LRO

Reviewed by: Ariel Ehrenberg <aehrenberg@nvidia.com>, Slava Shwartsman <slavash@nvidia.com>
Sponsored by: NVidia networking
MFC after: 1 week

show more ...

mlx5: correct the predicate asserted in __predict_true()

Reviewed by: Ariel Ehrenberg <aehrenberg@nvidia.com>, Slava Shwartsman <slavash@nvidia.com>
Sponsored by: NVidia networking
MFC after: 1 week

mlx5: recalculate tcp checksum for ipv6 hw lro coalesced packet

Reviewed by: Ariel Ehrenberg <aehrenberg@nvidia.com>, Slava Shwartsman <slavash@nvidia.com>
Sponsored by: NVidia networking
MFC after:

mlx5: recalculate tcp checksum for ipv6 hw lro coalesced packet

Reviewed by: Ariel Ehrenberg <aehrenberg@nvidia.com>, Slava Shwartsman <slavash@nvidia.com>
Sponsored by: NVidia networking
MFC after: 1 week

show more ...

mlx5: recalculate tcp checksum for ipv4 hw lro coalesced packet

Reviewed by: Ariel Ehrenberg <aehrenberg@nvidia.com>, Slava Shwartsman <slavash@nvidia.com>
Sponsored by: NVidia networking
MFC after:

mlx5: recalculate tcp checksum for ipv4 hw lro coalesced packet

Reviewed by: Ariel Ehrenberg <aehrenberg@nvidia.com>, Slava Shwartsman <slavash@nvidia.com>
Sponsored by: NVidia networking
MFC after: 1 week

show more ...

mlx5_en: correct recalculation of the ipv4 checksum for hw lro packet

The call to in_cksum_skip() did not skipped the ethernet header.

Reviewed by: Ariel Ehrenberg <aehrenberg@nvidia.com>, Slava Sh

mlx5_en: correct recalculation of the ipv4 checksum for hw lro packet

The call to in_cksum_skip() did not skipped the ethernet header.

Reviewed by: Ariel Ehrenberg <aehrenberg@nvidia.com>, Slava Shwartsman <slavash@nvidia.com>
Sponsored by: NVidia networking
MFC after: 1 week

show more ...

tcp: extend the use of the th_flags accessor function

Formally, there are 12 bits for TCP header flags.
Use the accessor functions in more (kernel) places.

No functional change.

Reviewed By: cc, #

tcp: extend the use of the th_flags accessor function

Formally, there are 12 bits for TCP header flags.
Use the accessor functions in more (kernel) places.

No functional change.

Reviewed By: cc, #transport, cy, glebius, #iflib, kbowling
Sponsored by: NetApp, Inc.
Differential Revision: https://reviews.freebsd.org/D47063

show more ...

mlx5en: do now waste ipsec_accel_in_tag on non-ipsec packets rx

Do not prepend ipsec tags into mbuf head when preparing rx wqe, store it
separately. Only prepend (and clear the store) when received

mlx5en: do now waste ipsec_accel_in_tag on non-ipsec packets rx

Do not prepend ipsec tags into mbuf head when preparing rx wqe, store it
separately. Only prepend (and clear the store) when received packed was
indeed offloaded by ipsec engine. Then we do not need to refill tags
for slots that received non-ipsec packets.

This should solve some minimal degradation of the rx CPU usage due to
unneeded tag allocation for each packet.

Sponsored by: NVidia networking

show more ...

mlx5en: add IPSEC_OFFLOAD support

Right now, only IPv4 transport mode, with aes-gcm ESP, is supported.
Driver also cooperates with NAT-T, and obeys socket policies, which
makes IKEd like StrongSwan

mlx5en: add IPSEC_OFFLOAD support

Right now, only IPv4 transport mode, with aes-gcm ESP, is supported.
Driver also cooperates with NAT-T, and obeys socket policies, which
makes IKEd like StrongSwan working.

Sponsored by: NVIDIA networking

show more ...

sys: Remove $FreeBSD$: two-line .h pattern

Remove /^\s*\*\n \*\s+\$FreeBSD\$$\n/

Mechanically convert mlx5en(4) to IfAPI

Reviewed by: zlei
Sponsored by: Juniper Networks, Inc.
Differential Revision: https://reviews.freebsd.org/D38595

pfil: add pfil_mem_{in,out}() and retire pfil_run_hooks()

The 0b70e3e78b0 changed the original design of a single entry point
into pfil(9) chains providing separate functions for the filtering
point

pfil: add pfil_mem_{in,out}() and retire pfil_run_hooks()

The 0b70e3e78b0 changed the original design of a single entry point
into pfil(9) chains providing separate functions for the filtering
points that always provide mbufs and know the direction of a flow.
The motivation was to reduce branching. The logical continuation
would be to do the same for the filtering points that always provide
a memory pointer and retire the single entry point.

o Hooks now provide two functions: one for mbufs and optional for
memory pointers.
o pfil_hook_args() has a new member and pfil_add_hook() has a
requirement to zero out uninitialized data. Bump PFIL_VERSION.
o As it was before, a hook function for a memory pointer may realloc
into an mbuf. Such mbuf would be returned via a pointer that must
be provided in argument.
o The only hook that supports memory pointers is ipfw:default-link.
It is rewritten to provide two functions.
o All remaining uses of pfil_run_hooks() are converted to
pfil_mem_in().
o Transparent union of pfil_packet_t and tricks to fix pointer
alignment are retired. Internal pfil_realloc() reduces down to
m_devget() and thus is retired, too.

Reviewed by: mjg, ocochard
Differential revision: https://reviews.freebsd.org/D37977

show more ...

mlx5 M_TSTMP accuracy looses quite a bit of precision so lets fix it.

The way that the clock is synchronized between the system and the current mlx5 for the purposes of the M_TSTMP
being carried we

mlx5 M_TSTMP accuracy looses quite a bit of precision so lets fix it.

The way that the clock is synchronized between the system and the current mlx5 for the purposes of the M_TSTMP
being carried we loose a lot of precision. Instead lets change the math that calculates this to separate out
the seconds/nanoseconds and operate on the two values so we don't get overflow instead of just
shifting the value down and loosing precision.

Reviewed by: kib, hselasky
Sponsored by: Netflix Inc
Differential Revision: https://reviews.freebsd.org/D36327

show more ...

mlx5en(4): Set the leaf network interface field in the mbuf packet header.

This will be used for TLS RX.

Submitted by: jhb@
Differential revision: https://reviews.freebsd.org/D32356
Sponsored by: N

mlx5en(4): Set the leaf network interface field in the mbuf packet header.

This will be used for TLS RX.

Submitted by: jhb@
Differential revision: https://reviews.freebsd.org/D32356
Sponsored by: NVIDIA Networking

show more ...

mlx5en: Improve CQE error debugging.

MFC after: 1 week
Sponsored by: NVIDIA Networking

mlx5en: Implement TLS RX support.

TLS RX support is modeled after TLS TX support. The basic structures and layouts
are almost identical, except that the send tag created filters RX traffic and
not T

mlx5en: Implement TLS RX support.

TLS RX support is modeled after TLS TX support. The basic structures and layouts
are almost identical, except that the send tag created filters RX traffic and
not TX traffic.

The TLS RX tag keeps track of past TLS records up to a certain limit,
approximately 1 Gbyte of TCP data. TLS records of same length are joined
into a single database record.

Regularly the HW is queried for TLS RX progress information. The TCP sequence
number gotten from the HW is then matches against the database of TLS TCP
sequence number records and lengths. If a match is found a static params WQE
is queued on the IQ and the hardware should immediately resume decrypting TLS
data until the next non-sequential TCP packet arrives.

Offloading TLS RX data is supported for untagged, prio-tagged, and
regular VLAN traffic.

MFC after: 1 week
Sponsored by: NVIDIA Networking

show more ...

mlx5en: Properly account for no-checksum on tunneled packets.

MFC after: 1 week
Sponsored by: NVIDIA Networking

mlx5en: Implement support for internal queues, IQ.

Internal send queues are regular sendqueues which are reserved for WQE commands
towards the hardware and firmware. These queues typically carry res

mlx5en: Implement support for internal queues, IQ.

Internal send queues are regular sendqueues which are reserved for WQE commands
towards the hardware and firmware. These queues typically carry resync
information for ongoing TLS RX connections and when changing schedule queues
for rate limited connections.

The internal queue, IQ, code is more or less a stripped down copy
of the existing SQ managing code with exception of:

1) An optional single segment memory buffer which can be read or
written as a whole by the hardware, may be provided.
2) An optional completion callback for all transmit operations, may
be provided.
3) Does not support mbufs.

MFC after: 1 week
Sponsored by: NVIDIA Networking

show more ...

mlx5en: Patch to inhibit transmit doorbell writes during packet reception.

During packet reception the network stack frequently transmit data in
response to TCP window updates. To reduce the number

mlx5en: Patch to inhibit transmit doorbell writes during packet reception.

During packet reception the network stack frequently transmit data in
response to TCP window updates. To reduce the number of transmit doorbells
needed, inhibit all transmit doorbells designated for the same channel until
after the reception of packets for the given channel is completed.

While at it slightly refactor the mlx5e_tx_notify_hw() function:

1) The doorbell information is always stored into sq->doorbell.d64 .
No need to pass a separate pointer to this variable.

2) Move checks for skipping doorbell writes inside this function.

MFC after: 1 week
Sponsored by: NVIDIA Networking

show more ...

mlx5en: idiomatic use of preprocessor, in particular paths

MFC after: 1 week
Sponsored by: NVIDIA Networking

mlx5en: normalize use of the opt_*.h files

MFC after: 1 week
Sponsored by: NVIDIA Networking

mlx5en: handle offloaded Rx checksums calculated for tunneled packets

Reviewed by: hselasky
Sponsored by: Mellanox Technologies/NVidia Networking
MFC after: 1 week

Report EQE data upon CQ completion in mlx5core.

Report EQE data upon CQ completion to let upper layers use this data.

Linux commit:
4e0e2ea1886afe8c001971ff767f6670312a9b04

MFC after: 1 week
Spons

Report EQE data upon CQ completion in mlx5core.

Report EQE data upon CQ completion to let upper layers use this data.

Linux commit:
4e0e2ea1886afe8c001971ff767f6670312a9b04

MFC after: 1 week
Sponsored by: Mellanox Technologies // NVIDIA Networking

show more ...