mlx5en: do now waste ipsec_accel_in_tag on non-ipsec packets rx

Do not prepend ipsec tags into mbuf head when preparing rx wqe, store it
separately. Only prepend (and clear the store) when received

mlx5en: do now waste ipsec_accel_in_tag on non-ipsec packets rx

Do not prepend ipsec tags into mbuf head when preparing rx wqe, store it
separately. Only prepend (and clear the store) when received packed was
indeed offloaded by ipsec engine. Then we do not need to refill tags
for slots that received non-ipsec packets.

This should solve some minimal degradation of the rx CPU usage due to
unneeded tag allocation for each packet.

Sponsored by: NVidia networking

show more ...

mlx5en: stop including mlx5_accel/ipsec.h from en.h

This creates a circular dependency preventing inline functions from
ipsec.h from using en.h definitions.

Sponsored by: NVidia networking

mlx5en: add IPSEC_OFFLOAD support

Right now, only IPv4 transport mode, with aes-gcm ESP, is supported.
Driver also cooperates with NAT-T, and obeys socket policies, which
makes IKEd like StrongSwan

mlx5en: add IPSEC_OFFLOAD support

Right now, only IPv4 transport mode, with aes-gcm ESP, is supported.
Driver also cooperates with NAT-T, and obeys socket policies, which
makes IKEd like StrongSwan working.

Sponsored by: NVIDIA networking

show more ...

sys: Remove $FreeBSD$: two-line .h pattern

Remove /^\s*\*\n \*\s+\$FreeBSD\$$\n/

mlx5en(4): Don't wait for receive queue to fill up with mbufs during open channels.

Failure to get mbufs may be transient.
Don't permanently fail to open the channels due to lack of mbufs.
This also

mlx5en(4): Don't wait for receive queue to fill up with mbufs during open channels.

Failure to get mbufs may be transient.
Don't permanently fail to open the channels due to lack of mbufs.
This also makes modifying channel parameters faster.

MFC after: 1 week
Sponsored by: NVIDIA Networking

show more ...

mlx5: Make mlx5_del_flow_rule() NULL safe.

This change factors out repeated NULL checks.

No functional change intended.

MFC after: 1 week
Sponsored by: NVIDIA Networking

Mechanically convert mlx5en(4) to IfAPI

Reviewed by: zlei
Sponsored by: Juniper Networks, Inc.
Differential Revision: https://reviews.freebsd.org/D38595

mlx5 M_TSTMP accuracy looses quite a bit of precision so lets fix it.

The way that the clock is synchronized between the system and the current mlx5 for the purposes of the M_TSTMP
being carried we

mlx5 M_TSTMP accuracy looses quite a bit of precision so lets fix it.

The way that the clock is synchronized between the system and the current mlx5 for the purposes of the M_TSTMP
being carried we loose a lot of precision. Instead lets change the math that calculates this to separate out
the seconds/nanoseconds and operate on the two values so we don't get overflow instead of just
shifting the value down and loosing precision.

Reviewed by: kib, hselasky
Sponsored by: Netflix Inc
Differential Revision: https://reviews.freebsd.org/D36327

show more ...

mlx5en(4): Fix a few typos in source code comments

- s/persistant/persistent/

MFC after: 3 days

mlx5en: Improve CQE error debugging.

MFC after: 1 week
Sponsored by: NVIDIA Networking

mlx5en: Improve RX- and TX- TLS refcounting.

Use the send tag refcounting mechanism to refcount the RX- and TX- TLS
send tags. Then it is no longer needed to wait for refcounts to reach
zero when de

mlx5en: Improve RX- and TX- TLS refcounting.

Use the send tag refcounting mechanism to refcount the RX- and TX- TLS
send tags. Then it is no longer needed to wait for refcounts to reach
zero when destroying RX- and TX- TLS send tags as a result of pending
data or WQE commands.

This also ensures that when TX-TLS and rate limiting is used at the same
time, the underlying SQ is not prematurely destroyed.

MFC after: 1 week
Sponsored by: NVIDIA Networking

show more ...

mlx5en: Implement TLS RX support.

TLS RX support is modeled after TLS TX support. The basic structures and layouts
are almost identical, except that the send tag created filters RX traffic and
not T

mlx5en: Implement TLS RX support.

TLS RX support is modeled after TLS TX support. The basic structures and layouts
are almost identical, except that the send tag created filters RX traffic and
not TX traffic.

The TLS RX tag keeps track of past TLS records up to a certain limit,
approximately 1 Gbyte of TCP data. TLS records of same length are joined
into a single database record.

Regularly the HW is queried for TLS RX progress information. The TCP sequence
number gotten from the HW is then matches against the database of TLS TCP
sequence number records and lengths. If a match is found a static params WQE
is queued on the IQ and the hardware should immediately resume decrypting TLS
data until the next non-sequential TCP packet arrives.

Offloading TLS RX data is supported for untagged, prio-tagged, and
regular VLAN traffic.

MFC after: 1 week
Sponsored by: NVIDIA Networking

show more ...

mlx5en: Implement one RQT object per channel.

These objects will eventually be used to switch TLS RX traffic.

MFC after: 1 week
Sponsored by: NVIDIA Networking

mlx5: Implement flow steering helper functions for TCP sockets.

This change adds convenience functions to setup a flow steering rule based on
a TCP socket. The helper function gets all the address i

mlx5: Implement flow steering helper functions for TCP sockets.

This change adds convenience functions to setup a flow steering rule based on
a TCP socket. The helper function gets all the address information from the
socket and returns a steering rule, to be used with HW TLS RX offload.

MFC after: 1 week
Sponsored by: NVIDIA Networking

show more ...

mlx5en: Create and destroy all flow tables and rules when the network interface attaches and detaches.

Previously flow steering tables and rules were only created and destroyed
at link up and down e

mlx5en: Create and destroy all flow tables and rules when the network interface attaches and detaches.

Previously flow steering tables and rules were only created and destroyed
at link up and down events, respectivly. Due to new requirements for adding
TLS RX flow tables and rules, the main flow steering table must always be
available as there are permanent redirections from the TLS RX flow table
to the vlan flow table.

MFC after: 1 week
Sponsored by: NVIDIA Networking

show more ...

mlx5/mlx5en: Add SQ remap support

Add support to map an SQ to a specific schedule queue using a
special WQE as performance enhancement.

SQ remap operation is handled by a privileged internal queue,

mlx5/mlx5en: Add SQ remap support

Add support to map an SQ to a specific schedule queue using a
special WQE as performance enhancement.

SQ remap operation is handled by a privileged internal queue, IQ,
and the mapping is enabled from one rate to another.

The transition from paced to non-paced should however always go
through FW.

MFC after: 1 week
Sponsored by: NVIDIA Networking

show more ...

mlx5en: Implement support for internal queues, IQ.

Internal send queues are regular sendqueues which are reserved for WQE commands
towards the hardware and firmware. These queues typically carry res

mlx5en: Implement support for internal queues, IQ.

Internal send queues are regular sendqueues which are reserved for WQE commands
towards the hardware and firmware. These queues typically carry resync
information for ongoing TLS RX connections and when changing schedule queues
for rate limited connections.

The internal queue, IQ, code is more or less a stripped down copy
of the existing SQ managing code with exception of:

1) An optional single segment memory buffer which can be read or
written as a whole by the hardware, may be provided.
2) An optional completion callback for all transmit operations, may
be provided.
3) Does not support mbufs.

MFC after: 1 week
Sponsored by: NVIDIA Networking

show more ...

mlx5en: Implement dummy receive queue, RQ, for dropping packets.

What is a drop RQ and why is it needed?

The RSS indirection table, also called the RQT, selects the
destination RQ based on the rece

mlx5en: Implement dummy receive queue, RQ, for dropping packets.

What is a drop RQ and why is it needed?

The RSS indirection table, also called the RQT, selects the
destination RQ based on the receive queue number, RQN. The RQT is
frequently referred to by flow steering rules to distribute traffic
among multiple RQs. The problem is that the RQs cannot be destroyed
before the RQT referring them is destroyed too. Further, TLS RX
rules may still be referring to the RQT even if the link went
down. Because there is no magic RQN for dropping packets, we create
a dummy RQ, also called drop RQ, which sole purpose is to drop all
received packets. When the link goes down this RQN is filled in all
RQT entries, of the main RQT, so the real RQs which are about to be
destroyed can be released and the TLS RX rules can be sustained.

MFC after: 1 week
Sponsored by: NVIDIA Networking

show more ...

mlx5en: Patch to inhibit transmit doorbell writes during packet reception.

During packet reception the network stack frequently transmit data in
response to TCP window updates. To reduce the number

mlx5en: Patch to inhibit transmit doorbell writes during packet reception.

During packet reception the network stack frequently transmit data in
response to TCP window updates. To reduce the number of transmit doorbells
needed, inhibit all transmit doorbells designated for the same channel until
after the reception of packets for the given channel is completed.

While at it slightly refactor the mlx5e_tx_notify_hw() function:

1) The doorbell information is always stored into sq->doorbell.d64 .
No need to pass a separate pointer to this variable.

2) Move checks for skipping doorbell writes inside this function.

MFC after: 1 week
Sponsored by: NVIDIA Networking

show more ...

mlx5en: idiomatic use of preprocessor, in particular paths

MFC after: 1 week
Sponsored by: NVIDIA Networking

mlx5en: normalize use of the opt_*.h files

MFC after: 1 week
Sponsored by: NVIDIA Networking

Add a switch structure for send tags.

Move the type and function pointers for operations on existing send
tags (modify, query, next, free) out of 'struct ifnet' and into a new
'struct if_snd_tag_sw'

Add a switch structure for send tags.

Move the type and function pointers for operations on existing send
tags (modify, query, next, free) out of 'struct ifnet' and into a new
'struct if_snd_tag_sw'. A pointer to this structure is added to the
generic part of send tags and is initialized by m_snd_tag_init()
(which now accepts a switch structure as a new argument in place of
the type).

Previously, device driver ifnet methods switched on the type to call
type-specific functions. Now, those type-specific functions are saved
in the switch structure and invoked directly. In addition, this more
gracefully permits multiple implementations of the same tag within a
driver. In particular, NIC TLS for future Chelsio adapters will use a
different implementation than the existing NIC TLS support for T6
adapters.

Reviewed by: gallatin, hselasky, kib (older version)
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D31572

show more ...

ibcore: Kernel space update based on Linux 5.7-rc1.

Overview:

This is the first stage of a RDMA stack upgrade introducing kernel
changes only based on Linux 5.7-rc1.

This patch is based on about f

ibcore: Kernel space update based on Linux 5.7-rc1.

Overview:

This is the first stage of a RDMA stack upgrade introducing kernel
changes only based on Linux 5.7-rc1.

This patch is based on about four main areas of work:
- Update of the IB uobjects system:
- The memory holding so-called AH, CQ, PD, SRQ and UCONTEXT objects
is now managed by ibcore. This also require some changes in the
kernel verbs API. The updated verbs changes are typically about
initialize and deinitialize objects, and remove allocation and
free of memory.

- Update of the uverbs IOCTL framework:
- The parsing and handling of user-space commands has been
completely refactored to integrate with the updated IB uobjects
system.

- Various changes and updates to the generic uverbs interfaces in
device drivers including the new uAPI surface.

- The mlx5_ib_devx.c in mlx5ib and related mlx5 core changes.

Dependencies:

- The mlx4ib driver code has been updated with the minimum changes
needed.

- The mlx5ib driver code has been updated with the minimum changes
needed including DV support.

Compatibility:

- All user-space facing APIs are backwards compatible after this
change.

- All kernel-space facing RDMA APIs are backwards compatible after
this change, with exception of ib_create_ah() and ib_destroy_ah()
which takes a new flag.

- The "ib_device_ops" structure exist, but only contains the driver ID
and some structure sizes.

Differences from Linux:

- Infiniband drivers must use the INIT_IB_DEVICE_OPS() macro to set
the sizes needed for allocating various IB objects, when adding
IB device instances.

Security:

- PRIV_NET_RAW is needed to use raw ethernet transmit features.
- PRIV_DRIVER is needed to use other privileged operations.

Based on upstream Linux, Torvalds (5.7-rc1):
8632e9b5645bbc2331d21d892b0d6961c1a08429

MFC after: 1 week
Reviewed by: kib
Differential Revision: https://reviews.freebsd.org/D31149
Sponsored by: NVIDIA Networking

show more ...

mlx5en: Allow binding channels to CPUs when RSS is not enabled.

MFC after: 1 week
Submitted by: Netflix
Reviewed by: kib
Sponsored by: Mellanox Technologies // NVIDIA Networking

mlx5en: Allocate per-channel doorbells.

To avoid congestion on the same PCI memory register space when
traffic consists mostly of small packets.

MFC after: 1 week
Reviewed by: kib
Sponsored by: Mel

mlx5en: Allocate per-channel doorbells.

To avoid congestion on the same PCI memory register space when
traffic consists mostly of small packets.

MFC after: 1 week
Reviewed by: kib
Sponsored by: Mellanox Technologies // NVIDIA Networking

show more ...