sys: Remove $FreeBSD$: one-line sh pattern

Remove /^\s*#[#!]?\s*\$FreeBSD\$.*$\n/

LinuxKPI: define LINUXKPI_INCLUDES for module builds as well

While for in-kernel we already have LINUXKPI_INCLUDES in kern.pre.mk
for kmod builds we've not had a common define to use leading to vari

LinuxKPI: define LINUXKPI_INCLUDES for module builds as well

While for in-kernel we already have LINUXKPI_INCLUDES in kern.pre.mk
for kmod builds we've not had a common define to use leading to various
spellings of include paths.

In order for the include list to be expanded more easily in the future,
e.g., adding the "dummy" includes (for all) and to harmonize code,
duplicate LINUXKPI_INCLUDES to kmod.mk and use it for all module Makefiles.

MFC after: 1 week
Reviewed by: hselasky
Differential Revision: https://reviews.freebsd.org/D36683

show more ...

mlx5en: Implement TLS RX support.

TLS RX support is modeled after TLS TX support. The basic structures and layouts
are almost identical, except that the send tag created filters RX traffic and
not T

mlx5en: Implement TLS RX support.

TLS RX support is modeled after TLS TX support. The basic structures and layouts
are almost identical, except that the send tag created filters RX traffic and
not TX traffic.

The TLS RX tag keeps track of past TLS records up to a certain limit,
approximately 1 Gbyte of TCP data. TLS records of same length are joined
into a single database record.

Regularly the HW is queried for TLS RX progress information. The TCP sequence
number gotten from the HW is then matches against the database of TLS TCP
sequence number records and lengths. If a match is found a static params WQE
is queued on the IQ and the hardware should immediately resume decrypting TLS
data until the next non-sequential TCP packet arrives.

Offloading TLS RX data is supported for untagged, prio-tagged, and
regular VLAN traffic.

MFC after: 1 week
Sponsored by: NVIDIA Networking

show more ...

mlx5en: Implement support for internal queues, IQ.

Internal send queues are regular sendqueues which are reserved for WQE commands
towards the hardware and firmware. These queues typically carry res

mlx5en: Implement support for internal queues, IQ.

Internal send queues are regular sendqueues which are reserved for WQE commands
towards the hardware and firmware. These queues typically carry resync
information for ongoing TLS RX connections and when changing schedule queues
for rate limited connections.

The internal queue, IQ, code is more or less a stripped down copy
of the existing SQ managing code with exception of:

1) An optional single segment memory buffer which can be read or
written as a whole by the hardware, may be provided.
2) An optional completion callback for all transmit operations, may
be provided.
3) Does not support mbufs.

MFC after: 1 week
Sponsored by: NVIDIA Networking

show more ...

ibcore: Kernel space update based on Linux 5.7-rc1.

Overview:

This is the first stage of a RDMA stack upgrade introducing kernel
changes only based on Linux 5.7-rc1.

This patch is based on about f

ibcore: Kernel space update based on Linux 5.7-rc1.

Overview:

This is the first stage of a RDMA stack upgrade introducing kernel
changes only based on Linux 5.7-rc1.

This patch is based on about four main areas of work:
- Update of the IB uobjects system:
- The memory holding so-called AH, CQ, PD, SRQ and UCONTEXT objects
is now managed by ibcore. This also require some changes in the
kernel verbs API. The updated verbs changes are typically about
initialize and deinitialize objects, and remove allocation and
free of memory.

- Update of the uverbs IOCTL framework:
- The parsing and handling of user-space commands has been
completely refactored to integrate with the updated IB uobjects
system.

- Various changes and updates to the generic uverbs interfaces in
device drivers including the new uAPI surface.

- The mlx5_ib_devx.c in mlx5ib and related mlx5 core changes.

Dependencies:

- The mlx4ib driver code has been updated with the minimum changes
needed.

- The mlx5ib driver code has been updated with the minimum changes
needed including DV support.

Compatibility:

- All user-space facing APIs are backwards compatible after this
change.

- All kernel-space facing RDMA APIs are backwards compatible after
this change, with exception of ib_create_ah() and ib_destroy_ah()
which takes a new flag.

- The "ib_device_ops" structure exist, but only contains the driver ID
and some structure sizes.

Differences from Linux:

- Infiniband drivers must use the INIT_IB_DEVICE_OPS() macro to set
the sizes needed for allocating various IB objects, when adding
IB device instances.

Security:

- PRIV_NET_RAW is needed to use raw ethernet transmit features.
- PRIV_DRIVER is needed to use other privileged operations.

Based on upstream Linux, Torvalds (5.7-rc1):
8632e9b5645bbc2331d21d892b0d6961c1a08429

MFC after: 1 week
Reviewed by: kib
Differential Revision: https://reviews.freebsd.org/D31149
Sponsored by: NVIDIA Networking

show more ...

Introduce LINUXKPI_GENSRCS.

Centralize the list of generated files required by linuxkpi consumers,
into the common variable. This way, consumers that use the variable
are insulated from possible ch

Introduce LINUXKPI_GENSRCS.

Centralize the list of generated files required by linuxkpi consumers,
into the common variable. This way, consumers that use the variable
are insulated from possible changes in the list.

Reviewed by: hselasky, imp
Sponsored by: Mellanox Technologies
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D24137

show more ...

Add pci_iov_if.h header as a dependency for Linuxkpi consumers.

Sponsored by: Mellanox Technologies
MFC after: 2 weeks

Implement hardware TLS via send tags for mlx5en(4), which is supported by
ConnectX-6 DX.

Currently TLS v1.2 and v1.3 with AES 128/256 crypto over TCP/IP (v4
and v6) is supported.

A per PCI device U

Implement hardware TLS via send tags for mlx5en(4), which is supported by
ConnectX-6 DX.

Currently TLS v1.2 and v1.3 with AES 128/256 crypto over TCP/IP (v4
and v6) is supported.

A per PCI device UMA zone is used to manage the memory of the send
tags. To optimize performance some crypto contexts may be cached by
the UMA zone, until the UMA zone finishes the memory of the given send
tag.

An asynchronous task is used manage setup of the send tags towards the
firmware. Most importantly setting the AES 128/256 bit pre-shared keys
for the crypto context.

Updating the state of the AES crypto engine and encrypting data, is
all done in the fast path. Each send tag tracks the TCP sequence
number in order to detect non-contiguous blocks of data, which may
require a dump of prior unencrypted data, to restore the crypto state
prior to wire transmission.

Statistics counters have been added to count the amount of TLS data
transmitted in total, and the amount of TLS data which has been dumped
prior to transmission. When non-contiguous TCP sequence numbers are
detected, the software needs to dump the beginning of the current TLS
record up until the point of retransmission. All TLS counters utilize
the counter(9) API.

In order to enable hardware TLS offload the following sysctls must be set:
kern.ipc.mb_use_ext_pgs=1
kern.ipc.tls.ifnet.permitted=1
kern.ipc.tls.enable=1

Sponsored by: Mellanox Technologies

show more ...

Merge ^/head r352764 through r353315.

Import Linux code to query/set buffer state in mlx5en(4).

Submitted by: kib@
MFC after: 3 days
Sponsored by: Mellanox Technologies

MFHead @347527

Sponsored by: The FreeBSD Foundation

Add support for Dynamic Interrupt Moderation, DIM, in mlx5en(4).

Add support for DIM based on Linux,
with some minor adaptions specific to FreeBSD.

Linux commit
f97c3dc3c0e8d23a5c4357d182afeef4c67f

Add support for Dynamic Interrupt Moderation, DIM, in mlx5en(4).

Add support for DIM based on Linux,
with some minor adaptions specific to FreeBSD.

Linux commit
f97c3dc3c0e8d23a5c4357d182afeef4c67f5c33

MFC after: 3 days
Sponsored by: Mellanox Technologies

show more ...

Merge ^/head r340918 through r341763.

mlx5fpga: Initial code import.

Submitted by: kib@
Approved by: hselasky (mentor)
MFC after: 1 week
Sponsored by: Mellanox Technologies

Add support for hardware rate limiting to mlx5en(4).

The hardware rate limiting feature is enabled by the RATELIMIT kernel
option. Please refer to ifconfig(8) and the txrtlmt option and the
SO_MAX_P

Add support for hardware rate limiting to mlx5en(4).

The hardware rate limiting feature is enabled by the RATELIMIT kernel
option. Please refer to ifconfig(8) and the txrtlmt option and the
SO_MAX_PACING_RATE set socket option for more information. This
feature is compatible with hardware transmit send offload, TSO.

A set of sysctl(8) knobs under dev.mce.<N>.rate_limit are provided to
setup the ratelimit table and also to fine tune various rate limit
related parameters.

Sponsored by: Mellanox Technologies

show more ...

sys/modules: normalize .CURDIR-relative paths to SRCTOP

This simplifies make output/logic

Tested with: `cd sys/modules; make ALL_MODULES=` on amd64
MFC after: 1 month
Sponsored by: Dell EMC Isilon

Merge ^/head r303250 through r304235.

Switch to the new block based LRO input function for the mlx5en
driver. This change significantly increases the overall RX aggregation
ratio for heavily loaded networks handling 10-80 thousand simult

Switch to the new block based LRO input function for the mlx5en
driver. This change significantly increases the overall RX aggregation
ratio for heavily loaded networks handling 10-80 thousand simultaneous
connections.

Remove the turbo LRO code and all references to it which has now been
superceeded by the tcp_lro_queue_mbuf() function.

Tested by: Netflix
Sponsored by: Mellanox Technologies
MFC after: 1 week

show more ...

MFH r289384-r293170

Sponsored by: The FreeBSD Foundation

MFH @r292599

This includes the pluggable TCP framework and other chnages to the
netstack to track for VNET stability.

Security: The FreeBSD Foundation

Add full support for Receive Side Scaling, RSS, to the mlx5en
driver. This includes binding all interrupt and worker threads
according to the RSS configuration, setting up correct Toeplitz
hashing ke

Add full support for Receive Side Scaling, RSS, to the mlx5en
driver. This includes binding all interrupt and worker threads
according to the RSS configuration, setting up correct Toeplitz
hashing keys as given by RSS and setting the correct mbuf
hashtype for all received traffic.

MFC after: 1 week
Sponsored by: Mellanox Technologies
Differential Revision: https://reviews.freebsd.org/D4410

show more ...

Catch up with head (r291075).

Add mlx5 and mlx5en driver(s) for ConnectX-4 and ConnectX-4LX cards
from Mellanox Technologies. The current driver supports ethernet
speeds up to and including 100 GBit/s. Infiniband support will be

Add mlx5 and mlx5en driver(s) for ConnectX-4 and ConnectX-4LX cards
from Mellanox Technologies. The current driver supports ethernet
speeds up to and including 100 GBit/s. Infiniband support will be
done later.

The code added is not compiled by default, which will be done by a
separate commit.

Sponsored by: Mellanox Technologies
MFC after: 2 weeks

show more ...