OpenCrypto: Convert sessions to opaque handles instead of integers

Track session objects in the framework, and pass handles between the
framework (OCF), consumers, and drivers. Avoid redundancy and

OpenCrypto: Convert sessions to opaque handles instead of integers

Track session objects in the framework, and pass handles between the
framework (OCF), consumers, and drivers. Avoid redundancy and complexity in
individual drivers by allocating session memory in the framework and
providing it to drivers in ::newsession().

Session handles are no longer integers with information encoded in various
high bits. Use of the CRYPTO_SESID2FOO() macros should be replaced with the
appropriate crypto_ses2foo() function on the opaque session handle.

Convert OCF drivers (in particular, cryptosoft, as well as myriad others) to
the opaque handle interface. Discard existing session tracking as much as
possible (quick pass). There may be additional code ripe for deletion.

Convert OCF consumers (ipsec, geom_eli, krb5, cryptodev) to handle-style
interface. The conversion is largely mechnical.

The change is documented in crypto.9.

Inspired by
https://lists.freebsd.org/pipermail/freebsd-arch/2018-January/018835.html .

No objection from: ae (ipsec portion)
Reported by: jhb

show more ...

Don't overflow the ipad[] array when clearing the remainder.

After the auth key is copied into the ipad[] array, any remaining bytes
are cleared to zero (in case the key is shorter than one block si

Don't overflow the ipad[] array when clearing the remainder.

After the auth key is copied into the ipad[] array, any remaining bytes
are cleared to zero (in case the key is shorter than one block size).
The full block size was used as the length of the zero rather than the
size of the remaining ipad[]. In practice this overflow was harmless as
it could only clear bytes in the following opad[] array which is
initialized with a copy of ipad[] in the next statement.

Sponsored by: Chelsio Communications

show more ...

Move ccr_aes_getdeckey() from ccr(4) to the cxgbe(4) driver.

This routine will also be used by the TOE module to manage TLS keys.

Sponsored by: Chelsio Communications

Store IV in output buffer in GCM software fallback when requested.

Properly honor the lack of the CRD_F_IV_PRESENT flag in the GCM
software fallback case for encryption requests.

Submitted by: Hars

Store IV in output buffer in GCM software fallback when requested.

Properly honor the lack of the CRD_F_IV_PRESENT flag in the GCM
software fallback case for encryption requests.

Submitted by: Harsh Jain @ Chelsio
Sponsored by: Chelsio Communications

show more ...

Don't read or generate an IV until all error checking is complete.

In particular, this avoids edge cases where a generated IV might be
written into the output buffer even though the request is faile

Don't read or generate an IV until all error checking is complete.

In particular, this avoids edge cases where a generated IV might be
written into the output buffer even though the request is failed with
an error.

Sponsored by: Chelsio Communications

show more ...

Expand the software fallback for GCM to cover more cases.

- Extend ccr_gcm_soft() to handle requests with a non-empty payload.
While here, switch to allocating the GMAC context instead of placing

Expand the software fallback for GCM to cover more cases.

- Extend ccr_gcm_soft() to handle requests with a non-empty payload.
While here, switch to allocating the GMAC context instead of placing
it on the stack since it is over 1KB in size.
- Allow ccr_gcm() to return a special error value (EMSGSIZE) which
triggers a fallback to ccr_gcm_soft(). Move the existing empty
payload check into ccr_gcm() and change a few other cases
(e.g. large AAD) to fallback to software via EMSGSIZE as well.
- Add a new 'sw_fallback' stat to count the number of requests
processed via the software fallback.

Submitted by: Harsh Jain @ Chelsio (original version)
Sponsored by: Chelsio Communications

show more ...

Clamp DSGL entries to a length of 2KB.

This works around an issue in the T6 that can result in DMA engine
stalls if an error occurs while processing a DSGL entry with a length
larger than 2KB.

Subm

Clamp DSGL entries to a length of 2KB.

This works around an issue in the T6 that can result in DMA engine
stalls if an error occurs while processing a DSGL entry with a length
larger than 2KB.

Submitted by: Harsh Jain @ Chelsio
Sponsored by: Chelsio Communications

show more ...

Fail crypto requests when the resulting work request is too large.

Most crypto requests will not trigger this condition, but a request
with a highly-fragmented data buffer (and a resulting "large" S

Fail crypto requests when the resulting work request is too large.

Most crypto requests will not trigger this condition, but a request
with a highly-fragmented data buffer (and a resulting "large" S/G
list) could trigger it.

Sponsored by: Chelsio Communications

show more ...

Don't discard AAD and IV output data for AEAD requests.

The T6 can hang when processing certain AEAD requests if the request
sets a flag asking the crypto engine to discard the input IV and AAD
rath

Don't discard AAD and IV output data for AEAD requests.

The T6 can hang when processing certain AEAD requests if the request
sets a flag asking the crypto engine to discard the input IV and AAD
rather than copying them into the output buffer. The existing driver
always discards the IV and AAD as we do not need it. As a workaround,
allocate a single "dummy" buffer when the ccr driver attaches and
change all AEAD requests to write the IV and AAD to this scratch
buffer. The contents of the scratch buffer are never used (similar to
"bogus_page"), and it is ok for multiple in-flight requests to share
this dummy buffer.

Submitted by: Harsh Jain @ Chelsio (original version)
Sponsored by: Chelsio Communications

show more ...

Reject requests with AAD and IV larger than 511 bytes.

The T6 crypto engine's control messages only support a total AAD
length (including the prefixed IV) of 511 bytes. Reject requests with
large A

Reject requests with AAD and IV larger than 511 bytes.

The T6 crypto engine's control messages only support a total AAD
length (including the prefixed IV) of 511 bytes. Reject requests with
large AAD rather than returning incorrect results.

Sponsored by: Chelsio Communications

show more ...

Always set the IV location to IV_NOP.

The firmware ignores this field in the FW_CRYPTO_LOOKASIDE_WR work
request.

Submitted by: Harsh Jain @ Chelsio
Sponsored by: Chelsio Communications

Always store the IV in the immediate portion of a work request.

Combined authentication-encryption and GCM requests already stored the
IV in the immediate explicitly. This extends this behavior to

Always store the IV in the immediate portion of a work request.

Combined authentication-encryption and GCM requests already stored the
IV in the immediate explicitly. This extends this behavior to block
cipher requests to work around a firmware bug. While here, simplify
the AEAD and GCM handlers to not include always-true conditions.

Submitted by: Harsh Jain @ Chelsio
Sponsored by: Chelsio Communications

show more ...

Revert r327828, r327949, r327953, r328016-r328026, r328041:
Uses of mallocarray(9).

The use of mallocarray(9) has rocketed the required swap to build FreeBSD.
This is likely caused by the allocation

Revert r327828, r327949, r327953, r328016-r328026, r328041:
Uses of mallocarray(9).

The use of mallocarray(9) has rocketed the required swap to build FreeBSD.
This is likely caused by the allocation size attributes which put extra pressure
on the compiler.

Given that most of these checks are superfluous we have to choose better
where to use mallocarray(9). We still have more uses of mallocarray(9) but
hopefully this is enough to bring swap usage to a reasonable level.

Reported by: wosch
PR: 225197

show more ...

dev: make some use of mallocarray(9).

Focus on code where we are doing multiplications within malloc(9). None of
these is likely to overflow, however the change is still useful as some
static checke

dev: make some use of mallocarray(9).

Focus on code where we are doing multiplications within malloc(9). None of
these is likely to overflow, however the change is still useful as some
static checkers can benefit from the allocation attributes we use for
mallocarray.

This initial sweep only covers malloc(9) calls with M_NOWAIT. No good
reason but I started doing the changes before r327796 and at that time it
was convenient to make sure the sorrounding code could handle NULL values.

show more ...

Merge ^/head r323559 through r325504.

MFhead@r323635

Fix some incorrect sysctl pointers for some error stats.

The bad_session, sglist_error, and process_error sysctl nodes were
returning the value of the pad_error node instead of the appropriate
error

Fix some incorrect sysctl pointers for some error stats.

The bad_session, sglist_error, and process_error sysctl nodes were
returning the value of the pad_error node instead of the appropriate
error counters.

Sponsored by: Chelsio Communications

show more ...

Merge ^/head r319548 through r319778.

Fix the software fallback for GCM to validate the existing tag for decrypts.

Sponsored by: Chelsio Communications

Add explicit handling for requests with an empty payload.

- For HMAC requests, construct a special input buffer to request an empty
hash result.
- For plain cipher requests and requests that chain

Add explicit handling for requests with an empty payload.

- For HMAC requests, construct a special input buffer to request an empty
hash result.
- For plain cipher requests and requests that chain an AES cipher with an
HMAC, fail with EINVAL if there is no cipher payload. If needed in
the future, chained requests that only contain AAD could be serviced as
HMAC-only requests.
- For GCM requests, the hardware does not support generating the tag for
an AAD-only request. Instead, complete these requests synchronously
in software on the assumption that such requests are rare.

Sponsored by: Chelsio Communications

show more ...

Merge ^/head r318964 through r319164.

Fail large requests with EFBIG.

The adapter firmware in general does not accept PDUs larger than 64k - 1
bytes in size. Sending crypto requests larger than this size result in
hangs or incorrect ou

Fail large requests with EFBIG.

The adapter firmware in general does not accept PDUs larger than 64k - 1
bytes in size. Sending crypto requests larger than this size result in
hangs or incorrect output, so reject them with EFBIG. For requests
chaining an AES cipher with an HMAC, the firmware appears to require
slightly smaller requests (around 512 bytes).

Sponsored by: Chelsio Communications

show more ...

Merge ^/head r318380 through r318559.

Add a driver for the Chelsio T6 crypto accelerator engine.

The ccr(4) driver supports use of the crypto accelerator engine on
Chelsio T6 NICs in "lookaside" mode via the opencrypto framework.

Curre

Add a driver for the Chelsio T6 crypto accelerator engine.

The ccr(4) driver supports use of the crypto accelerator engine on
Chelsio T6 NICs in "lookaside" mode via the opencrypto framework.

Currently, the driver supports AES-CBC, AES-CTR, AES-GCM, and AES-XTS
cipher algorithms as well as the SHA1-HMAC, SHA2-256-HMAC, SHA2-384-HMAC,
and SHA2-512-HMAC authentication algorithms. The driver also supports
chaining one of AES-CBC, AES-CTR, or AES-XTS with an authentication
algorithm for encrypt-then-authenticate operations.

Note that this driver is still under active development and testing and
may not yet be ready for production use. It does pass the tests in
tests/sys/opencrypto with the exception that the AES-GCM implementation
in the driver does not yet support requests with a zero byte payload.

To use this driver currently, the "uwire" configuration must be used
along with explicitly enabling support for lookaside crypto capabilities
in the cxgbe(4) driver. These can be done by setting the following
tunables before loading the cxgbe(4) driver:

hw.cxgbe.config_file=uwire
hw.cxgbe.cryptocaps_allowed=-1

MFC after: 1 month
Relnotes: yes
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D10763

show more ...