ossl: Fix some bugs in the fallback AES-GCM implementation

gcm_*_aesni() are used when the AVX512 implementation is not available.
Fix two bugs which manifest when handling operations spanning multi

ossl: Fix some bugs in the fallback AES-GCM implementation

gcm_*_aesni() are used when the AVX512 implementation is not available.
Fix two bugs which manifest when handling operations spanning multiple
segments:
- Avoid underflow when the length of the input is smaller than the
residual.
- In gcm_decrypt_aesni(), ensure that we begin the operation at the
right offset into the input and output buffers.

Reviewed by: jhb
Fixes: 9b1d87286c78 ("ossl: Add a fallback AES-GCM implementation using AES-NI")
MFC after: 3 days
Differential Revision: https://reviews.freebsd.org/D42838

show more ...

ossl: Add a fallback AES-GCM implementation using AES-NI

This lets one use ossl(4) for AES-GCM operations on contemporary amd64
platforms. A kernel benchmark indicates that this gives roughly
equiv

ossl: Add a fallback AES-GCM implementation using AES-NI

This lets one use ossl(4) for AES-GCM operations on contemporary amd64
platforms. A kernel benchmark indicates that this gives roughly
equivalent throughput to aesni(4) for various buffer sizes.

Bulk processing is done in aesni-gcm-x86_64.S, the rest is handled in a
C wrapper ported from OpenSSL's gcm128.c.

Sponsored by: Stormshield
Sponsored by: Klara, Inc.
Reviewed by: jhb
MFC after: 3 months
Differential Revision: https://reviews.freebsd.org/D39967

show more ...

ossl: Add a VAES-based AES-GCM implementation for amd64

aes-gcm-avx512.S is generated from OpenSSL 3.1 and implements AES-GCM.
ossl_x86.c detects whether the CPU implements the required AVX512
instr

ossl: Add a VAES-based AES-GCM implementation for amd64

aes-gcm-avx512.S is generated from OpenSSL 3.1 and implements AES-GCM.
ossl_x86.c detects whether the CPU implements the required AVX512
instructions; if not, the ossl(4) module does not provide an AES-GCM
implementation. The VAES implementation increases throughput for all
buffer sizes in both directions, up to 2x for sufficiently large
buffers.

The "process" implementation is in two parts: a generic OCF layer in
ossl_aes.c that calls a set of MD functions to do the heavy lifting.
The intent there is to make it possible to add other implementations for
other platforms, e.g., to reduce the diff required for D37421.

A follow-up commit will add a fallback path to legacy AES-NI, so that
ossl(4) can be used in preference to aesni(4) on all amd64 platforms.
In the long term we would like to replace aesni(4) and armv8crypto(4)
with ossl(4).

Note, currently this implementation will not be selected by default
since aesni(4) and ossl(4) return the same probe priority for crypto
sessions, and the opencrypto framework selects the first registered
implementation to break a tie. Since aesni(4) is compiled into the
kernel, aesni(4) wins. A separate change may modify ossl(4) to have
priority.

Sponsored by: Stormshield
Sponsored by: Klara, Inc.
Reviewed by: jhb
MFC after: 3 months
Differential Revision: https://reviews.freebsd.org/D39783

show more ...