| c0a44d17 | 01-Sep-2026 |
Eric Biggers <ebiggers@kernel.org> |
xor: Add AVX-512 optimized xor_gen()
Add an implementation of xor_gen() using AVX-512.
It uses 512-bit vectors, i.e. ZMM registers. It also uses the vpternlogq instruction to do three-input XORs w
xor: Add AVX-512 optimized xor_gen()
Add an implementation of xor_gen() using AVX-512.
It uses 512-bit vectors, i.e. ZMM registers. It also uses the vpternlogq instruction to do three-input XORs when applicable.
It's enabled on x86_64 CPUs that have AVX512F && !PREFER_YMM. In practice that means:
- AMD Zen 4 and later (client and server) - Intel Sapphire Rapids and later (server) - Intel Rocket Lake (client) - Intel Nova Lake and later (client)
The !PREFER_YMM condition excludes the older AVX-512 implementations in Intel Skylake Server and Intel Ice Lake. They could run this code, but they're known to have overly-eager downclocking when ZMM registers are used. This is the same policy that the crypto and CRC code uses.
Results from xor_benchmark on AMD Ryzen 9 9950X (Zen 5):
Before:
# xor_benchmark: 4096 bytes 16384 bytes # xor_benchmark: 4 disks: 256 GB/s 185 GB/s # xor_benchmark: 5 disks: 233 GB/s 165 GB/s # xor_benchmark: 6 disks: 243 GB/s 174 GB/s # xor_benchmark: 7 disks: 244 GB/s 176 GB/s # xor_benchmark: 8 disks: 265 GB/s 179 GB/s # xor_benchmark: 10 disks: 249 GB/s 175 GB/s # xor_benchmark: 12 disks: 219 GB/s 173 GB/s # xor_benchmark: 15 disks: 206 GB/s 177 GB/s # xor_benchmark: 16 disks: 212 GB/s 180 GB/s # xor_benchmark: 32 disks: 193 GB/s 177 GB/s
After:
# xor_benchmark: 4096 bytes 16384 bytes # xor_benchmark: 4 disks: 492 GB/s 202 GB/s # xor_benchmark: 5 disks: 436 GB/s 191 GB/s # xor_benchmark: 6 disks: 466 GB/s 193 GB/s # xor_benchmark: 7 disks: 491 GB/s 194 GB/s # xor_benchmark: 8 disks: 513 GB/s 205 GB/s # xor_benchmark: 10 disks: 454 GB/s 199 GB/s # xor_benchmark: 12 disks: 330 GB/s 204 GB/s # xor_benchmark: 15 disks: 266 GB/s 198 GB/s # xor_benchmark: 16 disks: 243 GB/s 177 GB/s # xor_benchmark: 32 disks: 187 GB/s 165 GB/s
Signed-off-by: Eric Biggers <ebiggers@kernel.org> Signed-off-by: Borislav Petkov (AMD) <bp@alien8.de> Reviewed-by: David Laight <david.laight.linux@gmail.com> Reviewed-by: Christoph Hellwig <hch@lst.de> Acked-by: Christoph Hellwig <hch@lst.de> Link: https://patch.msgid.link/20260901145740.42337-9-ebiggers@kernel.org
show more ...
|
| 80dcf0a7 | 27-Mar-2026 |
Christoph Hellwig <hch@lst.de> |
xor: pass the entire operation to the low-level ops
Currently the high-level xor code chunks up all operations into small units for only up to 1 + 4 vectors, and passes it to four different methods.
xor: pass the entire operation to the low-level ops
Currently the high-level xor code chunks up all operations into small units for only up to 1 + 4 vectors, and passes it to four different methods. This means the FPU/vector context is entered and left a lot for wide stripes, and a lot of indirect expensive indirect calls are performed. Switch to passing the entire gen_xor request to the low-level ops, and provide a macro to dispatch it to the existing helper.
This reduce the number of indirect calls and FPU/vector context switches by a factor approaching nr_stripes / 4, and also reduces source and binary code size.
Link: https://lkml.kernel.org/r/20260327061704.3707577-27-hch@lst.de Signed-off-by: Christoph Hellwig <hch@lst.de> Reviewed-by: Eric Biggers <ebiggers@kernel.org> Tested-by: Eric Biggers <ebiggers@kernel.org> Cc: Albert Ou <aou@eecs.berkeley.edu> Cc: Alexander Gordeev <agordeev@linux.ibm.com> Cc: Alexandre Ghiti <alex@ghiti.fr> Cc: Andreas Larsson <andreas@gaisler.com> Cc: Anton Ivanov <anton.ivanov@cambridgegreys.com> Cc: Ard Biesheuvel <ardb@kernel.org> Cc: Arnd Bergmann <arnd@arndb.de> Cc: "Borislav Petkov (AMD)" <bp@alien8.de> Cc: Catalin Marinas <catalin.marinas@arm.com> Cc: Chris Mason <clm@fb.com> Cc: Christian Borntraeger <borntraeger@linux.ibm.com> Cc: Dan Williams <dan.j.williams@intel.com> Cc: David S. Miller <davem@davemloft.net> Cc: David Sterba <dsterba@suse.com> Cc: Heiko Carstens <hca@linux.ibm.com> Cc: Herbert Xu <herbert@gondor.apana.org.au> Cc: "H. Peter Anvin" <hpa@zytor.com> Cc: Huacai Chen <chenhuacai@kernel.org> Cc: Ingo Molnar <mingo@redhat.com> Cc: Jason A. Donenfeld <jason@zx2c4.com> Cc: Johannes Berg <johannes@sipsolutions.net> Cc: Li Nan <linan122@huawei.com> Cc: Madhavan Srinivasan <maddy@linux.ibm.com> Cc: Magnus Lindholm <linmag7@gmail.com> Cc: Matt Turner <mattst88@gmail.com> Cc: Michael Ellerman <mpe@ellerman.id.au> Cc: Nicholas Piggin <npiggin@gmail.com> Cc: Palmer Dabbelt <palmer@dabbelt.com> Cc: Richard Henderson <richard.henderson@linaro.org> Cc: Richard Weinberger <richard@nod.at> Cc: Russell King <linux@armlinux.org.uk> Cc: Song Liu <song@kernel.org> Cc: Sven Schnelle <svens@linux.ibm.com> Cc: Ted Ts'o <tytso@mit.edu> Cc: Vasily Gorbik <gor@linux.ibm.com> Cc: WANG Xuerui <kernel@xen0n.name> Cc: Will Deacon <will@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
show more ...
|