[PATCH v2 00/19] crypto: x86 - fix RCU stalls

From: Robert Elliott
Date: Wed Oct 12 2022 - 18:00:05 EST


This series fixes the RCU stalls triggered by the x86 crypto
modules discussed in
https://lore.kernel.org/all/MW5PR84MB18426EBBA3303770A8BC0BDFAB759@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/

Two root causes were:
- too much data processed between kernel_fpu_begin and
kernel_fpu_end calls (which are heavily used by the x86
optimized drivers)
- tcrypt not calling cond_resched during speed test loops

These problems have always been lurking, but improving the
loading of the x86/sha512 module led to it happening a lot
during boot when using SHA-512 for module signature checking.

Fixing these problems makes it safer to improve loading
the rest of the x86 modules like the sha512 module.

This series only handles the x86 modules.

Testing
=======
The most effective testing was by enabling
CONFIG_CRYPTO_MANAGER_EXTRA_TESTS=y

which creates random test vectors and compares the results
of the CPU-optimized function to the generic function,
and running two threads of repeated modprobe commands
to exercise those tests:
watch -n 0 modprobe tcrypt mode=200
watch -n 0 ./tcrypt_sweep

where tcrypt_sweep walks through all the test modes:
#!/usr/bin/perl
use strict;

my @modes;

open SOURCE, "<", "/home/me/linux/crypto/tcrypt.c" or die $!;
while (<SOURCE>) {
if (/^\s+case ([0-9]+):$/) {
push @modes, $1;
}
}
close SOURCE;

foreach (@modes) {
print "$_ ";

# don't run mode 300, which runs 301-399
# don't run mode 400, which runs 401-499
if (($_ eq "0") || ($_ eq "300") || ($_ eq "400")) {
system "echo \"===== Skipping special modprobe tcrypt mode=$_\" > /dev/kmsg";
} else {
system "echo \"Running modprobe tcrypt mode=$_\" > /dev/kmsg";
system "modprobe tcrypt mode=$_";
}
}



Robert Elliott (19):
crypto: tcrypt - test crc32
crypto: tcrypt - test nhpoly1305
crypto: tcrypt - reschedule during cycles speed tests
crypto: x86/sha - limit FPU preemption
crypto: x86/crc - limit FPU preemption
crypto: x86/sm3 - limit FPU preemption
crypto: x86/ghash - restructure FPU context saving
crypto: x86/ghash - limit FPU preemption
crypto: x86 - use common macro for FPU limit
crypto: x86/sha1, sha256 - load based on CPU features
crypto: x86/crc - load based on CPU features
crypto: x86/sm3 - load based on CPU features
crypto: x86/ghash - load based on CPU features
crypto: x86 - load based on CPU features
crypto: x86 - add pr_fmt to all modules
crypto: x86 - print CPU optimized loaded messages
crypto: x86 - standardize suboptimal prints
crypto: x86 - standardize not loaded prints
crypto: x86/sha - register only the best function

arch/x86/crypto/aegis128-aesni-glue.c | 21 ++-
arch/x86/crypto/aesni-intel_glue.c | 31 ++--
arch/x86/crypto/aria_aesni_avx_glue.c | 19 +-
arch/x86/crypto/blake2s-glue.c | 34 +++-
arch/x86/crypto/blowfish_glue.c | 19 +-
arch/x86/crypto/camellia_aesni_avx2_glue.c | 25 ++-
arch/x86/crypto/camellia_aesni_avx_glue.c | 24 ++-
arch/x86/crypto/camellia_glue.c | 20 ++-
arch/x86/crypto/cast5_avx_glue.c | 21 ++-
arch/x86/crypto/cast6_avx_glue.c | 21 ++-
arch/x86/crypto/chacha_glue.c | 35 +++-
arch/x86/crypto/crc32-pclmul_asm.S | 6 +-
arch/x86/crypto/crc32-pclmul_glue.c | 37 ++--
arch/x86/crypto/crc32c-intel_glue.c | 51 ++++--
arch/x86/crypto/crct10dif-pclmul_glue.c | 54 ++++--
arch/x86/crypto/curve25519-x86_64.c | 27 ++-
arch/x86/crypto/des3_ede_glue.c | 16 +-
arch/x86/crypto/ghash-clmulni-intel_glue.c | 40 +++--
arch/x86/crypto/nhpoly1305-avx2-glue.c | 27 ++-
arch/x86/crypto/nhpoly1305-sse2-glue.c | 23 ++-
arch/x86/crypto/poly1305_glue.c | 64 +++++--
arch/x86/crypto/polyval-clmulni_glue.c | 14 +-
arch/x86/crypto/serpent_avx2_glue.c | 25 ++-
arch/x86/crypto/serpent_avx_glue.c | 21 ++-
arch/x86/crypto/serpent_sse2_glue.c | 19 +-
arch/x86/crypto/sha1_ssse3_glue.c | 188 +++++++++++--------
arch/x86/crypto/sha256_ssse3_glue.c | 198 ++++++++++++---------
arch/x86/crypto/sha512_ssse3_glue.c | 154 +++++++++-------
arch/x86/crypto/sm3_avx_glue.c | 52 +++++-
arch/x86/crypto/sm4_aesni_avx2_glue.c | 25 ++-
arch/x86/crypto/sm4_aesni_avx_glue.c | 23 ++-
arch/x86/crypto/twofish_avx_glue.c | 25 ++-
arch/x86/crypto/twofish_glue.c | 19 +-
arch/x86/crypto/twofish_glue_3way.c | 26 ++-
crypto/tcrypt.c | 56 +++---
35 files changed, 1060 insertions(+), 400 deletions(-)

--
2.37.3