143cb5451SMauro Carvalho Chehab.. SPDX-License-Identifier: GPL-2.0 243cb5451SMauro Carvalho Chehab 343cb5451SMauro Carvalho Chehab========================== 443cb5451SMauro Carvalho ChehabRCU Torture Test Operation 543cb5451SMauro Carvalho Chehab========================== 643cb5451SMauro Carvalho Chehab 743cb5451SMauro Carvalho Chehab 843cb5451SMauro Carvalho ChehabCONFIG_RCU_TORTURE_TEST 943cb5451SMauro Carvalho Chehab======================= 1043cb5451SMauro Carvalho Chehab 1143cb5451SMauro Carvalho ChehabThe CONFIG_RCU_TORTURE_TEST config option is available for all RCU 1243cb5451SMauro Carvalho Chehabimplementations. It creates an rcutorture kernel module that can 1343cb5451SMauro Carvalho Chehabbe loaded to run a torture test. The test periodically outputs 1443cb5451SMauro Carvalho Chehabstatus messages via printk(), which can be examined via the dmesg 1543cb5451SMauro Carvalho Chehabcommand (perhaps grepping for "torture"). The test is started 1643cb5451SMauro Carvalho Chehabwhen the module is loaded, and stops when the module is unloaded. 1743cb5451SMauro Carvalho Chehab 1843cb5451SMauro Carvalho ChehabModule parameters are prefixed by "rcutorture." in 1943cb5451SMauro Carvalho ChehabDocumentation/admin-guide/kernel-parameters.txt. 2043cb5451SMauro Carvalho Chehab 2143cb5451SMauro Carvalho ChehabOutput 2243cb5451SMauro Carvalho Chehab====== 2343cb5451SMauro Carvalho Chehab 2443cb5451SMauro Carvalho ChehabThe statistics output is as follows:: 2543cb5451SMauro Carvalho Chehab 2643cb5451SMauro Carvalho Chehab rcu-torture:--- Start of test: nreaders=16 nfakewriters=4 stat_interval=30 verbose=0 test_no_idle_hz=1 shuffle_interval=3 stutter=5 irqreader=1 fqs_duration=0 fqs_holdoff=0 fqs_stutter=3 test_boost=1/0 test_boost_interval=7 test_boost_duration=4 2743cb5451SMauro Carvalho Chehab rcu-torture: rtc: (null) ver: 155441 tfle: 0 rta: 155441 rtaf: 8884 rtf: 155440 rtmbe: 0 rtbe: 0 rtbke: 0 rtbre: 0 rtbf: 0 rtb: 0 nt: 3055767 2843cb5451SMauro Carvalho Chehab rcu-torture: Reader Pipe: 727860534 34213 0 0 0 0 0 0 0 0 0 2943cb5451SMauro Carvalho Chehab rcu-torture: Reader Batch: 727877838 17003 0 0 0 0 0 0 0 0 0 3043cb5451SMauro Carvalho Chehab rcu-torture: Free-Block Circulation: 155440 155440 155440 155440 155440 155440 155440 155440 155440 155440 0 3143cb5451SMauro Carvalho Chehab rcu-torture:--- End of test: SUCCESS: nreaders=16 nfakewriters=4 stat_interval=30 verbose=0 test_no_idle_hz=1 shuffle_interval=3 stutter=5 irqreader=1 fqs_duration=0 fqs_holdoff=0 fqs_stutter=3 test_boost=1/0 test_boost_interval=7 test_boost_duration=4 3243cb5451SMauro Carvalho Chehab 3343cb5451SMauro Carvalho ChehabThe command "dmesg | grep torture:" will extract this information on 3443cb5451SMauro Carvalho Chehabmost systems. On more esoteric configurations, it may be necessary to 3543cb5451SMauro Carvalho Chehabuse other commands to access the output of the printk()s used by 3643cb5451SMauro Carvalho Chehabthe RCU torture test. The printk()s use KERN_ALERT, so they should 3743cb5451SMauro Carvalho Chehabbe evident. ;-) 3843cb5451SMauro Carvalho Chehab 3943cb5451SMauro Carvalho ChehabThe first and last lines show the rcutorture module parameters, and the 4043cb5451SMauro Carvalho Chehablast line shows either "SUCCESS" or "FAILURE", based on rcutorture's 4143cb5451SMauro Carvalho Chehabautomatic determination as to whether RCU operated correctly. 4243cb5451SMauro Carvalho Chehab 4343cb5451SMauro Carvalho ChehabThe entries are as follows: 4443cb5451SMauro Carvalho Chehab 4543cb5451SMauro Carvalho Chehab* "rtc": The hexadecimal address of the structure currently visible 4643cb5451SMauro Carvalho Chehab to readers. 4743cb5451SMauro Carvalho Chehab 4843cb5451SMauro Carvalho Chehab* "ver": The number of times since boot that the RCU writer task 4943cb5451SMauro Carvalho Chehab has changed the structure visible to readers. 5043cb5451SMauro Carvalho Chehab 5143cb5451SMauro Carvalho Chehab* "tfle": If non-zero, indicates that the "torture freelist" 5243cb5451SMauro Carvalho Chehab containing structures to be placed into the "rtc" area is empty. 5343cb5451SMauro Carvalho Chehab This condition is important, since it can fool you into thinking 5443cb5451SMauro Carvalho Chehab that RCU is working when it is not. :-/ 5543cb5451SMauro Carvalho Chehab 5643cb5451SMauro Carvalho Chehab* "rta": Number of structures allocated from the torture freelist. 5743cb5451SMauro Carvalho Chehab 5843cb5451SMauro Carvalho Chehab* "rtaf": Number of allocations from the torture freelist that have 5943cb5451SMauro Carvalho Chehab failed due to the list being empty. It is not unusual for this 6043cb5451SMauro Carvalho Chehab to be non-zero, but it is bad for it to be a large fraction of 6143cb5451SMauro Carvalho Chehab the value indicated by "rta". 6243cb5451SMauro Carvalho Chehab 6343cb5451SMauro Carvalho Chehab* "rtf": Number of frees into the torture freelist. 6443cb5451SMauro Carvalho Chehab 6543cb5451SMauro Carvalho Chehab* "rtmbe": A non-zero value indicates that rcutorture believes that 6643cb5451SMauro Carvalho Chehab rcu_assign_pointer() and rcu_dereference() are not working 6743cb5451SMauro Carvalho Chehab correctly. This value should be zero. 6843cb5451SMauro Carvalho Chehab 6943cb5451SMauro Carvalho Chehab* "rtbe": A non-zero value indicates that one of the rcu_barrier() 7043cb5451SMauro Carvalho Chehab family of functions is not working correctly. 7143cb5451SMauro Carvalho Chehab 7243cb5451SMauro Carvalho Chehab* "rtbke": rcutorture was unable to create the real-time kthreads 7343cb5451SMauro Carvalho Chehab used to force RCU priority inversion. This value should be zero. 7443cb5451SMauro Carvalho Chehab 7543cb5451SMauro Carvalho Chehab* "rtbre": Although rcutorture successfully created the kthreads 7643cb5451SMauro Carvalho Chehab used to force RCU priority inversion, it was unable to set them 7743cb5451SMauro Carvalho Chehab to the real-time priority level of 1. This value should be zero. 7843cb5451SMauro Carvalho Chehab 7943cb5451SMauro Carvalho Chehab* "rtbf": The number of times that RCU priority boosting failed 8043cb5451SMauro Carvalho Chehab to resolve RCU priority inversion. 8143cb5451SMauro Carvalho Chehab 8243cb5451SMauro Carvalho Chehab* "rtb": The number of times that rcutorture attempted to force 8343cb5451SMauro Carvalho Chehab an RCU priority inversion condition. If you are testing RCU 8443cb5451SMauro Carvalho Chehab priority boosting via the "test_boost" module parameter, this 8543cb5451SMauro Carvalho Chehab value should be non-zero. 8643cb5451SMauro Carvalho Chehab 8743cb5451SMauro Carvalho Chehab* "nt": The number of times rcutorture ran RCU read-side code from 8843cb5451SMauro Carvalho Chehab within a timer handler. This value should be non-zero only 8943cb5451SMauro Carvalho Chehab if you specified the "irqreader" module parameter. 9043cb5451SMauro Carvalho Chehab 9143cb5451SMauro Carvalho Chehab* "Reader Pipe": Histogram of "ages" of structures seen by readers. 9243cb5451SMauro Carvalho Chehab If any entries past the first two are non-zero, RCU is broken. 9343cb5451SMauro Carvalho Chehab And rcutorture prints the error flag string "!!!" to make sure 9443cb5451SMauro Carvalho Chehab you notice. The age of a newly allocated structure is zero, 9543cb5451SMauro Carvalho Chehab it becomes one when removed from reader visibility, and is 9643cb5451SMauro Carvalho Chehab incremented once per grace period subsequently -- and is freed 9743cb5451SMauro Carvalho Chehab after passing through (RCU_TORTURE_PIPE_LEN-2) grace periods. 9843cb5451SMauro Carvalho Chehab 9943cb5451SMauro Carvalho Chehab The output displayed above was taken from a correctly working 10043cb5451SMauro Carvalho Chehab RCU. If you want to see what it looks like when broken, break 10143cb5451SMauro Carvalho Chehab it yourself. ;-) 10243cb5451SMauro Carvalho Chehab 10343cb5451SMauro Carvalho Chehab* "Reader Batch": Another histogram of "ages" of structures seen 10443cb5451SMauro Carvalho Chehab by readers, but in terms of counter flips (or batches) rather 10543cb5451SMauro Carvalho Chehab than in terms of grace periods. The legal number of non-zero 10643cb5451SMauro Carvalho Chehab entries is again two. The reason for this separate view is that 10743cb5451SMauro Carvalho Chehab it is sometimes easier to get the third entry to show up in the 10843cb5451SMauro Carvalho Chehab "Reader Batch" list than in the "Reader Pipe" list. 10943cb5451SMauro Carvalho Chehab 11043cb5451SMauro Carvalho Chehab* "Free-Block Circulation": Shows the number of torture structures 11143cb5451SMauro Carvalho Chehab that have reached a given point in the pipeline. The first element 11243cb5451SMauro Carvalho Chehab should closely correspond to the number of structures allocated, 11343cb5451SMauro Carvalho Chehab the second to the number that have been removed from reader view, 11443cb5451SMauro Carvalho Chehab and all but the last remaining to the corresponding number of 11543cb5451SMauro Carvalho Chehab passes through a grace period. The last entry should be zero, 11643cb5451SMauro Carvalho Chehab as it is only incremented if a torture structure's counter 11743cb5451SMauro Carvalho Chehab somehow gets incremented farther than it should. 11843cb5451SMauro Carvalho Chehab 11943cb5451SMauro Carvalho ChehabDifferent implementations of RCU can provide implementation-specific 12043cb5451SMauro Carvalho Chehabadditional information. For example, Tree SRCU provides the following 12143cb5451SMauro Carvalho Chehabadditional line:: 12243cb5451SMauro Carvalho Chehab 12343cb5451SMauro Carvalho Chehab srcud-torture: Tree SRCU per-CPU(idx=0): 0(35,-21) 1(-4,24) 2(1,1) 3(-26,20) 4(28,-47) 5(-9,4) 6(-10,14) 7(-14,11) T(1,6) 12443cb5451SMauro Carvalho Chehab 12543cb5451SMauro Carvalho ChehabThis line shows the per-CPU counter state, in this case for Tree SRCU 12643cb5451SMauro Carvalho Chehabusing a dynamically allocated srcu_struct (hence "srcud-" rather than 12743cb5451SMauro Carvalho Chehab"srcu-"). The numbers in parentheses are the values of the "old" and 12843cb5451SMauro Carvalho Chehab"current" counters for the corresponding CPU. The "idx" value maps the 12943cb5451SMauro Carvalho Chehab"old" and "current" values to the underlying array, and is useful for 13043cb5451SMauro Carvalho Chehabdebugging. The final "T" entry contains the totals of the counters. 13143cb5451SMauro Carvalho Chehab 13243cb5451SMauro Carvalho ChehabUsage on Specific Kernel Builds 13343cb5451SMauro Carvalho Chehab=============================== 13443cb5451SMauro Carvalho Chehab 13543cb5451SMauro Carvalho ChehabIt is sometimes desirable to torture RCU on a specific kernel build, 13643cb5451SMauro Carvalho Chehabfor example, when preparing to put that kernel build into production. 13743cb5451SMauro Carvalho ChehabIn that case, the kernel should be built with CONFIG_RCU_TORTURE_TEST=m 13843cb5451SMauro Carvalho Chehabso that the test can be started using modprobe and terminated using rmmod. 13943cb5451SMauro Carvalho Chehab 14043cb5451SMauro Carvalho ChehabFor example, the following script may be used to torture RCU:: 14143cb5451SMauro Carvalho Chehab 14243cb5451SMauro Carvalho Chehab #!/bin/sh 14343cb5451SMauro Carvalho Chehab 14443cb5451SMauro Carvalho Chehab modprobe rcutorture 14543cb5451SMauro Carvalho Chehab sleep 3600 14643cb5451SMauro Carvalho Chehab rmmod rcutorture 14743cb5451SMauro Carvalho Chehab dmesg | grep torture: 14843cb5451SMauro Carvalho Chehab 14943cb5451SMauro Carvalho ChehabThe output can be manually inspected for the error flag of "!!!". 15043cb5451SMauro Carvalho ChehabOne could of course create a more elaborate script that automatically 15143cb5451SMauro Carvalho Chehabchecked for such errors. The "rmmod" command forces a "SUCCESS", 15243cb5451SMauro Carvalho Chehab"FAILURE", or "RCU_HOTPLUG" indication to be printk()ed. The first 15343cb5451SMauro Carvalho Chehabtwo are self-explanatory, while the last indicates that while there 15443cb5451SMauro Carvalho Chehabwere no RCU failures, CPU-hotplug problems were detected. 15543cb5451SMauro Carvalho Chehab 15643cb5451SMauro Carvalho Chehab 15743cb5451SMauro Carvalho ChehabUsage on Mainline Kernels 15843cb5451SMauro Carvalho Chehab========================= 15943cb5451SMauro Carvalho Chehab 16043cb5451SMauro Carvalho ChehabWhen using rcutorture to test changes to RCU itself, it is often 16143cb5451SMauro Carvalho Chehabnecessary to build a number of kernels in order to test that change 16243cb5451SMauro Carvalho Chehabacross a broad range of combinations of the relevant Kconfig options 16343cb5451SMauro Carvalho Chehaband of the relevant kernel boot parameters. In this situation, use 16443cb5451SMauro Carvalho Chehabof modprobe and rmmod can be quite time-consuming and error-prone. 16543cb5451SMauro Carvalho Chehab 16643cb5451SMauro Carvalho ChehabTherefore, the tools/testing/selftests/rcutorture/bin/kvm.sh 16743cb5451SMauro Carvalho Chehabscript is available for mainline testing for x86, arm64, and 16843cb5451SMauro Carvalho Chehabpowerpc. By default, it will run the series of tests specified by 16943cb5451SMauro Carvalho Chehabtools/testing/selftests/rcutorture/configs/rcu/CFLIST, with each test 17043cb5451SMauro Carvalho Chehabrunning for 30 minutes within a guest OS using a minimal userspace 17143cb5451SMauro Carvalho Chehabsupplied by an automatically generated initrd. After the tests are 17243cb5451SMauro Carvalho Chehabcomplete, the resulting build products and console output are analyzed 17343cb5451SMauro Carvalho Chehabfor errors and the results of the runs are summarized. 17443cb5451SMauro Carvalho Chehab 17543cb5451SMauro Carvalho ChehabOn larger systems, rcutorture testing can be accelerated by passing the 17643cb5451SMauro Carvalho Chehab--cpus argument to kvm.sh. For example, on a 64-CPU system, "--cpus 43" 17743cb5451SMauro Carvalho Chehabwould use up to 43 CPUs to run tests concurrently, which as of v5.4 would 17843cb5451SMauro Carvalho Chehabcomplete all the scenarios in two batches, reducing the time to complete 17943cb5451SMauro Carvalho Chehabfrom about eight hours to about one hour (not counting the time to build 18043cb5451SMauro Carvalho Chehabthe sixteen kernels). The "--dryrun sched" argument will not run tests, 18143cb5451SMauro Carvalho Chehabbut rather tell you how the tests would be scheduled into batches. This 18243cb5451SMauro Carvalho Chehabcan be useful when working out how many CPUs to specify in the --cpus 18343cb5451SMauro Carvalho Chehabargument. 18443cb5451SMauro Carvalho Chehab 18543cb5451SMauro Carvalho ChehabNot all changes require that all scenarios be run. For example, a change 18643cb5451SMauro Carvalho Chehabto Tree SRCU might run only the SRCU-N and SRCU-P scenarios using the 18743cb5451SMauro Carvalho Chehab--configs argument to kvm.sh as follows: "--configs 'SRCU-N SRCU-P'". 188fb91e42fSCharles HanLarge systems can run multiple copies of the full set of scenarios, 18943cb5451SMauro Carvalho Chehabfor example, a system with 448 hardware threads can run five instances 19043cb5451SMauro Carvalho Chehabof the full set concurrently. To make this happen:: 19143cb5451SMauro Carvalho Chehab 19243cb5451SMauro Carvalho Chehab kvm.sh --cpus 448 --configs '5*CFLIST' 19343cb5451SMauro Carvalho Chehab 19443cb5451SMauro Carvalho ChehabAlternatively, such a system can run 56 concurrent instances of a single 19543cb5451SMauro Carvalho Chehabeight-CPU scenario:: 19643cb5451SMauro Carvalho Chehab 19743cb5451SMauro Carvalho Chehab kvm.sh --cpus 448 --configs '56*TREE04' 19843cb5451SMauro Carvalho Chehab 19943cb5451SMauro Carvalho ChehabOr 28 concurrent instances of each of two eight-CPU scenarios:: 20043cb5451SMauro Carvalho Chehab 20143cb5451SMauro Carvalho Chehab kvm.sh --cpus 448 --configs '28*TREE03 28*TREE04' 20243cb5451SMauro Carvalho Chehab 20343cb5451SMauro Carvalho ChehabOf course, each concurrent instance will use memory, which can be 20443cb5451SMauro Carvalho Chehablimited using the --memory argument, which defaults to 512M. Small 20543cb5451SMauro Carvalho Chehabvalues for memory may require disabling the callback-flooding tests 20643cb5451SMauro Carvalho Chehabusing the --bootargs parameter discussed below. 20743cb5451SMauro Carvalho Chehab 20843cb5451SMauro Carvalho ChehabSometimes additional debugging is useful, and in such cases the --kconfig 2090c208a79SPaul E. McKenneyparameter to kvm.sh may be used, for example, ``--kconfig 'CONFIG_RCU_EQS_DEBUG=y'``. 2100c208a79SPaul E. McKenneyIn addition, there are the --gdb, --kasan, and --kcsan parameters. 2110c208a79SPaul E. McKenneyNote that --gdb limits you to one scenario per kvm.sh run and requires 2120c208a79SPaul E. McKenneythat you have another window open from which to run ``gdb`` as instructed 2130c208a79SPaul E. McKenneyby the script. 21443cb5451SMauro Carvalho Chehab 21543cb5451SMauro Carvalho ChehabKernel boot arguments can also be supplied, for example, to control 21643cb5451SMauro Carvalho Chehabrcutorture's module parameters. For example, to test a change to RCU's 21743cb5451SMauro Carvalho ChehabCPU stall-warning code, use "--bootargs 'rcutorture.stall_cpu=30'". 21843cb5451SMauro Carvalho ChehabThis will of course result in the scripting reporting a failure, namely 219c4af9e00SRandy Dunlapthe resulting RCU CPU stall warning. As noted above, reducing memory may 22043cb5451SMauro Carvalho Chehabrequire disabling rcutorture's callback-flooding tests:: 22143cb5451SMauro Carvalho Chehab 22243cb5451SMauro Carvalho Chehab kvm.sh --cpus 448 --configs '56*TREE04' --memory 128M \ 22343cb5451SMauro Carvalho Chehab --bootargs 'rcutorture.fwd_progress=0' 22443cb5451SMauro Carvalho Chehab 22543cb5451SMauro Carvalho ChehabSometimes all that is needed is a full set of kernel builds. This is 2260c208a79SPaul E. McKenneywhat the --buildonly parameter does. 22743cb5451SMauro Carvalho Chehab 2280c208a79SPaul E. McKenneyThe --duration parameter can override the default run time of 30 minutes. 2290c208a79SPaul E. McKenneyFor example, ``--duration 2d`` would run for two days, ``--duration 3h`` 2300c208a79SPaul E. McKenneywould run for three hours, ``--duration 5m`` would run for five minutes, 2310c208a79SPaul E. McKenneyand ``--duration 45s`` would run for 45 seconds. This last can be useful 2320c208a79SPaul E. McKenneyfor tracking down rare boot-time failures. 2330c208a79SPaul E. McKenney 2340c208a79SPaul E. McKenneyFinally, the --trust-make parameter allows each kernel build to reuse what 2350c208a79SPaul E. McKenneyit can from the previous kernel build. Please note that without the 2360c208a79SPaul E. McKenney--trust-make parameter, your tags files may be demolished. 23743cb5451SMauro Carvalho Chehab 23843cb5451SMauro Carvalho ChehabThere are additional more arcane arguments that are documented in the 23943cb5451SMauro Carvalho Chehabsource code of the kvm.sh script. 24043cb5451SMauro Carvalho Chehab 24143cb5451SMauro Carvalho ChehabIf a run contains failures, the number of buildtime and runtime failures 24243cb5451SMauro Carvalho Chehabis listed at the end of the kvm.sh output, which you really should redirect 24343cb5451SMauro Carvalho Chehabto a file. The build products and console output of each run is kept in 24443cb5451SMauro Carvalho Chehabtools/testing/selftests/rcutorture/res in timestamped directories. A 24543cb5451SMauro Carvalho Chehabgiven directory can be supplied to kvm-find-errors.sh in order to have 24643cb5451SMauro Carvalho Chehabit cycle you through summaries of errors and full error logs. For example:: 24743cb5451SMauro Carvalho Chehab 24843cb5451SMauro Carvalho Chehab tools/testing/selftests/rcutorture/bin/kvm-find-errors.sh \ 24943cb5451SMauro Carvalho Chehab tools/testing/selftests/rcutorture/res/2020.01.20-15.54.23 25043cb5451SMauro Carvalho Chehab 25143cb5451SMauro Carvalho ChehabHowever, it is often more convenient to access the files directly. 25243cb5451SMauro Carvalho ChehabFiles pertaining to all scenarios in a run reside in the top-level 25343cb5451SMauro Carvalho Chehabdirectory (2020.01.20-15.54.23 in the example above), while per-scenario 25443cb5451SMauro Carvalho Chehabfiles reside in a subdirectory named after the scenario (for example, 25543cb5451SMauro Carvalho Chehab"TREE04"). If a given scenario ran more than once (as in "--configs 25643cb5451SMauro Carvalho Chehab'56*TREE04'" above), the directories corresponding to the second and 25743cb5451SMauro Carvalho Chehabsubsequent runs of that scenario include a sequence number, for example, 25843cb5451SMauro Carvalho Chehab"TREE04.2", "TREE04.3", and so on. 25943cb5451SMauro Carvalho Chehab 26043cb5451SMauro Carvalho ChehabThe most frequently used file in the top-level directory is testid.txt. 26143cb5451SMauro Carvalho ChehabIf the test ran in a git repository, then this file contains the commit 26243cb5451SMauro Carvalho Chehabthat was tested and any uncommitted changes in diff format. 26343cb5451SMauro Carvalho Chehab 26443cb5451SMauro Carvalho ChehabThe most frequently used files in each per-scenario-run directory are: 26543cb5451SMauro Carvalho Chehab 26643cb5451SMauro Carvalho Chehab.config: 26743cb5451SMauro Carvalho Chehab This file contains the Kconfig options. 26843cb5451SMauro Carvalho Chehab 26943cb5451SMauro Carvalho ChehabMake.out: 27043cb5451SMauro Carvalho Chehab This contains build output for a specific scenario. 27143cb5451SMauro Carvalho Chehab 27243cb5451SMauro Carvalho Chehabconsole.log: 27343cb5451SMauro Carvalho Chehab This contains the console output for a specific scenario. 27443cb5451SMauro Carvalho Chehab This file may be examined once the kernel has booted, but 27543cb5451SMauro Carvalho Chehab it might not exist if the build failed. 27643cb5451SMauro Carvalho Chehab 27743cb5451SMauro Carvalho Chehabvmlinux: 27843cb5451SMauro Carvalho Chehab This contains the kernel, which can be useful with tools like 27943cb5451SMauro Carvalho Chehab objdump and gdb. 28043cb5451SMauro Carvalho Chehab 28143cb5451SMauro Carvalho ChehabA number of additional files are available, but are less frequently used. 28243cb5451SMauro Carvalho ChehabMany are intended for debugging of rcutorture itself or of its scripting. 28343cb5451SMauro Carvalho Chehab 28443cb5451SMauro Carvalho ChehabAs of v5.4, a successful run with the default set of scenarios produces 28543cb5451SMauro Carvalho Chehabthe following summary at the end of the run on a 12-CPU system:: 28643cb5451SMauro Carvalho Chehab 28743cb5451SMauro Carvalho Chehab SRCU-N ------- 804233 GPs (148.932/s) [srcu: g10008272 f0x0 ] 28843cb5451SMauro Carvalho Chehab SRCU-P ------- 202320 GPs (37.4667/s) [srcud: g1809476 f0x0 ] 28943cb5451SMauro Carvalho Chehab SRCU-t ------- 1122086 GPs (207.794/s) [srcu: g0 f0x0 ] 29043cb5451SMauro Carvalho Chehab SRCU-u ------- 1111285 GPs (205.794/s) [srcud: g1 f0x0 ] 29143cb5451SMauro Carvalho Chehab TASKS01 ------- 19666 GPs (3.64185/s) [tasks: g0 f0x0 ] 29243cb5451SMauro Carvalho Chehab TASKS02 ------- 20541 GPs (3.80389/s) [tasks: g0 f0x0 ] 29343cb5451SMauro Carvalho Chehab TASKS03 ------- 19416 GPs (3.59556/s) [tasks: g0 f0x0 ] 29443cb5451SMauro Carvalho Chehab TINY01 ------- 836134 GPs (154.84/s) [rcu: g0 f0x0 ] n_max_cbs: 34198 29543cb5451SMauro Carvalho Chehab TINY02 ------- 850371 GPs (157.476/s) [rcu: g0 f0x0 ] n_max_cbs: 2631 29643cb5451SMauro Carvalho Chehab TREE01 ------- 162625 GPs (30.1157/s) [rcu: g1124169 f0x0 ] 29743cb5451SMauro Carvalho Chehab TREE02 ------- 333003 GPs (61.6672/s) [rcu: g2647753 f0x0 ] n_max_cbs: 35844 29843cb5451SMauro Carvalho Chehab TREE03 ------- 306623 GPs (56.782/s) [rcu: g2975325 f0x0 ] n_max_cbs: 1496497 29943cb5451SMauro Carvalho Chehab CPU count limited from 16 to 12 30043cb5451SMauro Carvalho Chehab TREE04 ------- 246149 GPs (45.5831/s) [rcu: g1695737 f0x0 ] n_max_cbs: 434961 30143cb5451SMauro Carvalho Chehab TREE05 ------- 314603 GPs (58.2598/s) [rcu: g2257741 f0x2 ] n_max_cbs: 193997 30243cb5451SMauro Carvalho Chehab TREE07 ------- 167347 GPs (30.9902/s) [rcu: g1079021 f0x0 ] n_max_cbs: 478732 30343cb5451SMauro Carvalho Chehab CPU count limited from 16 to 12 30443cb5451SMauro Carvalho Chehab TREE09 ------- 752238 GPs (139.303/s) [rcu: g13075057 f0x0 ] n_max_cbs: 99011 3050c208a79SPaul E. McKenney 3060c208a79SPaul E. McKenney 3070c208a79SPaul E. McKenneyRepeated Runs 3080c208a79SPaul E. McKenney============= 3090c208a79SPaul E. McKenney 3100c208a79SPaul E. McKenneySuppose that you are chasing down a rare boot-time failure. Although you 3110c208a79SPaul E. McKenneycould use kvm.sh, doing so will rebuild the kernel on each run. If you 3120c208a79SPaul E. McKenneyneed (say) 1,000 runs to have confidence that you have fixed the bug, 3130c208a79SPaul E. McKenneythese pointless rebuilds can become extremely annoying. 3140c208a79SPaul E. McKenney 3150c208a79SPaul E. McKenneyThis is why kvm-again.sh exists. 3160c208a79SPaul E. McKenney 3170c208a79SPaul E. McKenneySuppose that a previous kvm.sh run left its output in this directory:: 3180c208a79SPaul E. McKenney 3190c208a79SPaul E. McKenney tools/testing/selftests/rcutorture/res/2022.11.03-11.26.28 3200c208a79SPaul E. McKenney 321*6151b9c8SFederico VagaThen this run can be re-run without rebuilding as follow:: 3220c208a79SPaul E. McKenney 3230c208a79SPaul E. McKenney kvm-again.sh tools/testing/selftests/rcutorture/res/2022.11.03-11.26.28 3240c208a79SPaul E. McKenney 3250c208a79SPaul E. McKenneyA few of the original run's kvm.sh parameters may be overridden, perhaps 3260c208a79SPaul E. McKenneymost notably --duration and --bootargs. For example:: 3270c208a79SPaul E. McKenney 3280c208a79SPaul E. McKenney kvm-again.sh tools/testing/selftests/rcutorture/res/2022.11.03-11.26.28 \ 3290c208a79SPaul E. McKenney --duration 45s 3300c208a79SPaul E. McKenney 3310c208a79SPaul E. McKenneywould re-run the previous test, but for only 45 seconds, thus facilitating 3320c208a79SPaul E. McKenneytracking down the aforementioned rare boot-time failure. 3330c208a79SPaul E. McKenney 3340c208a79SPaul E. McKenney 3350c208a79SPaul E. McKenneyDistributed Runs 3360c208a79SPaul E. McKenney================ 3370c208a79SPaul E. McKenney 3380c208a79SPaul E. McKenneyAlthough kvm.sh is quite useful, its testing is confined to a single 3390c208a79SPaul E. McKenneysystem. It is not all that hard to use your favorite framework to cause 3400c208a79SPaul E. McKenney(say) 5 instances of kvm.sh to run on your 5 systems, but this will very 3410c208a79SPaul E. McKenneylikely unnecessarily rebuild kernels. In addition, manually distributing 3420c208a79SPaul E. McKenneythe desired rcutorture scenarios across the available systems can be 3430c208a79SPaul E. McKenneypainstaking and error-prone. 3440c208a79SPaul E. McKenney 3450c208a79SPaul E. McKenneyAnd this is why the kvm-remote.sh script exists. 3460c208a79SPaul E. McKenney 3470c208a79SPaul E. McKenneyIf you the following command works:: 3480c208a79SPaul E. McKenney 3490c208a79SPaul E. McKenney ssh system0 date 3500c208a79SPaul E. McKenney 3510c208a79SPaul E. McKenneyand if it also works for system1, system2, system3, system4, and system5, 3520c208a79SPaul E. McKenneyand all of these systems have 64 CPUs, you can type:: 3530c208a79SPaul E. McKenney 3540c208a79SPaul E. McKenney kvm-remote.sh "system0 system1 system2 system3 system4 system5" \ 3550c208a79SPaul E. McKenney --cpus 64 --duration 8h --configs "5*CFLIST" 3560c208a79SPaul E. McKenney 3570c208a79SPaul E. McKenneyThis will build each default scenario's kernel on the local system, then 3580c208a79SPaul E. McKenneyspread each of five instances of each scenario over the systems listed, 3590c208a79SPaul E. McKenneyrunning each scenario for eight hours. At the end of the runs, the 3600c208a79SPaul E. McKenneyresults will be gathered, recorded, and printed. Most of the parameters 3610c208a79SPaul E. McKenneythat kvm.sh will accept can be passed to kvm-remote.sh, but the list of 3620c208a79SPaul E. McKenneysystems must come first. 3630c208a79SPaul E. McKenney 3640c208a79SPaul E. McKenneyThe kvm.sh ``--dryrun scenarios`` argument is useful for working out 3650c208a79SPaul E. McKenneyhow many scenarios may be run in one batch across a group of systems. 3660c208a79SPaul E. McKenney 3670c208a79SPaul E. McKenneyYou can also re-run a previous remote run in a manner similar to kvm.sh: 3680c208a79SPaul E. McKenney 3690c208a79SPaul E. McKenney kvm-remote.sh "system0 system1 system2 system3 system4 system5" \ 3700c208a79SPaul E. McKenney tools/testing/selftests/rcutorture/res/2022.11.03-11.26.28-remote \ 3710c208a79SPaul E. McKenney --duration 24h 3720c208a79SPaul E. McKenney 373c4af9e00SRandy DunlapIn this case, most of the kvm-again.sh parameters may be supplied following 3740c208a79SPaul E. McKenneythe pathname of the old run-results directory. 375