1====================== 2Linux Kernel Selftests 3====================== 4 5The kernel contains a set of "self tests" under the tools/testing/selftests/ 6directory. These are intended to be small tests to exercise individual code 7paths in the kernel. Tests are intended to be run after building, installing 8and booting a kernel. 9 10Kselftest from mainline can be run on older stable kernels. Running tests 11from mainline offers the best coverage. Several test rings run mainline 12kselftest suite on stable releases. The reason is that when a new test 13gets added to test existing code to regression test a bug, we should be 14able to run that test on an older kernel. Hence, it is important to keep 15code that can still test an older kernel and make sure it skips the test 16gracefully on newer releases. 17 18You can find additional information on Kselftest framework, how to 19write new tests using the framework on Kselftest wiki: 20 21https://kselftest.wiki.kernel.org/ 22 23On some systems, hot-plug tests could hang forever waiting for cpu and 24memory to be ready to be offlined. A special hot-plug target is created 25to run the full range of hot-plug tests. In default mode, hot-plug tests run 26in safe mode with a limited scope. In limited mode, cpu-hotplug test is 27run on a single cpu as opposed to all hotplug capable cpus, and memory 28hotplug test is run on 2% of hotplug capable memory instead of 10%. 29 30kselftest runs as a userspace process. Tests that can be written/run in 31userspace may wish to use the `Test Harness`_. Tests that need to be 32run in kernel space may wish to use a `Test Module`_. 33 34Documentation on the tests 35========================== 36 37For documentation on the kselftests themselves, see: 38 39.. toctree:: 40 41 testing-devices 42 43Running the selftests (hotplug tests are run in limited mode) 44============================================================= 45 46To build the tests:: 47 48 $ make headers 49 $ make -C tools/testing/selftests 50 51To run the tests:: 52 53 $ make -C tools/testing/selftests run_tests 54 55To build and run the tests with a single command, use:: 56 57 $ make kselftest 58 59Note that some tests will require root privileges. 60 61Kselftest supports saving output files in a separate directory and then 62running tests. To locate output files in a separate directory two syntaxes 63are supported. In both cases the working directory must be the root of the 64kernel src. This is applicable to "Running a subset of selftests" section 65below. 66 67To build, save output files in a separate directory with O= :: 68 69 $ make O=/tmp/kselftest kselftest 70 71To build, save output files in a separate directory with KBUILD_OUTPUT :: 72 73 $ export KBUILD_OUTPUT=/tmp/kselftest; make kselftest 74 75The O= assignment takes precedence over the KBUILD_OUTPUT environment 76variable. 77 78The above commands by default run the tests and print full pass/fail report. 79Kselftest supports "summary" option to make it easier to understand the test 80results. Please find the detailed individual test results for each test in 81/tmp/testname file(s) when summary option is specified. This is applicable 82to "Running a subset of selftests" section below. 83 84To run kselftest with summary option enabled :: 85 86 $ make summary=1 kselftest 87 88Running a subset of selftests 89============================= 90 91You can use the "TARGETS" variable on the make command line to specify 92single test to run, or a list of tests to run. 93 94To run only tests targeted for a single subsystem:: 95 96 $ make -C tools/testing/selftests TARGETS=ptrace run_tests 97 98You can specify multiple tests to build and run:: 99 100 $ make TARGETS="size timers" kselftest 101 102To build, save output files in a separate directory with O= :: 103 104 $ make O=/tmp/kselftest TARGETS="size timers" kselftest 105 106To build, save output files in a separate directory with KBUILD_OUTPUT :: 107 108 $ export KBUILD_OUTPUT=/tmp/kselftest; make TARGETS="size timers" kselftest 109 110Additionally you can use the "SKIP_TARGETS" variable on the make command 111line to specify one or more targets to exclude from the TARGETS list. 112 113To run all tests but a single subsystem:: 114 115 $ make -C tools/testing/selftests SKIP_TARGETS=ptrace run_tests 116 117You can specify multiple tests to skip:: 118 119 $ make SKIP_TARGETS="size timers" kselftest 120 121You can also specify a restricted list of tests to run together with a 122dedicated skiplist:: 123 124 $ make TARGETS="breakpoints size timers" SKIP_TARGETS=size kselftest 125 126See the top-level tools/testing/selftests/Makefile for the list of all 127possible targets. 128 129Requiring all targets to build successfully 130=========================================== 131 132By default, the build succeeds as long as at least one target builds 133without error. Set ``FORCE_TARGETS=1`` to instead require every target to 134build successfully; make will abort as soon as any target fails:: 135 136 $ make -C tools/testing/selftests FORCE_TARGETS=1 137 138This applies to both the ``all`` and ``install`` targets and is useful in 139CI environments where a silent partial build would be misleading. 140 141Running the full range hotplug selftests 142======================================== 143 144To build the hotplug tests:: 145 146 $ make -C tools/testing/selftests hotplug 147 148To run the hotplug tests:: 149 150 $ make -C tools/testing/selftests run_hotplug 151 152Note that some tests will require root privileges. 153 154 155Install selftests 156================= 157 158You can use the "install" target of "make" (which calls the `kselftest_install.sh` 159tool) to install selftests in the default location (`tools/testing/selftests/kselftest_install`), 160or in a user specified location via the `INSTALL_PATH` "make" variable. 161 162To install selftests in default location:: 163 164 $ make -C tools/testing/selftests install 165 166To install selftests in a user specified location:: 167 168 $ make -C tools/testing/selftests install INSTALL_PATH=/some/other/path 169 170Running installed selftests 171=========================== 172 173Found in the install directory, as well as in the Kselftest tarball, 174is a script named `run_kselftest.sh` to run the tests. 175 176You can simply do the following to run the installed Kselftests. Please 177note some tests will require root privileges:: 178 179 $ cd kselftest_install 180 $ ./run_kselftest.sh 181 182To see the list of available tests, the `-l` option can be used:: 183 184 $ ./run_kselftest.sh -l 185 186The `-c` option can be used to run all the tests from a test collection, or 187the `-t` option for specific single tests. Either can be used multiple times:: 188 189 $ ./run_kselftest.sh -c size -c seccomp -t timers:posix_timers -t timer:nanosleep 190 191For other features see the script usage output, seen with the `-h` option. 192 193Timeout for selftests 194===================== 195 196Selftests are designed to be quick and so a default timeout is used of 45 197seconds for each test. Tests can override the default timeout by adding 198a settings file in their directory and set a timeout variable there to the 199configured a desired upper timeout for the test. Only a few tests override 200the timeout with a value higher than 45 seconds, selftests strives to keep 201it that way. Timeouts in selftests are not considered fatal because the 202system under which a test runs may change and this can also modify the 203expected time it takes to run a test. If you have control over the systems 204which will run the tests you can configure a test runner on those systems to 205use a greater or lower timeout on the command line as with the `-o` or 206the `--override-timeout` argument. For example to use 165 seconds instead 207one would use:: 208 209 $ ./run_kselftest.sh --override-timeout 165 210 211You can look at the TAP output to see if you ran into the timeout. Test 212runners which know a test must run under a specific time can then optionally 213treat these timeouts then as fatal. 214 215Packaging selftests 216=================== 217 218In some cases packaging is desired, such as when tests need to run on a 219different system. To package selftests, run:: 220 221 $ make -C tools/testing/selftests gen_tar 222 223This generates a tarball in the `INSTALL_PATH/kselftest-packages` directory. By 224default, `.gz` format is used. The tar compression format can be overridden by 225specifying a `FORMAT` make variable. Any value recognized by `tar's auto-compress`_ 226option is supported, such as:: 227 228 $ make -C tools/testing/selftests gen_tar FORMAT=.xz 229 230`make gen_tar` invokes `make install` so you can use it to package a subset of 231tests by using variables specified in `Running a subset of selftests`_ 232section:: 233 234 $ make -C tools/testing/selftests gen_tar TARGETS="size" FORMAT=.xz 235 236.. _tar's auto-compress: https://www.gnu.org/software/tar/manual/html_node/gzip.html#auto_002dcompress 237 238Contributing new tests 239====================== 240 241In general, the rules for selftests are 242 243 * Do as much as you can if you're not root; 244 245 * Don't take too long; 246 247 * Don't break the build on any architecture, and 248 249 * Don't cause the top-level "make run_tests" to fail if your feature is 250 unconfigured. 251 252 * The output of tests must conform to the TAP standard to ensure high 253 testing quality and to capture failures/errors with specific details. 254 The kselftest.h and kselftest_harness.h headers provide wrappers for 255 outputting test results. These wrappers should be used for pass, 256 fail, exit, and skip messages. CI systems can easily parse TAP output 257 messages to detect test results. 258 259Contributing new tests (details) 260================================ 261 262 * In your Makefile, use facilities from lib.mk by including it instead of 263 reinventing the wheel. Specify flags and binaries generation flags on 264 need basis before including lib.mk. :: 265 266 CFLAGS = $(KHDR_INCLUDES) 267 TEST_GEN_PROGS := close_range_test 268 include ../lib.mk 269 270 * Use TEST_GEN_XXX if such binaries or files are generated during 271 compiling. 272 273 TEST_PROGS, TEST_GEN_PROGS mean it is the executable tested by 274 default. 275 276 TEST_GEN_MODS_DIR should be used by tests that require modules to be built 277 before the test starts. The variable will contain the name of the directory 278 containing the modules. 279 280 TEST_CUSTOM_PROGS should be used by tests that require custom build 281 rules and prevent common build rule use. 282 283 TEST_PROGS are for test shell scripts. Please ensure shell script has 284 its exec bit set. Otherwise, lib.mk run_tests will generate a warning. 285 286 TEST_CUSTOM_PROGS and TEST_PROGS will be run by common run_tests. 287 288 TEST_PROGS_EXTENDED, TEST_GEN_PROGS_EXTENDED mean it is the 289 executable which is not tested by default. 290 291 TEST_FILES, TEST_GEN_FILES mean it is the file which is used by 292 test. 293 294 TEST_INCLUDES is similar to TEST_FILES, it lists files which should be 295 included when exporting or installing the tests, with the following 296 differences: 297 298 * symlinks to files in other directories are preserved 299 * the part of paths below tools/testing/selftests/ is preserved when 300 copying the files to the output directory 301 302 TEST_INCLUDES is meant to list dependencies located in other directories of 303 the selftests hierarchy. 304 305 * First use the headers inside the kernel source and/or git repo, and then the 306 system headers. Headers for the kernel release as opposed to headers 307 installed by the distro on the system should be the primary focus to be able 308 to find regressions. Use KHDR_INCLUDES in Makefile to include headers from 309 the kernel source. 310 311 * If a test needs specific kernel config options enabled, add a config file in 312 the test directory to enable them. 313 314 e.g: tools/testing/selftests/android/config 315 316 * Create a .gitignore file inside test directory and add all generated objects 317 in it. 318 319 * Add new test name in TARGETS in selftests/Makefile:: 320 321 TARGETS += android 322 323 * All changes should pass:: 324 325 kselftest-{all,install,clean,gen_tar} 326 kselftest-{all,install,clean,gen_tar} O=abo_path 327 kselftest-{all,install,clean,gen_tar} O=rel_path 328 make -C tools/testing/selftests {all,install,clean,gen_tar} 329 make -C tools/testing/selftests {all,install,clean,gen_tar} O=abs_path 330 make -C tools/testing/selftests {all,install,clean,gen_tar} O=rel_path 331 332Test Module 333=========== 334 335Kselftest tests the kernel from userspace. Sometimes things need 336testing from within the kernel, one method of doing this is to create a 337test module. We can tie the module into the kselftest framework by 338using a shell script test runner. ``kselftest/module.sh`` is designed 339to facilitate this process. There is also a header file provided to 340assist writing kernel modules that are for use with kselftest: 341 342- ``tools/testing/selftests/kselftest_module.h`` 343- ``tools/testing/selftests/kselftest/module.sh`` 344 345Note that test modules should taint the kernel with TAINT_TEST. This will 346happen automatically for modules which are in the ``tools/testing/`` 347directory, or for modules which use the ``kselftest_module.h`` header above. 348Otherwise, you'll need to add ``MODULE_INFO(test, "Y")`` to your module 349source. selftests which do not load modules typically should not taint the 350kernel, but in cases where a non-test module is loaded, TEST_TAINT can be 351applied from userspace by writing to ``/proc/sys/kernel/tainted``. 352 353How to use 354---------- 355 356Here we show the typical steps to create a test module and tie it into 357kselftest. We use kselftests for lib/ as an example. 358 3591. Create the test module 360 3612. Create the test script that will run (load/unload) the module 362 e.g. ``tools/testing/selftests/lib/bitmap.sh`` 363 3643. Add line to config file e.g. ``tools/testing/selftests/lib/config`` 365 3664. Add test script to makefile e.g. ``tools/testing/selftests/lib/Makefile`` 367 3685. Verify it works: 369 370.. code-block:: sh 371 372 # Assumes you have booted a fresh build of this kernel tree 373 cd /path/to/linux/tree 374 make kselftest-merge 375 make modules 376 sudo make modules_install 377 make TARGETS=lib kselftest 378 379Example Module 380-------------- 381 382A bare bones test module might look like this: 383 384.. code-block:: c 385 386 // SPDX-License-Identifier: GPL-2.0+ 387 388 #define pr_fmt(fmt) KBUILD_MODNAME ": " fmt 389 390 #include "../tools/testing/selftests/kselftest_module.h" 391 392 KSTM_MODULE_GLOBALS(); 393 394 /* 395 * Kernel module for testing the foobinator 396 */ 397 398 static int __init test_function() 399 { 400 ... 401 } 402 403 static void __init selftest(void) 404 { 405 KSTM_CHECK_ZERO(do_test_case("", 0)); 406 } 407 408 KSTM_MODULE_LOADERS(test_foo); 409 MODULE_AUTHOR("John Developer <jd@fooman.org>"); 410 MODULE_LICENSE("GPL"); 411 MODULE_INFO(test, "Y"); 412 413Example test script 414------------------- 415 416.. code-block:: sh 417 418 #!/bin/bash 419 # SPDX-License-Identifier: GPL-2.0+ 420 $(dirname $0)/../kselftest/module.sh "foo" test_foo 421 422 423Test Harness 424============ 425 426The kselftest_harness.h file contains useful helpers to build tests. The 427test harness is for userspace testing, for kernel space testing see `Test 428Module`_ above. 429 430The tests from tools/testing/selftests/seccomp/seccomp_bpf.c can be used as 431example. 432 433Example 434------- 435 436.. kernel-doc:: tools/testing/selftests/kselftest_harness.h 437 :doc: example 438 439 440Helpers 441------- 442 443.. kernel-doc:: tools/testing/selftests/kselftest_harness.h 444 :functions: TH_LOG TEST TEST_SIGNAL FIXTURE FIXTURE_DATA FIXTURE_SETUP 445 FIXTURE_TEARDOWN TEST_F TEST_HARNESS_MAIN FIXTURE_VARIANT 446 FIXTURE_VARIANT_ADD 447 448Operators 449--------- 450 451.. kernel-doc:: tools/testing/selftests/kselftest_harness.h 452 :doc: operators 453 454.. kernel-doc:: tools/testing/selftests/kselftest_harness.h 455 :functions: ASSERT_EQ ASSERT_NE ASSERT_LT ASSERT_LE ASSERT_GT ASSERT_GE 456 ASSERT_NULL ASSERT_TRUE ASSERT_NULL ASSERT_TRUE ASSERT_FALSE 457 ASSERT_STREQ ASSERT_STRNE EXPECT_EQ EXPECT_NE EXPECT_LT 458 EXPECT_LE EXPECT_GT EXPECT_GE EXPECT_NULL EXPECT_TRUE 459 EXPECT_FALSE EXPECT_STREQ EXPECT_STRNE 460