1====================== 2Linux Kernel Selftests 3====================== 4 5The kernel contains a set of "self tests" under the tools/testing/selftests/ 6directory. These are intended to be small tests to exercise individual code 7paths in the kernel. Tests are intended to be run after building, installing 8and booting a kernel. 9 10Kselftest from mainline can be run on older stable kernels. Running tests 11from mainline offers the best coverage. Several test rings run mainline 12kselftest suite on stable releases. The reason is that when a new test 13gets added to test existing code to regression test a bug, we should be 14able to run that test on an older kernel. Hence, it is important to keep 15code that can still test an older kernel and make sure it skips the test 16gracefully on newer releases. 17 18On some systems, hot-plug tests could hang forever waiting for cpu and 19memory to be ready to be offlined. A special hot-plug target is created 20to run the full range of hot-plug tests. In default mode, hot-plug tests run 21in safe mode with a limited scope. In limited mode, cpu-hotplug test is 22run on a single cpu as opposed to all hotplug capable cpus, and memory 23hotplug test is run on 2% of hotplug capable memory instead of 10%. 24 25kselftest runs as a userspace process. Tests that can be written/run in 26userspace may wish to use the `Test Harness`_. Tests that need to be 27run in kernel space may wish to use a `Test Module`_. 28 29Documentation on the tests 30========================== 31 32For documentation on the kselftests themselves, see: 33 34.. toctree:: 35 36 testing-devices 37 38Running the selftests (hotplug tests are run in limited mode) 39============================================================= 40 41To build the tests:: 42 43 $ make headers 44 $ make -C tools/testing/selftests 45 46To run the tests:: 47 48 $ make -C tools/testing/selftests run_tests 49 50To build and run the tests with a single command, use:: 51 52 $ make kselftest 53 54Note that some tests will require root privileges. 55 56Kselftest supports saving output files in a separate directory and then 57running tests. To locate output files in a separate directory two syntaxes 58are supported. In both cases the working directory must be the root of the 59kernel src. This is applicable to "Running a subset of selftests" section 60below. 61 62To build, save output files in a separate directory with O= :: 63 64 $ make O=/tmp/kselftest kselftest 65 66To build, save output files in a separate directory with KBUILD_OUTPUT :: 67 68 $ export KBUILD_OUTPUT=/tmp/kselftest; make kselftest 69 70The O= assignment takes precedence over the KBUILD_OUTPUT environment 71variable. 72 73The above commands by default run the tests and print full pass/fail report. 74Kselftest supports "summary" option to make it easier to understand the test 75results. Please find the detailed individual test results for each test in 76/tmp/testname file(s) when summary option is specified. This is applicable 77to "Running a subset of selftests" section below. 78 79To run kselftest with summary option enabled :: 80 81 $ make summary=1 kselftest 82 83Running a subset of selftests 84============================= 85 86You can use the "TARGETS" variable on the make command line to specify 87single test to run, or a list of tests to run. 88 89To run only tests targeted for a single subsystem:: 90 91 $ make -C tools/testing/selftests TARGETS=ptrace run_tests 92 93You can specify multiple tests to build and run:: 94 95 $ make TARGETS="size timers" kselftest 96 97To build, save output files in a separate directory with O= :: 98 99 $ make O=/tmp/kselftest TARGETS="size timers" kselftest 100 101To build, save output files in a separate directory with KBUILD_OUTPUT :: 102 103 $ export KBUILD_OUTPUT=/tmp/kselftest; make TARGETS="size timers" kselftest 104 105Additionally you can use the "SKIP_TARGETS" variable on the make command 106line to specify one or more targets to exclude from the TARGETS list. 107 108To run all tests but a single subsystem:: 109 110 $ make -C tools/testing/selftests SKIP_TARGETS=ptrace run_tests 111 112You can specify multiple tests to skip:: 113 114 $ make SKIP_TARGETS="size timers" kselftest 115 116You can also specify a restricted list of tests to run together with a 117dedicated skiplist:: 118 119 $ make TARGETS="breakpoints size timers" SKIP_TARGETS=size kselftest 120 121See the top-level tools/testing/selftests/Makefile for the list of all 122possible targets. 123 124Requiring all targets to build successfully 125=========================================== 126 127By default, the build succeeds as long as at least one target builds 128without error. Set ``FORCE_TARGETS=1`` to instead require every target to 129build successfully; make will abort as soon as any target fails:: 130 131 $ make -C tools/testing/selftests FORCE_TARGETS=1 132 133This applies to both the ``all`` and ``install`` targets and is useful in 134CI environments where a silent partial build would be misleading. 135 136Running the full range hotplug selftests 137======================================== 138 139To build the hotplug tests:: 140 141 $ make -C tools/testing/selftests hotplug 142 143To run the hotplug tests:: 144 145 $ make -C tools/testing/selftests run_hotplug 146 147Note that some tests will require root privileges. 148 149 150Install selftests 151================= 152 153You can use the "install" target of "make" (which calls the `kselftest_install.sh` 154tool) to install selftests in the default location (`tools/testing/selftests/kselftest_install`), 155or in a user specified location via the `INSTALL_PATH` "make" variable. 156 157To install selftests in default location:: 158 159 $ make -C tools/testing/selftests install 160 161To install selftests in a user specified location:: 162 163 $ make -C tools/testing/selftests install INSTALL_PATH=/some/other/path 164 165Running installed selftests 166=========================== 167 168Found in the install directory, as well as in the Kselftest tarball, 169is a script named `run_kselftest.sh` to run the tests. 170 171You can simply do the following to run the installed Kselftests. Please 172note some tests will require root privileges:: 173 174 $ cd kselftest_install 175 $ ./run_kselftest.sh 176 177To see the list of available tests, the `-l` option can be used:: 178 179 $ ./run_kselftest.sh -l 180 181The `-c` option can be used to run all the tests from a test collection, or 182the `-t` option for specific single tests. Either can be used multiple times:: 183 184 $ ./run_kselftest.sh -c size -c seccomp -t timers:posix_timers -t timer:nanosleep 185 186For other features see the script usage output, seen with the `-h` option. 187 188Timeout for selftests 189===================== 190 191Selftests are designed to be quick and so a default timeout is used of 45 192seconds for each test. Tests can override the default timeout by adding 193a settings file in their directory and set a timeout variable there to the 194configured a desired upper timeout for the test. Only a few tests override 195the timeout with a value higher than 45 seconds, selftests strives to keep 196it that way. Timeouts in selftests are not considered fatal because the 197system under which a test runs may change and this can also modify the 198expected time it takes to run a test. If you have control over the systems 199which will run the tests you can configure a test runner on those systems to 200use a greater or lower timeout on the command line as with the `-o` or 201the `--override-timeout` argument. For example to use 165 seconds instead 202one would use:: 203 204 $ ./run_kselftest.sh --override-timeout 165 205 206You can look at the TAP output to see if you ran into the timeout. Test 207runners which know a test must run under a specific time can then optionally 208treat these timeouts then as fatal. 209 210Packaging selftests 211=================== 212 213In some cases packaging is desired, such as when tests need to run on a 214different system. To package selftests, run:: 215 216 $ make -C tools/testing/selftests gen_tar 217 218This generates a tarball in the `INSTALL_PATH/kselftest-packages` directory. By 219default, `.gz` format is used. The tar compression format can be overridden by 220specifying a `FORMAT` make variable. Any value recognized by `tar's auto-compress`_ 221option is supported, such as:: 222 223 $ make -C tools/testing/selftests gen_tar FORMAT=.xz 224 225`make gen_tar` invokes `make install` so you can use it to package a subset of 226tests by using variables specified in `Running a subset of selftests`_ 227section:: 228 229 $ make -C tools/testing/selftests gen_tar TARGETS="size" FORMAT=.xz 230 231.. _tar's auto-compress: https://www.gnu.org/software/tar/manual/html_node/gzip.html#auto_002dcompress 232 233Contributing new tests 234====================== 235 236In general, the rules for selftests are 237 238 * Do as much as you can if you're not root; 239 240 * Don't take too long; 241 242 * Don't break the build on any architecture, and 243 244 * Don't cause the top-level "make run_tests" to fail if your feature is 245 unconfigured. 246 247 * The output of tests must conform to the TAP standard to ensure high 248 testing quality and to capture failures/errors with specific details. 249 The kselftest.h and kselftest_harness.h headers provide wrappers for 250 outputting test results. These wrappers should be used for pass, 251 fail, exit, and skip messages. CI systems can easily parse TAP output 252 messages to detect test results. 253 254Contributing new tests (details) 255================================ 256 257 * In your Makefile, use facilities from lib.mk by including it instead of 258 reinventing the wheel. Specify flags and binaries generation flags on 259 need basis before including lib.mk. :: 260 261 CFLAGS = $(KHDR_INCLUDES) 262 TEST_GEN_PROGS := close_range_test 263 include ../lib.mk 264 265 * Use TEST_GEN_XXX if such binaries or files are generated during 266 compiling. 267 268 TEST_PROGS, TEST_GEN_PROGS mean it is the executable tested by 269 default. 270 271 TEST_GEN_MODS_DIR should be used by tests that require modules to be built 272 before the test starts. The variable will contain the name of the directory 273 containing the modules. 274 275 TEST_CUSTOM_PROGS should be used by tests that require custom build 276 rules and prevent common build rule use. 277 278 TEST_PROGS are for test shell scripts. Please ensure shell script has 279 its exec bit set. Otherwise, lib.mk run_tests will generate a warning. 280 281 TEST_CUSTOM_PROGS and TEST_PROGS will be run by common run_tests. 282 283 TEST_PROGS_EXTENDED, TEST_GEN_PROGS_EXTENDED mean it is the 284 executable which is not tested by default. 285 286 TEST_FILES, TEST_GEN_FILES mean it is the file which is used by 287 test. 288 289 TEST_INCLUDES is similar to TEST_FILES, it lists files which should be 290 included when exporting or installing the tests, with the following 291 differences: 292 293 * symlinks to files in other directories are preserved 294 * the part of paths below tools/testing/selftests/ is preserved when 295 copying the files to the output directory 296 297 TEST_INCLUDES is meant to list dependencies located in other directories of 298 the selftests hierarchy. 299 300 * First use the headers inside the kernel source and/or git repo, and then the 301 system headers. Headers for the kernel release as opposed to headers 302 installed by the distro on the system should be the primary focus to be able 303 to find regressions. Use KHDR_INCLUDES in Makefile to include headers from 304 the kernel source. 305 306 * If a test needs specific kernel config options enabled, add a config file in 307 the test directory to enable them. 308 309 e.g: tools/testing/selftests/android/config 310 311 * Create a .gitignore file inside test directory and add all generated objects 312 in it. 313 314 * Add new test name in TARGETS in selftests/Makefile:: 315 316 TARGETS += android 317 318 * All changes should pass:: 319 320 kselftest-{all,install,clean,gen_tar} 321 kselftest-{all,install,clean,gen_tar} O=abo_path 322 kselftest-{all,install,clean,gen_tar} O=rel_path 323 make -C tools/testing/selftests {all,install,clean,gen_tar} 324 make -C tools/testing/selftests {all,install,clean,gen_tar} O=abs_path 325 make -C tools/testing/selftests {all,install,clean,gen_tar} O=rel_path 326 327Test Module 328=========== 329 330Kselftest tests the kernel from userspace. Sometimes things need 331testing from within the kernel, one method of doing this is to create a 332test module. We can tie the module into the kselftest framework by 333using a shell script test runner. ``kselftest/module.sh`` is designed 334to facilitate this process. There is also a header file provided to 335assist writing kernel modules that are for use with kselftest: 336 337- ``tools/testing/selftests/kselftest_module.h`` 338- ``tools/testing/selftests/kselftest/module.sh`` 339 340Note that test modules should taint the kernel with TAINT_TEST. This will 341happen automatically for modules which are in the ``tools/testing/`` 342directory, or for modules which use the ``kselftest_module.h`` header above. 343Otherwise, you'll need to add ``MODULE_INFO(test, "Y")`` to your module 344source. selftests which do not load modules typically should not taint the 345kernel, but in cases where a non-test module is loaded, TEST_TAINT can be 346applied from userspace by writing to ``/proc/sys/kernel/tainted``. 347 348How to use 349---------- 350 351Here we show the typical steps to create a test module and tie it into 352kselftest. We use kselftests for lib/ as an example. 353 3541. Create the test module 355 3562. Create the test script that will run (load/unload) the module 357 e.g. ``tools/testing/selftests/lib/bitmap.sh`` 358 3593. Add line to config file e.g. ``tools/testing/selftests/lib/config`` 360 3614. Add test script to makefile e.g. ``tools/testing/selftests/lib/Makefile`` 362 3635. Verify it works: 364 365.. code-block:: sh 366 367 # Assumes you have booted a fresh build of this kernel tree 368 cd /path/to/linux/tree 369 make kselftest-merge 370 make modules 371 sudo make modules_install 372 make TARGETS=lib kselftest 373 374Example Module 375-------------- 376 377A bare bones test module might look like this: 378 379.. code-block:: c 380 381 // SPDX-License-Identifier: GPL-2.0+ 382 383 #define pr_fmt(fmt) KBUILD_MODNAME ": " fmt 384 385 #include "../tools/testing/selftests/kselftest_module.h" 386 387 KSTM_MODULE_GLOBALS(); 388 389 /* 390 * Kernel module for testing the foobinator 391 */ 392 393 static int __init test_function() 394 { 395 ... 396 } 397 398 static void __init selftest(void) 399 { 400 KSTM_CHECK_ZERO(do_test_case("", 0)); 401 } 402 403 KSTM_MODULE_LOADERS(test_foo); 404 MODULE_AUTHOR("John Developer <jd@fooman.org>"); 405 MODULE_LICENSE("GPL"); 406 MODULE_INFO(test, "Y"); 407 408Example test script 409------------------- 410 411.. code-block:: sh 412 413 #!/bin/bash 414 # SPDX-License-Identifier: GPL-2.0+ 415 $(dirname $0)/../kselftest/module.sh "foo" test_foo 416 417 418Test Harness 419============ 420 421The kselftest_harness.h file contains useful helpers to build tests. The 422test harness is for userspace testing, for kernel space testing see `Test 423Module`_ above. 424 425The tests from tools/testing/selftests/seccomp/seccomp_bpf.c can be used as 426example. 427 428Example 429------- 430 431.. kernel-doc:: tools/testing/selftests/kselftest_harness.h 432 :doc: example 433 434 435Helpers 436------- 437 438.. kernel-doc:: tools/testing/selftests/kselftest_harness.h 439 :functions: TH_LOG TEST TEST_SIGNAL FIXTURE FIXTURE_DATA FIXTURE_SETUP 440 FIXTURE_TEARDOWN TEST_F TEST_HARNESS_MAIN FIXTURE_VARIANT 441 FIXTURE_VARIANT_ADD 442 443Operators 444--------- 445 446.. kernel-doc:: tools/testing/selftests/kselftest_harness.h 447 :doc: operators 448 449.. kernel-doc:: tools/testing/selftests/kselftest_harness.h 450 :functions: ASSERT_EQ ASSERT_NE ASSERT_LT ASSERT_LE ASSERT_GT ASSERT_GE 451 ASSERT_NULL ASSERT_TRUE ASSERT_NULL ASSERT_TRUE ASSERT_FALSE 452 ASSERT_STREQ ASSERT_STRNE EXPECT_EQ EXPECT_NE EXPECT_LT 453 EXPECT_LE EXPECT_GT EXPECT_GE EXPECT_NULL EXPECT_TRUE 454 EXPECT_FALSE EXPECT_STREQ EXPECT_STRNE 455