1.. SPDX-License-Identifier: GPL-2.0 2 3============================ 4Tips For Running KUnit Tests 5============================ 6 7Using ``kunit.py run`` ("kunit tool") 8===================================== 9 10Running from any directory 11-------------------------- 12 13It can be handy to create a bash function like: 14 15.. code-block:: bash 16 17 function run_kunit() { 18 ( cd "$(git rev-parse --show-toplevel)" && ./tools/testing/kunit/kunit.py run $@ ) 19 } 20 21.. note:: 22 Early versions of ``kunit.py`` (before 5.6) didn't work unless run from 23 the kernel root, hence the use of a subshell and ``cd``. 24 25Running a subset of tests 26------------------------- 27 28``kunit.py run`` accepts an optional glob argument to filter tests. Currently 29this only matches against suite names, but this may change in the future. 30 31Say that we wanted to run the sysctl tests, we could do so via: 32 33.. code-block:: bash 34 35 $ echo -e 'CONFIG_KUNIT=y\nCONFIG_KUNIT_ALL_TESTS=y' > .kunit/.kunitconfig 36 $ ./tools/testing/kunit/kunit.py run 'sysctl*' 37 38We're paying the cost of building more tests than we need this way, but it's 39easier than fiddling with ``.kunitconfig`` files or commenting out 40``kunit_suite``'s. 41 42However, if we wanted to define a set of tests in a less ad hoc way, the next 43tip is useful. 44 45Defining a set of tests 46----------------------- 47 48``kunit.py run`` (along with ``build``, and ``config``) supports a 49``--kunitconfig`` flag. So if you have a set of tests that you want to run on a 50regular basis (especially if they have other dependencies), you can create a 51specific ``.kunitconfig`` for them. 52 53E.g. kunit has one for its tests: 54 55.. code-block:: bash 56 57 $ ./tools/testing/kunit/kunit.py run --kunitconfig=lib/kunit/.kunitconfig 58 59Alternatively, if you're following the convention of naming your 60file ``.kunitconfig``, you can just pass in the dir, e.g. 61 62.. code-block:: bash 63 64 $ ./tools/testing/kunit/kunit.py run --kunitconfig=lib/kunit 65 66.. note:: 67 This is a relatively new feature (5.12+) so we don't have any 68 conventions yet about on what files should be checked in versus just 69 kept around locally. It's up to you and your maintainer to decide if a 70 config is useful enough to submit (and therefore have to maintain). 71 72.. note:: 73 Having ``.kunitconfig`` fragments in a parent and child directory is 74 iffy. There's discussion about adding an "import" statement in these 75 files to make it possible to have a top-level config run tests from all 76 child directories. But that would mean ``.kunitconfig`` files are no 77 longer just simple .config fragments. 78 79 One alternative would be to have kunit tool recursively combine configs 80 automagically, but tests could theoretically depend on incompatible 81 options, so handling that would be tricky. 82 83Generating code coverage reports under UML 84------------------------------------------ 85 86.. note:: 87 TODO(brendanhiggins@google.com): There are various issues with UML and 88 versions of gcc 7 and up. You're likely to run into missing ``.gcda`` 89 files or compile errors. 90 91This is different from the "normal" way of getting coverage information that is 92documented in Documentation/dev-tools/gcov.rst. 93 94Instead of enabling ``CONFIG_GCOV_KERNEL=y``, we can set these options: 95 96.. code-block:: none 97 98 CONFIG_DEBUG_KERNEL=y 99 CONFIG_DEBUG_INFO=y 100 CONFIG_GCOV=y 101 102 103Putting it together into a copy-pastable sequence of commands: 104 105.. code-block:: bash 106 107 # Append coverage options to the current config 108 $ echo -e "CONFIG_DEBUG_KERNEL=y\nCONFIG_DEBUG_INFO=y\nCONFIG_GCOV=y" >> .kunit/.kunitconfig 109 $ ./tools/testing/kunit/kunit.py run 110 # Extract the coverage information from the build dir (.kunit/) 111 $ lcov -t "my_kunit_tests" -o coverage.info -c -d .kunit/ 112 113 # From here on, it's the same process as with CONFIG_GCOV_KERNEL=y 114 # E.g. can generate an HTML report in a tmp dir like so: 115 $ genhtml -o /tmp/coverage_html coverage.info 116 117 118If your installed version of gcc doesn't work, you can tweak the steps: 119 120.. code-block:: bash 121 122 $ ./tools/testing/kunit/kunit.py run --make_options=CC=/usr/bin/gcc-6 123 $ lcov -t "my_kunit_tests" -o coverage.info -c -d .kunit/ --gcov-tool=/usr/bin/gcov-6 124 125 126Running tests manually 127====================== 128 129Running tests without using ``kunit.py run`` is also an important use case. 130Currently it's your only option if you want to test on architectures other than 131UML. 132 133As running the tests under UML is fairly straightforward (configure and compile 134the kernel, run the ``./linux`` binary), this section will focus on testing 135non-UML architectures. 136 137 138Running built-in tests 139---------------------- 140 141When setting tests to ``=y``, the tests will run as part of boot and print 142results to dmesg in TAP format. So you just need to add your tests to your 143``.config``, build and boot your kernel as normal. 144 145So if we compiled our kernel with: 146 147.. code-block:: none 148 149 CONFIG_KUNIT=y 150 CONFIG_KUNIT_EXAMPLE_TEST=y 151 152Then we'd see output like this in dmesg signaling the test ran and passed: 153 154.. code-block:: none 155 156 TAP version 14 157 1..1 158 # Subtest: example 159 1..1 160 # example_simple_test: initializing 161 ok 1 - example_simple_test 162 ok 1 - example 163 164Running tests as modules 165------------------------ 166 167Depending on the tests, you can build them as loadable modules. 168 169For example, we'd change the config options from before to 170 171.. code-block:: none 172 173 CONFIG_KUNIT=y 174 CONFIG_KUNIT_EXAMPLE_TEST=m 175 176Then after booting into our kernel, we can run the test via 177 178.. code-block:: none 179 180 $ modprobe kunit-example-test 181 182This will then cause it to print TAP output to stdout. 183 184.. note:: 185 The ``modprobe`` will *not* have a non-zero exit code if any test 186 failed (as of 5.13). But ``kunit.py parse`` would, see below. 187 188.. note:: 189 You can set ``CONFIG_KUNIT=m`` as well, however, some features will not 190 work and thus some tests might break. Ideally tests would specify they 191 depend on ``KUNIT=y`` in their ``Kconfig``'s, but this is an edge case 192 most test authors won't think about. 193 As of 5.13, the only difference is that ``current->kunit_test`` will 194 not exist. 195 196Pretty-printing results 197----------------------- 198 199You can use ``kunit.py parse`` to parse dmesg for test output and print out 200results in the same familiar format that ``kunit.py run`` does. 201 202.. code-block:: bash 203 204 $ ./tools/testing/kunit/kunit.py parse /var/log/dmesg 205 206 207Retrieving per suite results 208---------------------------- 209 210Regardless of how you're running your tests, you can enable 211``CONFIG_KUNIT_DEBUGFS`` to expose per-suite TAP-formatted results: 212 213.. code-block:: none 214 215 CONFIG_KUNIT=y 216 CONFIG_KUNIT_EXAMPLE_TEST=m 217 CONFIG_KUNIT_DEBUGFS=y 218 219The results for each suite will be exposed under 220``/sys/kernel/debug/kunit/<suite>/results``. 221So using our example config: 222 223.. code-block:: bash 224 225 $ modprobe kunit-example-test > /dev/null 226 $ cat /sys/kernel/debug/kunit/example/results 227 ... <TAP output> ... 228 229 # After removing the module, the corresponding files will go away 230 $ modprobe -r kunit-example-test 231 $ cat /sys/kernel/debug/kunit/example/results 232 /sys/kernel/debug/kunit/example/results: No such file or directory 233 234Generating code coverage reports 235-------------------------------- 236 237See Documentation/dev-tools/gcov.rst for details on how to do this. 238 239The only vaguely KUnit-specific advice here is that you probably want to build 240your tests as modules. That way you can isolate the coverage from tests from 241other code executed during boot, e.g. 242 243.. code-block:: bash 244 245 # Reset coverage counters before running the test. 246 $ echo 0 > /sys/kernel/debug/gcov/reset 247 $ modprobe kunit-example-test 248