1.. SPDX-License-Identifier: GPL-2.0 2 3=================================================== 4The Kernel Test Anything Protocol (KTAP), version 1 5=================================================== 6 7TAP, or the Test Anything Protocol is a format for specifying test results used 8by a number of projects. Its website and specification are found at this `link 9<https://testanything.org/>`_. The Linux Kernel largely uses TAP output for test 10results. However, Kernel testing frameworks have special needs for test results 11which don't align with the original TAP specification. Thus, a "Kernel TAP" 12(KTAP) format is specified to extend and alter TAP to support these use-cases. 13This specification describes the generally accepted format of KTAP as it is 14currently used in the kernel. 15 16KTAP test results describe a series of tests (which may be nested: i.e., test 17can have subtests), each of which can contain both diagnostic data -- e.g., log 18lines -- and a final result. The test structure and results are 19machine-readable, whereas the diagnostic data is unstructured and is there to 20aid human debugging. 21 22KTAP output is built from four different types of lines: 23 24- Version lines 25- Plan lines 26- Test case result lines 27- Diagnostic lines 28 29In general, valid KTAP output should also form valid TAP output, but some 30information, in particular nested test results, may be lost. Also note that 31there is a stagnant draft specification for TAP14, KTAP diverges from this in 32a couple of places (notably the "Subtest" header), which are described where 33relevant later in this document. 34 35Version lines 36------------- 37 38All KTAP-formatted results begin with a "version line" which specifies which 39version of the (K)TAP standard the result is compliant with. 40 41For example: 42 43- "KTAP version 1" 44- "TAP version 13" 45- "TAP version 14" 46 47Note that, in KTAP, subtests also begin with a version line, which denotes the 48start of the nested test results. This differs from TAP14, which uses a 49separate "Subtest" line. 50 51While, going forward, "KTAP version 1" should be used by compliant tests, it 52is expected that most parsers and other tooling will accept the other versions 53listed here for compatibility with existing tests and frameworks. 54 55Plan lines 56---------- 57 58A test plan provides the number of tests (or subtests) in the KTAP output. 59 60Plan lines must follow the format of "1..N" where N is the number of tests or subtests. 61Plan lines follow version lines to indicate the number of nested tests. 62 63While there are cases where the number of tests is not known in advance -- in 64which case the test plan may be omitted -- it is strongly recommended one is 65present where possible. 66 67Test case result lines 68---------------------- 69 70Test case result lines indicate the final status of a test. 71They are required and must have the format: 72 73.. code-block:: none 74 75 <result> <number> [<description>][ # [<directive>] [<diagnostic data>]] 76 77The result can be either "ok", which indicates the test case passed, 78or "not ok", which indicates that the test case failed. 79 80<number> represents the number of the test being performed. The first test must 81have the number 1 and the number then must increase by 1 for each additional 82subtest within the same test at the same nesting level. 83 84The description is a description of the test, generally the name of 85the test, and can be any string of characters other than # or a 86newline. The description is optional, but recommended. 87 88The directive and any diagnostic data is optional. If either are present, they 89must follow a hash sign, "#". 90 91A directive is a keyword that indicates a different outcome for a test other 92than passed and failed. The directive is optional, and consists of a single 93keyword preceding the diagnostic data. In the event that a parser encounters 94a directive it doesn't support, it should fall back to the "ok" / "not ok" 95result. 96 97Currently accepted directives are: 98 99- "SKIP", which indicates a test was skipped (note the result of the test case 100 result line can be either "ok" or "not ok" if the SKIP directive is used) 101- "TODO", which indicates that a test is not expected to pass at the moment, 102 e.g. because the feature it is testing is known to be broken. While this 103 directive is inherited from TAP, its use in the kernel is discouraged. 104- "XFAIL", which indicates that a test is expected to fail. This is similar 105 to "TODO", above, and is used by some kselftest tests. 106- “TIMEOUT”, which indicates a test has timed out (note the result of the test 107 case result line should be “not ok” if the TIMEOUT directive is used) 108- “ERROR”, which indicates that the execution of a test has failed due to a 109 specific error that is included in the diagnostic data. (note the result of 110 the test case result line should be “not ok” if the ERROR directive is used) 111 112The diagnostic data is a plain-text field which contains any additional details 113about why this result was produced. This is typically an error message for ERROR 114or failed tests, or a description of missing dependencies for a SKIP result. 115 116The diagnostic data field is optional, and results which have neither a 117directive nor any diagnostic data do not need to include the "#" field 118separator. 119 120Example result lines include:: 121 122 ok 1 test_case_name 123 124The test "test_case_name" passed. 125 126:: 127 128 not ok 1 test_case_name 129 130The test "test_case_name" failed. 131 132:: 133 134 ok 1 test # SKIP necessary dependency unavailable 135 136The test "test" was SKIPPED with the diagnostic message "necessary dependency 137unavailable". 138 139:: 140 141 not ok 1 test # TIMEOUT 30 seconds 142 143The test "test" timed out, with diagnostic data "30 seconds". 144 145:: 146 147 ok 5 check return code # rcode=0 148 149The test "check return code" passed, with additional diagnostic data “rcode=0” 150 151 152Diagnostic lines 153---------------- 154 155If tests wish to output any further information, they should do so using 156"diagnostic lines". Diagnostic lines are optional, freeform text, and are 157often used to describe what is being tested and any intermediate results in 158more detail than the final result and diagnostic data line provides. 159 160Diagnostic lines are formatted as "# <diagnostic_description>", where the 161description can be any string. Diagnostic lines can be anywhere in the test 162output. As a rule, diagnostic lines regarding a test are directly before the 163test result line for that test. 164 165Note that most tools will treat unknown lines (see below) as diagnostic lines, 166even if they do not start with a "#": this is to capture any other useful 167kernel output which may help debug the test. It is nevertheless recommended 168that tests always prefix any diagnostic output they have with a "#" character. 169 170Unknown lines 171------------- 172 173There may be lines within KTAP output that do not follow the format of one of 174the four formats for lines described above. This is allowed, however, they will 175not influence the status of the tests. 176 177This is an important difference from TAP. Kernel tests may print messages 178to the system console or a log file. Both of these destinations may contain 179messages either from unrelated kernel or userspace activity, or kernel 180messages from non-test code that is invoked by the test. The kernel code 181invoked by the test likely is not aware that a test is in progress and 182thus can not print the message as a diagnostic message. 183 184Nested tests 185------------ 186 187In KTAP, tests can be nested. This is done by having a test include within its 188output an entire set of KTAP-formatted results. This can be used to categorize 189and group related tests, or to split out different results from the same test. 190 191The "parent" test's result should consist of all of its subtests' results, 192starting with another KTAP version line and test plan, and end with the overall 193result. If one of the subtests fail, for example, the parent test should also 194fail. 195 196Additionally, all lines in a subtest should be indented. One level of 197indentation is two spaces: " ". The indentation should begin at the version 198line and should end before the parent test's result line. 199 200"Unknown lines" are not considered to be lines in a subtest and thus are 201allowed to be either indented or not indented. 202 203An example of a test with two nested subtests: 204 205:: 206 207 KTAP version 1 208 1..1 209 KTAP version 1 210 1..2 211 ok 1 test_1 212 not ok 2 test_2 213 # example failed 214 not ok 1 example 215 216An example format with multiple levels of nested testing: 217 218:: 219 220 KTAP version 1 221 1..2 222 KTAP version 1 223 1..2 224 KTAP version 1 225 1..2 226 not ok 1 test_1 227 ok 2 test_2 228 not ok 1 test_3 229 ok 2 test_4 # SKIP 230 not ok 1 example_test_1 231 ok 2 example_test_2 232 233 234Major differences between TAP and KTAP 235-------------------------------------- 236 237================================================== ========= =============== 238Feature TAP KTAP 239================================================== ========= =============== 240yaml and json in diagnosic message ok not recommended 241TODO directive ok not recognized 242allows an arbitrary number of tests to be nested no yes 243"Unknown lines" are in category of "Anything else" yes no 244"Unknown lines" are incorrect allowed 245================================================== ========= =============== 246 247The TAP14 specification does permit nested tests, but instead of using another 248nested version line, uses a line of the form 249"Subtest: <name>" where <name> is the name of the parent test. 250 251Example KTAP output 252-------------------- 253:: 254 255 KTAP version 1 256 1..1 257 KTAP version 1 258 1..3 259 KTAP version 1 260 1..1 261 # test_1: initializing test_1 262 ok 1 test_1 263 ok 1 example_test_1 264 KTAP version 1 265 1..2 266 ok 1 test_1 # SKIP test_1 skipped 267 ok 2 test_2 268 ok 2 example_test_2 269 KTAP version 1 270 1..3 271 ok 1 test_1 272 # test_2: FAIL 273 not ok 2 test_2 274 ok 3 test_3 # SKIP test_3 skipped 275 not ok 3 example_test_3 276 not ok 1 main_test 277 278This output defines the following hierarchy: 279 280A single test called "main_test", which fails, and has three subtests: 281 282- "example_test_1", which passes, and has one subtest: 283 284 - "test_1", which passes, and outputs the diagnostic message "test_1: initializing test_1" 285 286- "example_test_2", which passes, and has two subtests: 287 288 - "test_1", which is skipped, with the explanation "test_1 skipped" 289 - "test_2", which passes 290 291- "example_test_3", which fails, and has three subtests 292 293 - "test_1", which passes 294 - "test_2", which outputs the diagnostic line "test_2: FAIL", and fails. 295 - "test_3", which is skipped with the explanation "test_3 skipped" 296 297Note that the individual subtests with the same names do not conflict, as they 298are found in different parent tests. This output also exhibits some sensible 299rules for "bubbling up" test results: a test fails if any of its subtests fail. 300Skipped tests do not affect the result of the parent test (though it often 301makes sense for a test to be marked skipped if _all_ of its subtests have been 302skipped). 303 304See also: 305--------- 306 307- The TAP specification: 308 https://testanything.org/tap-version-13-specification.html 309- The (stagnant) TAP version 14 specification: 310 https://github.com/TestAnything/Specification/blob/tap-14-specification/specification.md 311- The kselftest documentation: 312 Documentation/dev-tools/kselftest.rst 313- The KUnit documentation: 314 Documentation/dev-tools/kunit/index.rst 315