1.. SPDX-License-Identifier: GPL-2.0 2 3====================================== 4Design of the objtool klp test harness 5====================================== 6 7tools/objtool/tests/ holds unit tests for the klp subcommands of 8objtool -- ``klp checksum``, ``klp diff``, ``klp post-link`` and 9``--klp-symids`` -- which together turn two builds of the kernel into a 10livepatch module. 11 12This document explains how the harness is built and why. For the rules to 13follow when adding a test, see klp-write-tests.txt. 14 15 16TL;DR 17===== 18 19One run covers one compiler and one architecture; CI runs the combinations. 20Build objtool first -- it needs libelf and libxxhash -- and the same ARCH is 21used for both steps. 22 23Natively, with gcc:: 24 25 make -C tools/objtool 26 make -C tools/objtool tests 27 28Natively, with clang -- LLVM=1 additionally selects the LLVM binutils:: 29 30 CC=clang make -C tools/objtool tests 31 LLVM=1 make -C tools/objtool tests 32 33Cross, with gcc -- an arm64 host running the x86 tests:: 34 35 ARCH=x86_64 CROSS_COMPILE=x86_64-linux-gnu- make -C tools/objtool 36 ARCH=x86_64 CROSS_COMPILE=x86_64-linux-gnu- make -C tools/objtool tests 37 38Cross, with clang. It defaults to the host triple however it is invoked, so 39--target= is what makes it emit x86; OBJCOPY is needed because BFD's is 40usually built for the host's target alone:: 41 42 ARCH=x86_64 make -C tools/objtool 43 ARCH=x86_64 CC="clang --target=x86_64-linux-gnu" OBJCOPY=llvm-objcopy \ 44 make -C tools/objtool tests 45 46A run ends with a totals line; anything other than fail:0 is a real result:: 47 48 # pass:48 fail:0 static-skip:1 probe-skip:0 xfail:0 xpass:0 49 50Useful extras:: 51 52 tools/objtool/tests/run-tests.sh basic # one test, by name 53 tools/objtool/tests/run-tests.sh --keep basic # and keep what it built 54 55 56Why unit tests are possible at all 57================================== 58 59klp-build is a pipeline: build the kernel twice, checksum both, diff them, 60link the result. Testing that end to end means two kernel builds per case, 61which is too slow to run often and too heavy to keep in the tree. 62 63Three properties make a much cheaper test possible. 64 65**objtool has no configuration-dependent logic.** It never reads ``.config``. 66Every ``CONFIG_`` string in its source is a comment or one error message, and 67its only build-time conditionals are driven by host libraries and the target 68architecture. Configuration reaches objtool through exactly two channels: the 69``objtool-args-$(CONFIG_*)`` lines in scripts/Makefile.lib, and the 70contents of the object handed to it. 71 72**The klp subcommands use none of the first channel.** Of objtool's options 73they consult three -- ``checksum``, ``debug_checksum``, ``dryrun`` -- all from 74their own command line. So klp behaviour varies with configuration *only* 75through the input object. 76 77**Therefore a test can reproduce any configuration's behaviour by reproducing 78its input.** Compile a small freestanding fixture with the flags that 79configuration would have used, and objtool cannot tell the difference. No 80kernel, no ``.config``, no object cache. 81 82The whole suite runs in a few seconds. 83 84 85Shape of a test 86=============== 87 88Each test compiles one fixture twice -- once plain, once with ``-DPATCHED`` -- 89runs ``klp checksum`` over both, diffs them, and asserts on properties of the 90output object:: 91 92 . "$(dirname "$0")/../lib.sh" 93 94 setup 95 build_pair basic.c 96 97 assert_input_symbol changed 98 run_diff 99 100 assert_patched changed 101 assert_not_patched untouched 102 103 pass "changed function cloned, unchanged function left alone" 104 105Assertions check properties, never recorded output. Codegen varies between 106compilers and versions, so a golden file would report churn rather than 107regressions. 108 109 110Layout 111====== 112 113:: 114 115 tools/objtool/tests/ 116 lib.sh the harness: everything a test may call 117 run-tests.sh selects, runs and classifies 118 generic/ 119 test-*.sh 120 fixtures/*.c 121 x86/ 122 test-*.sh 123 fixtures/*.c 124 125Which architecture a test is for is expressed by where it lives. The runner 126executes ``generic/`` plus the directory matching this architecture, so a test 127which cannot apply is not run rather than running in order to report that it 128did not. There is no ``x86_only`` helper, and no lookup letting an 129architecture fixture shadow a generic one: an architecture-specific test 130carries its own fixtures. 131 132Compilers cannot be expressed the same way, because CI varies ``CC`` over the 133same tree. A compiler requirement stays a declaration inside the test 134(``gcc_only``, ``clang_only``). 135 136 137The environment is established once 138=================================== 139 140Sourcing lib.sh runs ``klp_preflight``, which checks that objtool 141exists and has klp support, that ``$CC`` works, that the binutils are present, 142and which architecture this is. The answers are exported, so: 143 144* ``run-tests.sh`` sources lib.sh too, and therefore knows the 145 architecture before it chooses which tests to run; 146* each test inherits the answers rather than repeating the work; 147* a test run on its own establishes them for itself. 148 149Preflight answers only whether the suite can run at all. A suite which cannot 150run must not exit 0 looking like one which passed, so a missing objtool fails 151the whole run with a TAP ``Bail out!`` rather than skipping each test in turn. 152What a *particular* compiler can do is a different question, left to the test 153which cares. 154 155 156Outcomes 157======== 158 159Output is TAP. The distinction the harness cares most about is between kinds 160of skip, because a skip is how a suite quietly stops testing anything: 161 162``declared`` 163 The test said in advance it does not apply -- ``gcc_only`` on a clang run. 164 Expected indefinitely. 165 166``probe`` 167 The construct did not turn up in the built object this time. Weaker: one 168 which becomes permanent is a fixture that has stopped testing anything. 169 170``undeclared`` 171 Counted as a **failure**. A test which gives up for a reason it never 172 declared is a hole, not an outcome. 173 174``xfail``/``xpass`` come with them, so a known failure is reported rather than 175commented out, and one which starts passing says so instead of going quietly 176green. 177 178The runner classifies the TAP result line, not everything a test printed: 179objtool warns on stderr and that output is captured, so a stray line ahead of 180the result would otherwise leave the exit status to decide -- and an expected 181failure exits 0. 182 183A run ends with a totals line:: 184 185 # pass:48 fail:0 static-skip:1 probe-skip:0 xfail:0 xpass:0 186 187and reports what it left out:: 188 189 # not run: 5 tests in x86/ (this run is arm64) 190 191 192Working directories 193=================== 194 195A run gets one directory; each test gets a subdirectory of it, mirroring the 196source layout:: 197 198 /tmp/klp-tests.XXXXXXXX/ 199 generic/test-basic/{orig.o,patched.o,out.o,Module.symvers,...} 200 x86/test-kcfi/... 201 202``--keep`` leaves it and reports the one path. Otherwise each test removes its 203own directory and the runner ``rmdir``s the run's -- which fails if anything 204was left behind, so a test which dies without cleaning up is reported rather 205than silently leaking. 206 207 208Running 209======= 210 211:: 212 213 make -C tools/objtool # needs libelf and libxxhash 214 make -C tools/objtool tests 215 216 CC=clang make -C tools/objtool tests # the other toolchain 217 LLVM=1 make -C tools/objtool tests # and its binutils too 218 219 tools/objtool/tests/run-tests.sh --keep basic # one test, keep its objects 220 221A run covers one compiler and one architecture; CI runs the combinations. 222 223Cross-compiled runs 224------------------- 225 226objtool klp is built only where ARCH_HAS_KLP is set, which today means x86 -- 227so an arm64 machine cannot run any of this natively. It can run all of it 228cross, because objtool is a host tool that only reads and rewrites ELF, and 229the tests only compile fixtures and inspect the objects. Nothing has to 230execute target code. 231 232:: 233 234 ARCH=x86_64 CROSS_COMPILE=x86_64-linux-gnu- make -C tools/objtool 235 ARCH=x86_64 CROSS_COMPILE=x86_64-linux-gnu- make -C tools/objtool tests 236 237objtool itself stays a native binary: it is built with HOSTCC, not CC, so 238setting a cross compiler cannot produce one the host is unable to run. ARCH 239selects both the objtool target and the directory of tests to run. 240 241clang needs telling, since it defaults to the host triple however it is 242invoked. The fixtures include no kernel headers, so no sysroot is needed: 243 244:: 245 246 ARCH=x86_64 CC="clang --target=x86_64-linux-gnu" \ 247 OBJCOPY=llvm-objcopy make -C tools/objtool tests 248 249CROSS_COMPILE picks the binutils, and each can be overridden on its own. 250readelf reads any target and rarely needs overriding; BFD's objcopy is usually 251built for the host's alone, hence OBJCOPY=llvm-objcopy above, or install 252binutils-multiarch. 253 254Either readelf will do. The assertions read readelf's output, and the two 255spell some of it differently -- GNU prints "OS [0xff20]" for SHN_LIVEPATCH 256where llvm-readelf prints "OS[0xff20]" -- so they accept both. 257 258Getting this wrong is easy and the harness refuses rather than producing a 259misleading result. "CC=clang ARCH=x86_64" alone selects the x86 tests and 260then builds arm64 objects; preflight compiles a probe object, hands it to 261objtool, and stops the run if they disagree about the architecture, or if 262ARCH does not match what the compiler emits. 263 264 265What this does not cover 266======================== 267 268These are unit tests for objtool's klp subcommands. They do not build a 269kernel, do not run scripts/livepatch/klp-build, and do not load a 270livepatch. Behaviour which only appears when the kernel applies a patch -- 271the module loader refusing a relocation, late module patching ordering -- has 272to be tested by booting, and is out of scope here. 273