1# Unit tests 2 3> [!NOTE] 4> 5> This document is a draft. It will be updated as we gain experience writing 6> and running unit tests. 7 8This directory contains a unit testing framework for OpenZFS, and a collection 9of unit tests. 10 11## Building and running 12 13The unit tests are built by default as part of the regular userspace build, so 14you probably don’t have to do anything else. 15 16The easiest way to run the tests is to run `make unit`, which will run all the 17available tests. 18 19``` 20$ make unit 21 UNITTEST tests/unit/test_zap 22Running test suite with seed 0x9d36890b... 23zap.mock_microzap_sanity [ OK ] [ 0.00001088 / 0.00000939 CPU ] 24zap.mock_fatzap_sanity [ OK ] [ 0.00004281 / 0.00004257 CPU ] 25zap.zap_basic 26 type=micro [ OK ] [ 0.00001899 / 0.00001893 CPU ] 27 type=fat [ OK ] [ 0.00004174 / 0.00004135 CPU ] 284 of 4 (100%) tests successful, 0 (0%) test skipped. 29``` 30 31Running a single test binary is possible with the `T=` param to `make unit`. 32 33``` 34$ make unit T=zap 35 UNITTEST tests/unit/test_zap 36 ... 37``` 38 39The test binaries are just normal programs in `./tests/unit`, and can be run 40directly. This is useful for debugging with `gdb`. 41 42``` 43$ ./tests/unit/test_zap 44Running test suite with seed 0x18e131ac... 45... 46``` 47 48The test framework provides various options for controlling how the tests are 49run. Add the `--help` switch for more info. If using the make rule, options can 50be passed via the `TOPT=` param. 51 52### Building just for tests 53 54Recommended “minimum” build for just the unit tests, with additional debug to 55assist with understanding issues. 56 57``` 58./configure \ 59 --with-config=user \ 60 --enable-debug --enable-debuginfo \ 61 --disable-sysvinit --disable-systemd --disable-pam --disable-pyzfs 62make -j$(nproc) 63``` 64 65TODO: add `--with-config=unit` that disables _everything_ not needed for the 66tests 67 68### Generating a coverage report 69 70If `configure` was run with `--enable-code-coverage`, then two additional build 71targets are available that will run the requested tests and produce a report. 72 73The `unit-coverage` target runs `scripts/coverage_report.pl` to produce a 74coverage summary directly in text immediately after the test output, and is 75good for inclusion in log files and other build system output. 76 77``` 78$ make unit-coverage T=zap 79 UNITTEST tests/unit/test_zap 80Running test suite with seed 0xf51efca9... 81zap.mock_microzap_sanity [ OK ] [ 0.00000941 / 0.00000834 CPU ] 82zap.mock_fatzap_sanity [ OK ] [ 0.00005782 / 0.00005766 CPU ] 83... 84zap.cursor_release_one 85 type=micro [ OK ] [ 0.00001705 / 0.00001681 CPU ] 86 type=fat [ OK ] [ 0.00004748 / 0.00004738 CPU ] 8730 of 30 (100%) tests successful, 0 (0%) test skipped. 88Coverage: test_zap | By line | By branch | By function 89 | Rate% Total Hit | Rate% Total Hit | Rate% Total Hit 90module/zfs/u8_textprep.c | 0.0% 802 0 | 0.0% 510 0 | 0.0% 12 0 91module/zfs/zap.c | 33.9% 610 207 | 31.1% 238 74 | 23.0% 74 17 92module/zfs/zap_fat.c | 47.1% 665 313 | 29.8% 446 133 | 62.2% 37 23 93module/zfs/zap_impl.c | 57.8% 232 134 | 39.7% 146 58 | 72.0% 25 18 94module/zfs/zap_leaf.c | 60.9% 466 284 | 41.2% 216 89 | 78.3% 23 18 95module/zfs/zap_micro.c | 68.9% 238 164 | 41.5% 142 59 | 92.9% 14 13 96``` 97 98The `unit-coverage-html` will use `lcov` and `genhtml` to generate an 99interactive HTML report that also can show the specific source lines that are 100covered. 101 102``` 103$ make unit-coverage-html T=zap 104 UNITTEST tests/unit/test_zap 105Running test suite with seed 0x485bf2e2... 106zap.mock_microzap_sanity [ OK ] [ 0.00000935 / 0.00000794 CPU ] 107zap.mock_fatzap_sanity [ OK ] [ 0.00006050 / 0.00006025 CPU ] 108... 109zap.cursor_release_one 110 type=micro [ OK ] [ 0.00001785 / 0.00001767 CPU ] 111 type=fat [ OK ] [ 0.00005262 / 0.00005250 CPU ] 11230 of 30 (100%) tests successful, 0 (0%) test skipped. 113coverage results: 114file:///home/robn/code/zfs-unit/tests/unit/tests/unit/test_zap_coverage/index.ht 115ml 116``` 117 118Currently the coverage data will only be regenerated when the test binary 119itself changes. To force it, use `make unit-clean-local` to remove the coverage 120data. 121 122## Guidance for test writers 123 124### Top five 125 126* Only bring in the source files under test. 127* Use mocks to create the test scenario, then interrogate them to understand 128the result. 129* Prefer more smaller tests over fewer bigger ones. 130* Use coverage reports to guide test development. 131* Do the simplest possible thing. 132 133### Test structure 134 135Tests should be as simple and as readable as possible. When a test fails, we 136want to avoid the possibility that it could be the test itself at fault rather 137than the system under test. 138 139* Aim for one source file per subsystem or source concept (eg ZAP). 140* Aim for one test function per API call or logical behaviour 141 * Each “version” or “mode” of an API call or behaviour is a separate test 142 * Don’t test more than one thing in the same test; a test shouldn’t rely on 143 state or results from an earlier test 144* Use test parameters for “class“ or ”vtable” -type APIs, where each 145 implementation should respond to API calls the same way 146 147### Build system 148 149The build setup `tests/unit/Makefile.am` is very similar to the other 150userspace, however it has a couple of differences to make the run and coverage 151targets work more smoothly. 152 153* Name the test program `test_foo`. Almost always, you will have one source 154 file with the actual tests in it, called `test_foo.c`. 155* Add the program to `UNIT_TESTS`. `noinst_PROGRAMS` will be populated from it, 156 but this gives a specific name the run and coverage targets can use to 157 resolve the `T=` parameter to a specific test. 158* List the source files under test in `nodist_%C%_test_foo_SOURCES`, and the 159 source files for the test itself in `%C%_test_foo_SOURCES`. This is 160 important, as the coverage targets use `nodist_%C%_ ... _SOURCES` as the list 161 of objects to include in the coverage output. 162 163### Mocks 164 165A “mock” struct is a fake version of some data structure that the subsystem 166under test will accept and use as though it was a real one. 167 168* Make mock structs opaque. All uses from the test suite should be through 169 specific named accessor functions. 170* Name a mock struct for the struct it is mimicking, prefixed with `mock_`. eg 171 `mock_dnode_t` is the mock for `dnode_t`. 172* Access functions should be named for the struct, eg the function to create a 173 `mock_dnode_t` is `mock_dnode_t *mock_dnode_create(...)`. 174* `mock_*` functions should always use the mock type name in its signature, 175 never the original. 176* The mock object should always be directly castable to its real type and 177 vice-versa, ie a `mock_dnode_t *` is always usable wherever a `dnode_t *` 178 is (within the domain of the subsystem under test). 179 180This guidance pushes the programmer towards being explicit at the possible 181expense of concision. This is in service of keeping the tests reliable; in 182particular, if mocks require explicit casting to use, then there’s far less 183chance of either a mock or a real object being used incorrectly in the test, 184which can be confusing. 185 186### Unit testing framework 187 188[µnit](https://nemequ.github.io/munit/) (aka munit) is the unit test framework. 189It is a relatively niche choice, and arguably abandoned by upstream, but is 190well constructed with a thoughtful feature set and some useful properties: 191 192* Just two source files we can easily carry in the repo. 193* Portable, including to Windows. 194* Each test is run in a forked process, so a test failure will not corrupt the 195 rest of the test suite run 196* Parameterised tests. 197* A large suite of assertions and other useful functions that make it easy to 198 integrate with. 199 200All OpenZFS unit tests are ultimately targeting munit, so its expected that 201they will use various features as needed. However, we also supply our own 202facilities to extend those in useful ways. 203 204#### Local extensions 205 206`unit.h` provides a handful of macros. The majority of these are aliases for 207the much longer munit names for same function, eg `unit_true(n)` is an alias 208for `munit_assert_true(n)`, `unit_eq(a,b)` is an alias for 209`munit_assert_uint64(a, ==, b)`, and so on. These are there so that the 210assertions do not dominate the test visually, as we want it to be easier to 211focus on the details. 212 213Similarly, the `UINT_TEST` and `UNIT_PARAM` macros exist to help with test 214definition, as the casts are a little complicated. 215 216The goal is to keep this set relatively small, but all of munit is there for 217use, so do extend it if necessary. 218