1tdc - Linux Traffic Control (tc) unit testing suite 2 3Author: Lucas Bates - lucasb@mojatatu.com 4 5tdc is a Python script to load tc unit tests from a separate JSON file and 6execute them inside a network namespace dedicated to the task. 7 8 9REQUIREMENTS 10------------ 11 12* Minimum Python version of 3.4. Earlier 3.X versions may work but are not 13 guaranteed. 14 15* The kernel must have network namespace support if using nsPlugin 16 17* The kernel must have veth support available, as a veth pair is created 18 prior to running the tests when using nsPlugin. 19 20* The kernel must have the appropriate infrastructure enabled to run all tdc 21 unit tests. See the config file in this directory for minimum required 22 features. As new tests will be added, config options list will be updated. 23 24* All tc-related features being tested must be built in or available as 25 modules. To check what is required in current setup run: 26 ./tdc.py -c 27 28 Note: 29 In the current release, tdc run will abort due to a failure in setup or 30 teardown commands - which includes not being able to run a test simply 31 because the kernel did not support a specific feature. (This will be 32 handled in a future version - the current workaround is to run the tests 33 on specific test categories that your kernel supports) 34 35 36BEFORE YOU RUN 37-------------- 38 39The path to the tc executable that will be most commonly tested can be defined 40in the tdc_config.py file. Find the 'TC' entry in the NAMES dictionary and 41define the path. 42 43If you need to test a different tc executable on the fly, you can do so by 44using the -p option when running tdc: 45 ./tdc.py -p /path/to/tc 46 47 48RUNNING TDC 49----------- 50 51To use tdc, root privileges are required. This is because the 52commands being tested must be run as root. The code that enforces 53execution by root uid has been moved into a plugin (see PLUGIN 54ARCHITECTURE, below). 55 56Tests that use a network device should have nsPlugin.py listed as a 57requirement for that test. nsPlugin executes all commands within a 58network namespace and creates a veth pair which may be used in those test 59cases. To disable execution within the namespace, pass the -N option 60to tdc when starting a test run; the veth pair will still be created 61by the plugin. 62 63Running tdc without any arguments will run all tests. Refer to the section 64on command line arguments for more information, or run: 65 ./tdc.py -h 66 67tdc will list the test names as they are being run, and print a summary in 68TAP (Test Anything Protocol) format when they are done. If tests fail, 69output captured from the failing test will be printed immediately following 70the failed test in the TAP output. 71 72 73OVERVIEW OF TDC EXECUTION 74------------------------- 75 76One run of tests is considered a "test suite" (this will be refined in the 77future). A test suite has one or more test cases in it. 78 79A test case has four stages: 80 81 - setup 82 - execute 83 - verify 84 - teardown 85 86The setup and teardown stages can run zero or more commands. The setup 87stage does some setup if the test needs it. The teardown stage undoes 88the setup and returns the system to a "neutral" state so any other test 89can be run next. These two stages require any commands run to return 90success, but do not otherwise verify the results. 91 92The execute and verify stages each run one command. The execute stage 93tests the return code against one or more acceptable values. The 94verify stage checks the return code for success, and also compares 95the stdout with a regular expression. 96 97Each of the commands in any stage will run in a shell instance. 98 99 100USER-DEFINED CONSTANTS 101---------------------- 102 103The tdc_config.py file contains multiple values that can be altered to suit 104your needs. Any value in the NAMES dictionary can be altered without affecting 105the tests to be run. These values are used in the tc commands that will be 106executed as part of the test. More will be added as test cases require. 107 108Example: 109 $TC qdisc add dev $DEV1 ingress 110 111The NAMES values are used to substitute into the commands in the test cases. 112 113 114COMMAND LINE ARGUMENTS 115---------------------- 116 117Run tdc.py -h to see the full list of available arguments. 118 119usage: tdc.py [-h] [-p PATH] [-D DIR [DIR ...]] [-f FILE [FILE ...]] 120 [-c [CATG [CATG ...]]] [-e ID [ID ...]] [-l] [-s] [-i] [-v] [-N] 121 [-d DEVICE] [-P] [-n] [-V] 122 123Linux TC unit tests 124 125optional arguments: 126 -h, --help show this help message and exit 127 -p PATH, --path PATH The full path to the tc executable to use 128 -v, --verbose Show the commands that are being run 129 -N, --notap Suppress tap results for command under test 130 -d DEVICE, --device DEVICE 131 Execute test cases that use a physical device, where 132 DEVICE is its name. (If not defined, tests that require 133 a physical device will be skipped) 134 -P, --pause Pause execution just before post-suite stage 135 136selection: 137 select which test cases: files plus directories; filtered by categories 138 plus testids 139 140 -D DIR [DIR ...], --directory DIR [DIR ...] 141 Collect tests from the specified directory(ies) 142 (default [tc-tests]) 143 -f FILE [FILE ...], --file FILE [FILE ...] 144 Run tests from the specified file(s) 145 -c [CATG [CATG ...]], --category [CATG [CATG ...]] 146 Run tests only from the specified category/ies, or if 147 no category/ies is/are specified, list known 148 categories. 149 -e ID [ID ...], --execute ID [ID ...] 150 Execute the specified test cases with specified IDs 151 152action: 153 select action to perform on selected test cases 154 155 -l, --list List all test cases, or those only within the 156 specified category 157 -s, --show Display the selected test cases 158 -i, --id Generate ID numbers for new test cases 159 160netns: 161 options for nsPlugin (run commands in net namespace) 162 163 -N, --no-namespace 164 Do not run commands in a network namespace. 165 166valgrind: 167 options for valgrindPlugin (run command under test under Valgrind) 168 169 -V, --valgrind Run commands under valgrind 170 171 172PLUGIN ARCHITECTURE 173------------------- 174 175There is now a plugin architecture, and some of the functionality that 176was in the tdc.py script has been moved into the plugins. 177 178The plugins are in the directory plugin-lib. The are executed from 179directory plugins. Put symbolic links from plugins to plugin-lib, 180and name them according to the order you want them to run. This is not 181necessary if a test case being run requires a specific plugin to work. 182 183Example: 184 185bjb@bee:~/work/tc-testing$ ls -l plugins 186total 4 187lrwxrwxrwx 1 bjb bjb 27 Oct 4 16:12 10-rootPlugin.py -> ../plugin-lib/rootPlugin.py 188lrwxrwxrwx 1 bjb bjb 25 Oct 12 17:55 20-nsPlugin.py -> ../plugin-lib/nsPlugin.py 189-rwxr-xr-x 1 bjb bjb 0 Sep 29 15:56 __init__.py 190 191The plugins are a subclass of TdcPlugin, defined in TdcPlugin.py and 192must be called "SubPlugin" so tdc can find them. They are 193distinguished from each other in the python program by their module 194name. 195 196This base class supplies "hooks" to run extra functions. These hooks are as follows: 197 198pre- and post-suite 199pre- and post-case 200pre- and post-execute stage 201adjust-command (runs in all stages and receives the stage name) 202 203The pre-suite hook receives the number of tests and an array of test ids. 204This allows you to dump out the list of skipped tests in the event of a 205failure during setup or teardown stage. 206 207The pre-case hook receives the ordinal number and test id of the current test. 208 209The adjust-command hook receives the stage id (see list below) and the 210full command to be executed. This allows for last-minute adjustment 211of the command. 212 213The stages are identified by the following strings: 214 215 - pre (pre-suite) 216 - setup 217 - command 218 - verify 219 - teardown 220 - post (post-suite) 221 222 223To write a plugin, you need to inherit from TdcPlugin in 224TdcPlugin.py. To use the plugin, you have to put the 225implementation file in plugin-lib, and add a symbolic link to it from 226plugins. It will be detected at run time and invoked at the 227appropriate times. There are a few examples in the plugin-lib 228directory: 229 230 - rootPlugin.py: 231 implements the enforcement of running as root 232 - nsPlugin.py: 233 sets up a network namespace and runs all commands in that namespace, 234 while also setting up dummy devices to be used in testing. 235 - valgrindPlugin.py 236 runs each command in the execute stage under valgrind, 237 and checks for leaks. 238 This plugin will output an extra test for each test in the test file, 239 one is the existing output as to whether the test passed or failed, 240 and the other is a test whether the command leaked memory or not. 241 (This one is a preliminary version, it may not work quite right yet, 242 but the overall template is there and it should only need tweaks.) 243 - buildebpfPlugin.py: 244 builds all programs in $EBPFDIR. 245 246 247ACKNOWLEDGEMENTS 248---------------- 249 250Thanks to: 251 252Jamal Hadi Salim, for providing valuable test cases 253Keara Leibovitz, who wrote the CLI test driver that I used as a base for the 254 first version of the tc testing suite. This work was presented at 255 Netdev 1.2 Tokyo in October 2016. 256Samir Hussain, for providing help while I dove into Python for the first time 257 and being a second eye for this code. 258