xref: /linux/tools/testing/selftests/tc-testing/README (revision 06d07429858317ded2db7986113a9e0129cd599b)
176b903eeSLucas Batestdc - Linux Traffic Control (tc) unit testing suite
276b903eeSLucas Bates
376b903eeSLucas BatesAuthor: Lucas Bates - lucasb@mojatatu.com
476b903eeSLucas Bates
576b903eeSLucas Batestdc is a Python script to load tc unit tests from a separate JSON file and
676b903eeSLucas Batesexecute them inside a network namespace dedicated to the task.
776b903eeSLucas Bates
876b903eeSLucas Bates
976b903eeSLucas BatesREQUIREMENTS
1076b903eeSLucas Bates------------
1176b903eeSLucas Bates
12*d3fc4eeaSPedro Tammela*  Minimum Python version of 3.8.
1376b903eeSLucas Bates
14489ce2f4SLucas Bates*  The kernel must have network namespace support if using nsPlugin
1576b903eeSLucas Bates
1676b903eeSLucas Bates*  The kernel must have veth support available, as a veth pair is created
17489ce2f4SLucas Bates   prior to running the tests when using nsPlugin.
1876b903eeSLucas Bates
19b70f1f3aSKeara Leibovitz*  The kernel must have the appropriate infrastructure enabled to run all tdc
20b70f1f3aSKeara Leibovitz   unit tests. See the config file in this directory for minimum required
21b70f1f3aSKeara Leibovitz   features. As new tests will be added, config options list will be updated.
22b70f1f3aSKeara Leibovitz
2395ce14c3SBrenda J. Butler*  All tc-related features being tested must be built in or available as
2495ce14c3SBrenda J. Butler   modules.  To check what is required in current setup run:
2576b903eeSLucas Bates   ./tdc.py -c
2676b903eeSLucas Bates
2776b903eeSLucas Bates   Note:
2876b903eeSLucas Bates   In the current release, tdc run will abort due to a failure in setup or
2976b903eeSLucas Bates   teardown commands - which includes not being able to run a test simply
3076b903eeSLucas Bates   because the kernel did not support a specific feature. (This will be
3176b903eeSLucas Bates   handled in a future version - the current workaround is to run the tests
3276b903eeSLucas Bates   on specific test categories that your kernel supports)
3376b903eeSLucas Bates
3476b903eeSLucas Bates
3576b903eeSLucas BatesBEFORE YOU RUN
3676b903eeSLucas Bates--------------
3776b903eeSLucas Bates
3876b903eeSLucas BatesThe path to the tc executable that will be most commonly tested can be defined
3976b903eeSLucas Batesin the tdc_config.py file. Find the 'TC' entry in the NAMES dictionary and
4076b903eeSLucas Batesdefine the path.
4176b903eeSLucas Bates
4276b903eeSLucas BatesIf you need to test a different tc executable on the fly, you can do so by
4376b903eeSLucas Batesusing the -p option when running tdc:
4476b903eeSLucas Bates	./tdc.py -p /path/to/tc
4576b903eeSLucas Bates
4676b903eeSLucas Bates
4776b903eeSLucas BatesRUNNING TDC
4876b903eeSLucas Bates-----------
4976b903eeSLucas Bates
5095ce14c3SBrenda J. ButlerTo use tdc, root privileges are required.  This is because the
5195ce14c3SBrenda J. Butlercommands being tested must be run as root.  The code that enforces
5295ce14c3SBrenda J. Butlerexecution by root uid has been moved into a plugin (see PLUGIN
5395ce14c3SBrenda J. ButlerARCHITECTURE, below).
5476b903eeSLucas Bates
55489ce2f4SLucas BatesTests that use a network device should have nsPlugin.py listed as a
56489ce2f4SLucas Batesrequirement for that test. nsPlugin executes all commands within a
57489ce2f4SLucas Batesnetwork namespace and creates a veth pair which may be used in those test
58489ce2f4SLucas Batescases. To disable execution within the namespace, pass the -N option
59489ce2f4SLucas Batesto tdc when starting a test run; the veth pair will still be created
60489ce2f4SLucas Batesby the plugin.
6176b903eeSLucas Bates
6276b903eeSLucas BatesRunning tdc without any arguments will run all tests. Refer to the section
6376b903eeSLucas Bateson command line arguments for more information, or run:
6476b903eeSLucas Bates	./tdc.py -h
6576b903eeSLucas Bates
6676b903eeSLucas Batestdc will list the test names as they are being run, and print a summary in
6776b903eeSLucas BatesTAP (Test Anything Protocol) format when they are done. If tests fail,
6876b903eeSLucas Batesoutput captured from the failing test will be printed immediately following
6976b903eeSLucas Batesthe failed test in the TAP output.
7076b903eeSLucas Bates
7176b903eeSLucas Bates
7295ce14c3SBrenda J. ButlerOVERVIEW OF TDC EXECUTION
7395ce14c3SBrenda J. Butler-------------------------
7495ce14c3SBrenda J. Butler
7595ce14c3SBrenda J. ButlerOne run of tests is considered a "test suite" (this will be refined in the
7695ce14c3SBrenda J. Butlerfuture).  A test suite has one or more test cases in it.
7795ce14c3SBrenda J. Butler
7895ce14c3SBrenda J. ButlerA test case has four stages:
7995ce14c3SBrenda J. Butler
8095ce14c3SBrenda J. Butler  - setup
8195ce14c3SBrenda J. Butler  - execute
8295ce14c3SBrenda J. Butler  - verify
8395ce14c3SBrenda J. Butler  - teardown
8495ce14c3SBrenda J. Butler
8595ce14c3SBrenda J. ButlerThe setup and teardown stages can run zero or more commands.  The setup
8695ce14c3SBrenda J. Butlerstage does some setup if the test needs it.  The teardown stage undoes
8795ce14c3SBrenda J. Butlerthe setup and returns the system to a "neutral" state so any other test
8895ce14c3SBrenda J. Butlercan be run next.  These two stages require any commands run to return
8995ce14c3SBrenda J. Butlersuccess, but do not otherwise verify the results.
9095ce14c3SBrenda J. Butler
9195ce14c3SBrenda J. ButlerThe execute and verify stages each run one command.  The execute stage
9295ce14c3SBrenda J. Butlertests the return code against one or more acceptable values.  The
9395ce14c3SBrenda J. Butlerverify stage checks the return code for success, and also compares
9495ce14c3SBrenda J. Butlerthe stdout with a regular expression.
9595ce14c3SBrenda J. Butler
9695ce14c3SBrenda J. ButlerEach of the commands in any stage will run in a shell instance.
9795ce14c3SBrenda J. Butler
98*d3fc4eeaSPedro TammelaEach test is an atomic unit. A test that for whatever reason spans multiple test
99*d3fc4eeaSPedro Tammeladefinitions is a bug.
100*d3fc4eeaSPedro Tammela
101*d3fc4eeaSPedro TammelaA test that runs inside a namespace (requires "nsPlugin") will run in parallel
102*d3fc4eeaSPedro Tammelawith other tests.
103*d3fc4eeaSPedro Tammela
104*d3fc4eeaSPedro TammelaTests that use netdevsim or don't run inside a namespace run serially with regards
105*d3fc4eeaSPedro Tammelato each other.
106*d3fc4eeaSPedro Tammela
10795ce14c3SBrenda J. Butler
10876b903eeSLucas BatesUSER-DEFINED CONSTANTS
10976b903eeSLucas Bates----------------------
11076b903eeSLucas Bates
11176b903eeSLucas BatesThe tdc_config.py file contains multiple values that can be altered to suit
11276b903eeSLucas Batesyour needs. Any value in the NAMES dictionary can be altered without affecting
11376b903eeSLucas Batesthe tests to be run. These values are used in the tc commands that will be
11476b903eeSLucas Batesexecuted as part of the test. More will be added as test cases require.
11576b903eeSLucas Bates
11676b903eeSLucas BatesExample:
11776b903eeSLucas Bates	$TC qdisc add dev $DEV1 ingress
11876b903eeSLucas Bates
11995ce14c3SBrenda J. ButlerThe NAMES values are used to substitute into the commands in the test cases.
12095ce14c3SBrenda J. Butler
12176b903eeSLucas Bates
12276b903eeSLucas BatesCOMMAND LINE ARGUMENTS
12376b903eeSLucas Bates----------------------
12476b903eeSLucas Bates
12576b903eeSLucas BatesRun tdc.py -h to see the full list of available arguments.
12676b903eeSLucas Bates
12795ce14c3SBrenda J. ButlerPLUGIN ARCHITECTURE
12895ce14c3SBrenda J. Butler-------------------
12995ce14c3SBrenda J. Butler
13095ce14c3SBrenda J. ButlerThere is now a plugin architecture, and some of the functionality that
13195ce14c3SBrenda J. Butlerwas in the tdc.py script has been moved into the plugins.
13295ce14c3SBrenda J. Butler
13395ce14c3SBrenda J. ButlerThe plugins are in the directory plugin-lib.  The are executed from
13495ce14c3SBrenda J. Butlerdirectory plugins.  Put symbolic links from plugins to plugin-lib,
135489ce2f4SLucas Batesand name them according to the order you want them to run. This is not
136489ce2f4SLucas Batesnecessary if a test case being run requires a specific plugin to work.
13795ce14c3SBrenda J. Butler
13895ce14c3SBrenda J. ButlerExample:
13995ce14c3SBrenda J. Butler
14095ce14c3SBrenda J. Butlerbjb@bee:~/work/tc-testing$ ls -l plugins
14195ce14c3SBrenda J. Butlertotal 4
14295ce14c3SBrenda J. Butlerlrwxrwxrwx  1 bjb  bjb    27 Oct  4 16:12 10-rootPlugin.py -> ../plugin-lib/rootPlugin.py
14395ce14c3SBrenda J. Butlerlrwxrwxrwx  1 bjb  bjb    25 Oct 12 17:55 20-nsPlugin.py -> ../plugin-lib/nsPlugin.py
14495ce14c3SBrenda J. Butler-rwxr-xr-x  1 bjb  bjb     0 Sep 29 15:56 __init__.py
14595ce14c3SBrenda J. Butler
14695ce14c3SBrenda J. ButlerThe plugins are a subclass of TdcPlugin, defined in TdcPlugin.py and
14795ce14c3SBrenda J. Butlermust be called "SubPlugin" so tdc can find them.  They are
14895ce14c3SBrenda J. Butlerdistinguished from each other in the python program by their module
14995ce14c3SBrenda J. Butlername.
15095ce14c3SBrenda J. Butler
15195ce14c3SBrenda J. ButlerThis base class supplies "hooks" to run extra functions.  These hooks are as follows:
15295ce14c3SBrenda J. Butler
15395ce14c3SBrenda J. Butlerpre- and post-suite
15495ce14c3SBrenda J. Butlerpre- and post-case
15595ce14c3SBrenda J. Butlerpre- and post-execute stage
15695ce14c3SBrenda J. Butleradjust-command (runs in all stages and receives the stage name)
15795ce14c3SBrenda J. Butler
15895ce14c3SBrenda J. ButlerThe pre-suite hook receives the number of tests and an array of test ids.
15995ce14c3SBrenda J. ButlerThis allows you to dump out the list of skipped tests in the event of a
16095ce14c3SBrenda J. Butlerfailure during setup or teardown stage.
16195ce14c3SBrenda J. Butler
16295ce14c3SBrenda J. ButlerThe pre-case hook receives the ordinal number and test id of the current test.
16395ce14c3SBrenda J. Butler
16495ce14c3SBrenda J. ButlerThe adjust-command hook receives the stage id (see list below) and the
16595ce14c3SBrenda J. Butlerfull command to be executed.  This allows for last-minute adjustment
16695ce14c3SBrenda J. Butlerof the command.
16795ce14c3SBrenda J. Butler
16895ce14c3SBrenda J. ButlerThe stages are identified by the following strings:
16995ce14c3SBrenda J. Butler
17095ce14c3SBrenda J. Butler  - pre  (pre-suite)
17195ce14c3SBrenda J. Butler  - setup
17295ce14c3SBrenda J. Butler  - command
17395ce14c3SBrenda J. Butler  - verify
17495ce14c3SBrenda J. Butler  - teardown
17595ce14c3SBrenda J. Butler  - post (post-suite)
17695ce14c3SBrenda J. Butler
17795ce14c3SBrenda J. Butler
17895ce14c3SBrenda J. ButlerTo write a plugin, you need to inherit from TdcPlugin in
17995ce14c3SBrenda J. ButlerTdcPlugin.py.  To use the plugin, you have to put the
18095ce14c3SBrenda J. Butlerimplementation file in plugin-lib, and add a symbolic link to it from
18195ce14c3SBrenda J. Butlerplugins.  It will be detected at run time and invoked at the
18295ce14c3SBrenda J. Butlerappropriate times.  There are a few examples in the plugin-lib
18395ce14c3SBrenda J. Butlerdirectory:
18495ce14c3SBrenda J. Butler
18595ce14c3SBrenda J. Butler  - rootPlugin.py:
18695ce14c3SBrenda J. Butler      implements the enforcement of running as root
18795ce14c3SBrenda J. Butler  - nsPlugin.py:
188489ce2f4SLucas Bates      sets up a network namespace and runs all commands in that namespace,
189489ce2f4SLucas Bates      while also setting up dummy devices to be used in testing.
19095ce14c3SBrenda J. Butler  - valgrindPlugin.py
19195ce14c3SBrenda J. Butler      runs each command in the execute stage under valgrind,
19295ce14c3SBrenda J. Butler      and checks for leaks.
19395ce14c3SBrenda J. Butler      This plugin will output an extra test for each test in the test file,
19495ce14c3SBrenda J. Butler      one is the existing output as to whether the test passed or failed,
19595ce14c3SBrenda J. Butler      and the other is a test whether the command leaked memory or not.
19695ce14c3SBrenda J. Butler      (This one is a preliminary version, it may not work quite right yet,
19795ce14c3SBrenda J. Butler      but the overall template is there and it should only need tweaks.)
19876b903eeSLucas Bates
19976b903eeSLucas Bates
20076b903eeSLucas BatesACKNOWLEDGEMENTS
20176b903eeSLucas Bates----------------
20276b903eeSLucas Bates
20376b903eeSLucas BatesThanks to:
20476b903eeSLucas Bates
20576b903eeSLucas BatesJamal Hadi Salim, for providing valuable test cases
20676b903eeSLucas BatesKeara Leibovitz, who wrote the CLI test driver that I used as a base for the
20776b903eeSLucas Bates   first version of the tc testing suite. This work was presented at
20876b903eeSLucas Bates   Netdev 1.2 Tokyo in October 2016.
20976b903eeSLucas BatesSamir Hussain, for providing help while I dove into Python for the first time
21076b903eeSLucas Bates    and being a second eye for this code.
211