README.md
1RADIX Test Framework
2====================
3
4Purpose
5-------
6
7This directory contains the RADIX test framework, which is a six-dimension
8script-driven facility intended to facilitate execution of
9
10- multi-stream
11- multi-client
12- multi-server
13- multi-thread
14- multi-process (in future)
15- multi-node (in future)
16
17test vignettes for network protocol testing applications. While it is currently
18used for QUIC, it has been designed to be agnostic so that it can be adapted to
19other protocols in future if desired.
20
21In particular, unlike the older multistream test framework, it does not assume a
22single client and a single server. Examples of vignettes designed to be
23supported by the RADIX test framework in future include:
24
25- single client ↔ single server
26- multiple clients ↔ single server
27- single client ↔ multiple servers
28- multiple clients ↔ multiple servers
29
30“Multi-process” and “multi-node” means there has been some consideration
31given to support of multi-process and multi-node testing in the future, though
32this is not currently supported.
33
34Differences to `quic_multistream_test`
35--------------------------------------
36
37The RADIX test features the following improvements relative to the
38`quic_multistream_test` framework:
39
40- Due to usage of the new QUIC server API, opcodes are no longer duplicated
41 into “C” and “S” variants. There is symmetry between all endpoints in this
42 regard. The legacy `QUIC_TSERVER` facility is not used and will eventually
43 be retired once other test suites no longer rely on it.
44
45- The framework is not limited to two objects (client and server),
46 but can instead instantiate an arbitrary number of SSL objects and pass these
47 to operations. Other kinds of object could be supported in future if needed.
48
49- Scripts are now generated dynamically at launch time by functions rather
50 than hardcoding them into the executable. This has the advantage that scripts
51 can be generated dynamically. In particular, this allows very long
52 procedurally generated test scripts which would be impractical to write
53 by hand. This can be used for stress testing, for example.
54
55- As a result of the fact that scripts are now generated dynamically, the
56 in-memory representation of script operations has been improved as the ability
57 to write a script operation as an initializer list is no longer required. The
58 new representation is simpler, more compact, and more flexible. A stack-based
59 approach allows arbitrary argument types to be passed as operands to an
60 operation, rather than having a fixed number of operands of fixed type for all
61 script operations.
62
63- Scripts are now named, rather than numbered, giving more useful debug output,
64 and hopefully reducing the frequency of merge conflicts.
65
66- Debug logging has in general been significantly improved and has been designed
67 to be accessible and useful.
68
69- Logging of child threads is now buffered and printed after a test is complete,
70 which avoids non-deterministic interleaving of test output.
71
72- The number of core opcodes for the interpreter has been dramatically reduced
73 and now the vast majority of test operations are performed via `OP_FUNC`,
74 which is similar to `OP_CHECK`. This change is largely transparent to the
75 test developer.
76
77- In the future, multi-process or multi-node testing will be supported.
78 The expectation is that multi-node testing will be facilitated by having the
79 master process invoke a hook shell script which is responsible for propping up
80 and tearing down the additional nodes. Writing a suitable hook script will be
81 left as an exercise to the test infrastructure maintainer. This abstracts the
82 test framework from the details of a specific infrastructure environment and
83 its management tools (VMs, containers, etc.).
84
85- The core test interpreter is designed to be agnostic to QUIC and could be
86 used for testing other protocols in the future. There is a clean, layered
87 design which draws a clear distinction between the core interpreter,
88 protocol-specific bindings and support code, test operation definitions, and
89 script definitions.
90
91- It is no longer needed to explicitly set the ALPN using an opcode when
92 establishing a connection using QUIC. Since ALPN is required for QUIC, the
93 first opcode of every test was used to set the ALPN to the same string, which
94 was redundant. This is now done automatically.
95
96- An explicit `OP_END` is no longer needed.
97
98Architecture
99------------
100
101The RADIX test suite framework is built in four layers:
102
103- **TERP** ([terp.c](./terp.c)), a protocol-agnostic stack-based script
104 interpreter for interruptible execution of test vignettes;
105
106- the **QUIC bindings** ([quic_bindings.c](./quic_bindings.c)), which defines
107 QUIC-specific test framework;
108
109- the **QUIC operations** ([quic_ops.c](./quic_ops.c)), which define specific
110 test operations for TERP which can be invoked by QUIC unit tests on top of the
111 basic infrastructure defined by the QUIC bindings;
112
113- the QUIC unit tests ([quic_tests.c](./quic_tests.c)), which use the above
114 QUIC bindings.
115