1Guidelines for test developers 2============================== 3 4How to add recipes 5------------------ 6 7For any test that you want to perform, you write a script located in 8`test/recipes/`, named `{nn}-test_{name}.t`, 9where `{nn}` is a two digit number and 10`{name}` is a unique name of your choice. 11 12Please note that if a test involves a new testing executable, you will need to 13do some additions in test/build.info. Please refer to the section 14["Changes to test/build.info"](README.md#changes-to-testbuildinfo) below. 15 16Naming conventions 17------------------ 18 19A test executable is named `test/{name}test.c` 20 21A test recipe is named `test/recipes/{nn}-test_{name}.t`, where `{nn}` is a two 22digit number and `{name}` is a unique name of your choice. 23 24The number `{nn}` is (somewhat loosely) grouped as follows: 25 26 00-04 sanity, internal and essential API tests 27 05-09 individual symmetric cipher algorithms 28 10-14 math (bignum) 29 15-19 individual asymmetric cipher algorithms 30 20-24 openssl commands (some otherwise not tested) 31 25-29 certificate forms, generation and verification 32 30-35 engine and evp 33 60-79 APIs: 34 60 X509 subsystem 35 61 BIO subsystem 36 65 CMP subsystem 37 70 PACKET layer 38 80-89 "larger" protocols (CA, CMS, OCSP, SSL, TSA) 39 90-98 misc 40 99 most time consuming tests [such as test_fuzz] 41 42A recipe that just runs a test executable 43----------------------------------------- 44 45A script that just runs a program looks like this: 46 47 #! /usr/bin/env perl 48 49 use OpenSSL::Test::Simple; 50 51 simple_test("test_{name}", "{name}test", "{name}"); 52 53`{name}` is the unique name you have chosen for your test. 54 55The second argument to `simple_test` is the test executable, and `simple_test` 56expects it to be located in `test/` 57 58For documentation on `OpenSSL::Test::Simple`, 59do `perldoc util/perl/OpenSSL/Test/Simple.pm`. 60 61A recipe that runs a more complex test 62-------------------------------------- 63 64For more complex tests, you will need to read up on Test::More and 65OpenSSL::Test. Test::More is normally preinstalled, do `man Test::More` for 66documentation. For OpenSSL::Test, do `perldoc util/perl/OpenSSL/Test.pm`. 67 68A script to start from could be this: 69 70 #! /usr/bin/env perl 71 72 use strict; 73 use warnings; 74 use OpenSSL::Test; 75 76 setup("test_{name}"); 77 78 plan tests => 2; # The number of tests being performed 79 80 ok(test1, "test1"); 81 ok(test2, "test1"); 82 83 sub test1 84 { 85 # test feature 1 86 } 87 88 sub test2 89 { 90 # test feature 2 91 } 92 93Changes to test/build.info 94-------------------------- 95 96Whenever a new test involves a new test executable you need to do the 97following (at all times, replace {NAME} and {name} with the name of your 98test): 99 100 * add `{name}` to the list of programs under `PROGRAMS_NO_INST` 101 102 * create a three line description of how to build the test, you will have 103 to modify the include paths and source files if you don't want to use the 104 basic test framework: 105 106 SOURCE[{name}]={name}.c 107 INCLUDE[{name}]=.. ../include ../apps/include 108 DEPEND[{name}]=../libcrypto libtestutil.a 109 110Generic form of C test executables 111---------------------------------- 112 113 #include "testutil.h" 114 115 static int my_test(void) 116 { 117 int testresult = 0; /* Assume the test will fail */ 118 int observed; 119 120 observed = function(); /* Call the code under test */ 121 if (!TEST_int_eq(observed, 2)) /* Check the result is correct */ 122 goto end; /* Exit on failure - optional */ 123 124 testresult = 1; /* Mark the test case a success */ 125 end: 126 cleanup(); /* Any cleanup you require */ 127 return testresult; 128 } 129 130 int setup_tests(void) 131 { 132 ADD_TEST(my_test); /* Add each test separately */ 133 return 1; /* Indicates success. Return 0 */ 134 /* to produce an error with a */ 135 /* usage message and -1 for */ 136 /* failure to set up with no */ 137 /* usage message. */ 138 } 139 140You should use the `TEST_xxx` macros provided by `testutil.h` to test all failure 141conditions. These macros produce an error message in a standard format if the 142condition is not met (and nothing if the condition is met). Additional 143information can be presented with the `TEST_info` macro that takes a `printf` 144format string and arguments. `TEST_error` is useful for complicated conditions, 145it also takes a `printf` format string and argument. In all cases the `TEST_xxx` 146macros are guaranteed to evaluate their arguments exactly once. This means 147that expressions with side effects are allowed as parameters. Thus, 148 149 if (!TEST_ptr(ptr = OPENSSL_malloc(..))) 150 151works fine and can be used in place of: 152 153 ptr = OPENSSL_malloc(..); 154 if (!TEST_ptr(ptr)) 155 156The former produces a more meaningful message on failure than the latter. 157 158Note that the test infrastructure automatically sets up all required environment 159variables (such as `OPENSSL_MODULES`, `OPENSSL_CONF`, etc.) for the tests. 160Individual tests may choose to override the default settings as required. 161