1Using OpenSSL Tests 2=================== 3 4After a successful build, and before installing, the libraries should be tested. 5Run: 6 7 $ make test # Unix 8 $ mms test ! OpenVMS 9 $ nmake test # Windows 10 11**Warning:** you MUST run the tests from an unprivileged account 12(or disable your privileges temporarily if your platform allows it). 13 14If some tests fail, take a look at the section Test Failures below. 15 16Test Failures 17------------- 18 19If some tests fail, look at the output. There may be reasons for the failure 20that isn't a problem in OpenSSL itself (like an OS malfunction or a Perl issue). 21You may want increased verbosity, that can be accomplished like this: 22 23Full verbosity, showing full output of all successful and failed test cases 24(`make` macro `VERBOSE` or `V`): 25 26 $ make V=1 test # Unix 27 $ mms /macro=(V=1) test ! OpenVMS 28 $ nmake V=1 test # Windows 29 30Verbosity on failed (sub-)tests only 31(`VERBOSE_FAILURE` or `VF` or `REPORT_FAILURES`): 32 33 $ make test VF=1 34 35Verbosity on failed (sub-)tests, in addition progress on succeeded (sub-)tests 36(`VERBOSE_FAILURE_PROGRESS` or `VFP` or `REPORT_FAILURES_PROGRESS`): 37 38 $ make test VFP=1 39 40If you want to run just one or a few specific tests, you can use 41the make variable TESTS to specify them, like this: 42 43 $ make TESTS='test_rsa test_dsa' test # Unix 44 $ mms/macro="TESTS=test_rsa test_dsa" test ! OpenVMS 45 $ nmake TESTS="test_rsa test_dsa" test # Windows 46 47And of course, you can combine (Unix examples shown): 48 49 $ make test TESTS='test_rsa test_dsa' VF=1 50 $ make test TESTS="test_cmp_*" VFP=1 51 52You can find the list of available tests like this: 53 54 $ make list-tests # Unix 55 $ mms list-tests ! OpenVMS 56 $ nmake list-tests # Windows 57 58Have a look at the manual for the perl module Test::Harness to 59see what other HARNESS_* variables there are. 60 61To report a bug please open an issue on GitHub, at 62<https://github.com/openssl/openssl/issues>. 63 64For more details on how the `make` variables `TESTS` can be used, 65see section Running Selected Tests below. 66 67Running Selected Tests 68---------------------- 69 70The `make` variable `TESTS` supports a versatile set of space separated tokens 71with which you can specify a set of tests to be performed. With a "current 72set of tests" in mind, initially being empty, here are the possible tokens: 73 74 alltests The current set of tests becomes the whole set of available 75 tests (as listed when you do 'make list-tests' or similar). 76 77 xxx Adds the test 'xxx' to the current set of tests. 78 79 -xxx Removes 'xxx' from the current set of tests. If this is the 80 first token in the list, the current set of tests is first 81 assigned the whole set of available tests, effectively making 82 this token equivalent to TESTS="alltests -xxx". 83 84 nn Adds the test group 'nn' (which is a number) to the current 85 set of tests. 86 87 -nn Removes the test group 'nn' from the current set of tests. 88 If this is the first token in the list, the current set of 89 tests is first assigned the whole set of available tests, 90 effectively making this token equivalent to 91 TESTS="alltests -xxx". 92 93Also, all tokens except for "alltests" may have wildcards, such as *. 94(on Unix and Windows, BSD style wildcards are supported, while on VMS, 95it's VMS style wildcards) 96 97### Examples 98 99Run all tests except for the fuzz tests: 100 101 $ make TESTS='-test_fuzz*' test 102 103or, if you want to be explicit: 104 105 $ make TESTS='alltests -test_fuzz*' test 106 107Run all tests that have a name starting with "test_ssl" but not those 108starting with "test_ssl_": 109 110 $ make TESTS='test_ssl* -test_ssl_*' test 111 112Run only test group 10: 113 114 $ make TESTS='10' test 115 116Run all tests except the slow group (group 99): 117 118 $ make TESTS='-99' test 119 120Run all tests in test groups 80 to 99 except for tests in group 90: 121 122 $ make TESTS='[89]? -90' test 123 124To run specific fuzz tests you can use for instance: 125 126 $ make test TESTS='test_fuzz_cmp test_fuzz_cms' 127 128To stochastically verify that the algorithm that produces uniformly distributed 129random numbers is operating correctly (with a false positive rate of 0.01%): 130 131 $ ./util/wrap.sh test/bntest -stochastic 132 133Running Tests in Parallel 134------------------------- 135 136By default the test harness will execute the selected tests sequentially. 137Depending on the platform characteristics, running more than one test job in 138parallel may speed up test execution. 139This can be requested by setting the `HARNESS_JOBS` environment variable to a 140positive integer value. This specifies the maximum number of test jobs to run in 141parallel. 142 143Depending on the Perl version different strategies could be adopted to select 144which test recipes can be run in parallel. In recent versions of Perl, unless 145specified otherwise, any task can be run in parallel. Consult the documentation 146for `TAP::Harness` to know more. 147 148To run up to four tests in parallel at any given time: 149 150 $ make HARNESS_JOBS=4 test 151 152Randomisation of Test Ordering 153------------------------------ 154 155By default, the test harness will execute tests in the order they were added. 156By setting the `OPENSSL_TEST_RAND_ORDER` environment variable to zero, the 157test ordering will be randomised. If a randomly ordered test fails, the 158seed value used will be reported. Setting the `OPENSSL_TEST_RAND_ORDER` 159environment variable to this value will rerun the tests in the same 160order. This assures repeatability of randomly ordered test runs. 161This repeatability is independent of the operating system, processor or 162platform used. 163 164To randomise the test ordering: 165 166 $ make OPENSSL_TEST_RAND_ORDER=0 test 167 168To run the tests using the order defined by the random seed `42`: 169 170 $ make OPENSSL_TEST_RAND_ORDER=42 test 171