xref: /freebsd/crypto/openssl/test/README-external.md (revision 59c8e88e72633afbc47a4ace0d2170d00d51f7dc) 
1Running external test suites with OpenSSL
2=========================================
3
4It is possible to integrate external test suites into OpenSSL's `make test`.
5This capability is considered a developer option and does not work on all
6platforms.
7
8Python PYCA/Cryptography test suite
9===================================
10
11This python test suite runs cryptographic tests with a local OpenSSL build as
12the implementation.
13
14First checkout the `PYCA/Cryptography` module into `./pyca-cryptography` using:
15
16    $ git submodule update --init
17
18Then configure/build OpenSSL compatible with the python module:
19
20    $ ./config shared enable-external-tests
21    $ make
22
23The tests will run in a python virtual environment which requires virtualenv
24to be installed.
25
26    $ make test VERBOSE=1 TESTS=test_external_pyca
27
28Test failures and suppressions
29------------------------------
30
31Some tests target older (<=1.0.2) versions so will not run. Other tests target
32other crypto implementations so are not relevant. Currently no tests fail.
33
34krb5 test suite
35===============
36
37Much like the PYCA/Cryptography test suite, this builds and runs the krb5
38tests against the local OpenSSL build.
39
40You will need a git checkout of krb5 at the top level:
41
42    $ git clone https://github.com/krb5/krb5
43
44krb5's master has to pass this same CI, but a known-good version is
45krb5-1.15.1-final if you want to be sure.
46
47    $ cd krb5
48    $ git checkout krb5-1.15.1-final
49    $ cd ..
50
51OpenSSL must be built with external tests enabled:
52
53    $ ./config enable-external-tests
54    $ make
55
56krb5's tests will then be run as part of the rest of the suite, or can be
57explicitly run (with more debugging):
58
59    $ VERBOSE=1 make TESTS=test_external_krb5 test
60
61Test-failures suppressions
62--------------------------
63
64krb5 will automatically adapt its test suite to account for the configuration
65of your system.  Certain tests may require more installed packages to run.  No
66tests are expected to fail.
67
68GOST engine test suite
69===============
70
71Much like the PYCA/Cryptography test suite, this builds and runs the GOST engine
72tests against the local OpenSSL build.
73
74You will need a git checkout of gost-engine at the top level:
75
76    $ git submodule update --init
77
78Then configure/build OpenSSL enabling external tests:
79
80    $ ./config shared enable-external-tests
81    $ make
82
83GOST engine requires CMake for the build process.
84
85GOST engine tests will then be run as part of the rest of the suite, or can be
86explicitly run (with more debugging):
87
88    $ make test VERBOSE=1 TESTS=test_external_gost_engine
89
90Updating test suites
91====================
92
93To update the commit for any of the above test suites:
94
95- Make sure the submodules are cloned locally:
96
97    $ git submodule update --init --recursive
98
99- Enter subdirectory and pull from the repository (use a specific branch/tag if required):
100
101    $ cd `<submodule-dir>`
102    $ git pull origin master
103
104- Go to root directory, there should be a new git status:
105
106    $ cd ../
107    $ git status
108      ...
109      #       modified:   `<submodule-dir>` (new commits)
110      ...
111
112- Add/commit/push the update
113
114    $ git add `<submodule-dir>`
115    $ git commit -m `"Updated <submodule> to latest commit"`
116    $ git push
117