1.\" SPDX-License-Identifier: CDDL-1.0 2.\" 3.\" This file and its contents are supplied under the terms of the 4.\" Common Development and Distribution License ("CDDL"), version 1.0. 5.\" You may only use this file in accordance with the terms of version 6.\" 1.0 of the CDDL. 7.\" 8.\" A full copy of the text of the CDDL should have accompanied this 9.\" source. A copy of the CDDL is also available via the Internet at 10.\" http://www.illumos.org/license/CDDL. 11.\" 12.\" Copyright (c) 2012 by Delphix. All rights reserved. 13.\" 14.Dd May 26, 2021 15.Dt RUN 1 16.Os 17. 18.Sh NAME 19.Nm run 20.Nd find, execute, and log the results of tests 21.Sh SYNOPSIS 22.Nm 23.Op Fl dgq 24.Op Fl o Ar outputdir 25.Op Fl pP Ar script 26.Op Fl t seconds 27.Op Fl uxX Ar username 28.Ar pathname Ns No … 29.Pp 30.Nm 31.Fl w Ar runfile 32.Op Fl gq 33.Op Fl o Ar outputdir 34.Op Fl pP Ar script 35.Op Fl t seconds 36.Op Fl uxX Ar username 37.Ar pathname Ns No … 38.Pp 39.Nm 40.Fl c Ar runfile 41.Op Fl dq 42.Pp 43.Nm 44.Op Fl h 45. 46.Sh DESCRIPTION 47.Nm 48command has three basic modes of operation. 49With neither 50.Fl c 51nor 52.Fl w , 53.Nm 54processes the arguments provided on 55the command line, adding them to the list for this run. 56If a specified 57.Ar pathname 58is an executable file, it is added as a test. 59If a specified 60.Ar pathname 61is a directory, the behavior depends upon the presence of 62.Fl g . 63If 64.Fl g 65is specified, the directory is treated as a test group. 66See the section on 67.Sy Test Groups 68below. 69Without 70.Fl g , 71.Nm 72simply descends into the directory looking for executable files. 73The tests are then executed, and the results are logged. 74.Pp 75With 76.Fl w , 77.Nm 78finds tests in the manner described above. 79Rather than executing the tests and logging the results, the test configuration 80is stored in a 81.Ar runfile , 82which can be used in future invocations, or edited 83to modify which tests are executed and which options are applied. 84Options included on the command line with 85.Fl w 86become defaults in the 87.Ar runfile . 88.Pp 89With 90.Fl c , 91.Nm 92parses a 93.Ar runfile , 94which can specify a series of tests and test groups to be executed. 95The tests are then executed, and the results are logged. 96. 97.Ss Test Groups 98A test group is comprised of a set of executable files, all of which exist in 99one directory. 100The options specified on the command line or in a 101.Ar runfile 102apply to individual tests in the group. 103The exception is options pertaining to pre and post scripts, which act on all 104tests as a group. 105Rather than running before and after each test, 106these scripts are run only once each at the start and end of the test group. 107.Ss Test Execution 108The specified tests run serially, and are typically assigned results according 109to exit values. 110Tests that exit zero and non-zero are marked 111.Sy PASS 112and 113.Sy FAIL , 114respectively. 115When a pre script fails for a test group, only the post script is executed, 116and the remaining tests are marked 117.Sy SKIPPED . 118Any test that exceeds 119its 120.Ar timeout 121is terminated, and marked 122.Sy KILLED . 123.Pp 124By default, tests are executed with the credentials of the 125.Nm 126script. 127Executing tests with other credentials is done via 128.Xr sudo 1m , 129which must 130be configured to allow execution without prompting for a password. 131Environment variables from the calling shell are available to individual tests. 132During test execution, the working directory is changed to 133.Ar outputdir . 134. 135.Ss Output Logging 136By default, 137.Nm 138will print one line on standard output at the conclusion 139of each test indicating the test name, result and elapsed time. 140Additionally, for each invocation of 141.Nm , 142a directory is created using the ISO 8601 date format. 143Within this directory is a file named 144.Sy log 145containing all the 146test output with timestamps, and a directory for each test. 147Within the test directories, there is one file each for standard output, 148standard error and merged output. 149The default location for the 150.Ar outputdir 151is 152.Pa /var/tmp/test_results . 153.Ss "Runfiles" 154The 155.Ar runfile 156is an INI-style configuration file that describes a test run. 157The file has one section named 158.Sy DEFAULT , 159which contains configuration option 160names and their values in 161.Sy name No = Ar value 162format. 163The values in this section apply to all the subsequent sections, 164unless they are also specified there, in which case the default is overridden. 165The remaining section names are the absolute pathnames of files and directories, 166describing tests and test groups respectively. 167The legal option names are: 168.Bl -tag -width "tests = ['filename', …]" 169.It Sy outputdir No = Ar pathname 170The name of the directory that holds test logs. 171.It Sy pre No = Ar script 172Run 173.Ar script 174prior to the test or test group. 175.It Sy pre_user No = Ar username 176Execute the pre script as 177.Ar username . 178.It Sy post No = Ar script 179Run 180.Ar script 181after the test or test group. 182.It Sy post_user No = Ar username 183Execute the post script as 184.Ar username . 185.It Sy quiet No = Sy True Ns | Ns Sy False 186If 187.Sy True , 188only the results summary is printed to standard out. 189.It Sy tests No = [ Ns Ar 'filename' , No … ] 190Specify a list of 191.Ar filenames 192for this test group. 193Only the basename of the absolute path is required. 194This option is only valid for test groups, and each 195.Ar filename 196must be single quoted. 197.It Sy timeout No = Ar n 198A timeout value of 199.Ar n 200seconds. 201.It Sy user No = Ar username 202Execute the test or test group as 203.Ar username . 204.El 205. 206.Sh OPTIONS 207.Bl -tag -width "-o outputdir" 208.It Fl c Ar runfile 209Specify a 210.Ar runfile 211to be consumed by the run command. 212.It Fl d 213Dry run mode. 214Execute no tests, but print a description of each test that would have been run. 215.It Fl m 216Enable kmemleak reporting (Linux only) 217.It Fl g 218Create test groups from any directories found while searching for tests. 219.It Fl o Ar outputdir 220Specify the directory in which to write test results. 221.It Fl p Ar script 222Run 223.Ar script 224prior to any test or test group. 225.It Fl P Ar script 226Run 227.Ar script 228after any test or test group. 229.It Fl q 230Print only the results summary to the standard output. 231.It Fl s Ar script 232Run 233.Ar script 234as a failsafe after any test is killed. 235.It Fl S Ar username 236Execute the failsafe script as 237.Ar username . 238.It Fl t Ar n 239Specify a timeout value of 240.Ar n 241seconds per test. 242.It Fl u Ar username 243Execute tests or test groups as 244.Ar username . 245.It Fl w Ar runfile 246Specify the name of the 247.Ar runfile 248to create. 249.It Fl x Ar username 250Execute the pre script as 251.Ar username . 252.It Fl X Ar username 253Execute the post script as 254.Ar username . 255.El 256. 257.Sh EXAMPLES 258.Bl -tag -width "-h" 259.It Sy Example 1 : No Running ad-hoc tests . 260This example demonstrates the simplest invocation of 261.Nm . 262.Bd -literal 263.No % Nm run Ar my-tests 264Test: /home/jkennedy/my-tests/test-01 [00:02] [PASS] 265Test: /home/jkennedy/my-tests/test-02 [00:04] [PASS] 266Test: /home/jkennedy/my-tests/test-03 [00:01] [PASS] 267 268Results Summary 269PASS 3 270 271Running Time: 00:00:07 272Percent passed: 100.0% 273Log directory: /var/tmp/test_results/20120923T180654 274.Ed 275.It Sy Example 2 : No Creating a Ar runfile No for future use . 276This example demonstrates creating a 277.Ar runfile 278with non-default options. 279.Bd -literal 280.No % Nm run Fl p Ar setup Fl x Ar root Fl g Fl w Ar new-tests.run Ar new-tests 281.No % Nm cat Pa new-tests.run 282[DEFAULT] 283pre = setup 284post_user = 285quiet = False 286user = 287timeout = 60 288post = 289pre_user = root 290outputdir = /var/tmp/test_results 291 292[/home/jkennedy/new-tests] 293tests = ['test-01', 'test-02', 'test-03'] 294.Ed 295.El 296. 297.Sh SEE ALSO 298.Xr sudo 1m 299