1.\" Copyright 2011 The Kyua Authors. 2.\" All rights reserved. 3.\" 4.\" Redistribution and use in source and binary forms, with or without 5.\" modification, are permitted provided that the following conditions are 6.\" met: 7.\" 8.\" * Redistributions of source code must retain the above copyright 9.\" notice, this list of conditions and the following disclaimer. 10.\" * Redistributions in binary form must reproduce the above copyright 11.\" notice, this list of conditions and the following disclaimer in the 12.\" documentation and/or other materials provided with the distribution. 13.\" * Neither the name of Google Inc. nor the names of its contributors 14.\" may be used to endorse or promote products derived from this software 15.\" without specific prior written permission. 16.\" 17.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 18.\" "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT 19.\" LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR 20.\" A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT 21.\" OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, 22.\" SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT 23.\" LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, 24.\" DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY 25.\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT 26.\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 27.\" OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 28.Dd May 12, 2015 29.Dt KYUA 1 30.Os 31.Sh NAME 32.Nm kyua 33.Nd Testing framework for infrastructure software 34.Sh SYNOPSIS 35.Nm 36.Op Fl -config Ar file 37.Op Fl -logfile Ar file 38.Op Fl -loglevel Ar level 39.Op Fl -variable Ar name=value 40.Ar command 41.Op Ar command_options 42.Op Ar command_arguments 43.Sh DESCRIPTION 44.Em If you are here looking for details on how to run the test suite in 45.Pa /usr/tests 46.Em ( or 47.Pa __TESTSDIR__ ) , 48.Em please start by reading the 49.Xr tests 7 50.Em manual page that should be supplied by your system . 51.Pp 52Kyua is a testing framework for infrastructure software, originally 53designed to equip BSD-based operating systems with a test suite. 54This means that Kyua is lightweight and simple, and that Kyua integrates well 55with various build systems and continuous integration frameworks. 56.Pp 57Kyua features an expressive test suite definition language, a safe 58runtime engine for test suites and a powerful report generation engine. 59.Pp 60Kyua is for both developers and users, from the developer applying a 61simple fix to a library to the system administrator deploying a new 62release on a production machine. 63.Pp 64Kyua is able to execute test programs written with a plethora of testing 65libraries and languages. 66The test program library of choice is ATF, which 67.Nm Ns 's 68design originated from. 69However, framework-less test programs and TAP-compliant test programs can also 70be executed through 71.Nm 72.Ss Overview 73As can be observed in the synopsis, the interface of 74.Nm 75implements a common subcommand-based interface. 76The arguments to the tool specify, in this order: a set of common options 77that all the commands accept, a required 78.Ar command 79name that specifies what 80.Nm 81should do, and 82a set of possibly-optional 83.Ar command_options 84and 85.Ar command_arguments 86that are specific to the chosen command. 87.Pp 88The following options are recognized by all the commands. 89Keep in mind that these must always be specified before the command name. 90.Bl -tag -width XX 91.It Fl -config Ar path , Fl c Ar path 92Specifies the configuration file to process, which must be in the format 93described in 94.Xr kyua.conf 5 . 95The special value 96.Sq none 97explicitly disables the loading of any configuration file. 98.Pp 99Defaults to 100.Pa ~/.kyua/kyua.conf 101if it exists, otherwise to 102.Pa __CONFDIR__/kyua.conf 103if it exists, 104or else to 105.Sq none . 106.It Fl -logfile Ar path 107Specifies the location of the file to which 108.Nm 109will log run time events useful for postmortem debugging. 110.Pp 111The default depends on different environment variables as described in 112.Sx Logging , 113but typically the file will be stored within the user's home directory. 114.It Fl -loglevel Ar level 115Specifies the maximum logging level to record in the log file. 116See 117.Sx Logging 118for more details. 119.Pp 120The default is 121.Sq info . 122.It Fl -variable Ar name=value , Fl v Ar name=value 123Sets the 124.Ar name 125configuration variable to 126.Ar value . 127The values set through this option have preference over the values set in the 128configuration file. 129.Pp 130The specified variable can either be a builtin variable or a test-suite 131specific variable. 132See 133.Xr kyua.conf 5 134for more details. 135.El 136.Pp 137The following commands are generic and do not have any relation to the execution 138of tests or the inspection of their results: 139.Bl -tag -width reportXjunitXX -offset indent 140.It Ar about 141Shows general program information. 142See 143.Xr kyua-about 1 . 144.It Ar config 145Inspects the values of the configuration variables. 146See 147.Xr kyua-config 1 . 148.It Ar db-exec 149Executes an arbitrary SQL statement on a results file and prints the 150resulting table. 151See 152.Xr kyua-db-exec 1 . 153.It Ar help 154Shows usage information. 155See 156.Xr kyua-help 1 . 157.El 158.Pp 159The following commands are used to generate reports based on the data previously 160recorded in a results file: 161.Bl -tag -width reportXjunitXX -offset indent 162.It Ar report 163Generates a plaintext report. 164Combined with its 165.Fl -verbose 166flag and the ability to only display specific test cases, this command can also 167be used to debug test failures post-facto on the console. 168See 169.Xr kyua-report 1 . 170.It Ar report-html 171Generates an HTML report. 172See 173.Xr kyua-report-html 1 . 174.It Ar report-junit 175Generates a JUnit report. 176See 177.Xr kyua-report-junit 1 . 178.El 179.Pp 180The following commands are used to interact with a test suite: 181.Bl -tag -width reportXjunitXX -offset indent 182.It Ar debug 183Executes a single test case in a controlled environment for debugging purposes. 184See 185.Xr kyua-debug 1 . 186.It Ar list 187Lists test cases defined in a test suite by a 188.Xr kyuafile 5 189and, optionally, displays their metadata. 190See 191.Xr kyua-list 1 . 192.It Ar test 193Runs tests defined in a test suite by a 194.Xr kyuafile 5 . 195See 196.Xr kyua-test 1 . 197.El 198.Ss Logging 199.Nm 200has a logging facility that collects all kinds of events at run time. 201These events are always logged to a file so that the log is available when 202it is most needed: right after a non-reproducible problem happens. 203The only way to disable logging is by sending the log to 204.Pa /dev/null . 205.Pp 206The location of the log file can be manually specified with the 207.Fl -logfile 208option, which applies to all commands. 209If no file is explicitly specified, the location of the log files is chosen in 210this order: 211.Bl -enum -offset indent 212.It 213.Pa ${HOME}/.kyua/logs/ 214if 215.Va HOME 216is defined. 217.It 218.Pa ${TMPDIR}/ 219if 220.Va TMPDIR 221is defined. 222.It 223.Pa /tmp/ . 224.El 225.Pp 226And the default naming scheme of the log files is: 227.Sq <progname>.<timestamp>.log . 228.Pp 229The messages stored in the log file have a level (or severity) attached to 230them. 231These are: 232.Bl -tag -width warningXX -offset indent 233.It error 234Fatal error messages. 235The program generally terminates after these, either in a clean manner or by 236crashing. 237.It warning 238Non-fatal error messages. 239These generally report a condition that must be addressed but the application 240can continue to run. 241.It info 242Informational messages. 243These tell the user what the program was doing at a general level of 244operation. 245.It debug 246Detailed informational messages. 247These are often useful when debugging problems in the application, as they 248contain lots of internal details. 249.El 250.Pp 251The default log level is 252.Sq info 253unless explicitly overridden with 254.Fl -loglevel . 255.Pp 256The log file is a plain text file containing one line per log record. 257The format of each line is as follows: 258.Bd -literal -offset indent 259timestamp entry_type pid file:line: message 260.Ed 261.Pp 262.Ar entry_type 263can be one of: 264.Sq E 265for an error, 266.Sq W 267for a warning, 268.Sq I 269for an informational message and 270.Sq D 271for a debug message. 272.Ss Bug reporting 273If you think you have encountered a bug in 274.Nm , 275please take the time to let the developers know about it. 276This will ensure that the bug is addressed and potentially fixed in the next 277Kyua release. 278.Pp 279The first step in reporting a bug is to check if there already is a similar 280bug in the database. 281You can check what issues are currently in the database by going to: 282.Bd -literal -offset indent 283https://github.com/jmmv/kyua/issues/ 284.Ed 285.Pp 286If there is no existing issue that describes an issue similar to the 287one you are experiencing, you can open a new one by visiting: 288.Bd -literal -offset indent 289https://github.com/jmmv/kyua/issues/new/ 290.Ed 291.Pp 292When doing so, please include as much detail as possible. 293Among other things, explain what operating system and platform you are running 294.Nm 295on, what were you trying to do, what exact messages you saw on the screen, 296how did you expect the program to behave, and any other details that you 297may find relevant. 298.Pp 299Also, please include a copy of the log file corresponding to the problem 300you are experiencing. 301Unless you have changed the location of the log files, you can most likely 302find them in 303.Pa ~/.kyua/logs/ . 304If the problem is reproducible, it is good idea to regenerate the log file 305with an increased log level so as to provide more information. 306For example: 307.Bd -literal -offset indent 308$ kyua --logfile=problem.log --loglevel=debug \\ 309 [rest of the command line] 310.Ed 311.Sh ENVIRONMENT 312The following variables are recognized and can be freely tuned by the end user: 313.Bl -tag -width COLUMNSXX 314.It Va COLUMNS 315The width of the screen, in number of characters. 316.Nm 317uses this to wrap long lines. 318If not present, the width of the screen is determined from the terminal 319stdout is connected to, and, if the guessing fails, this defaults to infinity. 320.It Va HOME 321Path to the user's home directory. 322.Nm 323uses this location to determine paths to configuration files and default log 324files. 325.It Va TMPDIR 326Path to the system-wide temporary directory. 327.Nm 328uses this location to place the work directory of test cases, among other 329things. 330.Pp 331The default value of this variable depends on the operating system. 332In general, it is 333.Pa /tmp . 334.El 335.Pp 336The following variables are also recognized, but you should not need to set them 337during normal operation. 338They are only provided to override the value of built-in values, which is useful 339when testing 340.Nm 341itself: 342.Bl -tag -width KYUAXCONFDIRXX 343.It Va KYUA_CONFDIR 344Path to the system-wide configuration files for 345.Nm . 346.Pp 347Defaults to 348.Pa __CONFDIR__ . 349.It Va KYUA_DOCDIR 350Path to the location of installed documentation. 351.Pp 352Defaults to 353.Pa __DOCDIR__ . 354.It Va KYUA_MISCDIR 355Path to the location of the installed miscellaneous scripts and data 356files provided by 357.Nm . 358.Pp 359Defaults to 360.Pa __MISCDIR__ . 361.It Va KYUA_STOREDIR 362Path to the location of the installed store support files; e.g., the 363directory containing the SQL database schema. 364.Pp 365Defaults to 366.Pa __STOREDIR__ . 367.El 368.Sh FILES 369.Bl -tag -width XXXX 370.It Pa ~/.kyua/store/ 371Default location for the results files. 372.It Pa ~/.kyua/kyua.conf 373User-specific configuration file. 374.It Pa ~/.kyua/logs/ 375Default location for the collected log files. 376.It Pa __CONFDIR__/kyua.conf 377System-wide configuration file. 378.El 379.Sh EXIT STATUS 380.Nm 381returns 0 on success, 1 on a controlled error condition in the given 382subcommand, 2 on a general unexpected error and 3 on a usage error. 383.Pp 384The documentation of the subcommands in the corresponding manual pages only 385details the difference between a successful exit (0) and the detection of a 386controlled error (1). 387Even though when those manual pages do not describe any other exit statuses, 388codes above 1 can be returned. 389.Sh SEE ALSO 390.Xr kyua.conf 5 , 391.Xr kyuafile 5 , 392.Xr atf 7 , 393.Xr tests 7 394.Sh AUTHORS 395The original author of 396.Nm 397was 398.An Julio Merino . 399.Pp 400For more details on the people that made 401.Nm 402possible and the license terms, run: 403.Bd -literal -offset indent 404$ kyua about 405.Ed 406