1.\" $NetBSD: tests.kyua.7,v 1.2 2013/07/20 21:39:59 wiz Exp $ 2.\" 3.\" Copyright (c) 2010 The NetBSD Foundation, Inc. 4.\" All rights reserved. 5.\" 6.\" Redistribution and use in source and binary forms, with or without 7.\" modification, are permitted provided that the following conditions 8.\" are met: 9.\" 1. Redistributions of source code must retain the above copyright 10.\" notice, this list of conditions and the following disclaimer. 11.\" 2. Redistributions in binary form must reproduce the above copyright 12.\" notice, this list of conditions and the following disclaimer in the 13.\" documentation and/or other materials provided with the distribution. 14.\" 15.\" THIS SOFTWARE IS PROVIDED BY THE NETBSD FOUNDATION, INC. AND 16.\" CONTRIBUTORS ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, 17.\" INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF 18.\" MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. 19.\" IN NO EVENT SHALL THE FOUNDATION OR CONTRIBUTORS BE LIABLE FOR ANY 20.\" DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL 21.\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE 22.\" GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS 23.\" INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER 24.\" IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR 25.\" OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN 26.\" IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 27.\" 28.Dd August 19, 2020 29.Dt TESTS 7 30.Os 31.Sh NAME 32.Nm tests 33.Nd introduction to the 34.Fx 35Test Suite 36.Sh DESCRIPTION 37The 38.Fx 39Test Suite provides a collection of automated tests for two major purposes. 40On one hand, the test suite aids 41.Em developers 42to detect bugs and regressions when they modify the source tree. 43On the other hand, it allows 44.Em end users 45(and, in particular, system administrators) to verify that fresh installations 46of the 47.Fx 48operating system behave correctly on their hardware platform and also to ensure 49that the system does not suffer from regressions during regular operation and 50maintenance. 51.Pp 52The 53.Fx 54Test Suite can be found in the 55.Pa /usr/tests 56hierarchy. 57.Pp 58This manual page describes how to run the test suite and how to configure 59some of its optional features. 60For information on writing the tests, see 61.Xr atf 7 . 62.Ss Installing the test suite 63If the 64.Pa /usr/tests 65directory is missing, then you will have to enable the build of the test 66suite, rebuild your system and install the results. 67You can do so by setting 68.Sq WITH_TESTS=yes 69in your 70.Pa /etc/src.conf 71file (see 72.Xr src.conf 5 73for details) 74and rebuilding the system as described in 75.Xr build 7 . 76.Ss When to run the tests? 77Before diving into the details of how to run the test suite, here are some 78scenarios in which you should run it: 79.Bl -bullet -offset indent 80.It 81After a fresh installation of 82.Fx 83to ensure that the system works correctly on your hardware platform. 84.It 85After an upgrade of 86.Fx 87to a different version to ensure that the new code works well on your 88hardware platform and that the upgrade did not introduce regressions in your 89configuration. 90.It 91After modifying the source tree to detect any new bugs and/or regressions. 92.It 93Periodically, maybe from a 94.Xr cron 8 95job, to ensure that any changes to the system (such as the installation of 96third-party packages or manual modifications to configuration files) do not 97introduce unexpected failures. 98.El 99.Ss Running the tests 100Use the following command to run the whole test suite: 101.Bd -literal -offset indent 102$ kyua test -k /usr/tests/Kyuafile 103.Ed 104.Pp 105The above will iterate through all test programs in 106.Pa /usr/tests 107recursively, execute them, store their results and debugging data in Kyua's 108database (by default in 109.Pa ~/.kyua/store.db ) , 110and print a summary of the results. 111This summary includes a brief count of all total tests run and how many of 112them failed. 113.Pp 114It is possible to restrict which tests to run by providing their names in 115the command line. 116For example, this would execute the tests for the 117.Xr cp 1 118and 119.Xr cut 1 120utilities: 121.Bd -literal -offset indent 122$ kyua test -k /usr/tests/Kyuafile bin/cp usr.bin/cut 123.Ed 124.Ss Obtaining reports of the tests execution 125Additional information about the test results can be retrieved 126by using Kyua's various reporting commands. 127For example, the following would print a plain-text report of the executed 128tests and show which ones failed: 129.Bd -literal -offset indent 130$ kyua report --verbose -r <.db file from output of test> 131.Ed 132.Pp 133This example would generate an HTML report ready to be published on a 134web server: 135.Bd -literal -offset indent 136$ kyua report-html --output ~/public_html/tests 137.Ed 138.Pp 139For further details on the command-line interface of Kyua, please refer 140to its manual page 141.Xr kyua 1 . 142.Ss Configuring the tests 143Some test cases in the 144.Fx 145Test Suite require manual configuration by the administrator before they can be 146run. 147Unless certain properties are defined, the tests that require them will be 148skipped. 149.Pp 150Test suites are configured by defining their configuration 151variables in 152.Pa /etc/kyua/kyua.conf . 153The format of this file is detailed in 154.Xr kyua.conf 5 . 155.Pp 156The following configuration variables are available in the 157.Fx 158Test Suite: 159.Bl -tag -width "allow_sysctl_side_effects" 160.It allow_devfs_side_effects 161If defined, enables tests that may destroy and recreate semipermanent device 162nodes, like disk devices. 163Without this variable, tests may still create and destroy devices nodes that 164are normally transient, like /dev/tap* and /dev/pts*, as long as they clean 165them up afterwards. 166However, tests that require this variable have a relaxed cleanup requirement; 167they must recreate any devices that they destroyed, but not necessarily with 168the same devnames. 169.It allow_sysctl_side_effects 170Enables tests that change globally significant 171.Xr sysctl 8 172variables. 173The tests will undo any changes in their cleanup phases. 174.It disks 175Must be set to a space delimited list of disk device nodes. 176Tests that need destructive access to disks must use these devices. 177Tests are not required to preserve any data present on these disks. 178.It fibs 179Must be set to a space delimited list of FIBs (routing tables). 180Tests that need to modify a routing table may use any of these. 181Tests will cleanup any new routes that they create. 182.El 183.Ss What to do if something fails? 184If there is 185.Em any failure 186during the execution of the test suite, please consider reporting it to the 187.Fx 188developers so that the failure can be analyzed and fixed. 189To do so, either send a message to the appropriate mailing list or file a 190problem report. 191For more details please refer to: 192.Bl -bullet -offset indent -compact 193.It 194.Lk https://lists.freebsd.org/ "FreeBSD Mailing Lists" 195.It 196.Lk https://www.freebsd.org/support/ "Problem Reporting" 197.El 198.Sh FILES 199.Bl -tag -compact -width "/etc/kyua/kyua.conf" 200.It Pa /etc/kyua/kyua.conf 201System-wide configuration file for 202.Xr kyua 1 . 203.It Pa ~/.kyua/kyua.conf 204User-specific configuration file for 205.Xr kyua 1 ; 206overrides the system file. 207.It Pa ~/.kyua/store.db 208Default result database used by Kyua. 209.It Pa /usr/tests/ 210Location of the 211.Fx 212Test Suite. 213.It Pa /usr/tests/Kyuafile 214Top-level test suite definition file. 215.El 216.Sh SEE ALSO 217.Xr kyua 1 , 218.Xr atf 7 , 219.Xr build 7 , 220.Xr development 7 221.Sh HISTORY 222The 223.Fx 224Test Suite first appeared in 225.Fx 10.1 226and was installed by default in 227.Fx 11.0 . 228.Pp 229The 230.Nm 231manual page first appeared in 232.Nx 6.0 233and was later ported to 234.Fx 10.1 . 235.Pp 236The test driver, 237.Xr kyua 1 , 238was imported as part of the base system in 239.Fx 13.0 , 240previously being available only in 241.Xr ports 7 . 242.Sh AUTHORS 243.An Julio Merino Aq Mt jmmv@FreeBSD.org 244