xref: /freebsd/share/man/man7/tests.7 (revision 4b9d605768acabc460aa6dcfe8a1f8db35b16794)
1.\"	$NetBSD: tests.kyua.7,v 1.2 2013/07/20 21:39:59 wiz Exp $
2.\"
3.\" Copyright (c) 2010 The NetBSD Foundation, Inc.
4.\" All rights reserved.
5.\"
6.\" Redistribution and use in source and binary forms, with or without
7.\" modification, are permitted provided that the following conditions
8.\" are met:
9.\" 1. Redistributions of source code must retain the above copyright
10.\"    notice, this list of conditions and the following disclaimer.
11.\" 2. Redistributions in binary form must reproduce the above copyright
12.\"    notice, this list of conditions and the following disclaimer in the
13.\"    documentation and/or other materials provided with the distribution.
14.\"
15.\" THIS SOFTWARE IS PROVIDED BY THE NETBSD FOUNDATION, INC. AND
16.\" CONTRIBUTORS ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES,
17.\" INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
18.\" MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
19.\" IN NO EVENT SHALL THE FOUNDATION OR CONTRIBUTORS BE LIABLE FOR ANY
20.\" DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
21.\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE
22.\" GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
23.\" INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER
24.\" IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR
25.\" OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN
26.\" IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
27.\"
28.Dd August 19, 2020
29.Dt TESTS 7
30.Os
31.Sh NAME
32.Nm tests
33.Nd introduction to the
34.Fx
35Test Suite
36.Sh DESCRIPTION
37The
38.Fx
39Test Suite provides a collection of automated tests for two major purposes.
40On one hand, the test suite aids
41.Em developers
42to detect bugs and regressions when they modify the source tree.
43On the other hand, it allows
44.Em end users
45(and, in particular, system administrators) to verify that fresh installations
46of the
47.Fx
48operating system behave correctly on their hardware platform and also to ensure
49that the system does not suffer from regressions during regular operation and
50maintenance.
51.Pp
52The
53.Fx
54Test Suite can be found in the
55.Pa /usr/tests
56hierarchy.
57.Pp
58This manual page describes how to run the test suite and how to configure
59some of its optional features.
60For information on writing the tests, see
61.Xr atf 7 .
62.Ss Installing the test suite
63If the
64.Pa /usr/tests
65directory is missing, then you will have to enable the build of the test
66suite, rebuild your system and install the results.
67You can do so by setting
68.Sq WITH_TESTS=yes
69in your
70.Pa /etc/src.conf
71file (see
72.Xr src.conf 5
73for details)
74and rebuilding the system as described in
75.Xr build 7 .
76.Ss When to run the tests?
77Before diving into the details of how to run the test suite, here are some
78scenarios in which you should run it:
79.Bl -bullet -offset indent
80.It
81After a fresh installation of
82.Fx
83to ensure that the system works correctly on your hardware platform.
84.It
85After an upgrade of
86.Fx
87to a different version to ensure that the new code works well on your
88hardware platform and that the upgrade did not introduce regressions in your
89configuration.
90.It
91After modifying the source tree to detect any new bugs and/or regressions.
92.It
93Periodically, maybe from a
94.Xr cron 8
95job, to ensure that any changes to the system (such as the installation of
96third-party packages or manual modifications to configuration files) do not
97introduce unexpected failures.
98.El
99.Ss Running the tests
100Use the following command to run the whole test suite:
101.Bd -literal -offset indent
102$ kyua test -k /usr/tests/Kyuafile
103.Ed
104.Pp
105The above will iterate through all test programs in
106.Pa /usr/tests
107recursively, execute them, store their results and debugging data in Kyua's
108database (by default in
109.Pa ~/.kyua/store.db ) ,
110and print a summary of the results.
111This summary includes a brief count of all total tests run and how many of
112them failed.
113.Pp
114It is possible to restrict which tests to run by providing their names in
115the command line.
116For example, this would execute the tests for the
117.Xr cp 1
118and
119.Xr cut 1
120utilities:
121.Bd -literal -offset indent
122$ kyua test -k /usr/tests/Kyuafile bin/cp usr.bin/cut
123.Ed
124.Ss Obtaining reports of the tests execution
125Additional information about the test results can be retrieved
126by using Kyua's various reporting commands.
127For example, the following would print a plain-text report of the executed
128tests and show which ones failed:
129.Bd -literal -offset indent
130$ kyua report --verbose -r <.db file from output of test>
131.Ed
132.Pp
133This example would generate an HTML report ready to be published on a
134web server:
135.Bd -literal -offset indent
136$ kyua report-html --output ~/public_html/tests
137.Ed
138.Pp
139For further details on the command-line interface of Kyua, please refer
140to its manual page
141.Xr kyua 1 .
142.Ss Configuring the tests
143Some test cases in the
144.Fx
145Test Suite require manual configuration by the administrator before they can be
146run.
147Unless certain properties are defined, the tests that require them will be
148skipped.
149.Pp
150Test suites are configured by defining their configuration
151variables in
152.Pa /etc/kyua/kyua.conf .
153The format of this file is detailed in
154.Xr kyua.conf 5 .
155.Pp
156The following configuration variables are available in the
157.Fx
158Test Suite:
159.Bl -tag -width "allow_sysctl_side_effects"
160.It allow_devfs_side_effects
161If defined, enables tests that may destroy and recreate semipermanent device
162nodes, like disk devices.
163Without this variable, tests may still create and destroy devices nodes that
164are normally transient, like /dev/tap* and /dev/pts*, as long as they clean
165them up afterwards.
166However, tests that require this variable have a relaxed cleanup requirement;
167they must recreate any devices that they destroyed, but not necessarily with
168the same devnames.
169.It allow_sysctl_side_effects
170Enables tests that change globally significant
171.Xr sysctl 8
172variables.
173The tests will undo any changes in their cleanup phases.
174.It disks
175Must be set to a space delimited list of disk device nodes.
176Tests that need destructive access to disks must use these devices.
177Tests are not required to preserve any data present on these disks.
178.It fibs
179Must be set to a space delimited list of FIBs (routing tables).
180Tests that need to modify a routing table may use any of these.
181Tests will cleanup any new routes that they create.
182.El
183.Ss What to do if something fails?
184If there is
185.Em any failure
186during the execution of the test suite, please consider reporting it to the
187.Fx
188developers so that the failure can be analyzed and fixed.
189To do so, either send a message to the appropriate mailing list or file a
190problem report.
191For more details please refer to:
192.Bl -bullet -offset indent -compact
193.It
194.Lk https://lists.freebsd.org/ "FreeBSD Mailing Lists"
195.It
196.Lk https://www.freebsd.org/support/ "Problem Reporting"
197.El
198.Sh FILES
199.Bl -tag -compact -width "/etc/kyua/kyua.conf"
200.It Pa /etc/kyua/kyua.conf
201System-wide configuration file for
202.Xr kyua 1 .
203.It Pa ~/.kyua/kyua.conf
204User-specific configuration file for
205.Xr kyua 1 ;
206overrides the system file.
207.It Pa ~/.kyua/store.db
208Default result database used by Kyua.
209.It Pa /usr/tests/
210Location of the
211.Fx
212Test Suite.
213.It Pa /usr/tests/Kyuafile
214Top-level test suite definition file.
215.El
216.Sh SEE ALSO
217.Xr kyua 1 ,
218.Xr atf 7 ,
219.Xr build 7 ,
220.Xr development 7
221.Sh HISTORY
222The
223.Fx
224Test Suite first appeared in
225.Fx 10.1
226and was installed by default in
227.Fx 11.0 .
228.Pp
229The
230.Nm
231manual page first appeared in
232.Nx 6.0
233and was later ported to
234.Fx 10.1 .
235.Pp
236The test driver,
237.Xr kyua 1 ,
238was imported as part of the base system in
239.Fx 13.0 ,
240previously being available only in
241.Xr ports 7 .
242.Sh AUTHORS
243.An Julio Merino Aq Mt jmmv@FreeBSD.org
244