kyua: Add FreeBSD Jail execution environment support

A new Kyua concept is added -- "execution environment". A test can be
configured to be run within a specific environment. The test case
lifecycle

kyua: Add FreeBSD Jail execution environment support

A new Kyua concept is added -- "execution environment". A test can be
configured to be run within a specific environment. The test case
lifecycle is extended respectively:
- execenv init (creates a jail or does nothing for default
execenv="host")
- test exec
- cleanup exec (optional)
- execenv cleanup (removes a jail or does nothing for default
execenv="host")

The following new functionality is provided, from bottom to top:

1 ATF based tests

- The new "execenv" metadata property can be set to explicitly ask for
an execution environment: "host" or "jail". If it's not defined, as
all existing tests do, then it implicitly means "host".

- The new "execenv.jail.params" metadata property can be optionally
defined to ask Kyua to use specific jail(8) parameters during creation
of a temporary jail. An example is "vnet allow.raw_sockets".

Kyua implicitly adds "children.max" to "execenv_jail_params"
parameters with the maximum possible value. A test case can override
it.

2 Kyuafile

- The same new metadata properties can be defined on Kyuafile level:
"execenv" and "execenv_jail_params".

- Note that historically ATF uses dotted style of metadata naming, while
Kyua uses underscore style. Hence "execenv.jail.params" vs.
"execenv_jail_params".

3 kyua.conf, kyua CLI

- The new "execenvs" engine configuration variable can be set to a list
of execution environments to run only tests designed for. Tests of not
listed environments are skipped.

- By default, this variable lists all execution environments supported
by a Kyua binary, e.g. execenvs="host jail".

- This variable can be changed via "kyua.conf" or via kyua CLI's "-v"
parameter. For example, "kyua -v execenvs=host test" will run only
host-based tests and skip jail-based ones.

- Current value of this variable can be examined with "kyua config".

[markj] This feature has not landed upstream yet.
See the discussion in https://github.com/freebsd/kyua/pull/224 .
Having the ability to automatically jail tests allows many network tests
to run in parallel, giving a drastic speedup. So, let's import the
feature and start using it in main.

Signed-off-by: Igor Ostapenko <pm@igoro.pro>
Reviewed by: markj, kp
Tested by: markj, kp
MFC after: 3 months
Differential Revision: https://reviews.freebsd.org/D45865

show more ...

Remove residual blank line at start of Makefile

This is a residual of the $FreeBSD$ removal.

MFC After: 3 days (though I'll just run the command on the branches)
Sponsored by: Netflix

Remove $FreeBSD$: one-line sh pattern

Remove /^\s*#[#!]?\s*\$FreeBSD\$.*$\n/

Explicitly set CXXSTD to c++11 for old C++ code using std::auto_ptr<>.

GCC 12 defaults to C++17 which removes (not just deprecates)
std::auto_ptr<>. Trying to use CXXSTD of c++03 doesn't work with

Explicitly set CXXSTD to c++11 for old C++ code using std::auto_ptr<>.

GCC 12 defaults to C++17 which removes (not just deprecates)
std::auto_ptr<>. Trying to use CXXSTD of c++03 doesn't work with
libc++ headers, but c++11 does.

Reviewed by: brooks, imp, emaste
Differential Revision: https://reviews.freebsd.org/D37531

show more ...

Simplify and speed up the kyua build

Instead of having multiple kyua libraries, just include the files as part
of usr.bin/kyua. Previously, we would build each kyua source up to four
times: once as

Simplify and speed up the kyua build

Instead of having multiple kyua libraries, just include the files as part
of usr.bin/kyua. Previously, we would build each kyua source up to four
times: once as a .o file and once as a .pieo. Additionally, the kyua
libraries might be built again for compat32. As all the kyua libraries
amount to 102 C++ sources the build time is significant (especially when
using an assertions enabled compiler). This change ensures that we build
306 fewer .cpp source files as part of buildworld.

Reviewed By: brooks
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D30967

show more ...

pkgbase: Install atf and kyua in the tests package

While here make sure that all tests dirs are taggued correctly.

Reviewed by: bapt, emaste
Differential Revision: https://reviews.freebsd.org/D27714

kyua: Only install examples if requested

Reviewed by: brooks
Differential Revision: https://reviews.freebsd.org/D27638

Install a kyua.conf based on the one in devel/kyua.

The kyua.conf from examples doesn't match the expected config and
contains a lot of undesirable entries such as setting the architecture
to amd64

Install a kyua.conf based on the one in devel/kyua.

The kyua.conf from examples doesn't match the expected config and
contains a lot of undesirable entries such as setting the architecture
to amd64 explicitly.

Reported by: arichardson (missing config)
Reviewed by: emaste
Obtained from: CheriBSD
Differential Revision: https://reviews.freebsd.org/D24267

show more ...

Fix build with src on a noexec filesystem.

${SH} expands to nothing on src builds so this worked by accident. Use
a bare "sh" instead.

PR: 245086
Reported by: pkubaj
Sponsored by: DARPA

Revert r359385-r359387

As noted by brooks/emaste, this is the wrong approach to take.
Revert the changes so brooks can apply a more proper change.

Requested by: brooks, emaste

The .PATH throbbing I added in r359385 is no longer required

MFC with: r359385

Check in the generated copies of the manpages

These manpages were meant to be templated once per `configure` run.

Given that we're not bound by as many constants, e.g., `--prefix` isn't
generally c

Check in the generated copies of the manpages

These manpages were meant to be templated once per `configure` run.

Given that we're not bound by as many constants, e.g., `--prefix` isn't
generally changing for kyua in the base system, having to generate the
manpages each build seems slightly less than optimal.

In the event that one's build environment doesn't define `$SH`, the build
will also fail until this change is introduced.

Instead of jumping through hoops dealing with shells or permissions, let's
just cut to the chase and check the generated copies into the sourcebase
under usr.bin/kyua .

MFC with: r359260
Reported by: Julian Stacey <jhs@berklix.com>

show more ...

Install expected kyua docs.

The "kyua about" command assumes these files exist causing tests
supplied devel/kyua to fail.

Fix a bug defining the default KYUA_DOCDIR so the installed files can be
fo

Install expected kyua docs.

The "kyua about" command assumes these files exist causing tests
supplied devel/kyua to fail.

Fix a bug defining the default KYUA_DOCDIR so the installed files can be
found.

Reported by: jenkins tests
Reviewed by: lwhsu
Sponsored by: DARPA
Differential Revision: https://reviews.freebsd.org/D24187

show more ...

Import the kyua test framework.

Having kyua in the base system will simplify automated testing in CI and
eliminates bootstrapping issues on new platforms.

The build of kyua is controlled by WITH(OU

Import the kyua test framework.

Having kyua in the base system will simplify automated testing in CI and
eliminates bootstrapping issues on new platforms.

The build of kyua is controlled by WITH(OUT)_TESTS_SUPPORT.

Reviewed by: emaste
Obtained from: CheriBSD
Sponsored by: DARPA
Differential Revision: https://reviews.freebsd.org/D24103

show more ...