kyua: Do not count skipped as passed in test cmd

It changes output of 'kyua test' CLI command only. Hence, other outputs
like junit are kept intact for CI and other use cases. It's meant to
improve

kyua: Do not count skipped as passed in test cmd

It changes output of 'kyua test' CLI command only. Hence, other outputs
like junit are kept intact for CI and other use cases. It's meant to
improve UX of attended use cases.

The issue is that the following can be tricky to interpret:

222/222 passed (0 failed)

It can be read as all tests are passed, but it might be a summary line
of all tests skipped due to some requirement is not met.

It's reworked to easily distinguish such cases:

222/222 passed (0 broken, 0 failed, 0 skipped)
0/222 passed (0 broken, 0 failed, 222 skipped)

The overall formula is:

<actually passed>/<total> (<details about not actually passed ones>)

Suggested by: kp
Reviewed by: ngie, markj
Approved by: markj (mentor)
Differential Revision: https://reviews.freebsd.org/D46653

show more ...

kyua: Add FreeBSD Jail execution environment support

A new Kyua concept is added -- "execution environment". A test can be
configured to be run within a specific environment. The test case
lifecycle

kyua: Add FreeBSD Jail execution environment support

A new Kyua concept is added -- "execution environment". A test can be
configured to be run within a specific environment. The test case
lifecycle is extended respectively:
- execenv init (creates a jail or does nothing for default
execenv="host")
- test exec
- cleanup exec (optional)
- execenv cleanup (removes a jail or does nothing for default
execenv="host")

The following new functionality is provided, from bottom to top:

1 ATF based tests

- The new "execenv" metadata property can be set to explicitly ask for
an execution environment: "host" or "jail". If it's not defined, as
all existing tests do, then it implicitly means "host".

- The new "execenv.jail.params" metadata property can be optionally
defined to ask Kyua to use specific jail(8) parameters during creation
of a temporary jail. An example is "vnet allow.raw_sockets".

Kyua implicitly adds "children.max" to "execenv_jail_params"
parameters with the maximum possible value. A test case can override
it.

2 Kyuafile

- The same new metadata properties can be defined on Kyuafile level:
"execenv" and "execenv_jail_params".

- Note that historically ATF uses dotted style of metadata naming, while
Kyua uses underscore style. Hence "execenv.jail.params" vs.
"execenv_jail_params".

3 kyua.conf, kyua CLI

- The new "execenvs" engine configuration variable can be set to a list
of execution environments to run only tests designed for. Tests of not
listed environments are skipped.

- By default, this variable lists all execution environments supported
by a Kyua binary, e.g. execenvs="host jail".

- This variable can be changed via "kyua.conf" or via kyua CLI's "-v"
parameter. For example, "kyua -v execenvs=host test" will run only
host-based tests and skip jail-based ones.

- Current value of this variable can be examined with "kyua config".

[markj] This feature has not landed upstream yet.
See the discussion in https://github.com/freebsd/kyua/pull/224 .
Having the ability to automatically jail tests allows many network tests
to run in parallel, giving a drastic speedup. So, let's import the
feature and start using it in main.

Signed-off-by: Igor Ostapenko <pm@igoro.pro>
Reviewed by: markj, kp
Tested by: markj, kp
MFC after: 3 months
Differential Revision: https://reviews.freebsd.org/D45865

show more ...