xref: /freebsd/contrib/atf/NEWS (revision f5f7c05209ca2c3748fd8b27c5e80ffad49120eb)
1Major changes between releases                  Automated Testing Framework
2===========================================================================
3
4
5Changes in version 0.16
6***********************
7
8Experimental version released on July 10th, 2012.
9
10* Added a --enable-tools flag to configure to request the build of the
11  deprecated ATF tools, whose build is now disabled by default.  In order
12  to continue running tests, you should migrate to Kyua instead of enabling
13  the build of the deprecated tools.  The kyua-atf-compat package provides
14  transitional compatibility versions of atf-run and atf-report built on
15  top of Kyua.
16
17* Tweaked the ATF_TEST_CASE macro of atf-c++ so that the compiler can
18  detect defined but unused test cases.
19
20* PR bin/45859: Fixed some XSLT bugs that resulted in the tc-time and
21  tp-time XML tags leaking into the generated HTML file.  Also improved
22  the CSS file slightly to correct alignment and color issues with the
23  timestamps column.
24
25* Optimized atf-c++/macros.hpp so that GNU G++ consumes less memory during
26  compilation with GNU G++.
27
28* Flipped the default to building shared libraries for atf-c and atf-c++,
29  and started versioning them.  As a side-effect, this removes the
30  --enable-unstable-shared flag from configure that appears to not work any
31  more (under NetBSD).  Additionally, some distributions require the use of
32  shared libraries for proper dependency tracking (e.g. Fedora), so it is
33  better if we do the right versioning upstream.
34
35* Project hosting moved from an adhoc solution (custom web site and
36  Monotone repository) to Google Code (standard wiki and Git).  ATF now
37  lives in a subcomponent of the Kyua project.
38
39
40Changes in version 0.15
41***********************
42
43Experimental version released on January 16th, 2012.
44
45* Respect stdin in atf-check.  The previous release silenced stdin for any
46  processes spawned by atf, not only test programs, which caused breakage
47  in tests that pipe data through atf-check.
48
49* Performance improvements to atf-sh.
50
51* Enabled detection of unused parameters and variables in the code and
52  fixed all warnings.
53
54* Changed the behavior of "developer mode".  Compiler warnings are now
55  enabled unconditionally regardless of whether we are in developer mode or
56  not; developer mode is now only used to perform strict warning checks and
57  to enable assertions.  Additionally, developer mode is now only
58  automatically enabled when building from the repository, not for formal
59  releases.
60
61* Added new Autoconf M4 macros (ATF_ARG_WITH, ATF_CHECK_C and
62  ATF_CHECK_CXX) to provide a consistent way of defining a --with-arg flag
63  in configure scripts and detecting the presence of any of the ATF
64  bindings.  Note that ATF_CHECK_SH was already introduced in 0.14, but it
65  has now been modified to also honor --with-atf if instantiated.
66
67* Added timing support to atf-run / atf-report.
68
69* Added support for a 'require.memory' property, to specify the minimum
70  amount of physical memory needed by the test case to yield valid results.
71
72* PR bin/45690: Force an ISO-8859-1 encoding in the XML files generated by
73  atf-report so that invalid data in the output of test cases does not
74  mangle our report.
75
76
77Changes in version 0.14
78***********************
79
80Experimental version released on June 14th, 2011.
81
82* Added a pkg-config file for atf-sh and an aclocal file to ease the
83  detection of atf-sh from autoconf scripts.
84
85* Made the default test case body defined by atf_sh fail.  This is to
86  ensure that test cases are properly defined in test programs and helps
87  in catching typos in the names of the body functions.
88
89* PR bin/44882: Made atf-run connect the stdin of test cases to /dev/zero.
90  This provides more consistent results with "normal" execution (in
91  particular, when tests are executed detached from a terminal).
92
93* Made atf-run hardcode TZ=UTC for test cases.  It used to undefine TZ, but
94  that does not take into account that libc determines the current timezone
95  from a configuration file.
96
97* All test programs will now print a warning when they are not run through
98  atf-run(1) stating that this is unsupported and may deliver incorrect
99  results.
100
101* Added support for the 'require.files' test-case property.  This allows
102  test cases to specify installed files that must be present for the test
103  case to run.
104
105
106Changes in version 0.13
107***********************
108
109Experimental version released on March 31st, 2011.
110
111This is the first release after the creation of the Kyua project, a more
112modular and reliable replacement for ATF.  From now on, ATF will change to
113accomodate the transition to this new codebase, but ATF will still continue
114to see development in the short/medium term.  Check out the project page at
115http://code.google.com/p/kyua/ for more details.
116
117The changes in this release are:
118
119* Added support to run the tests with the Kyua runtime engine (kyua-cli), a
120  new package that aims to replace atf-run and atf-report.  The ATF tests
121  can be run with the new system by issuing a 'make installcheck-kyua' from
122  the top-level directory of the project (assuming the 'kyua' binary is
123  available during the configuration stage of ATF).
124
125* atf-run and atf-report are now in maintenance mode (but *not* deprecated
126  yet!).  Kyua already implements a new, much more reliable runtime engine
127  that provides similar features to these tools.  That said, it is not
128  complete yet so all development efforts should go towards it.
129
130* If GDB is installed, atf-run dumps the stack trace of crashing test
131  programs in an attempt to aid debugging.  Contributed by Antti Kantee.
132
133* Reverted default timeout change in previous release and reset its value
134  to 5 minutes.  This was causing several issues, specially when running
135  the existing NetBSD test suite in qemu.
136
137* Fixed the 'match' output checker in atf-check to properly validate the
138  last line of a file even if it does not have a newline.
139
140* Added the ATF_REQUIRE_IN and ATF_REQUIRE_NOT_IN macros to atf-c++ to
141  check for the presence (or lack thereof) of an element in a collection.
142
143* PR bin/44176: Fixed a race condition in atf-run that would crash atf-run
144  when the cleanup of a test case triggered asynchronous modifications to
145  its work directory (e.g. killing a daemon process that cleans up a pid
146  file in the work directory).
147
148* PR bin/44301: Fixed the sample XSLT file to report bogus test programs
149  instead of just listing them as having 0 test cases.
150
151
152Changes in version 0.12
153***********************
154
155Experimental version released on November 7th, 2010.
156
157* Added the ATF_REQUIRE_THROW_RE to atf-c++, which is the same as
158  ATF_REQUIRE_THROW but allows checking for the validity of the exception's
159  error message by means of a regular expression.
160
161* Added the ATF_REQUIRE_MATCH to atf-c++, which allows checking for a
162  regular expression match in a string.
163
164* Changed the default timeout for test cases from 5 minutes to 30 seconds.
165  30 seconds is long enough for virtually all tests to complete, and 5
166  minutes is a way too long pause in a test suite where a single test case
167  stalls.
168
169* Deprecated the use.fs property.  While this seemed like a good idea in
170  the first place to impose more control on what test cases can do, it
171  turns out to be bad.  First, use.fs=false prevents bogus test cases
172  from dumping core so after-the-fact debugging is harder.  Second,
173  supporting use.fs adds a lot of unnecessary complexity.  atf-run will
174  now ignore any value provided to use.fs and will allow test cases to
175  freely access the file system if they wish to.
176
177* Added the atf_tc_get_config_var_as_{bool,long}{,_wd} functions to the atf-c
178  library.  The 'text' module became private in 0.11 but was being used
179  externally to simplify the parsing of configuration variables.
180
181* Made atf-run recognize the 'unprivileged-user' configuration variable
182  and automatically drop root privileges when a test case sets
183  require.user=unprivileged.  Note that this is, by no means, done for
184  security purposes; this is just for user convenience; tests should, in
185  general, not be blindly run as root in the first place.
186
187
188Changes in version 0.11
189***********************
190
191Experimental version released on October 20th, 2010.
192
193* The ATF_CHECK* macros in atf-c++ were renamed to ATF_REQUIRE* to match
194  their counterparts in atf-c.
195
196* Clearly separated the modules in atf-c that are supposed to be public
197  from those that are implementation details.  The header files for the
198  internal modules are not installed any more.
199
200* Made the atf-check tool private.  It is only required by atf-sh and being
201  public has the danger of causing confusion.  Also, making it private
202  simplifies the public API of atf.
203
204* Changed atf-sh to enable per-command error checking (set -e) by default.
205  This catches many cases in which a test case is broken but it is not
206  reported as such because execution continues.
207
208* Fixed the XSTL and CSS stylesheets to support expected failures.
209
210
211Changes in version 0.10
212***********************
213
214Experimental version released on July 2nd, 2010.
215
216Miscellaneous features
217
218* Added expected failures support to test cases and atf-run.  These
219  include, for example, expected clean exits, expected reception of fatal
220  signals, expected timeouts and expected errors in condition checks.
221  These statuses can be used to denote test cases that are known to fail
222  due to a bug in the code they are testing.  atf-report reports these
223  tests separately but they do not count towards the failed test cases
224  amount.
225
226* Added the ATF_CHECK_ERRNO and ATF_REQUIRE_ERRNO to the C library to
227  allow easy checking of call failures that update errno.
228
229* Added the has.cleanup meta-data property to test caes that specifies
230  whether the test case has a cleanup routine or not; its value is
231  automatically set.  This property is read by atf-run to know if it has to
232  run the cleanup routine; skipping this run for every test case
233  significantly speeds up the run time of test suites.
234
235* Reversed the order of the ATF_CHECK_THROW macro in the C++ binding to
236  take the expected exception as the first argument and the statement to
237  execute as the second argument.
238
239Changes in atf-check
240
241* Changed atf-check to support negating the status and output checks by
242  prefixing them with not- and added support to specify multiple checkers
243  for stdout and stderr, not only one.
244
245* Added the match output checker to atf-check to look for regular
246  expressions in the stdout and stderr of commands.
247
248* Modified the exit checks in atf-check to support checking for the
249  reception of signals.
250
251Code simplifications and cleanups
252
253* Removed usage messages from test programs to simplify the
254  implementation of every binding by a significant amount.  They just now
255  refer the user to the appropriate manual page and do not attempt to wrap
256  lines on terminal boundaries.  Test programs are not supposed to be run
257  by users directly so this minor interface regression is not important.
258
259* Removed the atf-format internal utility, which is unused after the
260  change documented above.
261
262* Removed the atf-cleanup internal utility.  It has been unused since the
263  test case isolation was moved to atf-run in 0.8
264
265* Splitted the Makefile.am into smaller files for easier maintenance and
266  dropped the use of M4.  Only affects users building from the repository
267  sources.
268
269* Intermixed tests with the source files in the source tree to provide
270  them more visibility and easier access.  The tests directory is gone from
271  the source tree and tests are now suffixed by _test, not prefixed by t_.
272
273* Simplifications to the atf-c library: removed the io, tcr and ui
274  modules as they had become unnecessary after all simplifications
275  introduced since the 0.8 release.
276
277* Removed the application/X-atf-tcr format introduced in 0.8 release.
278  Tests now print a much simplified format that is easy to parse and nicer
279  to read by end users.  As a side effect, the default for test cases is
280  now to print their results to stdout unless otherwise stated by providing
281  the -r flag.
282
283* Removed XML distribution documents and replaced them with plain-text
284  documents.  They provided little value and introduced a lot of complexity
285  to the build system.
286
287* Simplified the output of atf-version by not attempting to print a
288  revision number when building form a distfile.  Makes the build system
289  easier to maintain.
290
291
292Changes in version 0.9
293**********************
294
295Experimental version released on June 3rd, 2010.
296
297* Added atf-sh, an interpreter to process test programs written using
298  the shell API.  This is not really a shell interpreter by itself though:
299  it is just a wrapper around the system shell that eases the loading of
300  the necessary ATF libraries.
301
302* Removed atf-compile in favour of atf-sh.
303
304* Added the use.fs metadata property to test case, which is used to
305  specify which test cases require file system access.  This is to
306  highlight dependencies on external resources more clearly and to speed up
307  the execution of test suites by skipping the creation of many unnecessary
308  work directories.
309
310* Fixed test programs to get a sane default value for their source
311  directory.  This means that it should not be necessary any more to pass
312  -s when running test programs that do not live in the current directory.
313
314* Defining test case headers became optional.  This is trivial to achieve
315  in shell-based tests but a bit ugly in C and C++.  In C, use the new
316  ATF_TC_WITHOUT_HEAD macro to define the test case, and in C++ use
317  ATF_TEST_CASE_WITHOUT_HEAD.
318
319
320Changes in version 0.8
321**********************
322
323Experimental version released on May 7th, 2010.
324
325* Test programs no longer run several test cases in a row.  The execution
326  of a test program now requires a test case name, and that single test
327  case is executed.  To execute several test cases, use the atf-run utility
328  as usual.
329
330* Test programs no longer fork a subprocess to isolate the execution of
331  test cases.  They run the test case code in-process, and a crash of the
332  test case will result in a crash of the test program.  This is to ease
333  debugging of faulty test cases.
334
335* Test programs no longer isolate their test cases.  This means that they
336  will not create temporary directories nor sanitize the environment any
337  more.  Yes: running a test case that depends on system state by hand will
338  most likely yield different results depending on where (machine,
339  directory, user environment, etc.) it is run.  Isolation has been moved
340  to atf-run.
341
342* Test programs no longer print a cryptic format (application/X-atf-tcs)
343  on a special file channel.  They can now print whatever they want on the
344  screen.  Because test programs can now only run one test case every time,
345  providing controlled output is not necessary any more.
346
347* Test programs no longer write their status into a special file
348  descriptor.  Instead, they create a file with the results, which is later
349  parsed by atf-run.  This changes the semantics of the -r flag.
350
351* atf-run has been adjusted to perform the test case isolation.  As a
352  result, there is now a single canonical place that implements the
353  isolation of test caes.  In previous releases, the three language
354  bindings (C, C++ and shell) had to be kept in sync with each other (read:
355  not a nice thing to do at all).  As a side effect of this change, writing
356  bindings for other languages will be much, much easier from now on.
357
358* atf-run forks test programs on a test case basis, instead of on a test
359  program basis as it did before.  This is to provide the test case
360  isolation that was before implemented by the test programs themselves.
361
362* Removed the atf-exec tool.  This was used to implement test case
363  isolation in atf-sh, but it is now unnecessary.
364
365* It is now optional to define the descr meta-data property.  It has been
366  proven to be mostly useless, because test cases often carry a descriptive
367  name of their own.
368
369
370Changes in version 0.7
371**********************
372
373Experimental version released on December 22nd, 2009.
374
375* Added build-time checks to atf-c and atf-c++.  A binding for atf-sh
376  will come later.
377
378* Migrated all build-time checks for header files to proper ATF tests.
379  This demonstrates the use of the new feature described above.
380
381* Added an internal API for child process management.
382
383* Converted all plain-text distribution documents to a Docbook canonical
384  version, and include pre-generated plain text and HTML copies in the
385  distribution file.
386
387* Simplified the contents of the Makefile.am by regenerating it from a
388  canonical Makefile.am.m4 source.  As a side-effect, some dependency
389  specifications were fixed.
390
391* Migrated all checks from the check target to installcheck, as these
392  require ATF to be installed.
393
394* Fixed sign comparison mismatches triggered by the now-enabled
395  -Wsign-compare.
396
397* Fixed many memory and object leaks.
398
399
400Changes in version 0.6
401**********************
402
403Experimental version released on January 18th, 2009.
404
405* Make atf-exec be able to kill its child process after a certain period
406  of time; this is controlled through the new -t option.
407
408* Change atf-sh to use atf-exec's -t option to control the test case's
409  timeouts, instead of doing it internally.  Same behavior as before, but
410  noticeably faster.
411
412* atf-exec's -g option and atf-killpg are gone due to the previous
413  change.
414
415* Added the atf-check(1) tool, a program that executes a given command
416  and checks its exit code against a known value and allows the management
417  of stdout and stderr in multiple ways.  This replaces the previous
418  atf_check function in the atf-sh library and exposes this functionality
419  to both atf-c and atf-c++.
420
421* Added the ATF_REQUIRE family of macros to the C interface.  These help
422  in checking for fatal test conditions.  The old ATF_CHECK macros now
423  perform non-fatal checks only.  I.e. by using ATF_CHECK, the test case
424  can now continue its execution and the failures will not be reported
425  until the end of the whole run.
426
427* Extended the amount of ATF_CHECK_* C macros with new ones to provide
428  more features to the developer.  These also have their corresponding
429  counterparts in the ATF_REQUIRE_* family.  The new macros (listing the
430  suffixes only) are: _EQ (replaces _EQUAL), _EQ_MSG, _STREQ and
431  _STREQ_MSG.
432
433
434Changes in version 0.5
435**********************
436
437Experimental version released on May 1st, 2008.
438
439* Clauses 3 and 4 of the BSD license used by the project were dropped.
440  All the code is now under a 2-clause BSD license compatible with the GNU
441  General Public License (GPL).
442
443* Added a C-only binding so that binary test programs do not need to be
444  tied to C++ at all.  This binding is now known as the atf-c library.
445
446* Renamed the C++ binding to atf-c++ for consistency with the new atf-c.
447
448* Renamed the POSIX shell binding to atf-sh for consistency with the new
449  atf-c and atf-c++.
450
451* Added a -w flag to test programs through which it is possible to
452  specify the work directory to be used.  This was possible in prior
453  releases by defining the workdir configuration variable (-v workdir=...),
454  but was a conceptually incorrect mechanism.
455
456* Test programs now preserve the execution order of test cases when they
457  are given in the command line.  Even those mentioned more than once are
458  executed multiple times to comply with the user's requests.
459
460
461Changes in version 0.4
462**********************
463
464Experimental version released on February 4th, 2008.
465
466* Added two new manual pages, atf-c++-api and atf-sh-api, describing the
467  C++ and POSIX shell interfaces used to write test programs.
468
469* Added a pkg-config file, useful to get the flags to build against the
470  C++ library or to easily detect the presence of ATF.
471
472* Added a way for test cases to require a specific architecture and/or
473  machine type through the new 'require.arch' and 'require.machine'
474  meta-data properties, respectively.
475
476* Added the 'timeout' property to test cases, useful to set an
477  upper-bound limit for the test's run time and thus prevent global test
478  program stalls due to the test case's misbehavior.
479
480* Added the atf-exec(1) internal utility, used to execute a command
481  after changing the process group it belongs to.
482
483* Added the atf-killpg(1) internal utility, used to kill process groups.
484
485* Multiple portability fixes.  Of special interest, full support for
486  SunOS (Solaris Express Developer Edition 2007/09) using the Sun Studio 12
487  C++ compiler.
488
489* Fixed a serious bug that prevented atf-run(1) from working at all
490  under Fedora 8 x86_64.  Due to the nature of the bug, other platforms
491  were likely affected too.
492
493
494Changes in version 0.3
495**********************
496
497Experimental version released on November 11th, 2007.
498
499* Added XML output support to atf-report.  This is accompanied by a DTD
500  for the format's structure and sample XSLT/CSS files to post-process this
501  output and convert it to a plain HTML report.
502
503* Changed atf-run to add system information to the report it generates.
504  This is currently used by atf-report's XML output only, and is later
505  printed in the HTML reports in a nice and useful summary table.  The user
506  and system administrator are allowed to tune this feature by means of
507  hooks.
508
509* Removed the test cases' 'isolated' property.  This was intended to
510  avoid touching the file system at all when running the related test case,
511  but this has not been true for a long while: some control files are
512  unconditionally required for several purposes, and we cannot easily get
513  rid of them.  This way we remove several critical and delicate pieces of
514  code.
515
516* Improved atf-report's CSV output format to include information about
517  test programs too.
518
519* Fixed the tests that used atf-compile to not require this tool as a
520  helper.  Avoids systems without build-time utilities to skip many tests
521  that could otherwise be run.  (E.g. NetBSD without the comp.tgz set
522  installed.)
523
524* Many general cleanups: Fixed many pieces of code marked as ugly and/or
525  incomplete.
526
527
528Changes in version 0.2
529**********************
530
531Experimental version released on September 20th, 2007.
532
533* Test cases now get a known umask on entry.
534
535* atf-run now detects many unexpected failures caused by test programs and
536  reports them as bogus tests.  atf-report is able to handle these new
537  errors and nicely reports them to the user.
538
539* All the data formats read and written by the tools have been
540  documented and cleaned up.  These include those grammars that define how
541  the different components communicate with each other as well as the
542  format of files written by the developers and users: the Atffiles and the
543  configuration files.
544
545* Added the atf-version tool, a utility that displays information about
546  the currently installed version of ATF.
547
548* Test cases can now define an optional cleanup routine to undo their
549  actions regardless of their exit status.
550
551* atf-report now summarizes the list of failed (bogus) test programs
552  when using the ticker output format.
553
554* Test programs now capture some termination signals and clean up any
555  temporary files before exiting the program.
556
557* Multiple bug fixes and improvements all around.
558
559
560Changes in version 0.1
561**********************
562
563Experimental version released on August 20th, 2007.
564
565* First public version.  This was released coinciding with the end of the
566  Google Summer of Code 2007 program.
567
568
569===========================================================================
570vim: filetype=text:textwidth=75:expandtab:shiftwidth=2:softtabstop=2
571