1Major changes between releases Automated Testing Framework 2=========================================================================== 3 4 5Changes in version 0.20 6*********************** 7 8Experimental version released on February 7th, 2014. 9 10This is the first release without the code for the deprecated tools. If 11you require such code, please fetch a copy of the 0.19 release and extract 12the 'tools' directory for your own consumption. 13 14* Removed the deprecated tools. This includes atf-config, atf-report, 15 atf-run and atf-version. 16 17* Issue #8: Fixed atf-c/macros_test:use test failures spotted by the clang 18 that ships with FreeBSD 11.0-CURRENT. 19 20 21Changes in version 0.19 22*********************** 23 24Experimental version released on February 7th, 2014. 25 26This is the last release to bundle the code for the deprecated tools. 27The next release will drop their code and will stop worrying about 28backwards compatibility between the ATF libraries and what the old tools 29may or may not support. 30 31If you still require the old tools for some reason, grab a copy of the 32'tools' directory now. The code in this directory is standalone and 33does not depend on any internal details of atf-c++ any longer. 34 35* Various fixes and improvements to support running as part of the FreeBSD 36 test suite. 37 38* Project hosting moved from Google Code (as a subproject of Kyua) to 39 GitHub (as a first-class project). The main reason for the change is 40 the suppression of binary downloads in Google Code on Jan 15th, 2014. 41 See https://github.com/jmmv/atf/ 42 43* Removed builtin help from atf-sh(1) and atf-check(1) for simplicity 44 reasons. In other words, their -h option is gone. 45 46* Moved the code of the deprecated tools into a 'tools' directory and 47 completely decoupled their code from the internals of atf-c++. The 48 reason for this is to painlessly allow a third-party to maintain a 49 copy of these tools after we delete them because upcoming changes to 50 atf-c++ would break the stale tools. 51 52 53Changes in version 0.18 54*********************** 55 56Experimental version released on November 16th, 2013. 57 58* Issue 45: Added require.memory support in atf-run for FreeBSD. 59 60* Fixed an issue with the handling of cin with libc++. 61 62* Issue 64: Fixed various mandoc formatting warnings. 63 64* NetBSD PR bin/48284: Made atf-check flush its progress message to 65 stdout so that an interrupted test case always shows the last message 66 being executed. 67 68* NetBSD PR bin/48285: Fixed atf_check examples in atf-sh-api(3). 69 70 71Changes in version 0.17 72*********************** 73 74Experimental version released on February 14th, 2013. 75 76* Added the atf_utils_cat_file, atf_utils_compare_file, 77 atf_utils_copy_file, atf_utils_create_file, atf_utils_file_exists, 78 atf_utils_fork, atf_utils_grep_file, atf_utils_grep_string, 79 atf_utils_readline, atf_utils_redirect and atf_utils_wait utility 80 functions to atf-c-api. Documented the already-public 81 atf_utils_free_charpp function. 82 83* Added the cat_file, compare_file, copy_file, create_file, file_exists, 84 fork, grep_collection, grep_file, grep_string, redirect and wait 85 functions to the atf::utils namespace of atf-c++-api. These are 86 wrappers around the same functions added to the atf-c-api library. 87 88* Added the ATF_CHECK_MATCH, ATF_CHECK_MATCH_MSG, ATF_REQUIRE_MATCH and 89 ATF_REQUIRE_MATCH_MSG macros to atf-c to simplify the validation of a 90 string against a regular expression. 91 92* Miscellaneous fixes for manpage typos and compilation problems with 93 clang. 94 95* Added caching of the results of those configure tests that rely on 96 executing a test program. This should help crossbuild systems by 97 providing a mechanism to pre-specify what the results should be. 98 99* PR bin/45690: Make atf-report convert any non-printable characters to 100 a plain-text representation (matching their corresponding hexadecimal 101 entities) in XML output files. This is to prevent the output of test 102 cases from breaking xsltproc later. 103 104 105Changes in version 0.16 106*********************** 107 108Experimental version released on July 10th, 2012. 109 110* Added a --enable-tools flag to configure to request the build of the 111 deprecated ATF tools, whose build is now disabled by default. In order 112 to continue running tests, you should migrate to Kyua instead of enabling 113 the build of the deprecated tools. The kyua-atf-compat package provides 114 transitional compatibility versions of atf-run and atf-report built on 115 top of Kyua. 116 117* Tweaked the ATF_TEST_CASE macro of atf-c++ so that the compiler can 118 detect defined but unused test cases. 119 120* PR bin/45859: Fixed some XSLT bugs that resulted in the tc-time and 121 tp-time XML tags leaking into the generated HTML file. Also improved 122 the CSS file slightly to correct alignment and color issues with the 123 timestamps column. 124 125* Optimized atf-c++/macros.hpp so that GNU G++ consumes less memory during 126 compilation with GNU G++. 127 128* Flipped the default to building shared libraries for atf-c and atf-c++, 129 and started versioning them. As a side-effect, this removes the 130 --enable-unstable-shared flag from configure that appears to not work any 131 more (under NetBSD). Additionally, some distributions require the use of 132 shared libraries for proper dependency tracking (e.g. Fedora), so it is 133 better if we do the right versioning upstream. 134 135* Project hosting moved from an adhoc solution (custom web site and 136 Monotone repository) to Google Code (standard wiki and Git). ATF now 137 lives in a subcomponent of the Kyua project. 138 139 140Changes in version 0.15 141*********************** 142 143Experimental version released on January 16th, 2012. 144 145* Respect stdin in atf-check. The previous release silenced stdin for any 146 processes spawned by atf, not only test programs, which caused breakage 147 in tests that pipe data through atf-check. 148 149* Performance improvements to atf-sh. 150 151* Enabled detection of unused parameters and variables in the code and 152 fixed all warnings. 153 154* Changed the behavior of "developer mode". Compiler warnings are now 155 enabled unconditionally regardless of whether we are in developer mode or 156 not; developer mode is now only used to perform strict warning checks and 157 to enable assertions. Additionally, developer mode is now only 158 automatically enabled when building from the repository, not for formal 159 releases. 160 161* Added new Autoconf M4 macros (ATF_ARG_WITH, ATF_CHECK_C and 162 ATF_CHECK_CXX) to provide a consistent way of defining a --with-arg flag 163 in configure scripts and detecting the presence of any of the ATF 164 bindings. Note that ATF_CHECK_SH was already introduced in 0.14, but it 165 has now been modified to also honor --with-atf if instantiated. 166 167* Added timing support to atf-run / atf-report. 168 169* Added support for a 'require.memory' property, to specify the minimum 170 amount of physical memory needed by the test case to yield valid results. 171 172* PR bin/45690: Force an ISO-8859-1 encoding in the XML files generated by 173 atf-report so that invalid data in the output of test cases does not 174 mangle our report. 175 176 177Changes in version 0.14 178*********************** 179 180Experimental version released on June 14th, 2011. 181 182* Added a pkg-config file for atf-sh and an aclocal file to ease the 183 detection of atf-sh from autoconf scripts. 184 185* Made the default test case body defined by atf_sh fail. This is to 186 ensure that test cases are properly defined in test programs and helps 187 in catching typos in the names of the body functions. 188 189* PR bin/44882: Made atf-run connect the stdin of test cases to /dev/zero. 190 This provides more consistent results with "normal" execution (in 191 particular, when tests are executed detached from a terminal). 192 193* Made atf-run hardcode TZ=UTC for test cases. It used to undefine TZ, but 194 that does not take into account that libc determines the current timezone 195 from a configuration file. 196 197* All test programs will now print a warning when they are not run through 198 atf-run(1) stating that this is unsupported and may deliver incorrect 199 results. 200 201* Added support for the 'require.files' test-case property. This allows 202 test cases to specify installed files that must be present for the test 203 case to run. 204 205 206Changes in version 0.13 207*********************** 208 209Experimental version released on March 31st, 2011. 210 211This is the first release after the creation of the Kyua project, a more 212modular and reliable replacement for ATF. From now on, ATF will change to 213accomodate the transition to this new codebase, but ATF will still continue 214to see development in the short/medium term. Check out the project page at 215http://code.google.com/p/kyua/ for more details. 216 217The changes in this release are: 218 219* Added support to run the tests with the Kyua runtime engine (kyua-cli), a 220 new package that aims to replace atf-run and atf-report. The ATF tests 221 can be run with the new system by issuing a 'make installcheck-kyua' from 222 the top-level directory of the project (assuming the 'kyua' binary is 223 available during the configuration stage of ATF). 224 225* atf-run and atf-report are now in maintenance mode (but *not* deprecated 226 yet!). Kyua already implements a new, much more reliable runtime engine 227 that provides similar features to these tools. That said, it is not 228 complete yet so all development efforts should go towards it. 229 230* If GDB is installed, atf-run dumps the stack trace of crashing test 231 programs in an attempt to aid debugging. Contributed by Antti Kantee. 232 233* Reverted default timeout change in previous release and reset its value 234 to 5 minutes. This was causing several issues, specially when running 235 the existing NetBSD test suite in qemu. 236 237* Fixed the 'match' output checker in atf-check to properly validate the 238 last line of a file even if it does not have a newline. 239 240* Added the ATF_REQUIRE_IN and ATF_REQUIRE_NOT_IN macros to atf-c++ to 241 check for the presence (or lack thereof) of an element in a collection. 242 243* PR bin/44176: Fixed a race condition in atf-run that would crash atf-run 244 when the cleanup of a test case triggered asynchronous modifications to 245 its work directory (e.g. killing a daemon process that cleans up a pid 246 file in the work directory). 247 248* PR bin/44301: Fixed the sample XSLT file to report bogus test programs 249 instead of just listing them as having 0 test cases. 250 251 252Changes in version 0.12 253*********************** 254 255Experimental version released on November 7th, 2010. 256 257* Added the ATF_REQUIRE_THROW_RE to atf-c++, which is the same as 258 ATF_REQUIRE_THROW but allows checking for the validity of the exception's 259 error message by means of a regular expression. 260 261* Added the ATF_REQUIRE_MATCH to atf-c++, which allows checking for a 262 regular expression match in a string. 263 264* Changed the default timeout for test cases from 5 minutes to 30 seconds. 265 30 seconds is long enough for virtually all tests to complete, and 5 266 minutes is a way too long pause in a test suite where a single test case 267 stalls. 268 269* Deprecated the use.fs property. While this seemed like a good idea in 270 the first place to impose more control on what test cases can do, it 271 turns out to be bad. First, use.fs=false prevents bogus test cases 272 from dumping core so after-the-fact debugging is harder. Second, 273 supporting use.fs adds a lot of unnecessary complexity. atf-run will 274 now ignore any value provided to use.fs and will allow test cases to 275 freely access the file system if they wish to. 276 277* Added the atf_tc_get_config_var_as_{bool,long}{,_wd} functions to the atf-c 278 library. The 'text' module became private in 0.11 but was being used 279 externally to simplify the parsing of configuration variables. 280 281* Made atf-run recognize the 'unprivileged-user' configuration variable 282 and automatically drop root privileges when a test case sets 283 require.user=unprivileged. Note that this is, by no means, done for 284 security purposes; this is just for user convenience; tests should, in 285 general, not be blindly run as root in the first place. 286 287 288Changes in version 0.11 289*********************** 290 291Experimental version released on October 20th, 2010. 292 293* The ATF_CHECK* macros in atf-c++ were renamed to ATF_REQUIRE* to match 294 their counterparts in atf-c. 295 296* Clearly separated the modules in atf-c that are supposed to be public 297 from those that are implementation details. The header files for the 298 internal modules are not installed any more. 299 300* Made the atf-check tool private. It is only required by atf-sh and being 301 public has the danger of causing confusion. Also, making it private 302 simplifies the public API of atf. 303 304* Changed atf-sh to enable per-command error checking (set -e) by default. 305 This catches many cases in which a test case is broken but it is not 306 reported as such because execution continues. 307 308* Fixed the XSTL and CSS stylesheets to support expected failures. 309 310 311Changes in version 0.10 312*********************** 313 314Experimental version released on July 2nd, 2010. 315 316Miscellaneous features 317 318* Added expected failures support to test cases and atf-run. These 319 include, for example, expected clean exits, expected reception of fatal 320 signals, expected timeouts and expected errors in condition checks. 321 These statuses can be used to denote test cases that are known to fail 322 due to a bug in the code they are testing. atf-report reports these 323 tests separately but they do not count towards the failed test cases 324 amount. 325 326* Added the ATF_CHECK_ERRNO and ATF_REQUIRE_ERRNO to the C library to 327 allow easy checking of call failures that update errno. 328 329* Added the has.cleanup meta-data property to test caes that specifies 330 whether the test case has a cleanup routine or not; its value is 331 automatically set. This property is read by atf-run to know if it has to 332 run the cleanup routine; skipping this run for every test case 333 significantly speeds up the run time of test suites. 334 335* Reversed the order of the ATF_CHECK_THROW macro in the C++ binding to 336 take the expected exception as the first argument and the statement to 337 execute as the second argument. 338 339Changes in atf-check 340 341* Changed atf-check to support negating the status and output checks by 342 prefixing them with not- and added support to specify multiple checkers 343 for stdout and stderr, not only one. 344 345* Added the match output checker to atf-check to look for regular 346 expressions in the stdout and stderr of commands. 347 348* Modified the exit checks in atf-check to support checking for the 349 reception of signals. 350 351Code simplifications and cleanups 352 353* Removed usage messages from test programs to simplify the 354 implementation of every binding by a significant amount. They just now 355 refer the user to the appropriate manual page and do not attempt to wrap 356 lines on terminal boundaries. Test programs are not supposed to be run 357 by users directly so this minor interface regression is not important. 358 359* Removed the atf-format internal utility, which is unused after the 360 change documented above. 361 362* Removed the atf-cleanup internal utility. It has been unused since the 363 test case isolation was moved to atf-run in 0.8 364 365* Splitted the Makefile.am into smaller files for easier maintenance and 366 dropped the use of M4. Only affects users building from the repository 367 sources. 368 369* Intermixed tests with the source files in the source tree to provide 370 them more visibility and easier access. The tests directory is gone from 371 the source tree and tests are now suffixed by _test, not prefixed by t_. 372 373* Simplifications to the atf-c library: removed the io, tcr and ui 374 modules as they had become unnecessary after all simplifications 375 introduced since the 0.8 release. 376 377* Removed the application/X-atf-tcr format introduced in 0.8 release. 378 Tests now print a much simplified format that is easy to parse and nicer 379 to read by end users. As a side effect, the default for test cases is 380 now to print their results to stdout unless otherwise stated by providing 381 the -r flag. 382 383* Removed XML distribution documents and replaced them with plain-text 384 documents. They provided little value and introduced a lot of complexity 385 to the build system. 386 387* Simplified the output of atf-version by not attempting to print a 388 revision number when building form a distfile. Makes the build system 389 easier to maintain. 390 391 392Changes in version 0.9 393********************** 394 395Experimental version released on June 3rd, 2010. 396 397* Added atf-sh, an interpreter to process test programs written using 398 the shell API. This is not really a shell interpreter by itself though: 399 it is just a wrapper around the system shell that eases the loading of 400 the necessary ATF libraries. 401 402* Removed atf-compile in favour of atf-sh. 403 404* Added the use.fs metadata property to test case, which is used to 405 specify which test cases require file system access. This is to 406 highlight dependencies on external resources more clearly and to speed up 407 the execution of test suites by skipping the creation of many unnecessary 408 work directories. 409 410* Fixed test programs to get a sane default value for their source 411 directory. This means that it should not be necessary any more to pass 412 -s when running test programs that do not live in the current directory. 413 414* Defining test case headers became optional. This is trivial to achieve 415 in shell-based tests but a bit ugly in C and C++. In C, use the new 416 ATF_TC_WITHOUT_HEAD macro to define the test case, and in C++ use 417 ATF_TEST_CASE_WITHOUT_HEAD. 418 419 420Changes in version 0.8 421********************** 422 423Experimental version released on May 7th, 2010. 424 425* Test programs no longer run several test cases in a row. The execution 426 of a test program now requires a test case name, and that single test 427 case is executed. To execute several test cases, use the atf-run utility 428 as usual. 429 430* Test programs no longer fork a subprocess to isolate the execution of 431 test cases. They run the test case code in-process, and a crash of the 432 test case will result in a crash of the test program. This is to ease 433 debugging of faulty test cases. 434 435* Test programs no longer isolate their test cases. This means that they 436 will not create temporary directories nor sanitize the environment any 437 more. Yes: running a test case that depends on system state by hand will 438 most likely yield different results depending on where (machine, 439 directory, user environment, etc.) it is run. Isolation has been moved 440 to atf-run. 441 442* Test programs no longer print a cryptic format (application/X-atf-tcs) 443 on a special file channel. They can now print whatever they want on the 444 screen. Because test programs can now only run one test case every time, 445 providing controlled output is not necessary any more. 446 447* Test programs no longer write their status into a special file 448 descriptor. Instead, they create a file with the results, which is later 449 parsed by atf-run. This changes the semantics of the -r flag. 450 451* atf-run has been adjusted to perform the test case isolation. As a 452 result, there is now a single canonical place that implements the 453 isolation of test caes. In previous releases, the three language 454 bindings (C, C++ and shell) had to be kept in sync with each other (read: 455 not a nice thing to do at all). As a side effect of this change, writing 456 bindings for other languages will be much, much easier from now on. 457 458* atf-run forks test programs on a test case basis, instead of on a test 459 program basis as it did before. This is to provide the test case 460 isolation that was before implemented by the test programs themselves. 461 462* Removed the atf-exec tool. This was used to implement test case 463 isolation in atf-sh, but it is now unnecessary. 464 465* It is now optional to define the descr meta-data property. It has been 466 proven to be mostly useless, because test cases often carry a descriptive 467 name of their own. 468 469 470Changes in version 0.7 471********************** 472 473Experimental version released on December 22nd, 2009. 474 475* Added build-time checks to atf-c and atf-c++. A binding for atf-sh 476 will come later. 477 478* Migrated all build-time checks for header files to proper ATF tests. 479 This demonstrates the use of the new feature described above. 480 481* Added an internal API for child process management. 482 483* Converted all plain-text distribution documents to a Docbook canonical 484 version, and include pre-generated plain text and HTML copies in the 485 distribution file. 486 487* Simplified the contents of the Makefile.am by regenerating it from a 488 canonical Makefile.am.m4 source. As a side-effect, some dependency 489 specifications were fixed. 490 491* Migrated all checks from the check target to installcheck, as these 492 require ATF to be installed. 493 494* Fixed sign comparison mismatches triggered by the now-enabled 495 -Wsign-compare. 496 497* Fixed many memory and object leaks. 498 499 500Changes in version 0.6 501********************** 502 503Experimental version released on January 18th, 2009. 504 505* Make atf-exec be able to kill its child process after a certain period 506 of time; this is controlled through the new -t option. 507 508* Change atf-sh to use atf-exec's -t option to control the test case's 509 timeouts, instead of doing it internally. Same behavior as before, but 510 noticeably faster. 511 512* atf-exec's -g option and atf-killpg are gone due to the previous 513 change. 514 515* Added the atf-check(1) tool, a program that executes a given command 516 and checks its exit code against a known value and allows the management 517 of stdout and stderr in multiple ways. This replaces the previous 518 atf_check function in the atf-sh library and exposes this functionality 519 to both atf-c and atf-c++. 520 521* Added the ATF_REQUIRE family of macros to the C interface. These help 522 in checking for fatal test conditions. The old ATF_CHECK macros now 523 perform non-fatal checks only. I.e. by using ATF_CHECK, the test case 524 can now continue its execution and the failures will not be reported 525 until the end of the whole run. 526 527* Extended the amount of ATF_CHECK_* C macros with new ones to provide 528 more features to the developer. These also have their corresponding 529 counterparts in the ATF_REQUIRE_* family. The new macros (listing the 530 suffixes only) are: _EQ (replaces _EQUAL), _EQ_MSG, _STREQ and 531 _STREQ_MSG. 532 533 534Changes in version 0.5 535********************** 536 537Experimental version released on May 1st, 2008. 538 539* Clauses 3 and 4 of the BSD license used by the project were dropped. 540 All the code is now under a 2-clause BSD license compatible with the GNU 541 General Public License (GPL). 542 543* Added a C-only binding so that binary test programs do not need to be 544 tied to C++ at all. This binding is now known as the atf-c library. 545 546* Renamed the C++ binding to atf-c++ for consistency with the new atf-c. 547 548* Renamed the POSIX shell binding to atf-sh for consistency with the new 549 atf-c and atf-c++. 550 551* Added a -w flag to test programs through which it is possible to 552 specify the work directory to be used. This was possible in prior 553 releases by defining the workdir configuration variable (-v workdir=...), 554 but was a conceptually incorrect mechanism. 555 556* Test programs now preserve the execution order of test cases when they 557 are given in the command line. Even those mentioned more than once are 558 executed multiple times to comply with the user's requests. 559 560 561Changes in version 0.4 562********************** 563 564Experimental version released on February 4th, 2008. 565 566* Added two new manual pages, atf-c++-api and atf-sh-api, describing the 567 C++ and POSIX shell interfaces used to write test programs. 568 569* Added a pkg-config file, useful to get the flags to build against the 570 C++ library or to easily detect the presence of ATF. 571 572* Added a way for test cases to require a specific architecture and/or 573 machine type through the new 'require.arch' and 'require.machine' 574 meta-data properties, respectively. 575 576* Added the 'timeout' property to test cases, useful to set an 577 upper-bound limit for the test's run time and thus prevent global test 578 program stalls due to the test case's misbehavior. 579 580* Added the atf-exec(1) internal utility, used to execute a command 581 after changing the process group it belongs to. 582 583* Added the atf-killpg(1) internal utility, used to kill process groups. 584 585* Multiple portability fixes. Of special interest, full support for 586 SunOS (Solaris Express Developer Edition 2007/09) using the Sun Studio 12 587 C++ compiler. 588 589* Fixed a serious bug that prevented atf-run(1) from working at all 590 under Fedora 8 x86_64. Due to the nature of the bug, other platforms 591 were likely affected too. 592 593 594Changes in version 0.3 595********************** 596 597Experimental version released on November 11th, 2007. 598 599* Added XML output support to atf-report. This is accompanied by a DTD 600 for the format's structure and sample XSLT/CSS files to post-process this 601 output and convert it to a plain HTML report. 602 603* Changed atf-run to add system information to the report it generates. 604 This is currently used by atf-report's XML output only, and is later 605 printed in the HTML reports in a nice and useful summary table. The user 606 and system administrator are allowed to tune this feature by means of 607 hooks. 608 609* Removed the test cases' 'isolated' property. This was intended to 610 avoid touching the file system at all when running the related test case, 611 but this has not been true for a long while: some control files are 612 unconditionally required for several purposes, and we cannot easily get 613 rid of them. This way we remove several critical and delicate pieces of 614 code. 615 616* Improved atf-report's CSV output format to include information about 617 test programs too. 618 619* Fixed the tests that used atf-compile to not require this tool as a 620 helper. Avoids systems without build-time utilities to skip many tests 621 that could otherwise be run. (E.g. NetBSD without the comp.tgz set 622 installed.) 623 624* Many general cleanups: Fixed many pieces of code marked as ugly and/or 625 incomplete. 626 627 628Changes in version 0.2 629********************** 630 631Experimental version released on September 20th, 2007. 632 633* Test cases now get a known umask on entry. 634 635* atf-run now detects many unexpected failures caused by test programs and 636 reports them as bogus tests. atf-report is able to handle these new 637 errors and nicely reports them to the user. 638 639* All the data formats read and written by the tools have been 640 documented and cleaned up. These include those grammars that define how 641 the different components communicate with each other as well as the 642 format of files written by the developers and users: the Atffiles and the 643 configuration files. 644 645* Added the atf-version tool, a utility that displays information about 646 the currently installed version of ATF. 647 648* Test cases can now define an optional cleanup routine to undo their 649 actions regardless of their exit status. 650 651* atf-report now summarizes the list of failed (bogus) test programs 652 when using the ticker output format. 653 654* Test programs now capture some termination signals and clean up any 655 temporary files before exiting the program. 656 657* Multiple bug fixes and improvements all around. 658 659 660Changes in version 0.1 661********************** 662 663Experimental version released on August 20th, 2007. 664 665* First public version. This was released coinciding with the end of the 666 Google Summer of Code 2007 program. 667 668 669=========================================================================== 670vim: filetype=text:textwidth=75:expandtab:shiftwidth=2:softtabstop=2 671