1Major changes between releases Automated Testing Framework 2=========================================================================== 3 4 5Changes in version 0.18 6*********************** 7 8Experimental version released on November 16th, 2013. 9 10* Issue 45: Added require.memory support in atf-run for FreeBSD. 11 12* Fixed an issue with the handling of cin with libc++. 13 14* Issue 64: Fixed various mandoc formatting warnings. 15 16* NetBSD PR bin/48284: Made atf-check flush its progress message to 17 stdout so that an interrupted test case always shows the last message 18 being executed. 19 20* NetBSD PR bin/48285: Fixed atf_check examples in atf-sh-api(3). 21 22 23Changes in version 0.17 24*********************** 25 26Experimental version released on February 14th, 2013. 27 28* Added the atf_utils_cat_file, atf_utils_compare_file, 29 atf_utils_copy_file, atf_utils_create_file, atf_utils_file_exists, 30 atf_utils_fork, atf_utils_grep_file, atf_utils_grep_string, 31 atf_utils_readline, atf_utils_redirect and atf_utils_wait utility 32 functions to atf-c-api. Documented the already-public 33 atf_utils_free_charpp function. 34 35* Added the cat_file, compare_file, copy_file, create_file, file_exists, 36 fork, grep_collection, grep_file, grep_string, redirect and wait 37 functions to the atf::utils namespace of atf-c++-api. These are 38 wrappers around the same functions added to the atf-c-api library. 39 40* Added the ATF_CHECK_MATCH, ATF_CHECK_MATCH_MSG, ATF_REQUIRE_MATCH and 41 ATF_REQUIRE_MATCH_MSG macros to atf-c to simplify the validation of a 42 string against a regular expression. 43 44* Miscellaneous fixes for manpage typos and compilation problems with 45 clang. 46 47* Added caching of the results of those configure tests that rely on 48 executing a test program. This should help crossbuild systems by 49 providing a mechanism to pre-specify what the results should be. 50 51* PR bin/45690: Make atf-report convert any non-printable characters to 52 a plain-text representation (matching their corresponding hexadecimal 53 entities) in XML output files. This is to prevent the output of test 54 cases from breaking xsltproc later. 55 56 57Changes in version 0.16 58*********************** 59 60Experimental version released on July 10th, 2012. 61 62* Added a --enable-tools flag to configure to request the build of the 63 deprecated ATF tools, whose build is now disabled by default. In order 64 to continue running tests, you should migrate to Kyua instead of enabling 65 the build of the deprecated tools. The kyua-atf-compat package provides 66 transitional compatibility versions of atf-run and atf-report built on 67 top of Kyua. 68 69* Tweaked the ATF_TEST_CASE macro of atf-c++ so that the compiler can 70 detect defined but unused test cases. 71 72* PR bin/45859: Fixed some XSLT bugs that resulted in the tc-time and 73 tp-time XML tags leaking into the generated HTML file. Also improved 74 the CSS file slightly to correct alignment and color issues with the 75 timestamps column. 76 77* Optimized atf-c++/macros.hpp so that GNU G++ consumes less memory during 78 compilation with GNU G++. 79 80* Flipped the default to building shared libraries for atf-c and atf-c++, 81 and started versioning them. As a side-effect, this removes the 82 --enable-unstable-shared flag from configure that appears to not work any 83 more (under NetBSD). Additionally, some distributions require the use of 84 shared libraries for proper dependency tracking (e.g. Fedora), so it is 85 better if we do the right versioning upstream. 86 87* Project hosting moved from an adhoc solution (custom web site and 88 Monotone repository) to Google Code (standard wiki and Git). ATF now 89 lives in a subcomponent of the Kyua project. 90 91 92Changes in version 0.15 93*********************** 94 95Experimental version released on January 16th, 2012. 96 97* Respect stdin in atf-check. The previous release silenced stdin for any 98 processes spawned by atf, not only test programs, which caused breakage 99 in tests that pipe data through atf-check. 100 101* Performance improvements to atf-sh. 102 103* Enabled detection of unused parameters and variables in the code and 104 fixed all warnings. 105 106* Changed the behavior of "developer mode". Compiler warnings are now 107 enabled unconditionally regardless of whether we are in developer mode or 108 not; developer mode is now only used to perform strict warning checks and 109 to enable assertions. Additionally, developer mode is now only 110 automatically enabled when building from the repository, not for formal 111 releases. 112 113* Added new Autoconf M4 macros (ATF_ARG_WITH, ATF_CHECK_C and 114 ATF_CHECK_CXX) to provide a consistent way of defining a --with-arg flag 115 in configure scripts and detecting the presence of any of the ATF 116 bindings. Note that ATF_CHECK_SH was already introduced in 0.14, but it 117 has now been modified to also honor --with-atf if instantiated. 118 119* Added timing support to atf-run / atf-report. 120 121* Added support for a 'require.memory' property, to specify the minimum 122 amount of physical memory needed by the test case to yield valid results. 123 124* PR bin/45690: Force an ISO-8859-1 encoding in the XML files generated by 125 atf-report so that invalid data in the output of test cases does not 126 mangle our report. 127 128 129Changes in version 0.14 130*********************** 131 132Experimental version released on June 14th, 2011. 133 134* Added a pkg-config file for atf-sh and an aclocal file to ease the 135 detection of atf-sh from autoconf scripts. 136 137* Made the default test case body defined by atf_sh fail. This is to 138 ensure that test cases are properly defined in test programs and helps 139 in catching typos in the names of the body functions. 140 141* PR bin/44882: Made atf-run connect the stdin of test cases to /dev/zero. 142 This provides more consistent results with "normal" execution (in 143 particular, when tests are executed detached from a terminal). 144 145* Made atf-run hardcode TZ=UTC for test cases. It used to undefine TZ, but 146 that does not take into account that libc determines the current timezone 147 from a configuration file. 148 149* All test programs will now print a warning when they are not run through 150 atf-run(1) stating that this is unsupported and may deliver incorrect 151 results. 152 153* Added support for the 'require.files' test-case property. This allows 154 test cases to specify installed files that must be present for the test 155 case to run. 156 157 158Changes in version 0.13 159*********************** 160 161Experimental version released on March 31st, 2011. 162 163This is the first release after the creation of the Kyua project, a more 164modular and reliable replacement for ATF. From now on, ATF will change to 165accomodate the transition to this new codebase, but ATF will still continue 166to see development in the short/medium term. Check out the project page at 167http://code.google.com/p/kyua/ for more details. 168 169The changes in this release are: 170 171* Added support to run the tests with the Kyua runtime engine (kyua-cli), a 172 new package that aims to replace atf-run and atf-report. The ATF tests 173 can be run with the new system by issuing a 'make installcheck-kyua' from 174 the top-level directory of the project (assuming the 'kyua' binary is 175 available during the configuration stage of ATF). 176 177* atf-run and atf-report are now in maintenance mode (but *not* deprecated 178 yet!). Kyua already implements a new, much more reliable runtime engine 179 that provides similar features to these tools. That said, it is not 180 complete yet so all development efforts should go towards it. 181 182* If GDB is installed, atf-run dumps the stack trace of crashing test 183 programs in an attempt to aid debugging. Contributed by Antti Kantee. 184 185* Reverted default timeout change in previous release and reset its value 186 to 5 minutes. This was causing several issues, specially when running 187 the existing NetBSD test suite in qemu. 188 189* Fixed the 'match' output checker in atf-check to properly validate the 190 last line of a file even if it does not have a newline. 191 192* Added the ATF_REQUIRE_IN and ATF_REQUIRE_NOT_IN macros to atf-c++ to 193 check for the presence (or lack thereof) of an element in a collection. 194 195* PR bin/44176: Fixed a race condition in atf-run that would crash atf-run 196 when the cleanup of a test case triggered asynchronous modifications to 197 its work directory (e.g. killing a daemon process that cleans up a pid 198 file in the work directory). 199 200* PR bin/44301: Fixed the sample XSLT file to report bogus test programs 201 instead of just listing them as having 0 test cases. 202 203 204Changes in version 0.12 205*********************** 206 207Experimental version released on November 7th, 2010. 208 209* Added the ATF_REQUIRE_THROW_RE to atf-c++, which is the same as 210 ATF_REQUIRE_THROW but allows checking for the validity of the exception's 211 error message by means of a regular expression. 212 213* Added the ATF_REQUIRE_MATCH to atf-c++, which allows checking for a 214 regular expression match in a string. 215 216* Changed the default timeout for test cases from 5 minutes to 30 seconds. 217 30 seconds is long enough for virtually all tests to complete, and 5 218 minutes is a way too long pause in a test suite where a single test case 219 stalls. 220 221* Deprecated the use.fs property. While this seemed like a good idea in 222 the first place to impose more control on what test cases can do, it 223 turns out to be bad. First, use.fs=false prevents bogus test cases 224 from dumping core so after-the-fact debugging is harder. Second, 225 supporting use.fs adds a lot of unnecessary complexity. atf-run will 226 now ignore any value provided to use.fs and will allow test cases to 227 freely access the file system if they wish to. 228 229* Added the atf_tc_get_config_var_as_{bool,long}{,_wd} functions to the atf-c 230 library. The 'text' module became private in 0.11 but was being used 231 externally to simplify the parsing of configuration variables. 232 233* Made atf-run recognize the 'unprivileged-user' configuration variable 234 and automatically drop root privileges when a test case sets 235 require.user=unprivileged. Note that this is, by no means, done for 236 security purposes; this is just for user convenience; tests should, in 237 general, not be blindly run as root in the first place. 238 239 240Changes in version 0.11 241*********************** 242 243Experimental version released on October 20th, 2010. 244 245* The ATF_CHECK* macros in atf-c++ were renamed to ATF_REQUIRE* to match 246 their counterparts in atf-c. 247 248* Clearly separated the modules in atf-c that are supposed to be public 249 from those that are implementation details. The header files for the 250 internal modules are not installed any more. 251 252* Made the atf-check tool private. It is only required by atf-sh and being 253 public has the danger of causing confusion. Also, making it private 254 simplifies the public API of atf. 255 256* Changed atf-sh to enable per-command error checking (set -e) by default. 257 This catches many cases in which a test case is broken but it is not 258 reported as such because execution continues. 259 260* Fixed the XSTL and CSS stylesheets to support expected failures. 261 262 263Changes in version 0.10 264*********************** 265 266Experimental version released on July 2nd, 2010. 267 268Miscellaneous features 269 270* Added expected failures support to test cases and atf-run. These 271 include, for example, expected clean exits, expected reception of fatal 272 signals, expected timeouts and expected errors in condition checks. 273 These statuses can be used to denote test cases that are known to fail 274 due to a bug in the code they are testing. atf-report reports these 275 tests separately but they do not count towards the failed test cases 276 amount. 277 278* Added the ATF_CHECK_ERRNO and ATF_REQUIRE_ERRNO to the C library to 279 allow easy checking of call failures that update errno. 280 281* Added the has.cleanup meta-data property to test caes that specifies 282 whether the test case has a cleanup routine or not; its value is 283 automatically set. This property is read by atf-run to know if it has to 284 run the cleanup routine; skipping this run for every test case 285 significantly speeds up the run time of test suites. 286 287* Reversed the order of the ATF_CHECK_THROW macro in the C++ binding to 288 take the expected exception as the first argument and the statement to 289 execute as the second argument. 290 291Changes in atf-check 292 293* Changed atf-check to support negating the status and output checks by 294 prefixing them with not- and added support to specify multiple checkers 295 for stdout and stderr, not only one. 296 297* Added the match output checker to atf-check to look for regular 298 expressions in the stdout and stderr of commands. 299 300* Modified the exit checks in atf-check to support checking for the 301 reception of signals. 302 303Code simplifications and cleanups 304 305* Removed usage messages from test programs to simplify the 306 implementation of every binding by a significant amount. They just now 307 refer the user to the appropriate manual page and do not attempt to wrap 308 lines on terminal boundaries. Test programs are not supposed to be run 309 by users directly so this minor interface regression is not important. 310 311* Removed the atf-format internal utility, which is unused after the 312 change documented above. 313 314* Removed the atf-cleanup internal utility. It has been unused since the 315 test case isolation was moved to atf-run in 0.8 316 317* Splitted the Makefile.am into smaller files for easier maintenance and 318 dropped the use of M4. Only affects users building from the repository 319 sources. 320 321* Intermixed tests with the source files in the source tree to provide 322 them more visibility and easier access. The tests directory is gone from 323 the source tree and tests are now suffixed by _test, not prefixed by t_. 324 325* Simplifications to the atf-c library: removed the io, tcr and ui 326 modules as they had become unnecessary after all simplifications 327 introduced since the 0.8 release. 328 329* Removed the application/X-atf-tcr format introduced in 0.8 release. 330 Tests now print a much simplified format that is easy to parse and nicer 331 to read by end users. As a side effect, the default for test cases is 332 now to print their results to stdout unless otherwise stated by providing 333 the -r flag. 334 335* Removed XML distribution documents and replaced them with plain-text 336 documents. They provided little value and introduced a lot of complexity 337 to the build system. 338 339* Simplified the output of atf-version by not attempting to print a 340 revision number when building form a distfile. Makes the build system 341 easier to maintain. 342 343 344Changes in version 0.9 345********************** 346 347Experimental version released on June 3rd, 2010. 348 349* Added atf-sh, an interpreter to process test programs written using 350 the shell API. This is not really a shell interpreter by itself though: 351 it is just a wrapper around the system shell that eases the loading of 352 the necessary ATF libraries. 353 354* Removed atf-compile in favour of atf-sh. 355 356* Added the use.fs metadata property to test case, which is used to 357 specify which test cases require file system access. This is to 358 highlight dependencies on external resources more clearly and to speed up 359 the execution of test suites by skipping the creation of many unnecessary 360 work directories. 361 362* Fixed test programs to get a sane default value for their source 363 directory. This means that it should not be necessary any more to pass 364 -s when running test programs that do not live in the current directory. 365 366* Defining test case headers became optional. This is trivial to achieve 367 in shell-based tests but a bit ugly in C and C++. In C, use the new 368 ATF_TC_WITHOUT_HEAD macro to define the test case, and in C++ use 369 ATF_TEST_CASE_WITHOUT_HEAD. 370 371 372Changes in version 0.8 373********************** 374 375Experimental version released on May 7th, 2010. 376 377* Test programs no longer run several test cases in a row. The execution 378 of a test program now requires a test case name, and that single test 379 case is executed. To execute several test cases, use the atf-run utility 380 as usual. 381 382* Test programs no longer fork a subprocess to isolate the execution of 383 test cases. They run the test case code in-process, and a crash of the 384 test case will result in a crash of the test program. This is to ease 385 debugging of faulty test cases. 386 387* Test programs no longer isolate their test cases. This means that they 388 will not create temporary directories nor sanitize the environment any 389 more. Yes: running a test case that depends on system state by hand will 390 most likely yield different results depending on where (machine, 391 directory, user environment, etc.) it is run. Isolation has been moved 392 to atf-run. 393 394* Test programs no longer print a cryptic format (application/X-atf-tcs) 395 on a special file channel. They can now print whatever they want on the 396 screen. Because test programs can now only run one test case every time, 397 providing controlled output is not necessary any more. 398 399* Test programs no longer write their status into a special file 400 descriptor. Instead, they create a file with the results, which is later 401 parsed by atf-run. This changes the semantics of the -r flag. 402 403* atf-run has been adjusted to perform the test case isolation. As a 404 result, there is now a single canonical place that implements the 405 isolation of test caes. In previous releases, the three language 406 bindings (C, C++ and shell) had to be kept in sync with each other (read: 407 not a nice thing to do at all). As a side effect of this change, writing 408 bindings for other languages will be much, much easier from now on. 409 410* atf-run forks test programs on a test case basis, instead of on a test 411 program basis as it did before. This is to provide the test case 412 isolation that was before implemented by the test programs themselves. 413 414* Removed the atf-exec tool. This was used to implement test case 415 isolation in atf-sh, but it is now unnecessary. 416 417* It is now optional to define the descr meta-data property. It has been 418 proven to be mostly useless, because test cases often carry a descriptive 419 name of their own. 420 421 422Changes in version 0.7 423********************** 424 425Experimental version released on December 22nd, 2009. 426 427* Added build-time checks to atf-c and atf-c++. A binding for atf-sh 428 will come later. 429 430* Migrated all build-time checks for header files to proper ATF tests. 431 This demonstrates the use of the new feature described above. 432 433* Added an internal API for child process management. 434 435* Converted all plain-text distribution documents to a Docbook canonical 436 version, and include pre-generated plain text and HTML copies in the 437 distribution file. 438 439* Simplified the contents of the Makefile.am by regenerating it from a 440 canonical Makefile.am.m4 source. As a side-effect, some dependency 441 specifications were fixed. 442 443* Migrated all checks from the check target to installcheck, as these 444 require ATF to be installed. 445 446* Fixed sign comparison mismatches triggered by the now-enabled 447 -Wsign-compare. 448 449* Fixed many memory and object leaks. 450 451 452Changes in version 0.6 453********************** 454 455Experimental version released on January 18th, 2009. 456 457* Make atf-exec be able to kill its child process after a certain period 458 of time; this is controlled through the new -t option. 459 460* Change atf-sh to use atf-exec's -t option to control the test case's 461 timeouts, instead of doing it internally. Same behavior as before, but 462 noticeably faster. 463 464* atf-exec's -g option and atf-killpg are gone due to the previous 465 change. 466 467* Added the atf-check(1) tool, a program that executes a given command 468 and checks its exit code against a known value and allows the management 469 of stdout and stderr in multiple ways. This replaces the previous 470 atf_check function in the atf-sh library and exposes this functionality 471 to both atf-c and atf-c++. 472 473* Added the ATF_REQUIRE family of macros to the C interface. These help 474 in checking for fatal test conditions. The old ATF_CHECK macros now 475 perform non-fatal checks only. I.e. by using ATF_CHECK, the test case 476 can now continue its execution and the failures will not be reported 477 until the end of the whole run. 478 479* Extended the amount of ATF_CHECK_* C macros with new ones to provide 480 more features to the developer. These also have their corresponding 481 counterparts in the ATF_REQUIRE_* family. The new macros (listing the 482 suffixes only) are: _EQ (replaces _EQUAL), _EQ_MSG, _STREQ and 483 _STREQ_MSG. 484 485 486Changes in version 0.5 487********************** 488 489Experimental version released on May 1st, 2008. 490 491* Clauses 3 and 4 of the BSD license used by the project were dropped. 492 All the code is now under a 2-clause BSD license compatible with the GNU 493 General Public License (GPL). 494 495* Added a C-only binding so that binary test programs do not need to be 496 tied to C++ at all. This binding is now known as the atf-c library. 497 498* Renamed the C++ binding to atf-c++ for consistency with the new atf-c. 499 500* Renamed the POSIX shell binding to atf-sh for consistency with the new 501 atf-c and atf-c++. 502 503* Added a -w flag to test programs through which it is possible to 504 specify the work directory to be used. This was possible in prior 505 releases by defining the workdir configuration variable (-v workdir=...), 506 but was a conceptually incorrect mechanism. 507 508* Test programs now preserve the execution order of test cases when they 509 are given in the command line. Even those mentioned more than once are 510 executed multiple times to comply with the user's requests. 511 512 513Changes in version 0.4 514********************** 515 516Experimental version released on February 4th, 2008. 517 518* Added two new manual pages, atf-c++-api and atf-sh-api, describing the 519 C++ and POSIX shell interfaces used to write test programs. 520 521* Added a pkg-config file, useful to get the flags to build against the 522 C++ library or to easily detect the presence of ATF. 523 524* Added a way for test cases to require a specific architecture and/or 525 machine type through the new 'require.arch' and 'require.machine' 526 meta-data properties, respectively. 527 528* Added the 'timeout' property to test cases, useful to set an 529 upper-bound limit for the test's run time and thus prevent global test 530 program stalls due to the test case's misbehavior. 531 532* Added the atf-exec(1) internal utility, used to execute a command 533 after changing the process group it belongs to. 534 535* Added the atf-killpg(1) internal utility, used to kill process groups. 536 537* Multiple portability fixes. Of special interest, full support for 538 SunOS (Solaris Express Developer Edition 2007/09) using the Sun Studio 12 539 C++ compiler. 540 541* Fixed a serious bug that prevented atf-run(1) from working at all 542 under Fedora 8 x86_64. Due to the nature of the bug, other platforms 543 were likely affected too. 544 545 546Changes in version 0.3 547********************** 548 549Experimental version released on November 11th, 2007. 550 551* Added XML output support to atf-report. This is accompanied by a DTD 552 for the format's structure and sample XSLT/CSS files to post-process this 553 output and convert it to a plain HTML report. 554 555* Changed atf-run to add system information to the report it generates. 556 This is currently used by atf-report's XML output only, and is later 557 printed in the HTML reports in a nice and useful summary table. The user 558 and system administrator are allowed to tune this feature by means of 559 hooks. 560 561* Removed the test cases' 'isolated' property. This was intended to 562 avoid touching the file system at all when running the related test case, 563 but this has not been true for a long while: some control files are 564 unconditionally required for several purposes, and we cannot easily get 565 rid of them. This way we remove several critical and delicate pieces of 566 code. 567 568* Improved atf-report's CSV output format to include information about 569 test programs too. 570 571* Fixed the tests that used atf-compile to not require this tool as a 572 helper. Avoids systems without build-time utilities to skip many tests 573 that could otherwise be run. (E.g. NetBSD without the comp.tgz set 574 installed.) 575 576* Many general cleanups: Fixed many pieces of code marked as ugly and/or 577 incomplete. 578 579 580Changes in version 0.2 581********************** 582 583Experimental version released on September 20th, 2007. 584 585* Test cases now get a known umask on entry. 586 587* atf-run now detects many unexpected failures caused by test programs and 588 reports them as bogus tests. atf-report is able to handle these new 589 errors and nicely reports them to the user. 590 591* All the data formats read and written by the tools have been 592 documented and cleaned up. These include those grammars that define how 593 the different components communicate with each other as well as the 594 format of files written by the developers and users: the Atffiles and the 595 configuration files. 596 597* Added the atf-version tool, a utility that displays information about 598 the currently installed version of ATF. 599 600* Test cases can now define an optional cleanup routine to undo their 601 actions regardless of their exit status. 602 603* atf-report now summarizes the list of failed (bogus) test programs 604 when using the ticker output format. 605 606* Test programs now capture some termination signals and clean up any 607 temporary files before exiting the program. 608 609* Multiple bug fixes and improvements all around. 610 611 612Changes in version 0.1 613********************** 614 615Experimental version released on August 20th, 2007. 616 617* First public version. This was released coinciding with the end of the 618 Google Summer of Code 2007 program. 619 620 621=========================================================================== 622vim: filetype=text:textwidth=75:expandtab:shiftwidth=2:softtabstop=2 623