1Major changes between releases Automated Testing Framework 2=========================================================================== 3 4 5Changes in version 0.16 6*********************** 7 8Experimental version released on July 10th, 2012. 9 10* Added a --enable-tools flag to configure to request the build of the 11 deprecated ATF tools, whose build is now disabled by default. In order 12 to continue running tests, you should migrate to Kyua instead of enabling 13 the build of the deprecated tools. The kyua-atf-compat package provides 14 transitional compatibility versions of atf-run and atf-report built on 15 top of Kyua. 16 17* Tweaked the ATF_TEST_CASE macro of atf-c++ so that the compiler can 18 detect defined but unused test cases. 19 20* PR bin/45859: Fixed some XSLT bugs that resulted in the tc-time and 21 tp-time XML tags leaking into the generated HTML file. Also improved 22 the CSS file slightly to correct alignment and color issues with the 23 timestamps column. 24 25* Optimized atf-c++/macros.hpp so that GNU G++ consumes less memory during 26 compilation with GNU G++. 27 28* Flipped the default to building shared libraries for atf-c and atf-c++, 29 and started versioning them. As a side-effect, this removes the 30 --enable-unstable-shared flag from configure that appears to not work any 31 more (under NetBSD). Additionally, some distributions require the use of 32 shared libraries for proper dependency tracking (e.g. Fedora), so it is 33 better if we do the right versioning upstream. 34 35* Project hosting moved from an adhoc solution (custom web site and 36 Monotone repository) to Google Code (standard wiki and Git). ATF now 37 lives in a subcomponent of the Kyua project. 38 39 40Changes in version 0.15 41*********************** 42 43Experimental version released on January 16th, 2012. 44 45* Respect stdin in atf-check. The previous release silenced stdin for any 46 processes spawned by atf, not only test programs, which caused breakage 47 in tests that pipe data through atf-check. 48 49* Performance improvements to atf-sh. 50 51* Enabled detection of unused parameters and variables in the code and 52 fixed all warnings. 53 54* Changed the behavior of "developer mode". Compiler warnings are now 55 enabled unconditionally regardless of whether we are in developer mode or 56 not; developer mode is now only used to perform strict warning checks and 57 to enable assertions. Additionally, developer mode is now only 58 automatically enabled when building from the repository, not for formal 59 releases. 60 61* Added new Autoconf M4 macros (ATF_ARG_WITH, ATF_CHECK_C and 62 ATF_CHECK_CXX) to provide a consistent way of defining a --with-arg flag 63 in configure scripts and detecting the presence of any of the ATF 64 bindings. Note that ATF_CHECK_SH was already introduced in 0.14, but it 65 has now been modified to also honor --with-atf if instantiated. 66 67* Added timing support to atf-run / atf-report. 68 69* Added support for a 'require.memory' property, to specify the minimum 70 amount of physical memory needed by the test case to yield valid results. 71 72* PR bin/45690: Force an ISO-8859-1 encoding in the XML files generated by 73 atf-report so that invalid data in the output of test cases does not 74 mangle our report. 75 76 77Changes in version 0.14 78*********************** 79 80Experimental version released on June 14th, 2011. 81 82* Added a pkg-config file for atf-sh and an aclocal file to ease the 83 detection of atf-sh from autoconf scripts. 84 85* Made the default test case body defined by atf_sh fail. This is to 86 ensure that test cases are properly defined in test programs and helps 87 in catching typos in the names of the body functions. 88 89* PR bin/44882: Made atf-run connect the stdin of test cases to /dev/zero. 90 This provides more consistent results with "normal" execution (in 91 particular, when tests are executed detached from a terminal). 92 93* Made atf-run hardcode TZ=UTC for test cases. It used to undefine TZ, but 94 that does not take into account that libc determines the current timezone 95 from a configuration file. 96 97* All test programs will now print a warning when they are not run through 98 atf-run(1) stating that this is unsupported and may deliver incorrect 99 results. 100 101* Added support for the 'require.files' test-case property. This allows 102 test cases to specify installed files that must be present for the test 103 case to run. 104 105 106Changes in version 0.13 107*********************** 108 109Experimental version released on March 31st, 2011. 110 111This is the first release after the creation of the Kyua project, a more 112modular and reliable replacement for ATF. From now on, ATF will change to 113accomodate the transition to this new codebase, but ATF will still continue 114to see development in the short/medium term. Check out the project page at 115http://code.google.com/p/kyua/ for more details. 116 117The changes in this release are: 118 119* Added support to run the tests with the Kyua runtime engine (kyua-cli), a 120 new package that aims to replace atf-run and atf-report. The ATF tests 121 can be run with the new system by issuing a 'make installcheck-kyua' from 122 the top-level directory of the project (assuming the 'kyua' binary is 123 available during the configuration stage of ATF). 124 125* atf-run and atf-report are now in maintenance mode (but *not* deprecated 126 yet!). Kyua already implements a new, much more reliable runtime engine 127 that provides similar features to these tools. That said, it is not 128 complete yet so all development efforts should go towards it. 129 130* If GDB is installed, atf-run dumps the stack trace of crashing test 131 programs in an attempt to aid debugging. Contributed by Antti Kantee. 132 133* Reverted default timeout change in previous release and reset its value 134 to 5 minutes. This was causing several issues, specially when running 135 the existing NetBSD test suite in qemu. 136 137* Fixed the 'match' output checker in atf-check to properly validate the 138 last line of a file even if it does not have a newline. 139 140* Added the ATF_REQUIRE_IN and ATF_REQUIRE_NOT_IN macros to atf-c++ to 141 check for the presence (or lack thereof) of an element in a collection. 142 143* PR bin/44176: Fixed a race condition in atf-run that would crash atf-run 144 when the cleanup of a test case triggered asynchronous modifications to 145 its work directory (e.g. killing a daemon process that cleans up a pid 146 file in the work directory). 147 148* PR bin/44301: Fixed the sample XSLT file to report bogus test programs 149 instead of just listing them as having 0 test cases. 150 151 152Changes in version 0.12 153*********************** 154 155Experimental version released on November 7th, 2010. 156 157* Added the ATF_REQUIRE_THROW_RE to atf-c++, which is the same as 158 ATF_REQUIRE_THROW but allows checking for the validity of the exception's 159 error message by means of a regular expression. 160 161* Added the ATF_REQUIRE_MATCH to atf-c++, which allows checking for a 162 regular expression match in a string. 163 164* Changed the default timeout for test cases from 5 minutes to 30 seconds. 165 30 seconds is long enough for virtually all tests to complete, and 5 166 minutes is a way too long pause in a test suite where a single test case 167 stalls. 168 169* Deprecated the use.fs property. While this seemed like a good idea in 170 the first place to impose more control on what test cases can do, it 171 turns out to be bad. First, use.fs=false prevents bogus test cases 172 from dumping core so after-the-fact debugging is harder. Second, 173 supporting use.fs adds a lot of unnecessary complexity. atf-run will 174 now ignore any value provided to use.fs and will allow test cases to 175 freely access the file system if they wish to. 176 177* Added the atf_tc_get_config_var_as_{bool,long}{,_wd} functions to the atf-c 178 library. The 'text' module became private in 0.11 but was being used 179 externally to simplify the parsing of configuration variables. 180 181* Made atf-run recognize the 'unprivileged-user' configuration variable 182 and automatically drop root privileges when a test case sets 183 require.user=unprivileged. Note that this is, by no means, done for 184 security purposes; this is just for user convenience; tests should, in 185 general, not be blindly run as root in the first place. 186 187 188Changes in version 0.11 189*********************** 190 191Experimental version released on October 20th, 2010. 192 193* The ATF_CHECK* macros in atf-c++ were renamed to ATF_REQUIRE* to match 194 their counterparts in atf-c. 195 196* Clearly separated the modules in atf-c that are supposed to be public 197 from those that are implementation details. The header files for the 198 internal modules are not installed any more. 199 200* Made the atf-check tool private. It is only required by atf-sh and being 201 public has the danger of causing confusion. Also, making it private 202 simplifies the public API of atf. 203 204* Changed atf-sh to enable per-command error checking (set -e) by default. 205 This catches many cases in which a test case is broken but it is not 206 reported as such because execution continues. 207 208* Fixed the XSTL and CSS stylesheets to support expected failures. 209 210 211Changes in version 0.10 212*********************** 213 214Experimental version released on July 2nd, 2010. 215 216Miscellaneous features 217 218* Added expected failures support to test cases and atf-run. These 219 include, for example, expected clean exits, expected reception of fatal 220 signals, expected timeouts and expected errors in condition checks. 221 These statuses can be used to denote test cases that are known to fail 222 due to a bug in the code they are testing. atf-report reports these 223 tests separately but they do not count towards the failed test cases 224 amount. 225 226* Added the ATF_CHECK_ERRNO and ATF_REQUIRE_ERRNO to the C library to 227 allow easy checking of call failures that update errno. 228 229* Added the has.cleanup meta-data property to test caes that specifies 230 whether the test case has a cleanup routine or not; its value is 231 automatically set. This property is read by atf-run to know if it has to 232 run the cleanup routine; skipping this run for every test case 233 significantly speeds up the run time of test suites. 234 235* Reversed the order of the ATF_CHECK_THROW macro in the C++ binding to 236 take the expected exception as the first argument and the statement to 237 execute as the second argument. 238 239Changes in atf-check 240 241* Changed atf-check to support negating the status and output checks by 242 prefixing them with not- and added support to specify multiple checkers 243 for stdout and stderr, not only one. 244 245* Added the match output checker to atf-check to look for regular 246 expressions in the stdout and stderr of commands. 247 248* Modified the exit checks in atf-check to support checking for the 249 reception of signals. 250 251Code simplifications and cleanups 252 253* Removed usage messages from test programs to simplify the 254 implementation of every binding by a significant amount. They just now 255 refer the user to the appropriate manual page and do not attempt to wrap 256 lines on terminal boundaries. Test programs are not supposed to be run 257 by users directly so this minor interface regression is not important. 258 259* Removed the atf-format internal utility, which is unused after the 260 change documented above. 261 262* Removed the atf-cleanup internal utility. It has been unused since the 263 test case isolation was moved to atf-run in 0.8 264 265* Splitted the Makefile.am into smaller files for easier maintenance and 266 dropped the use of M4. Only affects users building from the repository 267 sources. 268 269* Intermixed tests with the source files in the source tree to provide 270 them more visibility and easier access. The tests directory is gone from 271 the source tree and tests are now suffixed by _test, not prefixed by t_. 272 273* Simplifications to the atf-c library: removed the io, tcr and ui 274 modules as they had become unnecessary after all simplifications 275 introduced since the 0.8 release. 276 277* Removed the application/X-atf-tcr format introduced in 0.8 release. 278 Tests now print a much simplified format that is easy to parse and nicer 279 to read by end users. As a side effect, the default for test cases is 280 now to print their results to stdout unless otherwise stated by providing 281 the -r flag. 282 283* Removed XML distribution documents and replaced them with plain-text 284 documents. They provided little value and introduced a lot of complexity 285 to the build system. 286 287* Simplified the output of atf-version by not attempting to print a 288 revision number when building form a distfile. Makes the build system 289 easier to maintain. 290 291 292Changes in version 0.9 293********************** 294 295Experimental version released on June 3rd, 2010. 296 297* Added atf-sh, an interpreter to process test programs written using 298 the shell API. This is not really a shell interpreter by itself though: 299 it is just a wrapper around the system shell that eases the loading of 300 the necessary ATF libraries. 301 302* Removed atf-compile in favour of atf-sh. 303 304* Added the use.fs metadata property to test case, which is used to 305 specify which test cases require file system access. This is to 306 highlight dependencies on external resources more clearly and to speed up 307 the execution of test suites by skipping the creation of many unnecessary 308 work directories. 309 310* Fixed test programs to get a sane default value for their source 311 directory. This means that it should not be necessary any more to pass 312 -s when running test programs that do not live in the current directory. 313 314* Defining test case headers became optional. This is trivial to achieve 315 in shell-based tests but a bit ugly in C and C++. In C, use the new 316 ATF_TC_WITHOUT_HEAD macro to define the test case, and in C++ use 317 ATF_TEST_CASE_WITHOUT_HEAD. 318 319 320Changes in version 0.8 321********************** 322 323Experimental version released on May 7th, 2010. 324 325* Test programs no longer run several test cases in a row. The execution 326 of a test program now requires a test case name, and that single test 327 case is executed. To execute several test cases, use the atf-run utility 328 as usual. 329 330* Test programs no longer fork a subprocess to isolate the execution of 331 test cases. They run the test case code in-process, and a crash of the 332 test case will result in a crash of the test program. This is to ease 333 debugging of faulty test cases. 334 335* Test programs no longer isolate their test cases. This means that they 336 will not create temporary directories nor sanitize the environment any 337 more. Yes: running a test case that depends on system state by hand will 338 most likely yield different results depending on where (machine, 339 directory, user environment, etc.) it is run. Isolation has been moved 340 to atf-run. 341 342* Test programs no longer print a cryptic format (application/X-atf-tcs) 343 on a special file channel. They can now print whatever they want on the 344 screen. Because test programs can now only run one test case every time, 345 providing controlled output is not necessary any more. 346 347* Test programs no longer write their status into a special file 348 descriptor. Instead, they create a file with the results, which is later 349 parsed by atf-run. This changes the semantics of the -r flag. 350 351* atf-run has been adjusted to perform the test case isolation. As a 352 result, there is now a single canonical place that implements the 353 isolation of test caes. In previous releases, the three language 354 bindings (C, C++ and shell) had to be kept in sync with each other (read: 355 not a nice thing to do at all). As a side effect of this change, writing 356 bindings for other languages will be much, much easier from now on. 357 358* atf-run forks test programs on a test case basis, instead of on a test 359 program basis as it did before. This is to provide the test case 360 isolation that was before implemented by the test programs themselves. 361 362* Removed the atf-exec tool. This was used to implement test case 363 isolation in atf-sh, but it is now unnecessary. 364 365* It is now optional to define the descr meta-data property. It has been 366 proven to be mostly useless, because test cases often carry a descriptive 367 name of their own. 368 369 370Changes in version 0.7 371********************** 372 373Experimental version released on December 22nd, 2009. 374 375* Added build-time checks to atf-c and atf-c++. A binding for atf-sh 376 will come later. 377 378* Migrated all build-time checks for header files to proper ATF tests. 379 This demonstrates the use of the new feature described above. 380 381* Added an internal API for child process management. 382 383* Converted all plain-text distribution documents to a Docbook canonical 384 version, and include pre-generated plain text and HTML copies in the 385 distribution file. 386 387* Simplified the contents of the Makefile.am by regenerating it from a 388 canonical Makefile.am.m4 source. As a side-effect, some dependency 389 specifications were fixed. 390 391* Migrated all checks from the check target to installcheck, as these 392 require ATF to be installed. 393 394* Fixed sign comparison mismatches triggered by the now-enabled 395 -Wsign-compare. 396 397* Fixed many memory and object leaks. 398 399 400Changes in version 0.6 401********************** 402 403Experimental version released on January 18th, 2009. 404 405* Make atf-exec be able to kill its child process after a certain period 406 of time; this is controlled through the new -t option. 407 408* Change atf-sh to use atf-exec's -t option to control the test case's 409 timeouts, instead of doing it internally. Same behavior as before, but 410 noticeably faster. 411 412* atf-exec's -g option and atf-killpg are gone due to the previous 413 change. 414 415* Added the atf-check(1) tool, a program that executes a given command 416 and checks its exit code against a known value and allows the management 417 of stdout and stderr in multiple ways. This replaces the previous 418 atf_check function in the atf-sh library and exposes this functionality 419 to both atf-c and atf-c++. 420 421* Added the ATF_REQUIRE family of macros to the C interface. These help 422 in checking for fatal test conditions. The old ATF_CHECK macros now 423 perform non-fatal checks only. I.e. by using ATF_CHECK, the test case 424 can now continue its execution and the failures will not be reported 425 until the end of the whole run. 426 427* Extended the amount of ATF_CHECK_* C macros with new ones to provide 428 more features to the developer. These also have their corresponding 429 counterparts in the ATF_REQUIRE_* family. The new macros (listing the 430 suffixes only) are: _EQ (replaces _EQUAL), _EQ_MSG, _STREQ and 431 _STREQ_MSG. 432 433 434Changes in version 0.5 435********************** 436 437Experimental version released on May 1st, 2008. 438 439* Clauses 3 and 4 of the BSD license used by the project were dropped. 440 All the code is now under a 2-clause BSD license compatible with the GNU 441 General Public License (GPL). 442 443* Added a C-only binding so that binary test programs do not need to be 444 tied to C++ at all. This binding is now known as the atf-c library. 445 446* Renamed the C++ binding to atf-c++ for consistency with the new atf-c. 447 448* Renamed the POSIX shell binding to atf-sh for consistency with the new 449 atf-c and atf-c++. 450 451* Added a -w flag to test programs through which it is possible to 452 specify the work directory to be used. This was possible in prior 453 releases by defining the workdir configuration variable (-v workdir=...), 454 but was a conceptually incorrect mechanism. 455 456* Test programs now preserve the execution order of test cases when they 457 are given in the command line. Even those mentioned more than once are 458 executed multiple times to comply with the user's requests. 459 460 461Changes in version 0.4 462********************** 463 464Experimental version released on February 4th, 2008. 465 466* Added two new manual pages, atf-c++-api and atf-sh-api, describing the 467 C++ and POSIX shell interfaces used to write test programs. 468 469* Added a pkg-config file, useful to get the flags to build against the 470 C++ library or to easily detect the presence of ATF. 471 472* Added a way for test cases to require a specific architecture and/or 473 machine type through the new 'require.arch' and 'require.machine' 474 meta-data properties, respectively. 475 476* Added the 'timeout' property to test cases, useful to set an 477 upper-bound limit for the test's run time and thus prevent global test 478 program stalls due to the test case's misbehavior. 479 480* Added the atf-exec(1) internal utility, used to execute a command 481 after changing the process group it belongs to. 482 483* Added the atf-killpg(1) internal utility, used to kill process groups. 484 485* Multiple portability fixes. Of special interest, full support for 486 SunOS (Solaris Express Developer Edition 2007/09) using the Sun Studio 12 487 C++ compiler. 488 489* Fixed a serious bug that prevented atf-run(1) from working at all 490 under Fedora 8 x86_64. Due to the nature of the bug, other platforms 491 were likely affected too. 492 493 494Changes in version 0.3 495********************** 496 497Experimental version released on November 11th, 2007. 498 499* Added XML output support to atf-report. This is accompanied by a DTD 500 for the format's structure and sample XSLT/CSS files to post-process this 501 output and convert it to a plain HTML report. 502 503* Changed atf-run to add system information to the report it generates. 504 This is currently used by atf-report's XML output only, and is later 505 printed in the HTML reports in a nice and useful summary table. The user 506 and system administrator are allowed to tune this feature by means of 507 hooks. 508 509* Removed the test cases' 'isolated' property. This was intended to 510 avoid touching the file system at all when running the related test case, 511 but this has not been true for a long while: some control files are 512 unconditionally required for several purposes, and we cannot easily get 513 rid of them. This way we remove several critical and delicate pieces of 514 code. 515 516* Improved atf-report's CSV output format to include information about 517 test programs too. 518 519* Fixed the tests that used atf-compile to not require this tool as a 520 helper. Avoids systems without build-time utilities to skip many tests 521 that could otherwise be run. (E.g. NetBSD without the comp.tgz set 522 installed.) 523 524* Many general cleanups: Fixed many pieces of code marked as ugly and/or 525 incomplete. 526 527 528Changes in version 0.2 529********************** 530 531Experimental version released on September 20th, 2007. 532 533* Test cases now get a known umask on entry. 534 535* atf-run now detects many unexpected failures caused by test programs and 536 reports them as bogus tests. atf-report is able to handle these new 537 errors and nicely reports them to the user. 538 539* All the data formats read and written by the tools have been 540 documented and cleaned up. These include those grammars that define how 541 the different components communicate with each other as well as the 542 format of files written by the developers and users: the Atffiles and the 543 configuration files. 544 545* Added the atf-version tool, a utility that displays information about 546 the currently installed version of ATF. 547 548* Test cases can now define an optional cleanup routine to undo their 549 actions regardless of their exit status. 550 551* atf-report now summarizes the list of failed (bogus) test programs 552 when using the ticker output format. 553 554* Test programs now capture some termination signals and clean up any 555 temporary files before exiting the program. 556 557* Multiple bug fixes and improvements all around. 558 559 560Changes in version 0.1 561********************** 562 563Experimental version released on August 20th, 2007. 564 565* First public version. This was released coinciding with the end of the 566 Google Summer of Code 2007 program. 567 568 569=========================================================================== 570vim: filetype=text:textwidth=75:expandtab:shiftwidth=2:softtabstop=2 571