From 3b296d8e65f6f30074cb1046dcf98d2fbd1ede36 Mon Sep 17 00:00:00 2001 From: Russ Allbery Date: Wed, 25 Aug 2010 18:43:43 -0700 Subject: Add tests/HOWTO from C TAP Harness docs/writing-tests --- tests/HOWTO | 228 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 228 insertions(+) create mode 100644 tests/HOWTO (limited to 'tests/HOWTO') diff --git a/tests/HOWTO b/tests/HOWTO new file mode 100644 index 0000000..bc731b4 --- /dev/null +++ b/tests/HOWTO @@ -0,0 +1,228 @@ + Writing TAP Tests + +Introduction + + This is a guide for users of the C TAP Harness package or similar + TAP-based test harnesses explaining how to write tests. If your + package uses C TAP Harness as the test suite driver, you may want to + copy this document to an appropriate file name in your test suite as + documentation for contributors. + +About TAP + + TAP is the Test Anything Protocol, a protocol for communication + between test cases and a test harness. This is the protocol used by + Perl for its internal test suite and for nearly all Perl modules, + since it's the format used by the build tools for Perl modules to run + tests and report their results. + + A TAP-based test suite works with a somewhat different set of + assumptions than an xUnit test suite. In TAP, each test case is a + separate program. That program, when run, must produce output in the + following format: + + 1..4 + ok 1 - the first test + ok 2 + # a diagnostic, ignored by the harness + not ok 3 - a failing test + ok 4 # skip a skipped test + + The output should all go to standard output. The first line specifies + the number of tests to be run, and then each test produces output that + looks like either "ok " or "not ok " depending on whether the + test succeeded or failed. Additional information about the test can + be provided after the "ok " or "not ok ", but is optional. + Additional diagnostics and information can be provided in lines + beginning with a "#". + + Processing directives are supported after the "ok " or "not ok " + and start with a "#". The main one of interest is "# skip" which says + that the test was skipped rather than successful and optionally gives + the reason. Also supported is "# todo", which normally annotates a + failing test and indicates that test is expected to fail, optionally + providing a reason for why. + + There are three more special cases. First, the initial line stating + the number of tests to run, called the plan, may appear at the end of + the output instead of the beginning. This can be useful if the number + of tests to run is not known in advance. Second, a plan in the form: + + 1..0 # skip entire test case skipped + + can be given instead, which indicates that this entire test case has + been skipped (generally because it depends on facilities or optional + configuration which is not present). Finally, if the test case + encounters a fatal error, it should print the text: + + Bail out! + + on standard output, optionally followed by an error message, and then + exit. This tells the harness that the test aborted unexpectedly. + + The exit status of a successful test case should always be 0. The + harness will report the test as "dubious" if all the tests appeared to + succeed but it exited with a non-zero status. + +Writing TAP Tests + + Environment + + One of the special features of C TAP Harness is the environment that + it sets up for your test cases. If your test program is called under + the runtests driver, the environment variables SOURCE and BUILD will + be set to the top of the test directory in the source tree and the top + of the build tree, respectively. You can use those environment + variables to locate additional test data, programs and libraries built + as part of your software build, and other supporting information + needed by tests. + + The C and shell TAP libraries support a test_file_path() function, + which looks for a file under the build tree and then under the source + tree, using the BUILD and SOURCE environment variables, and return the + full path to the file. This can be used to locate supporting data + files. + + Perl + + Since TAP is the native test framework for Perl, writing TAP tests in + Perl is very easy and extremely well-supported. If you've never + written tests in Perl before, start by reading the documentation for + Test::Tutorial and Test::Simple, which walks you through the basics, + including the TAP output syntax. Then, the best Perl module to use + for serious testing is Test::More, which provides a lot of additional + functions over Test::Simple including support for skipping tests, + bailing out, and not planning tests in advance. See the documentation + of Test::More for all the details and lots of examples. + + C TAP Harness can run Perl test scripts directly and interpret the + results correctly, and similarly the Perl Test::Harness module can run + TAP tests written in other languages using, for example, the TAP + library that comes with C TAP Harness. However, the "prove" tool that + comes with Perl and runs tests makes some Perl-specific assumptions + that aren't always appropriate for packages that aren't written in + Perl. + + C + + C TAP Harness provides a basic TAP library that takes away most of the + pain of writing TAP test cases in C. A C test case should start with + a call to plan(), passing in the number of tests to run. Then, each + test should use is_int(), is_string(), is_double(), or is_hex() as + appropriate to compare expected and seen values, or ok() to do a + simpler boolean test. The is_*() functions take expected and seen + values and then a printf-style format string explaining the test + (which may be NULL). ok() takes a boolean and then the printf-style + string. + + Here's a complete example test program that uses the C TAP library: + + #include + + int + main(void) + { + plan(4); + + ok(1, "the first test"); + is_int(42, 42, NULL); + diag("a diagnostic, ignored by the harness"); + ok(0, "a failing test"); + skip("a skipped test"); + + return 0; + } + + This test program produces the output shown above in the section on + TAP and demonstrates most of the functions. The other functions of + interest are sysdiag() (like diag() but adds strerror() results), + bail() and sysbail() for fatal errors, skip_block() to skip a whole + block of tests, and skip_all() which is called instead of plan() to + skip an entire test case. + + The C TAP library also provides plan_lazy(), which can be called + instead of plan(). If plan_lazy() is called, the library will keep + track of how many test results are reported and will print out the + plan at the end of execution of the program. This should normally be + avoided since the test may appear to be successful even if it exits + prematurely, but it can make writing tests easier in some + circumstances. + + Complete API documentation for the basic C TAP library that comes with + C TAP Harness is available at: + + + + It's common to need additional test functions and utility functions + for your C tests, particularly if you have to set up and tear down a + test environment for your test programs, and it's useful to have them + all in the libtap library so that you only have to link your test + programs with one library. Rather than editing tap/basic.c and + tap/basic.h to add those additional functions, add additional *.c and + *.h files into the tap directory with the function implementations and + prototypes, and then add those additional objects to the library. + That way, you can update tap/basic.c and tap/basic.h from subsequent + releases of C TAP Harness without having to merge changes with your + own code. + + Libraries of additional useful TAP test functions are available in + rra-c-util at: + + + + Some of the code there is particularly useful when testing programs + that require Kerberos keys. + + If you implement new test functions that compare an expected and seen + value, it's best to name them is_ and take the expected + value, the seen value, and then a printf-style format string and + possible arguments to match the calling convention of the functions + provided by C TAP Harness. + + Shell + + C TAP Harness provides a library of shell functions to make it easier + to write TAP tests in shell. That library includes much of the same + functionality as the C TAP library, but takes its parameters in a + somewhat different order to make better use of shell features. + + The libtap.sh file should be installed in a directory named tap in + your test suite area. It can then be loaded by tests written in shell + using the environment set up by runtests with: + + . "$SOURCE"/tap/libtap.sh + + Here is a complete test case written in shell which produces the same + output as the TAP sample above: + + #!/bin/sh + + . "$SOURCE"/tap/libtap.sh + cd "$BUILD" + + plan 4 + ok 'the first test' true + ok '' [ 42 -eq 42 ] + diag a diagnostic, ignored by the harness + ok '' false + skip 'a skipped test' + + The shell framework doesn't provide the is_* functions, so you'll use + the ok function more. It takes a string describing the text and then + treats all of its remaining arguments as a condition, evaluated the + same way as the arguments to the "if" statement. If that condition + evaluates to true, the test passes; otherwise, the test fails. + + The plan, plan_lazy, diag, and bail functions work the same as with + the C library. skip takes a string and skips the next test with that + explanation. skip_block takes a count and a string and skips that + many tests with that explanation. skip_all takes an optional reason + and skips the entire test case. + + Since it's common for shell programs to want to test the output of + commands, there's an additional function ok_program provided by the + shell test library. It takes the test description string, the + expected exit status, the expected program output, and then treats the + rest of its arguments as the program to run. That program is run with + standard error and standard output combined, and then its exit status + and output are tested against the provided values. -- cgit v1.2.3 From 6e405825d27f6b409a99289ad23c69367f154f94 Mon Sep 17 00:00:00 2001 From: Russ Allbery Date: Wed, 27 Feb 2013 15:31:44 -0800 Subject: Update tests/HOWTO from C TAP Harness 1.12 Change-Id: Icd7381f4fbe3699dbd6e5459ddb59548f60e56a4 Reviewed-on: https://gerrit.stanford.edu/846 Reviewed-by: Russ Allbery Tested-by: Russ Allbery --- tests/HOWTO | 32 ++++++++++++++++++++++++++------ 1 file changed, 26 insertions(+), 6 deletions(-) (limited to 'tests/HOWTO') diff --git a/tests/HOWTO b/tests/HOWTO index bc731b4..5d38748 100644 --- a/tests/HOWTO +++ b/tests/HOWTO @@ -96,12 +96,11 @@ Writing TAP Tests of Test::More for all the details and lots of examples. C TAP Harness can run Perl test scripts directly and interpret the - results correctly, and similarly the Perl Test::Harness module can run - TAP tests written in other languages using, for example, the TAP - library that comes with C TAP Harness. However, the "prove" tool that - comes with Perl and runs tests makes some Perl-specific assumptions - that aren't always appropriate for packages that aren't written in - Perl. + results correctly, and similarly the Perl Test::Harness module and + prove command can run TAP tests written in other languages using, for + example, the TAP library that comes with C TAP Harness. You can, if + you wish, use the library that comes with C TAP Harness but use prove + instead of runtests for running the test suite. C @@ -117,6 +116,7 @@ Writing TAP Tests Here's a complete example test program that uses the C TAP library: + #include #include int @@ -226,3 +226,23 @@ Writing TAP Tests rest of its arguments as the program to run. That program is run with standard error and standard output combined, and then its exit status and output are tested against the provided values. + + A utility function, strip_colon_error, is provided that runs the + command given as its arguments and strips text following a colon and a + space from the output (unless there is no whitespace on the line + before the colon and the space, normally indicating a prefix of the + program name). This function can be used to wrap commands that are + expected to fail with output that has a system- or locale-specific + error message appended, such as the output of strerror(). + +License + + This file is part of the documentation of C TAP Harness, which can be + found at . + + Copyright 2010 Russ Allbery + + Copying and distribution of this file, with or without modification, + are permitted in any medium without royalty provided the copyright + notice and this notice are preserved. This file is offered as-is, + without any warranty. -- cgit v1.2.3