summaryrefslogtreecommitdiff
path: root/tests/HOWTO
diff options
context:
space:
mode:
authorRuss Allbery <eagle@eyrie.org>2018-05-27 20:59:59 -0700
committerRuss Allbery <eagle@eyrie.org>2018-05-27 20:59:59 -0700
commitbdcb3741db27d6b773ce7cdf05aab063a70ea100 (patch)
tree11e11bb646e0b58db6e23b59e3c856ea59ef3545 /tests/HOWTO
parent62a9101895a2d596bf91231f119d338d2025ff08 (diff)
Update to rra-c-util 7.2 and C TAP Harness 4.3
Update to rra-c-util 7.2: * Improve configure output for krb5-config testing. * Define UINT32_MAX for systems that don't have it. * Add SPDX-License-Identifier headers to all substantial source files. * Fix new warnings from GCC 7 and Clang warnings. * Require Test::Strict 0.25 or later to run those tests. * Fix off-by-one error in return-value checks for snprintf. * Use Autoconf to probe for supported warning flags. * Fix running module-version-t -u with current versions of Perl. * Use C_TAP_SOURCE and C_TAP_BUILD instead of SOURCE and BUILD. Update to C TAP Harness 4.3: * Add support for valgrind and libtool in test lists. * Report test failures as left and right, not wanted and expected. * Fix string comparisons with NULL pointers and the string "(null)". * Add SPDX-License-Identifier headers to all substantial source files. * Avoid zero-length realloc allocations in breallocarray. * Fix new warnings from GCC 7 and Clang warnings. * Use C_TAP_SOURCE and C_TAP_BUILD instead of SOURCE and BUILD.
Diffstat (limited to 'tests/HOWTO')
-rw-r--r--tests/HOWTO248
1 files changed, 0 insertions, 248 deletions
diff --git a/tests/HOWTO b/tests/HOWTO
deleted file mode 100644
index b94985d..0000000
--- a/tests/HOWTO
+++ /dev/null
@@ -1,248 +0,0 @@
- Writing TAP Tests
-
-Introduction
-
- This is a guide for users of the C TAP Harness package or similar
- TAP-based test harnesses explaining how to write tests. If your
- package uses C TAP Harness as the test suite driver, you may want to
- copy this document to an appropriate file name in your test suite as
- documentation for contributors.
-
-About TAP
-
- TAP is the Test Anything Protocol, a protocol for communication
- between test cases and a test harness. This is the protocol used by
- Perl for its internal test suite and for nearly all Perl modules,
- since it's the format used by the build tools for Perl modules to run
- tests and report their results.
-
- A TAP-based test suite works with a somewhat different set of
- assumptions than an xUnit test suite. In TAP, each test case is a
- separate program. That program, when run, must produce output in the
- following format:
-
- 1..4
- ok 1 - the first test
- ok 2
- # a diagnostic, ignored by the harness
- not ok 3 - a failing test
- ok 4 # skip a skipped test
-
- The output should all go to standard output. The first line specifies
- the number of tests to be run, and then each test produces output that
- looks like either "ok <n>" or "not ok <n>" depending on whether the
- test succeeded or failed. Additional information about the test can
- be provided after the "ok <n>" or "not ok <n>", but is optional.
- Additional diagnostics and information can be provided in lines
- beginning with a "#".
-
- Processing directives are supported after the "ok <n>" or "not ok <n>"
- and start with a "#". The main one of interest is "# skip" which says
- that the test was skipped rather than successful and optionally gives
- the reason. Also supported is "# todo", which normally annotates a
- failing test and indicates that test is expected to fail, optionally
- providing a reason for why.
-
- There are three more special cases. First, the initial line stating
- the number of tests to run, called the plan, may appear at the end of
- the output instead of the beginning. This can be useful if the number
- of tests to run is not known in advance. Second, a plan in the form:
-
- 1..0 # skip entire test case skipped
-
- can be given instead, which indicates that this entire test case has
- been skipped (generally because it depends on facilities or optional
- configuration which is not present). Finally, if the test case
- encounters a fatal error, it should print the text:
-
- Bail out!
-
- on standard output, optionally followed by an error message, and then
- exit. This tells the harness that the test aborted unexpectedly.
-
- The exit status of a successful test case should always be 0. The
- harness will report the test as "dubious" if all the tests appeared to
- succeed but it exited with a non-zero status.
-
-Writing TAP Tests
-
- Environment
-
- One of the special features of C TAP Harness is the environment that
- it sets up for your test cases. If your test program is called under
- the runtests driver, the environment variables SOURCE and BUILD will
- be set to the top of the test directory in the source tree and the top
- of the build tree, respectively. You can use those environment
- variables to locate additional test data, programs and libraries built
- as part of your software build, and other supporting information
- needed by tests.
-
- The C and shell TAP libraries support a test_file_path() function,
- which looks for a file under the build tree and then under the source
- tree, using the BUILD and SOURCE environment variables, and return the
- full path to the file. This can be used to locate supporting data
- files.
-
- Perl
-
- Since TAP is the native test framework for Perl, writing TAP tests in
- Perl is very easy and extremely well-supported. If you've never
- written tests in Perl before, start by reading the documentation for
- Test::Tutorial and Test::Simple, which walks you through the basics,
- including the TAP output syntax. Then, the best Perl module to use
- for serious testing is Test::More, which provides a lot of additional
- functions over Test::Simple including support for skipping tests,
- bailing out, and not planning tests in advance. See the documentation
- of Test::More for all the details and lots of examples.
-
- C TAP Harness can run Perl test scripts directly and interpret the
- results correctly, and similarly the Perl Test::Harness module and
- prove command can run TAP tests written in other languages using, for
- example, the TAP library that comes with C TAP Harness. You can, if
- you wish, use the library that comes with C TAP Harness but use prove
- instead of runtests for running the test suite.
-
- C
-
- C TAP Harness provides a basic TAP library that takes away most of the
- pain of writing TAP test cases in C. A C test case should start with
- a call to plan(), passing in the number of tests to run. Then, each
- test should use is_int(), is_string(), is_double(), or is_hex() as
- appropriate to compare expected and seen values, or ok() to do a
- simpler boolean test. The is_*() functions take expected and seen
- values and then a printf-style format string explaining the test
- (which may be NULL). ok() takes a boolean and then the printf-style
- string.
-
- Here's a complete example test program that uses the C TAP library:
-
- #include <stddef.h>
- #include <tap/basic.h>
-
- int
- main(void)
- {
- plan(4);
-
- ok(1, "the first test");
- is_int(42, 42, NULL);
- diag("a diagnostic, ignored by the harness");
- ok(0, "a failing test");
- skip("a skipped test");
-
- return 0;
- }
-
- This test program produces the output shown above in the section on
- TAP and demonstrates most of the functions. The other functions of
- interest are sysdiag() (like diag() but adds strerror() results),
- bail() and sysbail() for fatal errors, skip_block() to skip a whole
- block of tests, and skip_all() which is called instead of plan() to
- skip an entire test case.
-
- The C TAP library also provides plan_lazy(), which can be called
- instead of plan(). If plan_lazy() is called, the library will keep
- track of how many test results are reported and will print out the
- plan at the end of execution of the program. This should normally be
- avoided since the test may appear to be successful even if it exits
- prematurely, but it can make writing tests easier in some
- circumstances.
-
- Complete API documentation for the basic C TAP library that comes with
- C TAP Harness is available at:
-
- <http://www.eyrie.org/~eagle/software/c-tap-harness/>
-
- It's common to need additional test functions and utility functions
- for your C tests, particularly if you have to set up and tear down a
- test environment for your test programs, and it's useful to have them
- all in the libtap library so that you only have to link your test
- programs with one library. Rather than editing tap/basic.c and
- tap/basic.h to add those additional functions, add additional *.c and
- *.h files into the tap directory with the function implementations and
- prototypes, and then add those additional objects to the library.
- That way, you can update tap/basic.c and tap/basic.h from subsequent
- releases of C TAP Harness without having to merge changes with your
- own code.
-
- Libraries of additional useful TAP test functions are available in
- rra-c-util at:
-
- <http://www.eyrie.org/~eagle/software/rra-c-util/>
-
- Some of the code there is particularly useful when testing programs
- that require Kerberos keys.
-
- If you implement new test functions that compare an expected and seen
- value, it's best to name them is_<something> and take the expected
- value, the seen value, and then a printf-style format string and
- possible arguments to match the calling convention of the functions
- provided by C TAP Harness.
-
- Shell
-
- C TAP Harness provides a library of shell functions to make it easier
- to write TAP tests in shell. That library includes much of the same
- functionality as the C TAP library, but takes its parameters in a
- somewhat different order to make better use of shell features.
-
- The libtap.sh file should be installed in a directory named tap in
- your test suite area. It can then be loaded by tests written in shell
- using the environment set up by runtests with:
-
- . "$SOURCE"/tap/libtap.sh
-
- Here is a complete test case written in shell which produces the same
- output as the TAP sample above:
-
- #!/bin/sh
-
- . "$SOURCE"/tap/libtap.sh
- cd "$BUILD"
-
- plan 4
- ok 'the first test' true
- ok '' [ 42 -eq 42 ]
- diag a diagnostic, ignored by the harness
- ok '' false
- skip 'a skipped test'
-
- The shell framework doesn't provide the is_* functions, so you'll use
- the ok function more. It takes a string describing the text and then
- treats all of its remaining arguments as a condition, evaluated the
- same way as the arguments to the "if" statement. If that condition
- evaluates to true, the test passes; otherwise, the test fails.
-
- The plan, plan_lazy, diag, and bail functions work the same as with
- the C library. skip takes a string and skips the next test with that
- explanation. skip_block takes a count and a string and skips that
- many tests with that explanation. skip_all takes an optional reason
- and skips the entire test case.
-
- Since it's common for shell programs to want to test the output of
- commands, there's an additional function ok_program provided by the
- shell test library. It takes the test description string, the
- expected exit status, the expected program output, and then treats the
- rest of its arguments as the program to run. That program is run with
- standard error and standard output combined, and then its exit status
- and output are tested against the provided values.
-
- A utility function, strip_colon_error, is provided that runs the
- command given as its arguments and strips text following a colon and a
- space from the output (unless there is no whitespace on the line
- before the colon and the space, normally indicating a prefix of the
- program name). This function can be used to wrap commands that are
- expected to fail with output that has a system- or locale-specific
- error message appended, such as the output of strerror().
-
-License
-
- This file is part of the documentation of C TAP Harness, which can be
- found at <http://www.eyrie.org/~eagle/software/c-tap-harness/>.
-
- Copyright 2010 Russ Allbery <eagle@eyrie.org>
-
- Copying and distribution of this file, with or without modification,
- are permitted in any medium without royalty provided the copyright
- notice and this notice are preserved. This file is offered as-is,
- without any warranty.