48 POSIX-Style Testing

To make use of POSIX-style testing methodology, you can do:

     (use-modules (ice-9 testing-lib))

This module contains testing infrastructure (procedures and macros)
for evaluating the behavior of Scheme code fragments, collecting the
results, and reporting them.  The approach is generally inspired by
Greg, the GNUstep regression test environment, which is believed to
conform to the POSIX standard for test frameworks.


CORE FUNCTIONS

The function (run-test name expected-result thunk) is the heart of the
testing environment.  The first parameter NAME is a unique name for the
test to be executed (for an explanation of this parameter see below under
TEST NAMES).  The second parameter EXPECTED-RESULT is a boolean value
that indicates whether the corresponding test is expected to pass.  If
EXPECTED-RESULT is #t the test is expected to pass, if EXPECTED-RESULT is
#f the test is expected to fail.  Finally, THUNK is the function that
actually performs the test.  For example:

   (run-test "integer addition" #t (lambda () (= 2 (+ 1 1))))

To report success, THUNK should either return #t or throw 'pass.  To
report failure, THUNK should either return #f or throw 'fail.  If THUNK
returns a non boolean value or throws 'unresolved, this indicates that
the test did not perform as expected.  For example the property that was
to be tested could not be tested because something else went wrong.
THUNK may also throw 'untested to indicate that the test was deliberately
not performed, for example because the test case is not complete yet.
Finally, if THUNK throws 'unsupported, this indicates that this test
requires some feature that is not available in the configured testing
environment.  All other exceptions thrown by THUNK are considered as
errors.


Convenience macros for tests expected to pass or fail

* (pass-if name body) is a short form for
  (run-test name #t (lambda () body))
* (expect-fail name body) is a short form for
  (run-test name #f (lambda () body))

For example:

   (pass-if "integer addition" (= 2 (+ 1 1)))


Convenience macros to test for exceptions

The following macros take exception parameters which are pairs
(type . message), where type is a symbol that denotes an exception type
like 'wrong-type-arg or 'out-of-range, and message is a string holding a
regular expression that describes the error message for the exception
like "Argument .* out of range".

* (pass-if-exception name exception body) will pass if the execution of
  body causes the given exception to be thrown.  If no exception is
  thrown, the test fails.  If some other exception is thrown, is is an
  error.
* (expect-fail-exception name exception body) will pass unexpectedly if
  the execution of body causes the given exception to be thrown.  If no
  exception is thrown, the test fails expectedly.  If some other
  exception is thrown, it is an error.


Built-in Exceptions

         exception:out-of-range
         exception:unbound-var
         exception:wrong-num-args
         exception:wrong-type-arg


Skipping the rest of the file

Sometimes test granularity at the file level is useful; for example,
a configurable feature is simply not available.  In that case, you can
use `(skip-file! "SOME REASON")', which throws `skip-file'.  Your test
driver should catch this, as none of the reporters are able to.


TEST NAMES

Every test in the test suite has a unique name, to help
developers find tests that are failing (or unexpectedly passing),
and to help gather statistics.

A test name is a list of printable objects.  For example:
("ports.scm" "file" "read and write back list of strings")
("ports.scm" "pipe" "read")

Test names may contain arbitrary objects, but they always have
the following properties:
- Test names can be compared with EQUAL?.
- Test names can be reliably stored and retrieved with the standard WRITE
  and READ procedures; doing so preserves their identity.

For example:

   (pass-if "simple addition" (= 4 (+ 2 2)))

In that case, the test name is the list ("simple addition").

The WITH-TEST-PREFIX syntax and WITH-TEST-PREFIX* procedure establish
a prefix for the names of all tests whose results are reported
within their dynamic scope.  For example:

(begin
  (with-test-prefix "basic arithmetic"
    (pass-if "addition" (= (+ 2 2) 4))
    (pass-if "subtraction" (= (- 4 2) 2)))
  (pass-if "multiplication" (= (* 2 2) 4)))

In that example, the three test names are:
  ("basic arithmetic" "addition"),
  ("basic arithmetic" "subtraction"), and
  ("multiplication").

WITH-TEST-PREFIX can be nested.  Each WITH-TEST-PREFIX postpends
a new element to the current prefix:

(with-test-prefix "arithmetic"
  (with-test-prefix "addition"
    (pass-if "integer" (= (+ 2 2) 4))
    (pass-if "complex" (= (+ 2+3i 4+5i) 6+8i)))
  (with-test-prefix "subtraction"
    (pass-if "integer" (zero? (- 2 2)))
    (pass-if "complex" (= (- 2+3i 1+2i) 1+1i))))

The four test names here are:
  ("arithmetic" "addition" "integer")
  ("arithmetic" "addition" "complex")
  ("arithmetic" "subtraction" "integer")
  ("arithmetic" "subtraction" "complex")

To print a name for a human reader, we DISPLAY its elements,
separated by ": ".  So, the last set of test names would be
reported as:

  arithmetic: addition: integer
  arithmetic: addition: complex
  arithmetic: subtraction: integer
  arithmetic: subtraction: complex

The Guile benchmarks use with-test-prefix to include the name of
the source file containing the test in the test name, to help
developers to find failing tests, and to provide each file with its
own namespace.


REPORTERS

A reporter is a function which we apply to each test outcome.
Reporters can log results, print interesting results to the
standard output, collect statistics, etc.

A reporter function takes two mandatory arguments, RESULT and TEST, and
possibly additional arguments depending on RESULT; its return value
is ignored.  RESULT has one of the following forms:

pass         - The test named TEST passed.
               Additional arguments are ignored.
upass        - The test named TEST passed unexpectedly.
               Additional arguments are ignored.
fail         - The test named TEST failed.
               Additional arguments are ignored.
xfail        - The test named TEST failed, as expected.
               Additional arguments are ignored.
unresolved   - The test named TEST did not perform as expected, for
               example the property that was to be tested could not be
               tested because something else went wrong.
               Additional arguments are ignored.
untested     - The test named TEST was not actually performed, for
               example because the test case is not complete yet.
               Additional arguments are ignored.
unsupported  - The test named TEST requires some feature that is not
               available in the configured testing environment.
               Additional arguments are ignored.
error        - An error occurred while the test named TEST was
               performed.  Since this result means that the system caught
               an exception it could not handle, the exception arguments
               are passed as additional arguments.  As a special
               case, `misc-error' exceptions are output like so:
               "in PROCEDURE: SIMPLE-FORMATTED-ERROR-MESSAGE".

This library provides some standard reporters for logging results
to a file, reporting interesting results to the user, and
collecting totals.

You can use the REGISTER-REPORTER function and friends to add
whatever reporting functions you like.  If you don't register any
reporters, the library uses FULL-REPORTER, which simply writes
all results to the standard output.


MISC

If you use Scheme mode in Emacs:

 (put 'run-test 'scheme-indent-function 2)
 (mapcar '(lambda (sym) (put sym 'scheme-indent-function 1))
         '(with-test-prefix
           pass-if expect-fail
           pass-if-exception expect-fail-exception))