| Writing tests
+--------------

Each test case is expected to be an independent executable that returns an
exit status.  An executable can be shared between tests by using
command-line options to select test variations.  For example, the syscalls
directory uses a single executable "syscalls" which takes options to
indicate which syscall should be tested.  You should choose whichever
method works best for your set of tests.

The exit status is interpreted by the test harness (run.bash) as follows:

    0     - PASS, meaning that the test behaved as expected
    1     - FAIL, meaning that the test did not PASS
    other - ERROR, meaning that there was an error which prevented the test from
            being performed, such as a setup step which did not complete
            correctly, or a failed memory allocation, etc.

When writing bash tests using functions.bash (see below), the exit_pass, 
exit_fail and exit_error functions generate the appropriate exit status.

The test case is expected to generate debugging information to either stdout or
stderr.  The debugging information is captured by the harness and displayed only
when requested by --verbose, --debug, or investigation of run.log

Bash tests
----------
Tests written in bash can make use of common functions in utils/testcase.bash
and utils/functions.bash.  These are automatically in $PATH by way of run.bash,
so they can be loaded simply as:

    source testcase.bash

Testcase.bash includes functions meant to be used in any of the various
testcases.  In turn it sources functions.bash, which contains functions that are
used in run.bash and can also be used in the testcases.

If you augment functions.bash, beware that it is expected to behave a little
differently from the test cases.  Zero status is success, non-zero is error
(there is no distinction between fail/error in the utility functions).
More importantly, debugging output goes to stdout, error output goes to stderr.
The reason for this is that the common functions are also used by the test
harness, which wants to be able to /dev/null the debug info without losing the
error messages.

The best way to run individual test cases is by specifying them on the run.bash
command-line.  However it may be possible to run individual scripts outside of
the harness.  In that case, you'll need to augment PATH in your shell
environment:

    PATH=/path/to/utils:/path/to/utils/bin:$PATH

| Writing run.conf
+-----------------

This configuration file is a bash script that sourced by run.bash.  It
primarily consists of a list of tests to run, but can also contain
some variable and function settings.

The list of tests is built using the function "+" provided by
run.bash.  To add a test to the list:

  + testname [optional arguments]

Functions
---------
run_test(testcase)

    - default is: "./$@" (see run.bash)

show_test()

    - default is: prf "%-75s " "$*"

    - see syscalls/run.conf for an example how this can be changed to customize
      the output

startup_hook()
    
    - default is: true (see run.bash)

    - can be defined to do any special setup required by your
      tests.  Note this runs prior to looping through the tests,
      not prior to each test.  If you need special setup prior to
      running each test, put it in run_test

cleanup()

    - runs after looping through all the tests, manipulated by prepend_cleanup()
      and append_cleanup().  See section Test Cleanup.

| Writing mls syscall tests
+--------------------------

This section describes how to add new mls syscall tests to the test harness.

An mls testcase is configured using a "+" line in the run.conf file.  The "+"
line contains all the information the harness needs to run that particular
testcase.  At a minimum, this is the syscall name, permission to test, mls op
and expected operation result.  For example:

+ unlink perm=file_unlink expres=success mlsop=eq

Additionally, you may need to specify one or more of the following named
parameters:

    err:   The name of the expected error.  This param is required for all
           failure tests.

    flag:  Provides additional information to the test utility program about how
           to attempt to access the test object.  The syntax is determined by
           the test utility program.

    op:    For multiplexed syscalls, determines which function to invoke.  The
           syntax is determined by the test utility program.

The named parameters don't have to be in any particular order, but it would be
nice to maintain consistency in the test run output.

The function run_test() is the main driver for all mls syscall testcases.  Here
are the important things it does:

    - builds the full subject/object security contexts based on the specified
      mls op

      The values for $mlsop match the selinux mls policy keywords (eq, dom,
      domby, incomp). The compute_contexts() function builds a default subject
      and object context based on the supplied keyword.

      The subject context is built from a combination of the label specified
      with 'subj' and the result of calling utils/test_context for the syscall's
      helper utility.

      Typically the object context is built from a combination of the label
      specified with 'obj' and the user/role/type of the current process.  This
      context is then used with runcon to create the test object.  Some test
      objects, however, are created with a different type than the type of the
      process that created them (e.g. lspp_harness_tmp_t).  In this case, the
      object context must be updated after creating the object.

    - creates the test object and registers its cleanup

      Special functions for creating test objects are named create_<test object>.
      If a special creation function doesn't yet exist for your test object, you
      will need to create one.

      The first parameter to a create function is the name of a variable to set
      with the name or identifier for the new object.  An optional context
      parameter causes the create function to create the object with the
      specified context.

      Additional test setup, such as creating additional objects or setting up
      variables for augrok, is handled in the main create functions for a group
      of test objects, e.g. create_fs_objects().

    - invokes the test operation

      The default syntax for the test operation is:

        runcon $subj do_$syscall [$op] [$dirname] [$source] $target [$flag]

      where $op, $dirname, $source and $flag are only used if they have been
      set.  This syntax should suffice for most testcases.  If it does not work
      for your testcase, you can create your own function for invoking your test
      operation and set its name in $testfunc.

      The utilities do_$syscall live in the utils/bin directory.  If a utility
      does not already exist for the syscall you want to test, you will have to
      add one.  The utility's arguments should follow the default test operation
      syntax if possible, and they should have readable names.  For example, the
      argument "create" is a better choice than something like -c or 100.  This
      will make our run.conf and run.log more readable.  All utilities must
      print the following space-separated list before exiting:

            test result (0 | 1 for success or failure)
            exit value (syscall return code | errno for success or failure)
            pid for the current process

    - checks the result

      The function check_result() determines if the test operation's result was
      the expected one.  The values $testres (test result) and $exitval (exit
      value), which were output by the test utility, are compared with the
      values for $success and $err, which were specified for the testcase in
      run.conf.  In the expected success case, $exitval is ignored by
      check_result().

      Note: If the testcase result is not the expected result, it is considered
      a test ERROR.  If the audit record is not found, it is considered a test
      FAILURE.

    - verifies the audit record with augrok

      Special functions for verifying audit records are prefixed with augrok_.
      They call augrok with different arguments depending on variables set in
      the test object creation functions.

| Test Cleanup
+-------------

cleanup function
----------------
run.bash has a cleanup function defined in run.bash.  This function tries to
restore the system to the state prior to run.bash execution.  By default the
function contains the list of commands to revert the effects of the commands in
the startup function, which runs before the tests:

    parse_cmdline "$@"
    startup || die "startup failed"
    run_tests
    exit $?

The cleanup function runs automatically when run.bash terminates, either
normally or abnormally, by way of a trap:

    trap 'cleanup &>/dev/null; close_log; exit' 0
    trap 'cleanup; close_log; exit' 1 2 3 15
    
On a normal exit, not caused by a signal such as ctrl-c, run.bash hides the
output of cleanup.  The output is only displayed if run.bash exits by way of
ctrl-c, so that you can see it is working.  (Actually, it doesn't produce any
output presently, so it's the same either way...)

There are two helper functions for augmenting the cleanup function:
append_cleanup and prepend_cleanup.  The arguments to this function will be
evaluated in the context of the cleanup function when it runs, for example:

    prepend_cleanup "rm -f $mytempfile"

Most of the time prepend_cleanup should be used, since operations should be
popped off the stack in the same order they were pushed.

test_cleanup function
---------------------
The cleanup function defined in run.bash only runs when run.bash quits, not at
the end of each testcase.  For the latter, testcase.bash defines another
function: test_cleanup.  By default this function removes the temporary files
which testcase.bash creates: $localtmp, $tmp1 and $tmp2.

The test_cleanup function runs when the testcase finishes, either normally or by
a signal such as ctrl-c.  This is accomplished by running the testcase in
a subshell and defining the traps in the context of that subshell.  The call
tree looks like this (pseudocode)

run.bash
    cleanup() { ... }
    append_cleanup() { append to cleanup }
    prepend_cleanup() { prepend to cleanup }
    trap 'cleanup; close_log; exit' 0 1 2 15

    source run.conf     # note that prepend_cleanup and append_cleanup called
                        # directly from run.conf affect the global cleanup,
                        # which only runs when run.bash terminates

    subshell
        run_test runs   # defined in run.conf
            source testcase.bash
                test_cleanup() { ... }
                append_cleanup() { append to test_cleanup }
                prepend_cleanup() { prepend to test_cleanup }
                trap 'test_cleanup; exit' 0 1 2 15

            test runs   # note that prepend_cleanup and append_cleanup called
                        # within the same shell as run_test will affect
                        # test_cleanup

        subshell terminates
            test_cleanup runs

    run.bash terminates
        cleanup runs

NOTE: If the testcase runs as an external script instead of inline in run_test,
the middle of the call tree will look more like this:

    subshell
        run_test runs   # defined in run.conf
            test script runs
                source testcase.bash
                    test_cleanup() { ... }
                    append_cleanup() { append to test_cleanup }
                    prepend_cleanup() { prepend to test_cleanup }
                    trap 'test_cleanup; exit' 0 1 2 15

                test script terminates
                    test_cleanup runs

        subshell terminates
            no cleanup since traps were set in the external script

