diff options
Diffstat (limited to 'contrib/bluegnu2.0.3/doc/dejagnu.info-1')
| -rw-r--r-- | contrib/bluegnu2.0.3/doc/dejagnu.info-1 | 1163 |
1 files changed, 1163 insertions, 0 deletions
diff --git a/contrib/bluegnu2.0.3/doc/dejagnu.info-1 b/contrib/bluegnu2.0.3/doc/dejagnu.info-1 new file mode 100644 index 0000000..c11c3b4 --- /dev/null +++ b/contrib/bluegnu2.0.3/doc/dejagnu.info-1 @@ -0,0 +1,1163 @@ +This is Info file dejagnu.info, produced by Makeinfo version 1.68 from +the input file ./dejagnu.texi. + +START-INFO-DIR-ENTRY +* DejaGnu: (dejagnu). The GNU testing framework. +END-INFO-DIR-ENTRY + + Copyright (C) 92, 93, 94, 95, 1996 Free Software Foundation, Inc. + + Permission is granted to make and distribute verbatim copies of this +manual provided the copyright notice and this permission notice are +preserved on all copies. + + Permission is granted to copy and distribute modified versions of +this manual under the conditions for verbatim copying, provided also +that the entire resulting derived work is distributed under the terms +of a permission notice identical to this one. + + Permission is granted to copy and distribute translations of this +manual into another language, under the above conditions for modified +versions. + + +File: dejagnu.info, Node: Top, Next: Overview, Up: (dir) + +DejaGnu +******* + + DejaGnu is a framework for running test suites on software tools. + + This file describes version 1.3 of DejaGnu. + +* Menu: + +* Overview:: What is DejaGnu? +* What is New:: What is new in this release. +* Invoking runtest:: Using `runtest', the main test driver +* Customizing:: Setting `runtest' defaults +* Internals:: The DejaGnu implementation +* Tests:: How to write a test case +* Extending:: New tools, new targets, and new hosts +* Installation:: Configuring and Installing DejaGnu +* Index:: Index + + +File: dejagnu.info, Node: Overview, Next: What is New, Prev: Top, Up: Top + +What is DejaGnu? +**************** + + DejaGnu is a framework for testing other programs. Its purpose is to +provide a single front end for all tests. Beyond this, DejaGnu offers +several advantages for testing: + + 1. The flexibility and consistency of the DejaGnu framework make it + easy to write tests for any program. + + 2. DejaGnu provides a layer of abstraction which allows you to write + tests that are portable to any host or target where a program must + be tested. For instance, a test for GDB can run (from any Unix + based host) on any target architecture that DejaGnu supports. + Currently DejaGnu runs tests on several single board computers, + whose operating software ranges from just a boot monitor to a + full-fledged, Unix-like realtime OS. + + 3. All tests have the same output format. This makes it easy to + integrate testing into other software development processes. + DejaGnu's output is designed to be parsed by other filtering + script, and it is also human readable. + + DejaGnu is written in `expect', which in turn uses "Tcl"--Tool +command language. + + Running tests requires two things: the testing framework, and the +test suites themselves. Tests are usually written in `expect' using +Tcl, but you can also use a Tcl script to run a test suite that is not +based on `expect'. (`expect' script filenames conventionally use +`.exp' as a suffix; for example, the main implementation of the DejaGnu +test driver is in the file `runtest.exp'.) + +* Menu: + +* Running Tests:: A first look at running DejaGnu tests +* Sample Test:: What does a DejaGnu test case look like? +* Design Goals:: Goals behind DejaGnu +* Posix:: DejaGnu conforms to POSIX 1003.3 +* Future Directions:: Where is DejaGnu going? +* Tcl and Expect:: Reading more about Tcl and Expect + + +File: dejagnu.info, Node: What is New, Next: Invoking runtest, Prev: Overview, Up: Top + +What is new in this release ? +***************************** + + This release has a number of substantial changes over version 1.2. +The most visible change is that the version of expect and Tcl included +in the release are up-to-date with the current stable net releases. +Other changes are: + + 1. The config sub-system in DejaGnu has been completely redesigned. + It now supports testing on remote hosts as well as remote targets. + + 2. More builtin support for building target binaries with the correct + linker flags. Currently this only works with GCC, preferably with a + target support by `libgloss'. + + 3. Lots of little bug fixes from a year of heavy use here at Cygnus + Support. + + 4. DejaGnu now uses `autoconf' for configuration. + + 5. New test cases for DejaGnu have been added for the new features, + plus the "-tool" option bug in the 1.2 testsuite has been fixed. + + 6. The `--tool' option is now optional. + + 7. `runtest' when searching for test drivers ignores all directories + named SCCS, RCS, and CVS. + + 8. There is now a generic keyword based test harness that uses + comments in source code to control how each test case gets built + and run. + + 9. There is now some support for running a testsuite with multiple + passes. + + + +File: dejagnu.info, Node: Running Tests, Next: Sample Test, Up: Overview + +Running existing tests +====================== + + To run tests from an existing collection, first use `configure' as +usual to set up the source directory containing the tests. Then try +running + + make check + + If the `check' target exists, it usually saves you some trouble--for +instance, it can set up any auxiliary programs or other files needed by +the tests. + + Once you have run `make check' to build any auxiliary files, you +might want to call the test driver `runtest' directly to repeat the +tests. You may also have to call `runtest' directly for test +collections with no `check' target in the `Makefile'. + + Typically, you must use two command-line options: `--tool', to +specify which set of tests to run(1), and `--srcdir', to specify where +to find test directories. + + For example, if the directory `gdb/testsuite' contains a collection +of DejaGnu tests for GDB, you can run them like this: + + eg$ cd gdb/testsuite + eg$ runtest --tool gdb +*Test output follows, ending with:* + + === gdb Summary === + + # of expected passes 508 + # of expected failures 103 + /usr/latest/bin/gdb version 4.14.4 -nx + + You can use the option `--srcdir' to point to some other directory +containing a collection of tests: + + eg$ runtest --tool gdb --srcdir /devo/gdb/testsuite + + These examples assume a "native" configuration, where the same +computer runs both `runtest' and the tests themselves. When you have a +"cross" configuration, the tests run on a different computer, +controlled by the host running `runtest'. In this situation, you need +the option `--name' to specify the network address for the other +computer: + + eg$ runtest --tool gdb --name vx9.munist.com + + If you always use the same option values, you can record them in a +file called `site.exp', rather than typing them each time. *Note +Setting defaults for `runtest' options: Config Values. + + By default, `runtest' prints only the names of the tests it runs, +output from any tests that have unexpected results, and a summary +showing how many tests passed and how many failed. To display output +from all tests (whether or not they behave as expected), use the +`--all' option. For more verbose output about processes being run, +communication, and so on, use `--verbose'. To see even more output, use +multiple `--verbose' options. *Note Using `runtest': Invoking runtest, +for a more detailed explanation of each `runtest' option. + + Test output goes into two files in your current directory: summary +output in `TOOL.sum', and detailed output in `TOOL.log'. (TOOL refers +to the collection of tests; for example, after a run with `--tool gdb', +look for output files `gdb.sum' and `gdb.log'.) *Note The files +DejaGnu writes: Output Files. + + ---------- Footnotes ---------- + + (1) `--tool' selects a particular suite of tests, *not* the name of +the executable program to run. *Note Configuration dependent values: +Config Values, for information on the variables that you can use to +specify the names of programs to run. + + +File: dejagnu.info, Node: Sample Test, Next: Design Goals, Prev: Running Tests, Up: Overview + +What does a DejaGnu test look like? +=================================== + + Each DejaGnu test is an `expect' script; the tests vary widely in +complexity, depending on the nature of the tool and the feature tested. + + Here is a very simple GDB test--one of the simplest tests shipped +with DejaGnu (extracted from `gdb.t00/echo.exp'):(1) + + # send a string to the GDB stdin: + send "echo Hello world!\n" + + # inspect the GDB stdout for the correct reply, + # and determine whether the test passes or fails: + expect { + -re "Hello world.*$prompt $" { pass "Echo test" } + -re "$prompt $" { fail "Echo test" } + timeout { fail "(timeout) Echo test" } + } + + Though brief, this example is a complete test. It illustrates some +of the main features of DejaGnu test scripts: + + * The test case does not start the tested program (GDB in this case); + all test scripts for interactive tools can assume the + corresponding tool is running. + + * Comments start with `#'. + + * The main commands you use to control a tested program are `send' + (to give it commands) and `expect' (to analyze its responses). + + * The `expect' command uses a list of pairs; a pattern (regular + expression if `-re' specified), followed by an action to run if the + pattern matches output from the program. Only the action for the + *first* matching pattern will execute. + + * Test cases use the commands `pass' and `fail' to record the test + outcome. + + ---------- Footnotes ---------- + + (1) More recent GDB tests use the `gdb_test' procedure. An +equivalent test using that procedure is ` gdb_test "echo Hello world!" +"Hello world!" ' + + +File: dejagnu.info, Node: Design Goals, Next: Posix, Prev: Sample Test, Up: Overview + +Design goals +============ + + DejaGnu grew out of the internal needs of Cygnus Support. Cygnus +maintains and enhances a variety of free programs in many different +environments, and we needed a testing tool that: + + * is useful to developers while fixing bugs; + + * automates running many tests during a software release process; + + * is portable among a variety of host computers; + + * supports cross-development testing; + + * permits testing interactive programs, like GDB; and + + * permits testing batch oriented programs, like GCC. + + Some of the requirements proved challenging. For example, +interactive programs do not lend themselves very well to automated +testing. But all the requirements are important: for instance, it is +imperative to make sure that GDB works as well when cross-debugging as +it does in a native configuration. + + Probably the greatest challenge was testing in a cross-development +environment (which can be a real nightmare). Most cross-development +environments are customized by each developer. Even when buying +packaged boards from vendors there are many differences. The +communication interfaces vary from a serial line to ethernet. DejaGnu +was designed with a modular communication setup, so that each kind of +communication can be added as required, and supported thereafter. Once +a communication procedure is coded, any test can use it. Currently +DejaGnu can use `rsh', `rlogin', `telnet', `tip', `kermit', and +`mondfe' for remote communications. + + Julia Menapace first coined the term "Deja Gnu" to describe an +earlier testing framework at Cygnus Support. When we replaced it with +the Expect-based framework, it was like DejaGnu all over again... + + +File: dejagnu.info, Node: Posix, Next: Future Directions, Prev: Design Goals, Up: Overview + +A POSIX conforming test framework +================================= + + DejaGnu conforms to the POSIX standard for test frameworks. + + POSIX standard 1003.3 defines what a testing framework needs to +provide, in order to permit the creation of POSIX conformance test +suites. This standard is primarily oriented to running POSIX +conformance tests, but its requirements also support testing of features +not related to POSIX conformance. POSIX 1003.3 does not specify a +particular testing framework, but at this time there is only one other +POSIX conforming test framework: TET.(1) + + The POSIX documentation refers to "assertions". An assertion is a +description of behavior. For example, if a standard says "The sun +shall shine", a corresponding assertion might be "The sun is shining." +A test based on this assertion would pass or fail depending on whether +it is daytime or nighttime. It is important to note that the standard +being tested is never 1003.3; the standard being tested is some other +standard, for which the assertions were written. + + As there is no test suite to test *testing frameworks* for POSIX +1003.3 conformance, verifying conformance to this standard is done by +repeatedly reading the standard and experimenting. One of the main +things 1003.3 does specify is the set of allowed output messages, and +their definitions. Four messages are supported for a required feature +of POSIX conforming systems, and a fifth for a conditional feature. +DejaGnu supports the use of all five output messages; in this sense a +test suite that uses exactly these messages can be considered POSIX +conforming. These definitions specify the output of a test case: + +`PASS' + A test has succeeded. That is, it demonstrated that the assertion + is true. + +`XFAIL' + POSIX 1003.3 does not incorporate the notion of expected failures, + so `PASS', instead of `XPASS', must also be returned for test + cases which were expected to fail and did not. This means that + `PASS' is in some sense more ambiguous than if `XPASS' is also + used. For information on `XPASS' and `XFAIL', see *Note Using + `runtest': Invoking runtest. + +`FAIL' + A test *has* produced the bug it was intended to capture. That is, + it has demonstrated that the assertion is false. The `FAIL' + message is based on the test case only. Other messages are used to + indicate a failure of the framework. + + As with `PASS', POSIX tests must return `FAIL' rather than `XFAIL' + even if a failure was expected. + +`UNRESOLVED' + A test produced indeterminate results. Usually, this means the + test executed in an unexpected fashion; this outcome requires that + a human being go over results, to determine if the test should + have passed or failed. This message is also used for any test + that requires human intervention because it is beyond the + abilities of the testing framework. Any unresolved test should + resolved to `PASS' or `FAIL' before a test run can be considered + finished. + + Note that for POSIX, each assertion must produce a test result + code. If the test isn't actually run, it must produce `UNRESOLVED' + rather than just leaving that test out of the output. This means + that you have to be careful when writing tests, to not carelessly + use tcl statements like `return'--if you alter the flow of control + of the tcl code you must insure that every test still produces + some result code. + + Here are some of the ways a test may wind up `UNRESOLVED': + + * A test's execution is interrupted. + + * A test does not produce a clear result. This is usually + because there was an `ERROR' from DejaGnu while processing + the test, or because there were three or more `WARNING' + messages. Any `WARNING' or `ERROR' messages can invalidate + the output of the test. This usually requires a human being + to examine the output to determine what really happened--and + to improve the test case. + + * A test depends on a previous test, which fails. + + * The test was set up incorrectly. + +`UNTESTED' + A test was not run. This is a placeholder, used when there is no + real test case yet. + +The only remaining output message left is intended to test features that +are specified by the applicable POSIX standard as conditional: + +`UNSUPPORTED' + There is no support for the tested case. This may mean that a + conditional feature of an operating system, or of a compiler, is + not implemented. DejaGnu also uses this message when a testing + environment (often a "bare board" target) lacks basic support for + compiling or running the test case. For example, a test for the + system subroutine `gethostname' would never work on a target board + running only a boot monitor. + + DejaGnu uses the same output procedures to produce these messages for +all test suites, and these procedures are already known to conform to +POSIX 1003.3. For a DejaGnu test suite to conform to POSIX 1003.3, you +must avoid the `setup_xfail' procedure as described in the `PASS' +section above, and you must be careful to return `UNRESOLVED' where +appropriate, as described in the `UNRESOLVED' section above. + + ---------- Footnotes ---------- + + (1) TET was created by Unisoft for a consortium comprised of X/Open, +Unix International, and the Open Software Foundation. + + +File: dejagnu.info, Node: Future Directions, Next: Tcl and Expect, Prev: Posix, Up: Overview + +Future directions +================= + + In the near future, there are two parallel directions for DejaGnu +development. The first is to add support for more hosts and targets. + + The second would permit testing programs with a more complex +interface, whether text based or GUI based. Two components already +exist: a Tcl based X window toolkit, and a terminal package for +`expect'. Both of these could be merged into DejaGnu in a way that +permits testing programs that run in each environment. + + Meanwhile, we hope DejaGnu enables the creation of test suites for +conformance to ANSI C and C++, to POSIX, and to other standards. We +encourage you to make any test suites you create freely available, +under the same terms as DejaGnu itself. + + +File: dejagnu.info, Node: Tcl and Expect, Prev: Future Directions, Up: Overview + +Tcl and Expect +============== + + Tcl was introduced in a paper by John K. Ousterhout at the 1990 +Winter Usenix conference, `Tcl: An Embeddable Command Language'. That +paper is included in PostScript form in the `doc' subdirectory of the +Tcl distribution. The version of Tcl included in DejaGnu at this time is +Tcl 7.4p3. + + Don Libes introduced `expect' in his paper `expect: Curing Those +Uncontrollable Fits of Interaction' at the 1990 Summer Usenix +conference. The paper is included in PostScript form in the `expect' +distribution (as are several other papers about `expect'). The version +of expect included in DejaGnu at this time is expect 5.18.0. + + +File: dejagnu.info, Node: Invoking runtest, Next: Customizing, Prev: What is New, Up: Top + +Using `runtest' +*************** + + `runtest' is the executable test driver for DejaGnu. You can +specify two kinds of things on the `runtest' command line: command line +options, and Tcl variables for the test scripts. The options are +listed alphabetically below. + + `runtest' returns an exit code of `1' if any test has an unexpected +result; otherwise (if all tests pass or fail as expected) it returns +`0' as the exit code. + + `runtest' flags the outcome of each test as one of these cases. +(*Note A POSIX conforming test framework: Posix, for a discussion of +how POSIX specifies the meanings of these cases.) + +`PASS' + The most desirable outcome: the test succeeded, and was expected to + succeed. + +`XPASS' + A pleasant kind of failure: a test was expected to fail, but + succeeded. This may indicate progress; inspect the test case to + determine whether you should amend it to stop expecting failure. + +`FAIL' + A test failed, although it was expected to succeed. This may + indicate regress; inspect the test case and the failing software + to locate the bug. + +`XFAIL' + A test failed, but it was expected to fail. This result indicates + no change in a known bug. If a test fails because the operating + system where the test runs lacks some facility required by the + test, the outcome is `UNSUPPORTED' instead. + +`UNRESOLVED' + Output from a test requires manual inspection; the test suite + could not automatically determine the outcome. For example, your + tests can report this outcome is when a test does not complete as + expected. + +`UNTESTED' + A test case is not yet complete, and in particular cannot yet + produce a `PASS' or `FAIL'. You can also use this outcome in dummy + "tests" that note explicitly the absence of a real test case for a + particular property. + +`UNSUPPORTED' + A test depends on a conditionally available feature that does not + exist (in the configured testing environment). For example, you + can use this outcome to report on a test case that does not work + on a particular target because its operating system support does + not include a required subroutine. + + `runtest' may also display the following messages: + +`ERROR' + Indicates a major problem (detected by the test case itself) in + running the test. This is usually an unrecoverable error, such as + a missing file or loss of communication to the target. (POSIX + test suites should not emit this message; use `UNSUPPORTED', + `UNTESTED', or `UNRESOLVED' instead, as appropriate.) + +`WARNING' + Indicates a possible problem in running the test. Usually warnings + correspond to recoverable errors, or display an important message + about the following tests. + +`NOTE' + An informational message about the test case. + + This is the full set of command line options that `runtest' +recognizes. Arguments may be abbreviated to the shortest unique string. + + runtest --tool TOOL [ TESTSUITE.exp ... ] + [ TESTSUITE.exp="testfile1 ..." ] + [ TCLVAR=VALUE... ] + [ --all ] [ --baud BAUD-RATE ] [ --connect TYPE ] + [ --debug ] [ --help ] [ --host STRING ] + [ --mail "NAME ..." ] [ --name STRING ] + [ --name NAME ] [ --outdir PATH ] + [ --objdir PATH ] [ --reboot ] + [ --srcdir PATH ] [ --strace N ] + [ --target STRING --build STRING ] + [ -v | --verbose ] [ -V | --version ] [ --DN ] + +`--tool TOOL' + TOOL specifies what set of tests to run, and what initialization + module to use. TOOL is used *only* for these two purposes: it is + *not* used to name the executable program to test. Executable + tool names (and paths) are recorded in `site.exp' (*note + Configuration dependent values: Config Values.), and you can + override them by specifying Tcl variables on the command line. + + For example, including `--tool gcc' on the `runtest' command line + runs tests from all test subdirectories whose names match `gcc.*', + and uses one of the initialization modules named + `config/*-gcc.exp'. To specify the name of the compiler (perhaps + as an alternative path to what `runtest' would use by default), use + `GCC=BINNAME' on the `runtest' command line. + +`TESTSUITE.exp ...' + Specify the names of testsuites to run. By default, `runtest' + runs all tests for the tool, but you can restrict it to particular + testsuites by giving the names of the `.exp' `expect' scripts that + control them. + + TESTSUITE.exp may not include path information; use plain + filenames. + +`TESTFILE.exp="testfile1 ..."' + Specify a subset of tests in a suite to run. For compiler or + assembler tests, which often use a single `.exp' script covering + many different source files, this option allows you to further + restrict the tests by listing particular source files to compile. + Some tools even support wildcards here. The wildcards supported + depend upon the tool, but typically they are `?', `*', and + `[chars]'. + +`TCLVAR=VALUE' + You can define Tcl variables for use by your test scripts in the + same style used with `make' for environment variables. For + example, `runtest GDB=gdb.old' defines a variable called `GDB'; + when your scripts refer to `$GDB' in this run, they use the value + `gdb.old'. + + The default Tcl variables used for most tools are defined in the + main DejaGnu `Makefile'; their values are captured in the + `site.exp' file. *Note Configuration dependent values: Config + Values. + +`--all' + Display all test output. By default, `runtest' shows only the + output of tests that produce unexpected results; that is, tests + with status `FAIL' (unexpected failure), `XPASS' (unexpected + success), or `ERROR' (a severe error in the test case itself). + Specify `--all' to see output for tests with status `PASS' + (success, as expected) `XFAIL' (failure, as expected), or + `WARNING' (minor error in the test case itself). + +`--baud BAUD-RATE' +`-b BAUD-RATE' + Set the default baud rate to something other than 9600. (Some + serial interface programs, like `tip', use a separate + initialization file instead of this value.) + +`--connect TYPE' + Connect to a target testing environment as specified by TYPE, if + the target is not the computer running `runtest'. For example, use + `--connect' to change the program used to connect to a "bare + board" boot monitor. The choices for TYPE in the DejaGnu 1.0 + distribution are `rlogin', `telnet', `rsh', `tip', `kermit', and + `mondfe'. + + The default for this option depends on the configuration (*note + Remote targets supported: Cross Targets.). The default is chosen + to be the most convenient communication method available, but + often other alternatives work as well; you may find it useful to + try alternative connect methods if you suspect a communication + problem with your testing target. + +`--debug' + Turns on the `expect' internal debugging output. Debugging output + is displayed as part of the `runtest' output, and logged to a file + called `dbg.log'. The extra debugging output does *not* appear on + standard output, unless the verbose level is greater than 2 (for + instance, to see debug output immediately, specify `--debug -v + -v'). The debugging output shows all attempts at matching the test + output of the tool with the scripted patterns describing expected + output. The output generated with `--strace' also goes into + `dbg.log'. + +`--help' +`-he' + Prints out a short summary of the `runtest' options, then exits + (even if you also specify other options). + +`--host STRING' + STRING is a full configuration "triple" name as used by + `configure'. Use this option to override the default string + recorded by your configuration's choice of host. This choice does + not change how anything is actually configured unless -build is + also specified; it affects *only* DejaGnu procedures that compare + the host string with particular values. The procedures `ishost', + `istarget', `isnative', and `setup_xfail' are affected by + `--host'. In this usage, `host' refers to the machine that the + tests are to be run on, which may not be the same as the `build' + machine. If `--build' is also specified, then `--host' refers to + the machine that the tests wil, be run on, not the machine DejaGnu + is run on. + +`--build STRING' + STRING is a full configuration "triple" name as used by + `configure'. This is the type of machine DejaGnu and the tools to + be tested are built on. For a normal cross this is the same as the + host, but for a canadian cross, they are seperate. + +`--name NAME' + NAME is a name for the particular testing target machine (for + cross testing). If the testing target has IP network support (for + example, `RPC' or `NFS'), this is the network name for the target + itself. (NAME is *not the configuration string* you specify as a + target with `configure'; the `--name' option names a particular + target, rather than describing a class of targets.) For targets + that connect in other ways, the meaning of the NAME string depends + on the connection method. *Note Remote targets supported: Cross + Targets. + +`--name STRING' + Specify a network name of testing target or its host. The + particular names that are meaningful with `--name' will depend on + your site configuration, and on the connection protocol: for + example, `tip' connections require names from a serial line + configuration file (usually called `/etc/remote'), while `telnet' + connections use IP hostnames. + +`--objdir PATH' + Use PATH as the top directory containing any auxiliary compiled + test code. This defaults to `.'. Use this option to locate + pre-compiled test code. You can normally prepare any auxiliary + files needed with `make'. + +`--outdir PATH' + Write output logs in directory PATH. The default is `.', the + directory where you start `runtest'. This option affects only the + summary and the detailed log files `TOOL.sum' and `TOOL.log'. The + DejaGnu debug log `dbg.log' always appears (when requested) in the + local directory. + +`--reboot' + Reboot the target board when `runtest' initializes. Usually, when + running tests on a separate target board, it is safer to reboot + the target to be certain of its state. However, when developing + test scripts, rebooting takes a lot of time. + +`--srcdir PATH' + Use PATH as the top directory for test scripts to run. `runtest' + looks in this directory for any subdirectory whose name begins + with the toolname (specified with `--tool'). For instance, with + `--tool gdb', `runtest' uses tests in subdirectories `gdb.*' (with + the usual shell-like filename expansion). If you do not use + `--srcdir', `runtest' looks for test directories under the current + working directory. + +`--strace N' + Turn on internal tracing for `expect', to N levels deep. By + adjusting the level, you can control the extent to which your + output expands multi-level Tcl statements. This allows you to + ignore some levels of `case' or `if' statements. Each procedure + call or control structure counts as one "level". + + The output is recorded in the same file, `dbg.log', used for output + from `--debug'. + +`--target STRING' + Use this option to override the default setting (running native + tests). STRING is a full configuration "triple" name(1) as used + by `configure'. This option changes the configuration `runtest' + uses for the default tool names, and other setup information. + *Note Using `configure': (configure.info)Using configure, for + details about `configure' names. + +`--verbose' +`-v' + Turns on more output. Repeating this option increases the amount + of output displayed. Level one (`-v') is simply test output. Level + two (`-v -v') shows messages on options, configuration, and process + control. Verbose messages appear in the detailed (`*.log') log + file, but not in the summary (`*.sum') log file. + +`--version' +`-V' + Prints out the version numbers of DejaGnu, `expect' and Tcl, and + exits without running any tests. + +`-D0' +`-D1' + Start the internal Tcl debugger. The Tcl debugger supports + breakpoints, single stepping, and other common debugging + activities. (See `A Debugger for Tcl Applications' by Don Libes. + (2)) + + If you specify `-D1', the `expect' shell stops at a breakpoint as + soon as DejaGnu invokes it. + + If you specify `-D0', DejaGnu starts as usual, but you can enter + the debugger by sending an interrupt (e.g. by typing <C-c>). + + ---------- Footnotes ---------- + + (1) Configuration triples have the form `CPU-VENDOR-OS'. + + (2) Distributed in PostScript form with `expect' as the file +`expect/tcl-debug.ps'. + + +File: dejagnu.info, Node: Customizing, Next: Internals, Prev: Invoking runtest, Up: Top + +Setting `runtest' defaults +************************** + + The site configuration file, `site.exp', captures +configuration-dependent values and propagates them to the DejaGnu test +environment using Tcl variables. This ties the DejaGnu test scripts +into the `configure' and `make' programs. + + DejaGnu supports more than one `site.exp' file. The multiple +instances of `site.exp' are loaded in a fixed order built into DejaGnu +(the more local last). The first file loaded is the optional +`~/.dejagnurc', then the local files, and finally the global file. + + 1. There is am optional "master" `site.exp', capturing configuration + values that apply to DejaGnu across the board, in each + configuration-specific subdirectory of the DejaGnu library + directory. `runtest' loads these values first. *Note Configuring + and Installing DejaGnu: Installation. The master `site.exp' + contains the default values for all targets and hosts supported by + DejaGnu. This master file is identified by setting the environment + variable `DEJAGNU' to the name of the file. This is also refered + to as the "global" config file. + + 2. Any directory containing a configured test suite also has a + `site.exp', capturing configuration values specific to the tool + under test. Since `runtest' loads these values last, the + individual test configuration can either rely on and use, or + override, any of the global values from the "master" `site.exp'. + + You can usually generate or update the testsuite `site.exp' by + typing `make site.exp' in the test suite directory, after the test + suite is configured. + + 3. You can also have a file in your home directory called + `.dejagnurc'. This gets loaded first before the other config + files. Usually this is used for personal stuff, like setting + `all_flag' so all the output gets printed, or verbosity levels. + + You can further override the default values in a user-editable +section of any `site.exp', or by setting variables on the `runtest' +command line. + +* Menu: + +* Config Values:: Variables used in the configuration file. +* Master Config File:: The master configuration file. +* Local Config File:: The local configuration file. +* Personal Config File:: The personal configuration file. + + +File: dejagnu.info, Node: Config Values, Next: Master Config File, Up: Customizing + +Config Variables +---------------- + + DejaGnu uses a named array in Tcl to hold all the info for each +machine. In the case of a canadian cross, this means host information as +well as target information. The named array is called `target_info', +and it has two indices. The following fields are part of the array. + +`name' + The name of the target. (mostly for error messages) This should + also be the string used for this target's array. It should also + be the same as the linker script so we can find them dynamically. + This should be the same as the argument used for `push_target{}'. + +`ldflags' + This is the linker flags required to produce a fully linked + executable. For `libgloss' supported targets this is usually just + the name of the linker script. + +`config' + The target canonical for this target. This is used by some init + files to make sure the target is supported. + +`cflags' + The flags required to produce an object file from a source file. + +`connect' + This is the connectmode for this target. This is for both IP and + serial connections. Typically this is either `telnet', `rlogin', + or `rsh'. + +`target' + This is the hostname of the target. This is for TCP/IP based + connections, and is also used for version of tip that use + /etc/remote. + +`serial' + This is the serial port. This is typically /dev/tty? or com?:. + +`netport' + This is the IP port. This is commonly used for telneting to target + boards that are connected to a terminal server. In that case the + IP port specifies the which serial port to use. + +`baud' + This is the baud rate for a serial port connection. + +`x10' + This is the parameters for an x10 controller. These are simple + devices that let us power cycle or reset a target board remotely. + +`fileid' + This is the fileid or spawn id of of the connection. + +`prompt' + a glob style pattern to recognize the prompt. + +`abbrev' + abbreviation for tool init files. + +`ioport' + This is the port for I/O on dual port systems. In this + configuration, the main serial port `0' is usually used for stdin + and stdout, which the second serial port can be used for debugging. + + The first index into the array is the same value as used in the +`name' field. This is usually a short version of the name of the target +board. For an example, here's the settings I use for my `Motorola's' +`IDP' board and my `Motorola' 6U VME `MVME135-1' board. (both m68k +targets) + + # IDP board + set target_info(idp,name) "idp" + set target_info(idp,ldflags) "-Tidp.ld" + set target_info(idp,config) m68k-unknown-aout + set target_info(idp,cflags) "" + set target_info(idp,connect) telnet + set target_info(idp,target) "s7" + set target_info(idp,serial) "tstty7" + set target_info(idp,netport) "wharfrat:1007" + set target_info(idp,baud) "9600" + # MVME 135 board + set target_info(idp,name) "mvme" + set target_info(idp,ldflags) "-Tmvme.ld" + set target_info(idp,config) m68k-unknown-aout + set target_info(idp,cflags) "" + set target_info(idp,connect) telnet + set target_info(idp,target) "s8" + set target_info(idp,serial) "tstty8" + set target_info(idp,netport) "wharfrat:1008" + set target_info(idp,baud) "9600" + + DejaGnu can use this information to switch between multiple targets +in one test run. This is done through the use of the `push_target' +procedure, which is discussed elsewhere. + + This array can also hold information for a remote host, which is used +when testing a candain cross. In this case, the only thing different is +the index is just `host'. Here's the settings I use to run tests on my +NT machine while running DejaGnu on a Unix machine. (in this case a +Linux box) + + set target_info(host,name) "nt-host" + set target_info(host,config) "386-unknown-winnt" + set target_info(host,connect) "telnet" + set target_info(host,target) "ripple" + + There is more info on how to use these variables in the sections on +the config files. *Note Configuration Files: Master Config File. + + In the user editable second section of `site.exp', you can not only +override the configuration variables captured in the first section, but +also specify default values for all the `runtest' command line options. +Save for `--debug', `--help', and `--version', each command line +option has an associated Tcl variable. Use the Tcl `set' command to +specify a new default value (as for the configuration variables). The +following table describes the correspondence between command line +options and variables you can set in `site.exp'. *Note Running the +Tests: Invoking runtest, for explanations of the command-line options. + + runtest Tcl + option variable description + __________ ________ ___________________________________________ + + --all all_flag display all test results if set + + --baud baud set the default baud rate to something other + than 9600. + --connect connectmode `rlogin', `telnet', `rsh', + `kermit', `tip', or `mondfe' + + --outdir outdir directory for `TOOL.sum' and `TOOL.log' + + --objdir objdir directory for pre-compiled binaries + + --reboot reboot reboot the target if set to `"1"'; + do not reboot if set to `"0"' (the default) + + --srcdir srcdir directory of test subdirectories + + --strace tracelevel a number: Tcl trace depth + + --tool tool name of tool to test; identifies init, test subdir + + --verbose verbose verbosity level. As option, use multiple times; + as variable, set a number, 0 or greater + --target target_triplet The canonical configuration string for the target. + --host host_triplet The canonical configuration string for the host. + --build build_triplet The canonical configuration string for the + build host. + + +File: dejagnu.info, Node: Master Config File, Next: Local Config File, Prev: Config Values, Up: Customizing + +Master Config File +------------------ + + The master config file is where all the target specific config +variables get set for a whole site get set. The idea is that for a +centralized testing lab where people have to share a target between +multiple developers. There are settings for both remote targets and +remote hosts. Here's an example of a Master Config File (also called +the Global config file) for a *canadian cross*. A canadian cross is +when you build and test a cross compiler on a machine other than the +one it's to be hosted on. + + Here we have the config settings for our California office. Note that +all config values are site dependant. Here we have two sets of values +that we use for testing m68k-aout cross compilers. As both of these +target boards has a different debugging protocol, we test on both of +them in sequence. + + global CFLAGS + global CXXFLAGS + + case "$target_triplet" in { + { "native" } { + set target_abbrev unix + } + { "m68*-unknown-aout" } { + set target_abbrev "rom68k" + # IDP target # IDP board with rom68k monitor + set target_info(idp,name) "idp" + set target_info(idp,ldflags) "-Tidp.ld" + set target_info(idp,config) m68k-unknown-aout + set target_info(idp,cflags) "" + set target_info(idp,connect) telnet + set target_info(idp,target) "s7" + set target_info(idp,serial) "tstty12" + set target_info(idp,netport) "truckin:1007" + set target_info(idp,baud) "9600" + # MVME target # Motorola MVME 135 with BUG monitor + set target_info(mvme,name) "mvme" + set target_info(mvme,ldflags) "-Tmvme.ld" + set target_info(mvme,config) m68k-unknown-aout + set target_info(mvme,cflags) "" + set target_info(mvme,connect) telnet + set target_info(mvme,target) "s4" + set target_info(mvme,serial) "tstty8" + set target_info(mvme,netport) "truckin:1004" + set target_info(mvme,baud) "9600" + } + } + + In this case, we have support for several remote hosts for our +m68k-aout cross compiler. Typically the remote Unix hosts run DejaGnu +locally, but we also use them for debugging the testsuites when we find +problems in running on remote hosts. Expect won't run on NT, so DejaGnu +is run on the local build machine, and it'll connect to the NT host and +run all the tests for this cross compiler on that host. + + case "$host_triplet" in { + "native" { + } + "i?86-*-linux*" { # Linux host + set target_info(host,name) "linux-host" + set target_info(host,config) $host_triplet + set target_info(host,connect) rlogin + set target_info(host,target) chinadoll + } + "i?86-*-winnt # NT host + set target_info(host,name) "nt-host" + set target_info(host,config) i386-unknown-winnt + set target_info(host,connect) telnet + set target_info(host,target) ripple + } + "hppa*-hp-hpux*" { # HP-UX host + set target_info(host,name) "hpux-host" + set target_info(host,config) $host_triplet + set target_info(host,connect) rlogin + set target_info(host,target) slipknot + } + "sparc-sun-sunos*" { # SunOS (sun4) + set target_info(host,name) "sunos-host" + set target_info(host,config) $host_triplet + set target_info(host,connect) rlogin + set target_info(host,target) darkstar + } + } + + +File: dejagnu.info, Node: Local Config File, Next: Personal Config File, Prev: Master Config File, Up: Customizing + +Local Config File +----------------- + + It is usually more convenient to keep these "manual overrides" in the +`site.exp' local to each test directory, rather than in the "master" +`site.exp' in the DejaGnu library. + + All local `site.exp' usually files have two sections, separated by +comment text. The first section is the part that is generated by +`make'. It is essentially a collection of Tcl variable definitions +based on `Makefile' environment variables. Since they are generated by +`make', they contain the values as specified by `configure'. (You can +also customize these values by using the `--site' option to +`configure'.) In particular, this section contains the `Makefile' +variables for host and target configuration data. Do not edit this +first section; if you do, your changes are replaced next time you run +`make'. + + The first section starts with: + + ## these variables are automatically generated by make ## + # Do not edit here. If you wish to override these values + # add them to the last section + + In the second section, you can override any default values (locally +to DejaGnu) for all the variables. The second section can also contain +your preferred defaults for all the command line options to `runtest'. +This allows you to easily customize `runtest' for your preferences in +each configured test-suite tree, so that you need not type options +repeatedly on the command line. (The second section may also be empty, +if you do not wish to override any defaults.) + + The first section ends with this line: + + ## All variables above are generated by configure. Do Not Edit ## + + You can make any changes under this line. If you wish to redefine a +variable in the top section, then just put a duplicate value in this +second section. Usually the values defined in this config file are +related to the configuration of the test run. This is the ideal place to +set the variables `host_triplet', `build_triplet', `target_triplet'. +All other variables are tool dependant. ie for testing a compiler, the +value for CC might be set to a freshly built binary, as opposed to one +in the user's path. + + +File: dejagnu.info, Node: Personal Config File, Prev: Local Config File, Up: Customizing + +Personal Config File +-------------------- + + The personal config file is used to customize `runtest's' behaviour +for each person. It's typically used to set the user prefered setting +for verbosity, and any experimental Tcl procedures. My personal +`~/.dejagnurc' file looks like: + + set all_flag 1 + set RLOGIN /usr/ucb/rlogin + set RSH /usr/ucb/rsh + + Here I set `all_flag' so I see all the test cases that PASS along +with the ones that FAIL. I also set RLOGIN and `RSH' to the BSD +version. I have `kerberos' installed, and when I rlogin to a target +board, it usually isn't supported. So I use the non secure versions of +these programs rather than the default that's in my path. + + +File: dejagnu.info, Node: Internals, Next: Tests, Prev: Customizing, Up: Top + +The DejaGnu Implementation +************************** + + DejaGnu is entirely written in `expect', which uses Tcl as a command +language. `expect' serves as a very programmable shell; you can run +any program, as with the usual Unix command shells--but once the +program is started, your `expect' script has fully programmable control +of its input and output. This does not just apply to the programs +under test; `expect' can also run any auxiliary program, such as `diff' +or `sh', with full control over its input and output. + + DejaGnu itself is merely a framework for the set of test suites +distributed separately for each GNU tool. Future releases of GNU tools +will include even more tests, developed throughout the free software +community. + + `runtest' is the glue to tie together and manage the test scripts. +The `runtest' program is actually a simple Bourne shell script that +locates a copy of the `expect' shell and then starts the main Tcl code, +`runtest.exp'. `runtest.exp' itself has these essential functions: + + 1. Parse the command line options, load the library files, and load + the default configuration files. + + 2. Locating the individual test scripts. `runtest.exp' locates the + tests by exploiting a straightforward naming convention based on + the string you specify with the `--tool' option. + + 3. Providing an extended test environment, by defining additional Tcl + procedures beyond those already in `expect'. + + 4. Locating target-dependent functions, to standardize the test + environment across a wide variety of test platforms. + +* Menu: + +* Names:: Conventions for using tool names +* Init Module:: Initialization module +* DejaGnu Builtins:: DejaGnu provides these Tcl procedures +* Target Dependent:: Procedures supplied by the init module +* Cross Targets:: Remote targets supported +* Input Files:: The files DejaGnu depends on +* Output Files:: The files DejaGnu produces + |