diff options
Diffstat (limited to 'contrib/bluegnu2.0.3/doc/dejagnu.info-1')
| -rw-r--r-- | contrib/bluegnu2.0.3/doc/dejagnu.info-1 | 1163 |
1 files changed, 0 insertions, 1163 deletions
diff --git a/contrib/bluegnu2.0.3/doc/dejagnu.info-1 b/contrib/bluegnu2.0.3/doc/dejagnu.info-1 deleted file mode 100644 index c11c3b4..0000000 --- a/contrib/bluegnu2.0.3/doc/dejagnu.info-1 +++ /dev/null @@ -1,1163 +0,0 @@ -This is Info file dejagnu.info, produced by Makeinfo version 1.68 from -the input file ./dejagnu.texi. - -START-INFO-DIR-ENTRY -* DejaGnu: (dejagnu). The GNU testing framework. -END-INFO-DIR-ENTRY - - Copyright (C) 92, 93, 94, 95, 1996 Free Software Foundation, Inc. - - Permission is granted to make and distribute verbatim copies of this -manual provided the copyright notice and this permission notice are -preserved on all copies. - - Permission is granted to copy and distribute modified versions of -this manual under the conditions for verbatim copying, provided also -that the entire resulting derived work is distributed under the terms -of a permission notice identical to this one. - - Permission is granted to copy and distribute translations of this -manual into another language, under the above conditions for modified -versions. - - -File: dejagnu.info, Node: Top, Next: Overview, Up: (dir) - -DejaGnu -******* - - DejaGnu is a framework for running test suites on software tools. - - This file describes version 1.3 of DejaGnu. - -* Menu: - -* Overview:: What is DejaGnu? -* What is New:: What is new in this release. -* Invoking runtest:: Using `runtest', the main test driver -* Customizing:: Setting `runtest' defaults -* Internals:: The DejaGnu implementation -* Tests:: How to write a test case -* Extending:: New tools, new targets, and new hosts -* Installation:: Configuring and Installing DejaGnu -* Index:: Index - - -File: dejagnu.info, Node: Overview, Next: What is New, Prev: Top, Up: Top - -What is DejaGnu? -**************** - - DejaGnu is a framework for testing other programs. Its purpose is to -provide a single front end for all tests. Beyond this, DejaGnu offers -several advantages for testing: - - 1. The flexibility and consistency of the DejaGnu framework make it - easy to write tests for any program. - - 2. DejaGnu provides a layer of abstraction which allows you to write - tests that are portable to any host or target where a program must - be tested. For instance, a test for GDB can run (from any Unix - based host) on any target architecture that DejaGnu supports. - Currently DejaGnu runs tests on several single board computers, - whose operating software ranges from just a boot monitor to a - full-fledged, Unix-like realtime OS. - - 3. All tests have the same output format. This makes it easy to - integrate testing into other software development processes. - DejaGnu's output is designed to be parsed by other filtering - script, and it is also human readable. - - DejaGnu is written in `expect', which in turn uses "Tcl"--Tool -command language. - - Running tests requires two things: the testing framework, and the -test suites themselves. Tests are usually written in `expect' using -Tcl, but you can also use a Tcl script to run a test suite that is not -based on `expect'. (`expect' script filenames conventionally use -`.exp' as a suffix; for example, the main implementation of the DejaGnu -test driver is in the file `runtest.exp'.) - -* Menu: - -* Running Tests:: A first look at running DejaGnu tests -* Sample Test:: What does a DejaGnu test case look like? -* Design Goals:: Goals behind DejaGnu -* Posix:: DejaGnu conforms to POSIX 1003.3 -* Future Directions:: Where is DejaGnu going? -* Tcl and Expect:: Reading more about Tcl and Expect - - -File: dejagnu.info, Node: What is New, Next: Invoking runtest, Prev: Overview, Up: Top - -What is new in this release ? -***************************** - - This release has a number of substantial changes over version 1.2. -The most visible change is that the version of expect and Tcl included -in the release are up-to-date with the current stable net releases. -Other changes are: - - 1. The config sub-system in DejaGnu has been completely redesigned. - It now supports testing on remote hosts as well as remote targets. - - 2. More builtin support for building target binaries with the correct - linker flags. Currently this only works with GCC, preferably with a - target support by `libgloss'. - - 3. Lots of little bug fixes from a year of heavy use here at Cygnus - Support. - - 4. DejaGnu now uses `autoconf' for configuration. - - 5. New test cases for DejaGnu have been added for the new features, - plus the "-tool" option bug in the 1.2 testsuite has been fixed. - - 6. The `--tool' option is now optional. - - 7. `runtest' when searching for test drivers ignores all directories - named SCCS, RCS, and CVS. - - 8. There is now a generic keyword based test harness that uses - comments in source code to control how each test case gets built - and run. - - 9. There is now some support for running a testsuite with multiple - passes. - - - -File: dejagnu.info, Node: Running Tests, Next: Sample Test, Up: Overview - -Running existing tests -====================== - - To run tests from an existing collection, first use `configure' as -usual to set up the source directory containing the tests. Then try -running - - make check - - If the `check' target exists, it usually saves you some trouble--for -instance, it can set up any auxiliary programs or other files needed by -the tests. - - Once you have run `make check' to build any auxiliary files, you -might want to call the test driver `runtest' directly to repeat the -tests. You may also have to call `runtest' directly for test -collections with no `check' target in the `Makefile'. - - Typically, you must use two command-line options: `--tool', to -specify which set of tests to run(1), and `--srcdir', to specify where -to find test directories. - - For example, if the directory `gdb/testsuite' contains a collection -of DejaGnu tests for GDB, you can run them like this: - - eg$ cd gdb/testsuite - eg$ runtest --tool gdb -*Test output follows, ending with:* - - === gdb Summary === - - # of expected passes 508 - # of expected failures 103 - /usr/latest/bin/gdb version 4.14.4 -nx - - You can use the option `--srcdir' to point to some other directory -containing a collection of tests: - - eg$ runtest --tool gdb --srcdir /devo/gdb/testsuite - - These examples assume a "native" configuration, where the same -computer runs both `runtest' and the tests themselves. When you have a -"cross" configuration, the tests run on a different computer, -controlled by the host running `runtest'. In this situation, you need -the option `--name' to specify the network address for the other -computer: - - eg$ runtest --tool gdb --name vx9.munist.com - - If you always use the same option values, you can record them in a -file called `site.exp', rather than typing them each time. *Note -Setting defaults for `runtest' options: Config Values. - - By default, `runtest' prints only the names of the tests it runs, -output from any tests that have unexpected results, and a summary -showing how many tests passed and how many failed. To display output -from all tests (whether or not they behave as expected), use the -`--all' option. For more verbose output about processes being run, -communication, and so on, use `--verbose'. To see even more output, use -multiple `--verbose' options. *Note Using `runtest': Invoking runtest, -for a more detailed explanation of each `runtest' option. - - Test output goes into two files in your current directory: summary -output in `TOOL.sum', and detailed output in `TOOL.log'. (TOOL refers -to the collection of tests; for example, after a run with `--tool gdb', -look for output files `gdb.sum' and `gdb.log'.) *Note The files -DejaGnu writes: Output Files. - - ---------- Footnotes ---------- - - (1) `--tool' selects a particular suite of tests, *not* the name of -the executable program to run. *Note Configuration dependent values: -Config Values, for information on the variables that you can use to -specify the names of programs to run. - - -File: dejagnu.info, Node: Sample Test, Next: Design Goals, Prev: Running Tests, Up: Overview - -What does a DejaGnu test look like? -=================================== - - Each DejaGnu test is an `expect' script; the tests vary widely in -complexity, depending on the nature of the tool and the feature tested. - - Here is a very simple GDB test--one of the simplest tests shipped -with DejaGnu (extracted from `gdb.t00/echo.exp'):(1) - - # send a string to the GDB stdin: - send "echo Hello world!\n" - - # inspect the GDB stdout for the correct reply, - # and determine whether the test passes or fails: - expect { - -re "Hello world.*$prompt $" { pass "Echo test" } - -re "$prompt $" { fail "Echo test" } - timeout { fail "(timeout) Echo test" } - } - - Though brief, this example is a complete test. It illustrates some -of the main features of DejaGnu test scripts: - - * The test case does not start the tested program (GDB in this case); - all test scripts for interactive tools can assume the - corresponding tool is running. - - * Comments start with `#'. - - * The main commands you use to control a tested program are `send' - (to give it commands) and `expect' (to analyze its responses). - - * The `expect' command uses a list of pairs; a pattern (regular - expression if `-re' specified), followed by an action to run if the - pattern matches output from the program. Only the action for the - *first* matching pattern will execute. - - * Test cases use the commands `pass' and `fail' to record the test - outcome. - - ---------- Footnotes ---------- - - (1) More recent GDB tests use the `gdb_test' procedure. An -equivalent test using that procedure is ` gdb_test "echo Hello world!" -"Hello world!" ' - - -File: dejagnu.info, Node: Design Goals, Next: Posix, Prev: Sample Test, Up: Overview - -Design goals -============ - - DejaGnu grew out of the internal needs of Cygnus Support. Cygnus -maintains and enhances a variety of free programs in many different -environments, and we needed a testing tool that: - - * is useful to developers while fixing bugs; - - * automates running many tests during a software release process; - - * is portable among a variety of host computers; - - * supports cross-development testing; - - * permits testing interactive programs, like GDB; and - - * permits testing batch oriented programs, like GCC. - - Some of the requirements proved challenging. For example, -interactive programs do not lend themselves very well to automated -testing. But all the requirements are important: for instance, it is -imperative to make sure that GDB works as well when cross-debugging as -it does in a native configuration. - - Probably the greatest challenge was testing in a cross-development -environment (which can be a real nightmare). Most cross-development -environments are customized by each developer. Even when buying -packaged boards from vendors there are many differences. The -communication interfaces vary from a serial line to ethernet. DejaGnu -was designed with a modular communication setup, so that each kind of -communication can be added as required, and supported thereafter. Once -a communication procedure is coded, any test can use it. Currently -DejaGnu can use `rsh', `rlogin', `telnet', `tip', `kermit', and -`mondfe' for remote communications. - - Julia Menapace first coined the term "Deja Gnu" to describe an -earlier testing framework at Cygnus Support. When we replaced it with -the Expect-based framework, it was like DejaGnu all over again... - - -File: dejagnu.info, Node: Posix, Next: Future Directions, Prev: Design Goals, Up: Overview - -A POSIX conforming test framework -================================= - - DejaGnu conforms to the POSIX standard for test frameworks. - - POSIX standard 1003.3 defines what a testing framework needs to -provide, in order to permit the creation of POSIX conformance test -suites. This standard is primarily oriented to running POSIX -conformance tests, but its requirements also support testing of features -not related to POSIX conformance. POSIX 1003.3 does not specify a -particular testing framework, but at this time there is only one other -POSIX conforming test framework: TET.(1) - - The POSIX documentation refers to "assertions". An assertion is a -description of behavior. For example, if a standard says "The sun -shall shine", a corresponding assertion might be "The sun is shining." -A test based on this assertion would pass or fail depending on whether -it is daytime or nighttime. It is important to note that the standard -being tested is never 1003.3; the standard being tested is some other -standard, for which the assertions were written. - - As there is no test suite to test *testing frameworks* for POSIX -1003.3 conformance, verifying conformance to this standard is done by -repeatedly reading the standard and experimenting. One of the main -things 1003.3 does specify is the set of allowed output messages, and -their definitions. Four messages are supported for a required feature -of POSIX conforming systems, and a fifth for a conditional feature. -DejaGnu supports the use of all five output messages; in this sense a -test suite that uses exactly these messages can be considered POSIX -conforming. These definitions specify the output of a test case: - -`PASS' - A test has succeeded. That is, it demonstrated that the assertion - is true. - -`XFAIL' - POSIX 1003.3 does not incorporate the notion of expected failures, - so `PASS', instead of `XPASS', must also be returned for test - cases which were expected to fail and did not. This means that - `PASS' is in some sense more ambiguous than if `XPASS' is also - used. For information on `XPASS' and `XFAIL', see *Note Using - `runtest': Invoking runtest. - -`FAIL' - A test *has* produced the bug it was intended to capture. That is, - it has demonstrated that the assertion is false. The `FAIL' - message is based on the test case only. Other messages are used to - indicate a failure of the framework. - - As with `PASS', POSIX tests must return `FAIL' rather than `XFAIL' - even if a failure was expected. - -`UNRESOLVED' - A test produced indeterminate results. Usually, this means the - test executed in an unexpected fashion; this outcome requires that - a human being go over results, to determine if the test should - have passed or failed. This message is also used for any test - that requires human intervention because it is beyond the - abilities of the testing framework. Any unresolved test should - resolved to `PASS' or `FAIL' before a test run can be considered - finished. - - Note that for POSIX, each assertion must produce a test result - code. If the test isn't actually run, it must produce `UNRESOLVED' - rather than just leaving that test out of the output. This means - that you have to be careful when writing tests, to not carelessly - use tcl statements like `return'--if you alter the flow of control - of the tcl code you must insure that every test still produces - some result code. - - Here are some of the ways a test may wind up `UNRESOLVED': - - * A test's execution is interrupted. - - * A test does not produce a clear result. This is usually - because there was an `ERROR' from DejaGnu while processing - the test, or because there were three or more `WARNING' - messages. Any `WARNING' or `ERROR' messages can invalidate - the output of the test. This usually requires a human being - to examine the output to determine what really happened--and - to improve the test case. - - * A test depends on a previous test, which fails. - - * The test was set up incorrectly. - -`UNTESTED' - A test was not run. This is a placeholder, used when there is no - real test case yet. - -The only remaining output message left is intended to test features that -are specified by the applicable POSIX standard as conditional: - -`UNSUPPORTED' - There is no support for the tested case. This may mean that a - conditional feature of an operating system, or of a compiler, is - not implemented. DejaGnu also uses this message when a testing - environment (often a "bare board" target) lacks basic support for - compiling or running the test case. For example, a test for the - system subroutine `gethostname' would never work on a target board - running only a boot monitor. - - DejaGnu uses the same output procedures to produce these messages for -all test suites, and these procedures are already known to conform to -POSIX 1003.3. For a DejaGnu test suite to conform to POSIX 1003.3, you -must avoid the `setup_xfail' procedure as described in the `PASS' -section above, and you must be careful to return `UNRESOLVED' where -appropriate, as described in the `UNRESOLVED' section above. - - ---------- Footnotes ---------- - - (1) TET was created by Unisoft for a consortium comprised of X/Open, -Unix International, and the Open Software Foundation. - - -File: dejagnu.info, Node: Future Directions, Next: Tcl and Expect, Prev: Posix, Up: Overview - -Future directions -================= - - In the near future, there are two parallel directions for DejaGnu -development. The first is to add support for more hosts and targets. - - The second would permit testing programs with a more complex -interface, whether text based or GUI based. Two components already -exist: a Tcl based X window toolkit, and a terminal package for -`expect'. Both of these could be merged into DejaGnu in a way that -permits testing programs that run in each environment. - - Meanwhile, we hope DejaGnu enables the creation of test suites for -conformance to ANSI C and C++, to POSIX, and to other standards. We -encourage you to make any test suites you create freely available, -under the same terms as DejaGnu itself. - - -File: dejagnu.info, Node: Tcl and Expect, Prev: Future Directions, Up: Overview - -Tcl and Expect -============== - - Tcl was introduced in a paper by John K. Ousterhout at the 1990 -Winter Usenix conference, `Tcl: An Embeddable Command Language'. That -paper is included in PostScript form in the `doc' subdirectory of the -Tcl distribution. The version of Tcl included in DejaGnu at this time is -Tcl 7.4p3. - - Don Libes introduced `expect' in his paper `expect: Curing Those -Uncontrollable Fits of Interaction' at the 1990 Summer Usenix -conference. The paper is included in PostScript form in the `expect' -distribution (as are several other papers about `expect'). The version -of expect included in DejaGnu at this time is expect 5.18.0. - - -File: dejagnu.info, Node: Invoking runtest, Next: Customizing, Prev: What is New, Up: Top - -Using `runtest' -*************** - - `runtest' is the executable test driver for DejaGnu. You can -specify two kinds of things on the `runtest' command line: command line -options, and Tcl variables for the test scripts. The options are -listed alphabetically below. - - `runtest' returns an exit code of `1' if any test has an unexpected -result; otherwise (if all tests pass or fail as expected) it returns -`0' as the exit code. - - `runtest' flags the outcome of each test as one of these cases. -(*Note A POSIX conforming test framework: Posix, for a discussion of -how POSIX specifies the meanings of these cases.) - -`PASS' - The most desirable outcome: the test succeeded, and was expected to - succeed. - -`XPASS' - A pleasant kind of failure: a test was expected to fail, but - succeeded. This may indicate progress; inspect the test case to - determine whether you should amend it to stop expecting failure. - -`FAIL' - A test failed, although it was expected to succeed. This may - indicate regress; inspect the test case and the failing software - to locate the bug. - -`XFAIL' - A test failed, but it was expected to fail. This result indicates - no change in a known bug. If a test fails because the operating - system where the test runs lacks some facility required by the - test, the outcome is `UNSUPPORTED' instead. - -`UNRESOLVED' - Output from a test requires manual inspection; the test suite - could not automatically determine the outcome. For example, your - tests can report this outcome is when a test does not complete as - expected. - -`UNTESTED' - A test case is not yet complete, and in particular cannot yet - produce a `PASS' or `FAIL'. You can also use this outcome in dummy - "tests" that note explicitly the absence of a real test case for a - particular property. - -`UNSUPPORTED' - A test depends on a conditionally available feature that does not - exist (in the configured testing environment). For example, you - can use this outcome to report on a test case that does not work - on a particular target because its operating system support does - not include a required subroutine. - - `runtest' may also display the following messages: - -`ERROR' - Indicates a major problem (detected by the test case itself) in - running the test. This is usually an unrecoverable error, such as - a missing file or loss of communication to the target. (POSIX - test suites should not emit this message; use `UNSUPPORTED', - `UNTESTED', or `UNRESOLVED' instead, as appropriate.) - -`WARNING' - Indicates a possible problem in running the test. Usually warnings - correspond to recoverable errors, or display an important message - about the following tests. - -`NOTE' - An informational message about the test case. - - This is the full set of command line options that `runtest' -recognizes. Arguments may be abbreviated to the shortest unique string. - - runtest --tool TOOL [ TESTSUITE.exp ... ] - [ TESTSUITE.exp="testfile1 ..." ] - [ TCLVAR=VALUE... ] - [ --all ] [ --baud BAUD-RATE ] [ --connect TYPE ] - [ --debug ] [ --help ] [ --host STRING ] - [ --mail "NAME ..." ] [ --name STRING ] - [ --name NAME ] [ --outdir PATH ] - [ --objdir PATH ] [ --reboot ] - [ --srcdir PATH ] [ --strace N ] - [ --target STRING --build STRING ] - [ -v | --verbose ] [ -V | --version ] [ --DN ] - -`--tool TOOL' - TOOL specifies what set of tests to run, and what initialization - module to use. TOOL is used *only* for these two purposes: it is - *not* used to name the executable program to test. Executable - tool names (and paths) are recorded in `site.exp' (*note - Configuration dependent values: Config Values.), and you can - override them by specifying Tcl variables on the command line. - - For example, including `--tool gcc' on the `runtest' command line - runs tests from all test subdirectories whose names match `gcc.*', - and uses one of the initialization modules named - `config/*-gcc.exp'. To specify the name of the compiler (perhaps - as an alternative path to what `runtest' would use by default), use - `GCC=BINNAME' on the `runtest' command line. - -`TESTSUITE.exp ...' - Specify the names of testsuites to run. By default, `runtest' - runs all tests for the tool, but you can restrict it to particular - testsuites by giving the names of the `.exp' `expect' scripts that - control them. - - TESTSUITE.exp may not include path information; use plain - filenames. - -`TESTFILE.exp="testfile1 ..."' - Specify a subset of tests in a suite to run. For compiler or - assembler tests, which often use a single `.exp' script covering - many different source files, this option allows you to further - restrict the tests by listing particular source files to compile. - Some tools even support wildcards here. The wildcards supported - depend upon the tool, but typically they are `?', `*', and - `[chars]'. - -`TCLVAR=VALUE' - You can define Tcl variables for use by your test scripts in the - same style used with `make' for environment variables. For - example, `runtest GDB=gdb.old' defines a variable called `GDB'; - when your scripts refer to `$GDB' in this run, they use the value - `gdb.old'. - - The default Tcl variables used for most tools are defined in the - main DejaGnu `Makefile'; their values are captured in the - `site.exp' file. *Note Configuration dependent values: Config - Values. - -`--all' - Display all test output. By default, `runtest' shows only the - output of tests that produce unexpected results; that is, tests - with status `FAIL' (unexpected failure), `XPASS' (unexpected - success), or `ERROR' (a severe error in the test case itself). - Specify `--all' to see output for tests with status `PASS' - (success, as expected) `XFAIL' (failure, as expected), or - `WARNING' (minor error in the test case itself). - -`--baud BAUD-RATE' -`-b BAUD-RATE' - Set the default baud rate to something other than 9600. (Some - serial interface programs, like `tip', use a separate - initialization file instead of this value.) - -`--connect TYPE' - Connect to a target testing environment as specified by TYPE, if - the target is not the computer running `runtest'. For example, use - `--connect' to change the program used to connect to a "bare - board" boot monitor. The choices for TYPE in the DejaGnu 1.0 - distribution are `rlogin', `telnet', `rsh', `tip', `kermit', and - `mondfe'. - - The default for this option depends on the configuration (*note - Remote targets supported: Cross Targets.). The default is chosen - to be the most convenient communication method available, but - often other alternatives work as well; you may find it useful to - try alternative connect methods if you suspect a communication - problem with your testing target. - -`--debug' - Turns on the `expect' internal debugging output. Debugging output - is displayed as part of the `runtest' output, and logged to a file - called `dbg.log'. The extra debugging output does *not* appear on - standard output, unless the verbose level is greater than 2 (for - instance, to see debug output immediately, specify `--debug -v - -v'). The debugging output shows all attempts at matching the test - output of the tool with the scripted patterns describing expected - output. The output generated with `--strace' also goes into - `dbg.log'. - -`--help' -`-he' - Prints out a short summary of the `runtest' options, then exits - (even if you also specify other options). - -`--host STRING' - STRING is a full configuration "triple" name as used by - `configure'. Use this option to override the default string - recorded by your configuration's choice of host. This choice does - not change how anything is actually configured unless -build is - also specified; it affects *only* DejaGnu procedures that compare - the host string with particular values. The procedures `ishost', - `istarget', `isnative', and `setup_xfail' are affected by - `--host'. In this usage, `host' refers to the machine that the - tests are to be run on, which may not be the same as the `build' - machine. If `--build' is also specified, then `--host' refers to - the machine that the tests wil, be run on, not the machine DejaGnu - is run on. - -`--build STRING' - STRING is a full configuration "triple" name as used by - `configure'. This is the type of machine DejaGnu and the tools to - be tested are built on. For a normal cross this is the same as the - host, but for a canadian cross, they are seperate. - -`--name NAME' - NAME is a name for the particular testing target machine (for - cross testing). If the testing target has IP network support (for - example, `RPC' or `NFS'), this is the network name for the target - itself. (NAME is *not the configuration string* you specify as a - target with `configure'; the `--name' option names a particular - target, rather than describing a class of targets.) For targets - that connect in other ways, the meaning of the NAME string depends - on the connection method. *Note Remote targets supported: Cross - Targets. - -`--name STRING' - Specify a network name of testing target or its host. The - particular names that are meaningful with `--name' will depend on - your site configuration, and on the connection protocol: for - example, `tip' connections require names from a serial line - configuration file (usually called `/etc/remote'), while `telnet' - connections use IP hostnames. - -`--objdir PATH' - Use PATH as the top directory containing any auxiliary compiled - test code. This defaults to `.'. Use this option to locate - pre-compiled test code. You can normally prepare any auxiliary - files needed with `make'. - -`--outdir PATH' - Write output logs in directory PATH. The default is `.', the - directory where you start `runtest'. This option affects only the - summary and the detailed log files `TOOL.sum' and `TOOL.log'. The - DejaGnu debug log `dbg.log' always appears (when requested) in the - local directory. - -`--reboot' - Reboot the target board when `runtest' initializes. Usually, when - running tests on a separate target board, it is safer to reboot - the target to be certain of its state. However, when developing - test scripts, rebooting takes a lot of time. - -`--srcdir PATH' - Use PATH as the top directory for test scripts to run. `runtest' - looks in this directory for any subdirectory whose name begins - with the toolname (specified with `--tool'). For instance, with - `--tool gdb', `runtest' uses tests in subdirectories `gdb.*' (with - the usual shell-like filename expansion). If you do not use - `--srcdir', `runtest' looks for test directories under the current - working directory. - -`--strace N' - Turn on internal tracing for `expect', to N levels deep. By - adjusting the level, you can control the extent to which your - output expands multi-level Tcl statements. This allows you to - ignore some levels of `case' or `if' statements. Each procedure - call or control structure counts as one "level". - - The output is recorded in the same file, `dbg.log', used for output - from `--debug'. - -`--target STRING' - Use this option to override the default setting (running native - tests). STRING is a full configuration "triple" name(1) as used - by `configure'. This option changes the configuration `runtest' - uses for the default tool names, and other setup information. - *Note Using `configure': (configure.info)Using configure, for - details about `configure' names. - -`--verbose' -`-v' - Turns on more output. Repeating this option increases the amount - of output displayed. Level one (`-v') is simply test output. Level - two (`-v -v') shows messages on options, configuration, and process - control. Verbose messages appear in the detailed (`*.log') log - file, but not in the summary (`*.sum') log file. - -`--version' -`-V' - Prints out the version numbers of DejaGnu, `expect' and Tcl, and - exits without running any tests. - -`-D0' -`-D1' - Start the internal Tcl debugger. The Tcl debugger supports - breakpoints, single stepping, and other common debugging - activities. (See `A Debugger for Tcl Applications' by Don Libes. - (2)) - - If you specify `-D1', the `expect' shell stops at a breakpoint as - soon as DejaGnu invokes it. - - If you specify `-D0', DejaGnu starts as usual, but you can enter - the debugger by sending an interrupt (e.g. by typing <C-c>). - - ---------- Footnotes ---------- - - (1) Configuration triples have the form `CPU-VENDOR-OS'. - - (2) Distributed in PostScript form with `expect' as the file -`expect/tcl-debug.ps'. - - -File: dejagnu.info, Node: Customizing, Next: Internals, Prev: Invoking runtest, Up: Top - -Setting `runtest' defaults -************************** - - The site configuration file, `site.exp', captures -configuration-dependent values and propagates them to the DejaGnu test -environment using Tcl variables. This ties the DejaGnu test scripts -into the `configure' and `make' programs. - - DejaGnu supports more than one `site.exp' file. The multiple -instances of `site.exp' are loaded in a fixed order built into DejaGnu -(the more local last). The first file loaded is the optional -`~/.dejagnurc', then the local files, and finally the global file. - - 1. There is am optional "master" `site.exp', capturing configuration - values that apply to DejaGnu across the board, in each - configuration-specific subdirectory of the DejaGnu library - directory. `runtest' loads these values first. *Note Configuring - and Installing DejaGnu: Installation. The master `site.exp' - contains the default values for all targets and hosts supported by - DejaGnu. This master file is identified by setting the environment - variable `DEJAGNU' to the name of the file. This is also refered - to as the "global" config file. - - 2. Any directory containing a configured test suite also has a - `site.exp', capturing configuration values specific to the tool - under test. Since `runtest' loads these values last, the - individual test configuration can either rely on and use, or - override, any of the global values from the "master" `site.exp'. - - You can usually generate or update the testsuite `site.exp' by - typing `make site.exp' in the test suite directory, after the test - suite is configured. - - 3. You can also have a file in your home directory called - `.dejagnurc'. This gets loaded first before the other config - files. Usually this is used for personal stuff, like setting - `all_flag' so all the output gets printed, or verbosity levels. - - You can further override the default values in a user-editable -section of any `site.exp', or by setting variables on the `runtest' -command line. - -* Menu: - -* Config Values:: Variables used in the configuration file. -* Master Config File:: The master configuration file. -* Local Config File:: The local configuration file. -* Personal Config File:: The personal configuration file. - - -File: dejagnu.info, Node: Config Values, Next: Master Config File, Up: Customizing - -Config Variables ----------------- - - DejaGnu uses a named array in Tcl to hold all the info for each -machine. In the case of a canadian cross, this means host information as -well as target information. The named array is called `target_info', -and it has two indices. The following fields are part of the array. - -`name' - The name of the target. (mostly for error messages) This should - also be the string used for this target's array. It should also - be the same as the linker script so we can find them dynamically. - This should be the same as the argument used for `push_target{}'. - -`ldflags' - This is the linker flags required to produce a fully linked - executable. For `libgloss' supported targets this is usually just - the name of the linker script. - -`config' - The target canonical for this target. This is used by some init - files to make sure the target is supported. - -`cflags' - The flags required to produce an object file from a source file. - -`connect' - This is the connectmode for this target. This is for both IP and - serial connections. Typically this is either `telnet', `rlogin', - or `rsh'. - -`target' - This is the hostname of the target. This is for TCP/IP based - connections, and is also used for version of tip that use - /etc/remote. - -`serial' - This is the serial port. This is typically /dev/tty? or com?:. - -`netport' - This is the IP port. This is commonly used for telneting to target - boards that are connected to a terminal server. In that case the - IP port specifies the which serial port to use. - -`baud' - This is the baud rate for a serial port connection. - -`x10' - This is the parameters for an x10 controller. These are simple - devices that let us power cycle or reset a target board remotely. - -`fileid' - This is the fileid or spawn id of of the connection. - -`prompt' - a glob style pattern to recognize the prompt. - -`abbrev' - abbreviation for tool init files. - -`ioport' - This is the port for I/O on dual port systems. In this - configuration, the main serial port `0' is usually used for stdin - and stdout, which the second serial port can be used for debugging. - - The first index into the array is the same value as used in the -`name' field. This is usually a short version of the name of the target -board. For an example, here's the settings I use for my `Motorola's' -`IDP' board and my `Motorola' 6U VME `MVME135-1' board. (both m68k -targets) - - # IDP board - set target_info(idp,name) "idp" - set target_info(idp,ldflags) "-Tidp.ld" - set target_info(idp,config) m68k-unknown-aout - set target_info(idp,cflags) "" - set target_info(idp,connect) telnet - set target_info(idp,target) "s7" - set target_info(idp,serial) "tstty7" - set target_info(idp,netport) "wharfrat:1007" - set target_info(idp,baud) "9600" - # MVME 135 board - set target_info(idp,name) "mvme" - set target_info(idp,ldflags) "-Tmvme.ld" - set target_info(idp,config) m68k-unknown-aout - set target_info(idp,cflags) "" - set target_info(idp,connect) telnet - set target_info(idp,target) "s8" - set target_info(idp,serial) "tstty8" - set target_info(idp,netport) "wharfrat:1008" - set target_info(idp,baud) "9600" - - DejaGnu can use this information to switch between multiple targets -in one test run. This is done through the use of the `push_target' -procedure, which is discussed elsewhere. - - This array can also hold information for a remote host, which is used -when testing a candain cross. In this case, the only thing different is -the index is just `host'. Here's the settings I use to run tests on my -NT machine while running DejaGnu on a Unix machine. (in this case a -Linux box) - - set target_info(host,name) "nt-host" - set target_info(host,config) "386-unknown-winnt" - set target_info(host,connect) "telnet" - set target_info(host,target) "ripple" - - There is more info on how to use these variables in the sections on -the config files. *Note Configuration Files: Master Config File. - - In the user editable second section of `site.exp', you can not only -override the configuration variables captured in the first section, but -also specify default values for all the `runtest' command line options. -Save for `--debug', `--help', and `--version', each command line -option has an associated Tcl variable. Use the Tcl `set' command to -specify a new default value (as for the configuration variables). The -following table describes the correspondence between command line -options and variables you can set in `site.exp'. *Note Running the -Tests: Invoking runtest, for explanations of the command-line options. - - runtest Tcl - option variable description - __________ ________ ___________________________________________ - - --all all_flag display all test results if set - - --baud baud set the default baud rate to something other - than 9600. - --connect connectmode `rlogin', `telnet', `rsh', - `kermit', `tip', or `mondfe' - - --outdir outdir directory for `TOOL.sum' and `TOOL.log' - - --objdir objdir directory for pre-compiled binaries - - --reboot reboot reboot the target if set to `"1"'; - do not reboot if set to `"0"' (the default) - - --srcdir srcdir directory of test subdirectories - - --strace tracelevel a number: Tcl trace depth - - --tool tool name of tool to test; identifies init, test subdir - - --verbose verbose verbosity level. As option, use multiple times; - as variable, set a number, 0 or greater - --target target_triplet The canonical configuration string for the target. - --host host_triplet The canonical configuration string for the host. - --build build_triplet The canonical configuration string for the - build host. - - -File: dejagnu.info, Node: Master Config File, Next: Local Config File, Prev: Config Values, Up: Customizing - -Master Config File ------------------- - - The master config file is where all the target specific config -variables get set for a whole site get set. The idea is that for a -centralized testing lab where people have to share a target between -multiple developers. There are settings for both remote targets and -remote hosts. Here's an example of a Master Config File (also called -the Global config file) for a *canadian cross*. A canadian cross is -when you build and test a cross compiler on a machine other than the -one it's to be hosted on. - - Here we have the config settings for our California office. Note that -all config values are site dependant. Here we have two sets of values -that we use for testing m68k-aout cross compilers. As both of these -target boards has a different debugging protocol, we test on both of -them in sequence. - - global CFLAGS - global CXXFLAGS - - case "$target_triplet" in { - { "native" } { - set target_abbrev unix - } - { "m68*-unknown-aout" } { - set target_abbrev "rom68k" - # IDP target # IDP board with rom68k monitor - set target_info(idp,name) "idp" - set target_info(idp,ldflags) "-Tidp.ld" - set target_info(idp,config) m68k-unknown-aout - set target_info(idp,cflags) "" - set target_info(idp,connect) telnet - set target_info(idp,target) "s7" - set target_info(idp,serial) "tstty12" - set target_info(idp,netport) "truckin:1007" - set target_info(idp,baud) "9600" - # MVME target # Motorola MVME 135 with BUG monitor - set target_info(mvme,name) "mvme" - set target_info(mvme,ldflags) "-Tmvme.ld" - set target_info(mvme,config) m68k-unknown-aout - set target_info(mvme,cflags) "" - set target_info(mvme,connect) telnet - set target_info(mvme,target) "s4" - set target_info(mvme,serial) "tstty8" - set target_info(mvme,netport) "truckin:1004" - set target_info(mvme,baud) "9600" - } - } - - In this case, we have support for several remote hosts for our -m68k-aout cross compiler. Typically the remote Unix hosts run DejaGnu -locally, but we also use them for debugging the testsuites when we find -problems in running on remote hosts. Expect won't run on NT, so DejaGnu -is run on the local build machine, and it'll connect to the NT host and -run all the tests for this cross compiler on that host. - - case "$host_triplet" in { - "native" { - } - "i?86-*-linux*" { # Linux host - set target_info(host,name) "linux-host" - set target_info(host,config) $host_triplet - set target_info(host,connect) rlogin - set target_info(host,target) chinadoll - } - "i?86-*-winnt # NT host - set target_info(host,name) "nt-host" - set target_info(host,config) i386-unknown-winnt - set target_info(host,connect) telnet - set target_info(host,target) ripple - } - "hppa*-hp-hpux*" { # HP-UX host - set target_info(host,name) "hpux-host" - set target_info(host,config) $host_triplet - set target_info(host,connect) rlogin - set target_info(host,target) slipknot - } - "sparc-sun-sunos*" { # SunOS (sun4) - set target_info(host,name) "sunos-host" - set target_info(host,config) $host_triplet - set target_info(host,connect) rlogin - set target_info(host,target) darkstar - } - } - - -File: dejagnu.info, Node: Local Config File, Next: Personal Config File, Prev: Master Config File, Up: Customizing - -Local Config File ------------------ - - It is usually more convenient to keep these "manual overrides" in the -`site.exp' local to each test directory, rather than in the "master" -`site.exp' in the DejaGnu library. - - All local `site.exp' usually files have two sections, separated by -comment text. The first section is the part that is generated by -`make'. It is essentially a collection of Tcl variable definitions -based on `Makefile' environment variables. Since they are generated by -`make', they contain the values as specified by `configure'. (You can -also customize these values by using the `--site' option to -`configure'.) In particular, this section contains the `Makefile' -variables for host and target configuration data. Do not edit this -first section; if you do, your changes are replaced next time you run -`make'. - - The first section starts with: - - ## these variables are automatically generated by make ## - # Do not edit here. If you wish to override these values - # add them to the last section - - In the second section, you can override any default values (locally -to DejaGnu) for all the variables. The second section can also contain -your preferred defaults for all the command line options to `runtest'. -This allows you to easily customize `runtest' for your preferences in -each configured test-suite tree, so that you need not type options -repeatedly on the command line. (The second section may also be empty, -if you do not wish to override any defaults.) - - The first section ends with this line: - - ## All variables above are generated by configure. Do Not Edit ## - - You can make any changes under this line. If you wish to redefine a -variable in the top section, then just put a duplicate value in this -second section. Usually the values defined in this config file are -related to the configuration of the test run. This is the ideal place to -set the variables `host_triplet', `build_triplet', `target_triplet'. -All other variables are tool dependant. ie for testing a compiler, the -value for CC might be set to a freshly built binary, as opposed to one -in the user's path. - - -File: dejagnu.info, Node: Personal Config File, Prev: Local Config File, Up: Customizing - -Personal Config File --------------------- - - The personal config file is used to customize `runtest's' behaviour -for each person. It's typically used to set the user prefered setting -for verbosity, and any experimental Tcl procedures. My personal -`~/.dejagnurc' file looks like: - - set all_flag 1 - set RLOGIN /usr/ucb/rlogin - set RSH /usr/ucb/rsh - - Here I set `all_flag' so I see all the test cases that PASS along -with the ones that FAIL. I also set RLOGIN and `RSH' to the BSD -version. I have `kerberos' installed, and when I rlogin to a target -board, it usually isn't supported. So I use the non secure versions of -these programs rather than the default that's in my path. - - -File: dejagnu.info, Node: Internals, Next: Tests, Prev: Customizing, Up: Top - -The DejaGnu Implementation -************************** - - DejaGnu is entirely written in `expect', which uses Tcl as a command -language. `expect' serves as a very programmable shell; you can run -any program, as with the usual Unix command shells--but once the -program is started, your `expect' script has fully programmable control -of its input and output. This does not just apply to the programs -under test; `expect' can also run any auxiliary program, such as `diff' -or `sh', with full control over its input and output. - - DejaGnu itself is merely a framework for the set of test suites -distributed separately for each GNU tool. Future releases of GNU tools -will include even more tests, developed throughout the free software -community. - - `runtest' is the glue to tie together and manage the test scripts. -The `runtest' program is actually a simple Bourne shell script that -locates a copy of the `expect' shell and then starts the main Tcl code, -`runtest.exp'. `runtest.exp' itself has these essential functions: - - 1. Parse the command line options, load the library files, and load - the default configuration files. - - 2. Locating the individual test scripts. `runtest.exp' locates the - tests by exploiting a straightforward naming convention based on - the string you specify with the `--tool' option. - - 3. Providing an extended test environment, by defining additional Tcl - procedures beyond those already in `expect'. - - 4. Locating target-dependent functions, to standardize the test - environment across a wide variety of test platforms. - -* Menu: - -* Names:: Conventions for using tool names -* Init Module:: Initialization module -* DejaGnu Builtins:: DejaGnu provides these Tcl procedures -* Target Dependent:: Procedures supplied by the init module -* Cross Targets:: Remote targets supported -* Input Files:: The files DejaGnu depends on -* Output Files:: The files DejaGnu produces - |