diff options
Diffstat (limited to 'contrib/bluegnu2.0.3/doc/dejagnu.texi')
| -rw-r--r-- | contrib/bluegnu2.0.3/doc/dejagnu.texi | 3503 |
1 files changed, 0 insertions, 3503 deletions
diff --git a/contrib/bluegnu2.0.3/doc/dejagnu.texi b/contrib/bluegnu2.0.3/doc/dejagnu.texi deleted file mode 100644 index 0d787fa..0000000 --- a/contrib/bluegnu2.0.3/doc/dejagnu.texi +++ /dev/null @@ -1,3503 +0,0 @@ -o\input texinfo @c -*- Texinfo -*- -@finalout -@setfilename dejagnu.info -@c -@c This file documents the GNU Testing Framework ``DejaGnu'' -@c -@c Copyright (C) 92, 93, 94, 95, 1996 Free Software Foundation, Inc. -@c -@c This text may be freely distributed under the terms of the GNU -@c General Public License. -@c - -@c FIXME---MAIN TODO LIST! -@c -@c * Revisit organization. -@c -@c * discuss Tcl/expect basics---enough to get started (see seminar notes). -@c Maybe this would permit abbreviating appendices. - -@ifinfo -@format -START-INFO-DIR-ENTRY -* DejaGnu: (dejagnu). The GNU testing framework. -END-INFO-DIR-ENTRY -@end format -@end ifinfo - -@syncodeindex ky cp -@syncodeindex fn cp - -@setchapternewpage odd -@settitle DejaGnu Testing Framework -@titlepage -@title The DejaGnu Testing Framework -@subtitle for DejaGnu Version 1.3 -@sp 1 -@subtitle Jan 1996 -@author Rob Savoye -@page - -@tex -{\parskip=0pt \hfill Cygnus Support} -@end tex - -@vskip 0pt plus 1filll -Copyright @copyright{} 92, 93, 94, 95, 1996 Free Software Foundation, Inc. - -Permission is granted to make and distribute verbatim copies of -this manual provided the copyright notice and this permission notice -are preserved on all copies. - -@noindent -Permission is granted to copy and distribute modified versions of this -manual under the conditions for verbatim copying, provided also that -the entire resulting derived work is distributed under the terms of a -permission notice identical to this one. - -Permission is granted to copy and distribute translations of this manual -into another language, under the above conditions for modified versions. -@end titlepage - -@ifinfo -Copyright @copyright{} 92, 93, 94, 95, 1996 Free Software Foundation, Inc. - -Permission is granted to make and distribute verbatim copies of -this manual provided the copyright notice and this permission notice -are preserved on all copies. - -@ignore -Permission is granted to process this file through TeX and print the -results, provided the printed document carries a copying permission -notice identical to this one except for the removal of this paragraph -(this paragraph not being relevant to the printed manual). -@end ignore - -Permission is granted to copy and distribute modified versions of this -manual under the conditions for verbatim copying, provided also that -the entire resulting derived work is distributed under the terms of a -permission notice identical to this one. - -Permission is granted to copy and distribute translations of this manual -into another language, under the above conditions for modified versions. - -@node Top -@top DejaGnu - -DejaGnu is a framework for running test suites on software tools. - -This file describes version 1.3 of DejaGnu. - -@menu -* Overview:: What is DejaGnu? -* What is New:: What is new in this release. -* Invoking runtest:: Using `runtest', the main test driver -* Customizing:: Setting `runtest' defaults -* Internals:: The DejaGnu implementation -* Tests:: How to write a test case -* Extending:: New tools, new targets, and new hosts -* Installation:: Configuring and Installing DejaGnu -* Index:: Index -@end menu -@end ifinfo - -@iftex -@raggedbottom -@end iftex - -@node Overview -@chapter What is DejaGnu? -@cindex overview - -DejaGnu is a framework for testing other programs. Its purpose is to -provide a single front end for all tests. Beyond this, DejaGnu offers -several advantages for testing: - -@enumerate -@item -The flexibility and consistency of the DejaGnu framework make it easy -to write tests for any program. - -@item -DejaGnu provides a layer of abstraction which allows you to write tests -that are portable to any host or target where a program must be tested. -For instance, a test for GDB can run (from any Unix based host) on any -target architecture that DejaGnu supports. Currently DejaGnu runs tests -on several single board computers, whose operating software ranges from -just a boot monitor to a full-fledged, Unix-like realtime OS. - -@item -All tests have the same output format. This makes it easy to integrate -testing into other software development processes. DejaGnu's output is -designed to be parsed by other filtering script, and it is also human -readable. -@end enumerate - -DejaGnu is written in @code{expect}, which in turn uses @dfn{Tcl}---Tool -command language. - -@cindex @code{expect} script names -@kindex .exp -@cindex suffix, @code{expect} scripts -Running tests requires two things: the testing framework, and the test -suites themselves. Tests are usually written in @code{expect} using -Tcl, but you can also use a Tcl script to run a test suite that is not -based on @code{expect}. (@code{expect} script filenames conventionally -use @samp{.exp} as a suffix; for example, the main implementation of the -DejaGnu test driver is in the file @file{runtest.exp}.) - - -@menu -* Running Tests:: A first look at running DejaGnu tests -* Sample Test:: What does a DejaGnu test case look like? -* Design Goals:: Goals behind DejaGnu -* Posix:: DejaGnu conforms to POSIX 1003.3 -* Future Directions:: Where is DejaGnu going? -* Tcl and Expect:: Reading more about Tcl and Expect -@end menu - -@node What is New -@chapter What is new in this release ? -@cindex What is New - -This release has a number of substantial changes over version 1.2. The -most visible change is that the version of expect and Tcl included in -the release are up-to-date with the current stable net releases. Other -changes are: - -@enumerate -@item -@c FIXME: add a link to the config section -The config sub-system in DejaGnu has been completely redesigned. It now -supports testing on remote hosts as well as remote targets. - -@item -More builtin support for building target binaries with the correct -linker flags. Currently this only works with GCC, preferably with a -target support by @code{libgloss}. - -@item -Lots of little bug fixes from a year of heavy use here at Cygnus -Support. - -@item -DejaGnu now uses @code{autoconf} for configuration. - -@item -New test cases for DejaGnu have been added for the new features, plus -the "--tool" option bug in the 1.2 testsuite has been fixed. - -@item -The @code{--tool} option is now optional. - -@item -@code{runtest} when searching for test drivers ignores all directories -named SCCS, RCS, and CVS. - -@item -There is now a generic keyword based test harness that uses comments in -source code to control how each test case gets built and run. - -@item -There is now some support for running a testsuite with multiple passes. - -@end enumerate - -@node Running Tests -@section Running existing tests -@cindex existing tests, running -@cindex running tests -@cindex tests, running - -@kindex make check -To run tests from an existing collection, first use @code{configure} as -usual to set up the source directory containing the tests. Then try -running - -@example -make check -@end example - -@cindex @code{check} makefile target -If the @code{check} target exists, it usually saves you some -trouble---for instance, it can set up any auxiliary programs or other -files needed by the tests. - -@cindex auxiliary files, building -Once you have run @samp{make check} to build any auxiliary files, you -might want to call the test driver @code{runtest} directly to repeat the -tests. You may also have to call @code{runtest} directly for test -collections with no @code{check} target in the @file{Makefile}. - -@c force page break to avoid losing footnote to another page -@page -@cindex @code{runtest}, most common options -@cindex options for @code{runtest}, common -Typically, you must use two command-line options: @samp{--tool}, to -specify which set of tests to run@footnote{@samp{--tool} selects a -particular suite of tests, @emph{not} the name of the executable program -to run. @xref{Config Values,,Configuration dependent values}, for -information on the variables that you can use to specify the names of -programs to run.}, and @samp{--srcdir}, to specify where to find test -directories. - -For example, if the directory @file{gdb/testsuite} contains a collection -of DejaGnu tests for @sc{gdb}, you can run them like this: - -@example -eg$ cd gdb/testsuite -eg$ runtest --tool gdb -@exdent @emph{Test output follows, ending with:} - - === gdb Summary === - -# of expected passes 508 -# of expected failures 103 -/usr/latest/bin/gdb version 4.14.4 -nx -@end example - -You can use the option @samp{--srcdir} to point to some other directory -containing a collection of tests: - -@smallexample -eg$ runtest --tool gdb --srcdir /devo/gdb/testsuite -@end smallexample - -@cindex native configuration -@cindex cross configuration -These examples assume a @dfn{native} configuration, where the same -computer runs both @code{runtest} and the tests themselves. When you -have a @dfn{cross} configuration, the tests run on a different computer, -controlled by the host running @code{runtest}. In this situation, you -need the option @samp{--name} to specify the network address for the -other computer: - -@smallexample -eg$ runtest --tool gdb --name vx9.munist.com -@end smallexample - -If you always use the same option values, you can record them in a file -called @file{site.exp}, rather than typing them each time. @xref{Config -Values,,Setting defaults for @code{runtest} options}. - -By default, @code{runtest} prints only the names of the tests it runs, -output from any tests that have unexpected results, and a summary -showing how many tests passed and how many failed. To display output -from all tests (whether or not they behave as expected), use the -@samp{--all} option. For more verbose output about processes being run, -communication, and so on, use @samp{--verbose}. To see even more output, -use multiple @samp{--verbose} options. @xref{Invoking runtest,,Using -@code{runtest}}, for a more detailed explanation of each @code{runtest} -option. - -Test output goes into two files in your current directory: summary -output in @file{@var{tool}.sum}, and detailed output in -@file{@var{tool}.log}. (@var{tool} refers to the collection of tests; -for example, after a run with @samp{--tool gdb}, look for output files -@file{gdb.sum} and @file{gdb.log}.) @xref{Output Files,,The files -DejaGnu writes}. - -@node Sample Test -@section What does a DejaGnu test look like? - -@cindex example -Each DejaGnu test is an @code{expect} script; the tests vary widely in -complexity, depending on the nature of the tool and the feature tested. - -@kindex gdb.t00/echo.exp -@kindex echo.exp -Here is a very simple @sc{gdb} test---one of the simplest tests shipped -with DejaGnu (extracted from @file{gdb.t00/echo.exp}):@footnote{More -recent @sc{gdb} tests use the @samp{gdb_test} procedure. -An equivalent test using that procedure is @samp{ gdb_test "echo Hello -world!" "Hello world!" }} -@c FIXME! include xref in footnote, when gdb_test documented in some manual. -@c @xref{}. -@c Extra spaces in @samp above avoid running end ' against " inside. - -@cartouche -@smallexample -# send a string to the GDB stdin: -send "echo Hello world!\n" - -# inspect the GDB stdout for the correct reply, -# and determine whether the test passes or fails: -expect @{ - -re "Hello world.*$prompt $" @{ pass "Echo test" @} - -re "$prompt $" @{ fail "Echo test" @} - timeout @{ fail "(timeout) Echo test" @} - @} -@end smallexample -@end cartouche - -Though brief, this example is a complete test. It illustrates some of -the main features of DejaGnu test scripts: - -@itemize @bullet -@item -The test case does not start the tested program (@sc{gdb} in this case); -all test scripts for interactive tools can assume the corresponding tool -is running. - -@item -Comments start with @samp{#}. - -@item -The main commands you use to control a tested program are @code{send} -(to give it commands) and @code{expect} (to analyze its responses). - -@item -The @code{expect} command uses a list of pairs; a pattern (regular -expression if @samp{-re} specified), followed by an action to run if the -pattern matches output from the program. Only the action for the -@emph{first} matching pattern will execute. - -@item -Test cases use the commands @code{pass} and @code{fail} to record the -test outcome. -@end itemize - -@node Design Goals -@section Design goals -@cindex design goals - -DejaGnu grew out of the internal needs of Cygnus Support. Cygnus -maintains and enhances a variety of free programs in many different -environments, and we needed a testing tool that: - -@itemize @bullet -@item -is useful to developers while fixing bugs; - -@item -automates running many tests during a software release process; - -@item -is portable among a variety of host computers; - -@item -supports cross-development testing; - -@item -permits testing interactive programs, like @sc{gdb}; and - -@item -permits testing batch oriented programs, like @sc{gcc}. -@end itemize - -Some of the requirements proved challenging. For example, interactive -programs do not lend themselves very well to automated testing. But all -the requirements are important: for instance, it is imperative to make -sure that @sc{gdb} works as well when cross-debugging as it does in a -native configuration. - -Probably the greatest challenge was testing in a cross-development -environment (which can be a real nightmare). Most cross-development -environments are customized by each developer. Even when buying -packaged boards from vendors there are many differences. The -communication interfaces vary from a serial line to ethernet. DejaGnu -was designed with a modular communication setup, so that each kind of -communication can be added as required, and supported thereafter. Once -a communication procedure is coded, any test can use it. Currently -DejaGnu can use @code{rsh}, @code{rlogin}, @code{telnet}, @code{tip}, -@code{kermit}, and @code{mondfe} for remote communications. - -@cindex name ``DejaGnu'' -@cindex DejaGnu, the name -@cindex Menapace, Julia -Julia Menapace first coined the term ``Deja Gnu'' to describe an earlier -testing framework at Cygnus Support. When we replaced it with the -Expect-based framework, it was like DejaGnu all over again@dots{} - -@node Posix -@section A POSIX conforming test framework - -@cindex POSIX conformance -@cindex standard conformance: POSIX 1003.3 -DejaGnu conforms to the @sc{posix} standard for test frameworks. - -@cindex TET -@sc{posix} standard 1003.3 defines what a testing framework needs to -provide, in order to permit the creation of @sc{posix} conformance -test suites. This standard is primarily oriented to running @sc{posix} -conformance tests, but its requirements also support testing of features -not related to @sc{posix} conformance. @sc{posix} 1003.3 does not -specify a particular testing framework, but at this time there is only -one other @sc{posix} conforming test framework: -@sc{tet}.@footnote{@sc{tet} was created by Unisoft for a consortium -comprised of X/Open, Unix International, and the Open Software -Foundation.} - -The @sc{posix} documentation refers to @dfn{assertions}. An assertion -is a description of behavior. For example, if a standard says ``The sun -shall shine'', a corresponding assertion might be ``The sun is -shining.'' A test based on this assertion would pass or fail depending -on whether it is daytime or nighttime. It is important to note that the -standard being tested is never 1003.3; the standard being tested is some -other standard, for which the assertions were written. - -As there is no test suite to test @emph{testing frameworks} for -@sc{posix} 1003.3 conformance, verifying conformance to this standard is -done by repeatedly reading the standard and experimenting. One of the -main things 1003.3 does specify is the set of allowed output messages, -and their definitions. Four messages are supported for a required -feature of @sc{posix} conforming systems, and a fifth for a conditional -feature. DejaGnu supports the use of all five output messages; in this -sense a test suite that uses exactly these messages can be considered -@sc{posix} conforming. These definitions specify the output of a test -case: - -@ftable @code -@cindex success, POSIX definition -@item PASS -A test has succeeded. That is, it demonstrated that the assertion is true. - -@cindex XFAIL, avoiding for POSIX -@item XFAIL -@sc{posix} 1003.3 does not incorporate the notion of expected failures, -so @code{PASS}, instead of @code{XPASS}, must also be returned for test -cases which were expected to fail and did not. This means that -@code{PASS} is in some sense more ambiguous than if @code{XPASS} is also -used. For information on @code{XPASS} and @code{XFAIL}, see -@ref{Invoking runtest,,Using @code{runtest}}. - -@item FAIL -@cindex failure, POSIX definition -A test @emph{has} produced the bug it was intended to capture. That is, -it has demonstrated that the assertion is false. The @code{FAIL} -message is based on the test case only. Other messages are used to -indicate a failure of the framework. - -As with @code{PASS}, @sc{posix} tests must return @code{FAIL} rather -than @code{XFAIL} even if a failure was expected. - -@item UNRESOLVED -@cindex ambiguity, required for POSIX -A test produced indeterminate results. Usually, this means the test -executed in an unexpected fashion; this outcome requires that a human -being go over results, to determine if the test should have passed or -failed. This message is also used for any test that requires human -intervention because it is beyond the abilities of the testing -framework. Any unresolved test should resolved to @code{PASS} or -@code{FAIL} before a test run can be considered finished. - -Note that for @sc{posix}, each assertion must produce a test result -code. If the test isn't actually run, it must produce @code{UNRESOLVED} -rather than just leaving that test out of the output. This means that -you have to be careful when writing tests, to not carelessly use tcl -statements like @code{return}---if you alter the flow of control of the -tcl code you must insure that every test still produces some result code. - -Here are some of the ways a test may wind up @code{UNRESOLVED}: - -@itemize @bullet -@item -A test's execution is interrupted. - -@item -A test does not produce a clear result. This is usually because there -was an @code{ERROR} from DejaGnu while processing the test, or because there -were three or more @code{WARNING} messages. Any @code{WARNING} or -@code{ERROR} messages can invalidate the output of the test. This -usually requires a human being to examine the output to -determine what really happened---and to improve the test case. - -@item -A test depends on a previous test, which fails. - -@item -The test was set up incorrectly. -@end itemize - -@item UNTESTED -A test was not run. This is a placeholder, used when there is no -real test case yet. -@end ftable - -@noindent -The only remaining output message left is intended to test features that -are specified by the applicable @sc{posix} standard as conditional: - -@ftable @code -@item UNSUPPORTED -There is no support for the tested case. This may mean that a -conditional feature of an operating system, or of a compiler, is not -implemented. DejaGnu also uses this message when a testing environment -(often a ``bare board'' target) lacks basic support for compiling or -running the test case. For example, a test for the system subroutine -@code{gethostname} would never work on a target board running only a -boot monitor. -@end ftable - -DejaGnu uses the same output procedures to produce these messages for -all test suites, and these procedures are already known to conform to -@sc{posix} 1003.3. For a DejaGnu test suite to conform to @sc{posix} -1003.3, you must avoid the @code{setup_xfail} procedure as described in -the @code{PASS} section above, and you must be careful to return -@code{UNRESOLVED} where appropriate, as described in the -@code{UNRESOLVED} section above. - -@node Future Directions -@section Future directions -@cindex future directions - -In the near future, there are two parallel directions for DejaGnu -development. The first is to add support for more hosts and targets. - -The second would permit testing programs with a more complex interface, -whether text based or GUI based. Two components already exist: a Tcl -based X window toolkit, and a terminal package for @code{expect}. Both -of these could be merged into DejaGnu in a way that permits testing -programs that run in each environment. - -Meanwhile, we hope DejaGnu enables the creation of test suites for -conformance to @sc{ansi} C and C++, to @sc{posix}, and to other -standards. We encourage you to make any test suites you create freely -available, under the same terms as DejaGnu itself. - -@node Tcl and Expect -@section Tcl and Expect -@cindex tool command language -@cindex tcl -@cindex Ousterhout, John K. -Tcl was introduced in a paper by John K. Ousterhout at the 1990 Winter -Usenix conference, @cite{Tcl: An Embeddable Command Language}. That -paper is included in PostScript form in the @file{doc} subdirectory of -the Tcl distribution. The version of Tcl included in DejaGnu at this time is -Tcl 7.4p3. - -@cindex @code{expect} scripting language -@cindex Libes, Don -Don Libes introduced @code{expect} in his paper @cite{expect: Curing -Those Uncontrollable Fits of Interaction} at the 1990 Summer Usenix -conference. The paper is included in PostScript form in the -@code{expect} distribution (as are several other papers about -@code{expect}). The version of expect included in DejaGnu at this time -is expect 5.18.0. - -@node Invoking runtest -@chapter Using @code{runtest} -@cindex invoking -@cindex running -@cindex command line options -@cindex options - -@cindex @code{runtest} description -@cindex DejaGnu test driver -@code{runtest} is the executable test driver for DejaGnu. You can -specify two kinds of things on the @code{runtest} command line: command -line options, and Tcl variables for the test scripts. The options are -listed alphabetically below. - -@cindex exit code from @code{runtest} -@cindex @code{runtest} exit code -@code{runtest} returns an exit code of @code{1} if any test -has an unexpected result; otherwise (if all tests pass or fail as -expected) it returns @code{0} as the exit code. - -@code{runtest} flags the outcome of each test as one of these cases. -(@xref{Posix,,A POSIX conforming test framework}, for a discussion of -how @sc{posix} specifies the meanings of these cases.) - -@table @code -@item PASS -@kindex PASS -@cindex successful test -@cindex test, successful -The most desirable outcome: the test succeeded, and was expected to -succeed. - -@item XPASS -@kindex XPASS -@cindex successful test, unexpected -@cindex unexpected success -A pleasant kind of failure: a test was expected to fail, but succeeded. -This may indicate progress; inspect the test case to determine whether -you should amend it to stop expecting failure. - -@item FAIL -@kindex FAIL -@cindex failing test, unexpected -@cindex test, failing -A test failed, although it was expected to succeed. This may indicate -regress; inspect the test case and the failing software to locate the bug. - -@item XFAIL -@kindex XFAIL -@cindex expected failure -@cindex failing test, expected -A test failed, but it was expected to fail. This result indicates no -change in a known bug. If a test fails because the operating system -where the test runs lacks some facility required by the test, the -outcome is @code{UNSUPPORTED} instead. - -@item UNRESOLVED -@kindex UNRESOLVED -@cindex test, unresolved outcome -Output from a test requires manual inspection; the test suite could not -automatically determine the outcome. For example, your tests can report -this outcome is when a test does not complete as expected. - -@item UNTESTED -@kindex UNTESTED -@cindex untested properties -A test case is not yet complete, and in particular cannot yet produce a -@code{PASS} or @code{FAIL}. You can also use this outcome in dummy -``tests'' that note explicitly the absence of a real test case -for a particular property. - -@item UNSUPPORTED -@kindex UNSUPPORTED -@cindex unsupported test -@cindex test, unsupported -A test depends on a conditionally available feature that does not exist -(in the configured testing environment). For example, you can use this -outcome to report on a test case that does not work on a particular -target because its operating system support does not include a required -subroutine. -@end table - -@code{runtest} may also display the following messages: - -@table @code -@item ERROR -@kindex ERROR -@cindex problem, detected by test case -@cindex test case cannot run -Indicates a major problem (detected by the test case itself) in running -the test. This is usually an unrecoverable error, such as a missing file -or loss of communication to the target. (@sc{posix} test suites should -not emit this message; use @code{UNSUPPORTED}, @code{UNTESTED}, or -@code{UNRESOLVED} instead, as appropriate.) - -@item WARNING -@kindex WARNING -@cindex test case warnings -Indicates a possible problem in running the test. Usually warnings -correspond to recoverable errors, or display an important message about -the following tests. - -@item NOTE -@kindex NOTE -@cindex test case messages -An informational message about the test case. -@end table - -This is the full set of command line options that @code{runtest} -recognizes. Arguments may be abbreviated to the shortest unique string. - -@cindex @code{runtest} option list -@cindex option list, @code{runtest} -@smallexample -runtest --tool @var{tool} [ @var{testsuite}.exp @dots{} ] -[ @var{testsuite}.exp="testfile1 @dots{}" ] -[ @var{tclvar}=@var{value}@dots{} ] -[ --all ] [ --baud @var{baud-rate} ] [ --connect @var{type} ] -[ --debug ] [ --help ] [ --host @var{string} ] -[ --mail "@var{name} @dots{}" ] [ --name @var{string} ] -[ --name @var{name} ] [ --outdir @var{path} ] -[ --objdir @var{path} ] [ --reboot ] -[ --srcdir @var{path} ] [ --strace @var{n} ] -[ --target @var{string} --build @var{string} ] -[ -v | --verbose ] [ -V | --version ] [ --D@var{n} ] -@end smallexample - -@table @code -@item --tool @var{tool} -@cindex selecting tests for a tool -@cindex @code{--tool} (@code{runtest} option) -@var{tool} specifies what set of tests to run, and what initialization -module to use. @var{tool} is used @emph{only} for these two purposes: -it is @emph{not} used to name the executable program to test. -Executable tool names (and paths) are recorded in @file{site.exp} -(@pxref{Config Values,,Configuration dependent values}), and you can -override them by specifying Tcl variables on the command line. - -For example, including @samp{--tool gcc} on the @code{runtest} command -line runs tests from all test subdirectories whose names match -@file{gcc.*}, and uses one of the initialization modules named -@file{config/*-gcc.exp}. To specify the name of the compiler (perhaps -as an alternative path to what @code{runtest} would use by default), use -@samp{GCC=@var{binname}} on the @code{runtest} command line. - -@item @var{testsuite}.exp @dots{} -@cindex selecting a range of tests -@cindex tests, running specifically -@cindex naming tests to run -Specify the names of testsuites to run. -By default, @code{runtest} runs all tests for the tool, but you can -restrict it to particular testsuites by giving the names of the @samp{.exp} -@code{expect} scripts that control them. - -@var{testsuite}.exp may not include path information; use plain filenames. - -@item @var{testfile}.exp="testfile1 @dots{}" -@cindex selecting a range of tests -@cindex tests, running specifically -@cindex naming tests to run -Specify a subset of tests in a suite to run. -For compiler or assembler tests, which often use a single @samp{.exp} -script covering many different source files, this option allows you to -further restrict the tests by listing particular source files to compile. -Some tools even support wildcards here. The wildcards supported depend -upon the tool, but typically they are @code{?}, @code{*}, and @code{[chars]}. - -@item @var{tclvar}=@var{value} -@kindex @var{tclvar}=@var{value} -@cindex Tcl variables, defining for @code{runtest} -@cindex command line Tcl variable definition -@cindex @code{runtest}, variable defns on cmdline -You can define Tcl variables for use by your test scripts in the same -style used with @code{make} for environment variables. For example, -@samp{runtest GDB=gdb.old} defines a variable called @samp{GDB}; when -your scripts refer to @samp{$GDB} in this run, they use the value -@samp{gdb.old}. - -The default Tcl variables used for most tools are defined in the main -DejaGnu @code{Makefile}; their values are captured in the -@file{site.exp} file. @xref{Config Values,,Configuration dependent -values}. - -@item --all -@cindex @code{--all} (@code{runtest} option) -@cindex test output, displaying all -Display all test output. By default, @code{runtest} shows only the -output of tests that produce unexpected results; that is, tests with -status @samp{FAIL} (unexpected failure), @samp{XPASS} (unexpected -success), or @samp{ERROR} (a severe error in the test case itself). -Specify @samp{--all} to see output for tests with status @samp{PASS} -(success, as expected) @samp{XFAIL} (failure, as expected), or -@samp{WARNING} (minor error in the test case itself). - -@item --baud @var{baud-rate} -@itemx -b @var{baud-rate} -@cindex baud rate, specifying -@cindex bps, specifying -@cindex @code{--baud} (@code{runtest} option) -@cindex @code{-b} (@code{runtest} option) -Set the default baud rate to something other than 9600. (Some serial -interface programs, like @code{tip}, use a separate initialization file -instead of this value.) - -@item --connect @var{type} -@cindex connecting to target -@cindex @code{--connect} (@code{runtest} option) -@cindex remote testbed, connecting to -@cindex @code{rlogin}, remote testing via -@cindex @code{telnet}, remote testing via -@cindex @code{rsh}, remote testing via -@cindex @code{tip}, remote testing via -@cindex @code{kermit}, remote testing via -@cindex @code{mondfe}, remote testing via -@cindex remote testing via @code{rlogin} -@cindex remote testing via @code{telnet} -@cindex remote testing via @code{rsh} -@cindex remote testing via @code{tip} -@cindex remote testing via @code{kermit} -@cindex remote testing via @code{mondfe} -Connect to a target testing environment as specified by @var{type}, if -the target is not the computer running @code{runtest}. For example, use -@samp{--connect} to change the program used to connect to a ``bare -board'' boot monitor. The choices for @var{type} in the DejaGnu 1.0 -distribution are @samp{rlogin}, @samp{telnet}, @samp{rsh}, @samp{tip}, -@samp{kermit}, and @samp{mondfe}. - -@noindent -The default for this option depends on the configuration (@pxref{Cross -Targets,,Remote targets supported}). The default is chosen to be the -most convenient communication method available, but often other -alternatives work as well; you may find it useful to try alternative -connect methods if you suspect a communication problem with your testing -target. - -@item --debug -@cindex @code{--debug} (@code{runtest} option) -@cindex debug log for test cases -@cindex test cases, debug log -@cindex @code{dbg.log} file -Turns on the @code{expect} internal debugging output. Debugging output -is displayed as part of the @code{runtest} output, and logged to a file -called @file{dbg.log}. The extra debugging output does @emph{not} -appear on standard output, unless the verbose level is greater than 2 -(for instance, to see debug output immediately, specify @samp{--debug -v --v}). The debugging output shows all attempts at matching the test -output of the tool with the scripted patterns describing expected -output. The output generated with @samp{--strace} also goes into -@file{dbg.log}. - -@item --help -@itemx -he -@cindex @code{--help} (@code{runtest} option) -@cindex help with @code{runtest} -@cindex @code{runtest}, listing options -Prints out a short summary of the @code{runtest} options, then exits -(even if you also specify other options). - -@item --host @var{string} -@cindex @code{--host} (@code{runtest} option) -@cindex specifying the host config name -@cindex host config name, changing -@var{string} is a full configuration ``triple'' name as used by -@code{configure}. Use this option to override the default string -recorded by your configuration's choice of host. This choice does not -change how anything is actually configured unless --build is also -specified; it affects @emph{only} DejaGnu procedures that compare the -host string with particular values. The procedures @code{ishost}, -@code{istarget}, @code{isnative}, and @code{setup_xfail} are affected by -@samp{--host}. In this usage, @code{host} refers to the machine that the -tests are to be run on, which may not be the same as the @code{build} -machine. If @code{--build} is also specified, then @code{--host} refers -to the machine that the tests wil, be run on, not the machine DejaGnu is -run on. - -@item --build @var{string} -@cindex @code{--build} (@code{runtest} option) -@cindex specifying the build config name -@cindex build config name, changing -@var{string} is a full configuration ``triple'' name as used by -@code{configure}. This is the type of machine DejaGnu and the tools to -be tested are built on. For a normal cross this is the same as the host, -but for a canadian cross, they are seperate. - -@item --name @var{name} -@cindex specifying target name -@cindex target machine name -@cindex @code{--name} (@code{runtest} option) -@var{name} is a name for the particular testing target machine (for -cross testing). If the testing target has IP network support (for -example, @code{RPC} or @code{NFS}), this is the network name for the -target itself. (@var{name} is @emph{not the configuration string} you -specify as a target with @code{configure}; the @samp{--name} option -names a particular target, rather than describing a class of targets.) -For targets that connect in other ways, the meaning of the @var{name} -string depends on the connection method. @xref{Cross Targets,,Remote -targets supported}. - -@item --name @var{string} -@cindex remote test machine name -@cindex name for remote test machine -Specify a network name of testing target or its host. The particular -names that are meaningful with @samp{--name} will depend on your site -configuration, and on the connection protocol: for example, @code{tip} -connections require names from a serial line configuration file (usually -called @file{/etc/remote}), while @code{telnet} connections use IP -hostnames. - -@item --objdir @var{path} -@cindex @code{--objdir} (@code{runtest} option) -@cindex object directory -@cindex test programs, auxiliary -@cindex auxiliary test programs -Use @var{path} as the top directory containing any auxiliary compiled -test code. This defaults to @file{.}. Use this option to locate -pre-compiled test code. You can normally prepare any auxiliary files -needed with @code{make}. - -@item --outdir @var{path} -@cindex output directory -@cindex @code{--outdir} (@code{runtest} option) -@cindex log files, where to write -Write output logs in directory @var{path}. The default is @samp{.}, the -directory where you start @code{runtest}. This option affects only the -summary and the detailed log files @file{@var{tool}.sum} and -@file{@var{tool}.log}. The DejaGnu debug log @file{dbg.log} always -appears (when requested) in the local directory. - -@item --reboot -@cindex rebooting remote targets -@cindex @code{--reboot} (@code{runtest} option) -Reboot the target board when @code{runtest} initializes. -Usually, when running tests on a separate target board, it is safer to -reboot the target to be certain of its state. However, when developing -test scripts, rebooting takes a lot of time. - -@item --srcdir @var{path} -@cindex source directory -@cindex @code{--srcdir} (@code{runtest} option) -Use @var{path} as the top directory for test scripts to run. -@code{runtest} looks in this directory for any subdirectory whose name -begins with the toolname (specified with @samp{--tool}). For instance, -with @samp{--tool gdb}, @code{runtest} uses tests in subdirectories -@file{gdb.*} (with the usual shell-like filename expansion). If you do -not use @samp{--srcdir}, @code{runtest} looks for test directories under -the current working directory. - -@item --strace @var{n} -@cindex @code{--strace} (@code{runtest} option) -@cindex tracing Tcl commands -@cindex @code{expect} internal tracing -Turn on internal tracing for @code{expect}, to @var{n} levels deep. By -adjusting the level, you can control the extent to which your output -expands multi-level Tcl statements. This allows you to ignore some -levels of @code{case} or @code{if} statements. Each procedure call or -control structure counts as one ``level''. - -The output is recorded in the same file, @file{dbg.log}, used for output -from @samp{--debug}. - -@item --target @var{string} -@cindex @code{--target} (@code{runtest} option) -@cindex specifying the target configuration -@cindex target configuration, specifying -Use this option to override the default setting (running native tests). -@var{string} is a full configuration ``triple'' -name@footnote{Configuration triples have the form -@samp{@var{cpu}-@var{vendor}-@var{os}}.} as used by @code{configure}. -This option changes the configuration @code{runtest} uses for the -default tool names, and other setup information. @xref{Using -configure,,Using @code{configure}, configure.info, Cygnus configure}, -for details about @code{configure} names. - -@item --verbose -@itemx -v -@cindex @code{--verbose} (@code{runtest} option) -@cindex @code{-v} (@code{runtest} option) -@cindex turning on output -@cindex output, additional -Turns on more output. Repeating this option increases the amount of -output displayed. Level one (@samp{-v}) is simply test output. Level -two (@samp{-v -v}) shows messages on options, configuration, and process -control. Verbose messages appear in the detailed (@file{*.log}) log -file, but not in the summary (@file{*.sum}) log file. - -@item --version -@itemx -V -@cindex @code{-V} (@code{runtest} option) -@cindex @code{--version} (@code{runtest} option) -@cindex version numbers -Prints out the version numbers of DejaGnu, @code{expect} and Tcl, and -exits without running any tests. - -@item -D0 -@itemx -D1 -@cindex starting the tcl debugger -@cindex tcl debugger -@c FIXME!!! we should say a *lot* more about this debugger -Start the internal Tcl debugger. The Tcl debugger supports breakpoints, -single stepping, and other common debugging activities. (See @cite{A -Debugger for Tcl Applications} by Don Libes. @footnote{Distributed in -PostScript form with @code{expect} as the file@* -@file{expect/tcl-debug.ps}.}) - -If you specify @samp{-D1}, the @code{expect} shell stops at a breakpoint -as soon as DejaGnu invokes it. - -If you specify @samp{-D0}, DejaGnu starts as usual, but you can enter -the debugger by sending an interrupt (e.g. by typing @key{C-c}). -@end table - -@node Customizing -@chapter Setting @code{runtest} defaults - -@kindex site.exp -@cindex variables of DejaGnu, defaults -The site configuration file, @file{site.exp}, captures -configuration-dependent values and propagates them to the DejaGnu test -environment using Tcl variables. This ties the DejaGnu test scripts -into the @code{configure} and @code{make} programs. - -@cindex @file{site.exp}, multiple -@cindex overriding @file{site.exp} -DejaGnu supports more than one @file{site.exp} file. The multiple -instances of @file{site.exp} are loaded in a fixed order built into -DejaGnu (the more local last). The first file loaded is the optional -@code{~/.dejagnurc}, then the local files, and finally the global file. - -@enumerate -@item -There is am optional ``master'' @file{site.exp}, capturing configuration values -that apply to DejaGnu across the board, in each configuration-specific -subdirectory of the DejaGnu library directory. @code{runtest} loads -these values first. @xref{Installation,,Configuring and Installing -DejaGnu}. The master @file{site.exp} contains the default values for -all targets and hosts supported by DejaGnu. This master file is -identified by setting the environment variable @code{DEJAGNU} to the -name of the file. This is also refered to as the ``global'' config file. - -@item -Any directory containing a configured test suite also has a -@file{site.exp}, capturing configuration values specific to the tool -under test. Since @code{runtest} loads these values last, the -individual test configuration can either rely on and use, or override, -any of the global values from the ``master'' @file{site.exp}. - -You can usually generate or update the testsuite @file{site.exp} by -typing @samp{make site.exp} in the test suite directory, after the test -suite is configured. - -@item -You can also have a file in your home directory called -@code{.dejagnurc}. This gets loaded first before the other config -files. Usually this is used for personal stuff, like setting -@code{all_flag} so all the output gets printed, or verbosity levels. -@end enumerate - -You can further override the default values in a user-editable section -of any @file{site.exp}, or by setting variables on the @code{runtest} -command line. - -@menu -* Config Values:: Variables used in the configuration file. -* Master Config File:: The master configuration file. -* Local Config File:: The local configuration file. -* Personal Config File:: The personal configuration file. -@end menu - -@node Config Values, Master Config File, , Customizing -@subsection Config Variables -@cindex configuration dependent defaults -@cindex setting defaults for DejaGnu variables - -@c NOTE: default values are given via @code{"fubar"} rather than the -@c more conventional @samp{fubar} to permit a consistent and clear -@c notation for the empty string (@code{""}), which will work exactly as -@c typed. - -DejaGnu uses a named array in Tcl to hold all the info for each -machine. In the case of a canadian cross, this means host information as -well as target information. The named array is called -@code{target_info}, and it has two indices. The following fields are -part of the array. - -@table @code -@item name -The name of the target. (mostly for error messages) This -should also be the string used for this target's array. -It should also be the same as the linker script so we -can find them dynamically. This should be the same as the argument used -for @code{push_target@{@}}. - -@item ldflags -This is the linker flags required to produce a fully linked -executable. For @code{libgloss} supported targets this is usually just -the name of the linker script. - -@item config -The target canonical for this target. This is used by some init files to -make sure the target is supported. - -@item cflags -The flags required to produce an object file from a source file. - -@item connect -This is the connectmode for this target. This is for both IP and -serial connections. Typically this is either @code{telnet}, -@code{rlogin}, or @code{rsh}. - -@item target -This is the hostname of the target. This is for TCP/IP based connections, -and is also used for version of tip that use /etc/remote. - -@item serial -This is the serial port. This is typically /dev/tty? or com?:. - -@item netport -This is the IP port. This is commonly used for telneting to target -boards that are connected to a terminal server. In that case the IP port -specifies the which serial port to use. - -@item baud -This is the baud rate for a serial port connection. - -@item x10 -This is the parameters for an x10 controller. These are simple devices -that let us power cycle or reset a target board remotely. - -@item fileid -This is the fileid or spawn id of of the connection. - -@item prompt -a glob style pattern to recognize the prompt. - -@item abbrev -abbreviation for tool init files. - -@item ioport -This is the port for I/O on dual port systems. In this configuration, -the main serial port @code{0} is usually used for stdin and stdout, -which the second serial port can be used for debugging. -@end table - -The first index into the array is the same value as used in the -@code{name} field. This is usually a short version of the name of the -target board. For an example, here's the settings I use for my -@code{Motorola's} @code{IDP} board and my @code{Motorola} 6U VME -@code{MVME135-1} board. (both m68k targets) - -@cartouche -@smallexample -# IDP board -set target_info(idp,name) "idp" -set target_info(idp,ldflags) "-Tidp.ld" -set target_info(idp,config) m68k-unknown-aout -set target_info(idp,cflags) "" -set target_info(idp,connect) telnet -set target_info(idp,target) "s7" -set target_info(idp,serial) "tstty7" -set target_info(idp,netport) "wharfrat:1007" -set target_info(idp,baud) "9600" -# MVME 135 board -set target_info(idp,name) "mvme" -set target_info(idp,ldflags) "-Tmvme.ld" -set target_info(idp,config) m68k-unknown-aout -set target_info(idp,cflags) "" -set target_info(idp,connect) telnet -set target_info(idp,target) "s8" -set target_info(idp,serial) "tstty8" -set target_info(idp,netport) "wharfrat:1008" -set target_info(idp,baud) "9600" -@end smallexample -@end cartouche - -DejaGnu can use this information to switch between multiple targets in -one test run. This is done through the use of the @code{push_target} -procedure, which is discussed elsewhere. -@c FIXME: write that section and put an xref here - -This array can also hold information for a remote host, which is used -when testing a candain cross. In this case, the only thing different is -the index is just @code{host}. Here's the settings I use to run tests -on my NT machine while running DejaGnu on a Unix machine. (in this case -a Linux box) - -@cartouche -@smallexample -set target_info(host,name) "nt-host" -set target_info(host,config) "386-unknown-winnt" -set target_info(host,connect) "telnet" -set target_info(host,target) "ripple" -@end smallexample -@end cartouche - -There is more info on how to use these variables in the sections on the -config files. @xref{Master Config File,,Configuration Files}. - -@cindex option defaults -@cindex @code{runtest} option defaults -@cindex variables for option defaults -@cindex defaults, option -In the user editable second section of @file{site.exp}, you can not only -override the configuration variables captured in the first section, but -also specify default values for all the @code{runtest} command line -options. Save for @samp{--debug}, @samp{--help}, and @samp{--version}, -each command line option has an associated Tcl variable. Use the Tcl -@code{set} command to specify a new default value (as for the -configuration variables). The following table describes the -correspondence between command line options and variables you can set in -@file{site.exp}. @xref{Invoking runtest,,Running the Tests}, for -explanations of the command-line options. - -@kindex all_flag -@kindex baud -@kindex reboot -@kindex outdir -@kindex objdir -@kindex runtests -@kindex ignoretests -@kindex srcdir -@kindex tracelevel -@kindex targetname -@kindex connectmode -@kindex tool -@kindex target_triplet -@kindex host_triplet -@kindex build_triplet -@kindex verbose - -@cindex command line option variables -@cindex Tcl variables for option defaults -@cindex default options, controlling -@cindex options, Tcl variables for defaults - -@ifinfo -@display -runtest Tcl -option variable description -__________ ________ ___________________________________________ - ---all all_flag display all test results if set - ---baud baud set the default baud rate to something other - than 9600. ---connect connectmode @samp{rlogin}, @samp{telnet}, @samp{rsh}, - @samp{kermit}, @samp{tip}, or @samp{mondfe} - ---outdir outdir directory for @file{@var{tool}.sum} and @file{@var{tool}.log} - ---objdir objdir directory for pre-compiled binaries - ---reboot reboot reboot the target if set to @code{"1"}; - do not reboot if set to @code{"0"} (the default) - ---srcdir srcdir directory of test subdirectories - ---strace tracelevel a number: Tcl trace depth - ---tool tool name of tool to test; identifies init, test subdir - ---verbose verbose verbosity level. As option, use multiple times; - as variable, set a number, 0 or greater ---target target_triplet The canonical configuration string for the target. ---host host_triplet The canonical configuration string for the host. ---build build_triplet The canonical configuration string for the - build host. - -@end display -@end ifinfo - -@tex -\vbox{\halign{\hfil \tt #\quad &\quad\tt #\hfil &\hbox{\vtop{{\raggedright\parindent=0pt\parskip=5pt\hsize=2.75in\rm#\strut\par}}}\hfill\cr -\cr -{\it runtest}&{\it Tcl}\cr -{\it option}&{\it variable}&{\it description}\cr -\noalign{\hrule width\hsize}\cr ---all &all\_flag &display all test results if set\cr ---baud &baud &set the default baud rate to something other - than 9600.\cr ---connect &connectmode &@samp{rlogin}, @samp{telnet}, @samp{rsh}, - @samp{kermit}, @samp{tip}, or @samp{mondfe}\cr ---mail &mailing\_list&address list for mailing test output\cr ---name &targetname &network name of testing target or its host\cr ---outdir &outdir &directory for @file{@var{tool}.sum} and @file{@var{tool}.log}\cr ---objdir &objdir &directory for compiled binaries\cr ---reboot &reboot &reboot the target if set to @code{"1"}; -do not reboot if set to @code{"0"} (the default)\cr ---srcdir &srcdir &directory of test subdirectories\cr ---strace &tracelevel &a number: Tcl trace depth\cr ---tool &tool &name of tool to test; identifies init, test subdir\cr ---verbose &verbose &verbosity level. As option, use multiple times; - as variable, set a number, 0 or greater\cr ---target &target\_triplet - &The canonical configuration string for the target.\cr ---host &host\_triplet &The canonical configuration string for the host.\cr ---build &build\_triplet &The canonical configuration string for the - build host.\cr -}} -@end tex - -@node Master Config File, Local Config File, Config Values, Customizing -@subsection Master Config File -@cindex master @file{site.exp} -@cindex @file{site.exp} for all of DejaGnu -The master config file is where all the target specific config variables -get set for a whole site get set. The idea is that for a centralized -testing lab where people have to share a target between multiple -developers. There are settings for both remote targets and remote hosts. -Here's an example of a Master Config File (also called the Global config -file) for a @emph{canadian cross}. A canadian cross is when you build -and test a cross compiler on a machine other than the one it's to be -hosted on. - -Here we have the config settings for our California office. Note that -all config values are site dependant. Here we have two sets of values -that we use for testing m68k-aout cross compilers. As both of these -target boards has a different debugging protocol, we test on both of -them in sequence. - -@cartouche -@smallexample -global CFLAGS -global CXXFLAGS - -case "$target_triplet" in @{ - @{ "native" @} @{ - set target_abbrev unix - @} - @{ "m68*-unknown-aout" @} @{ - set target_abbrev "rom68k" - # IDP target # IDP board with rom68k monitor - set target_info(idp,name) "idp" - set target_info(idp,ldflags) "-Tidp.ld" - set target_info(idp,config) m68k-unknown-aout - set target_info(idp,cflags) "" - set target_info(idp,connect) telnet - set target_info(idp,target) "s7" - set target_info(idp,serial) "tstty12" - set target_info(idp,netport) "truckin:1007" - set target_info(idp,baud) "9600" - # MVME target # Motorola MVME 135 with BUG monitor - set target_info(mvme,name) "mvme" - set target_info(mvme,ldflags) "-Tmvme.ld" - set target_info(mvme,config) m68k-unknown-aout - set target_info(mvme,cflags) "" - set target_info(mvme,connect) telnet - set target_info(mvme,target) "s4" - set target_info(mvme,serial) "tstty8" - set target_info(mvme,netport) "truckin:1004" - set target_info(mvme,baud) "9600" - @} -@} -@end smallexample -@end cartouche - - In this case, we have support for several remote hosts for -our m68k-aout cross compiler. Typically the remote Unix hosts run -DejaGnu locally, but we also use them for debugging the testsuites when -we find problems in running on remote hosts. Expect won't run on NT, so -DejaGnu is run on the local build machine, and it'll connect to the NT -host and run all the tests for this cross compiler on that host. - -@smallexample -@cartouche -case "$host_triplet" in @{ - "native" @{ - @} - "i?86-*-linux*" @{ # Linux host - set target_info(host,name) "linux-host" - set target_info(host,config) $host_triplet - set target_info(host,connect) rlogin - set target_info(host,target) chinadoll - @} - "i?86-*-winnt # NT host - set target_info(host,name) "nt-host" - set target_info(host,config) i386-unknown-winnt - set target_info(host,connect) telnet - set target_info(host,target) ripple - @} - "hppa*-hp-hpux*" @{ # HP-UX host - set target_info(host,name) "hpux-host" - set target_info(host,config) $host_triplet - set target_info(host,connect) rlogin - set target_info(host,target) slipknot - @} - "sparc-sun-sunos*" @{ # SunOS (sun4) - set target_info(host,name) "sunos-host" - set target_info(host,config) $host_triplet - set target_info(host,connect) rlogin - set target_info(host,target) darkstar - @} -@} -@end cartouche -@end smallexample - -@node Local Config File, Personal Config File, Master Config File, Customizing -@subsection Local Config File -@cindex local @file{site.exp} -@cindex @file{site.exp} for each tool -It is usually more convenient to keep these ``manual overrides'' in the -@file{site.exp} local to each test directory, rather than in the -``master'' @file{site.exp} in the DejaGnu library. - -All local @file{site.exp} usually files have two sections, separated by -comment text. The first section is the part that is generated by -@code{make}. It is essentially a collection of Tcl variable definitions -based on @file{Makefile} environment variables. Since they are generated -by @code{make}, they contain the values as specified by -@code{configure}. (You can also customize these values by using the -@samp{--site} option to @code{configure}.) In particular, this section -contains the @file{Makefile} variables for host and target configuration -data. Do not edit this first section; if you do, your changes are replaced -next time you run @code{make}. - -The first section starts with: - -@cartouche -@smallexample -## these variables are automatically generated by make ## -# Do not edit here. If you wish to override these values -# add them to the last section -@end smallexample -@end cartouche - -In the second section, you can override any default values (locally to -DejaGnu) for all the variables. The -second section can also contain your preferred defaults for all the -command line options to @code{runtest}. This allows you to easily -customize @code{runtest} for your preferences in each configured -test-suite tree, so that you need not type options repeatedly on the -command line. (The second section may also be empty, if you do not wish -to override any defaults.) - -The first section ends with this line: - -@cartouche -@smallexample -## All variables above are generated by configure. Do Not Edit ## -@end smallexample -@end cartouche - -You can make any changes under this line. If you wish to redefine a -variable in the top section, then just put a duplicate value in this -second section. Usually the values defined in this config file are -related to the configuration of the test run. This is the ideal place to -set the variables @code{host_triplet}, @code{build_triplet}, -@code{target_triplet}. All other variables are tool dependant. ie for -testing a compiler, the value for @var{CC} might be set to a freshly -built binary, as opposed to one in the user's path. - -@node Personal Config File, , Local Config File, Customizing -@subsection Personal Config File -@cindex personal config @file{site.exp} -@cindex @file{site.exp} for each person -The personal config file is used to customize @code{runtest's} behaviour -for each person. It's typically used to set the user prefered setting -for verbosity, and any experimental Tcl procedures. My personal -@file{~/.dejagnurc} file looks like: - -@cartouche -@smallexample -set all_flag 1 -set RLOGIN /usr/ucb/rlogin -set RSH /usr/ucb/rsh -@end smallexample -@end cartouche - -Here I set @code{all_flag} so I see all the test cases that PASS along -with the ones that FAIL. I also set @var{RLOGIN} and @code{RSH} to the -BSD version. I have @code{kerberos} installed, and when I rlogin to a -target board, it usually isn't supported. So I use the non secure -versions of these programs rather than the default that's in my path. - -@node Internals -@chapter The DejaGnu Implementation -@cindex operating principles -@cindex internal details - -DejaGnu is entirely written in @code{expect}, which uses Tcl as a -command language. @code{expect} serves as a very programmable shell; -you can run any program, as with the usual Unix command shells---but -once the program is started, your @code{expect} script has fully -programmable control of its input and output. This does not just apply -to the programs under test; @code{expect} can also run any auxiliary -program, such as @code{diff} or @code{sh}, with full control over its -input and output. - -DejaGnu itself is merely a framework for the set of test suites -distributed separately for each @sc{gnu} tool. Future releases of -@sc{gnu} tools will include even more tests, developed throughout the -free software community. - -@kindex runtest.exp -@code{runtest} is the glue to tie together and manage the test scripts. -The @code{runtest} program is actually a simple Bourne shell script that -locates a copy of the @code{expect} shell and then starts the main Tcl -code, @code{runtest.exp}. @code{runtest.exp} itself has these essential -functions: - -@enumerate -@item -Parse the command line options, load the library files, and load the -default configuration files. - -@item -Locating the individual test scripts. @code{runtest.exp} locates the tests -by exploiting a straightforward naming convention based on the string -you specify with the @samp{--tool} option. - -@item -Providing an extended test environment, by defining additional Tcl -procedures beyond those already in @code{expect}. - -@item -Locating target-dependent functions, to standardize the test environment -across a wide variety of test platforms. -@end enumerate - -@menu -* Names:: Conventions for using tool names -* Init Module:: Initialization module -* DejaGnu Builtins:: DejaGnu provides these Tcl procedures -* Target Dependent:: Procedures supplied by the init module -* Cross Targets:: Remote targets supported -* Input Files:: The files DejaGnu depends on -* Output Files:: The files DejaGnu produces -@end menu - -@node Names -@section Conventions for using tool names - -@cindex @code{--tool} and naming conventions -@cindex tool names and naming conventions -@cindex naming conventions -DejaGnu uses @samp{$tool}, the name of the tool under test, to tie -together the testing configuration in a straightforward but flexible -way. If there is only one testsuite for a particular application, then -@samp{$tool} is optional. - -@samp{$tool} is @emph{not} used to invoke the tool, since sites that run -multiple configurations of a particular tool often call each -configuration by a different name. @code{runtest} uses the -configuration-dependent variables captured in @file{site.exp} to -determine how to call each tool. - -@cindex directory names and @code{--tool} -@cindex test directories, naming -@code{runtest} uses tool names to find directories containing tests. -@code{runtest} scans the source directory (specified with -@code{--srcdir}) for all directories whose names start with the tool -name. It is a common practice to put a period after the tool part of the -name. For instance, directories that start with -@samp{g++.} contain @sc{g++} tests. To add a new test, just put it in -any directory (create an entirely new directory, if you wish) whose name -follows this convention. - -@cindex @code{exp} filename suffix -@cindex test filename -@cindex filename for test files -A test is any file in an appropriately named subdirectory whose name -ends in @samp{.exp} (the conventional way of naming @code{expect} -scripts). These simple naming conventions make it as simple as possible -to install new tests: all you must do is put the test in the right -directory. - -@cindex order of tests -@cindex tests, running order -@code{runtest} sorts the tests in each subdirectory by name (using the -Tcl @code{lsort} command) and runs them in the resulting order. - -@node Init Module -@section Initialization module -@cindex tool initialization -@cindex setting up targets - -@c FIXME! should this node be merged with "Target dependent"? - -@cindex init file, purpose -@cindex starting interactive tools -@cindex initialization -The initialization module (or ``init file'') has two purposes: to -provide tool and target dependent procedures, and to start up an -interactive tool to the point where it is ready to operate. The latter -includes establishing communications with the target. All the tests for -interactive programs assume that the tool is already running and -communicating. Initialization modules for non-interactive programs may -only need to supply the support functions. - -@cindex init file name -@cindex name, initialization module -Each test suite directory must contain (in its @file{config} -subdirectory) a separate initialization module for each target. The -appropriate init file is can be named several ways. The prefered name is -the @emph{os} part of the canonical configuration name with @code{.exp} -as the suffix. An example would be that for an @code{m68k-coff} system, -the @code{target_os} part would be @code{coff}. The next way is for -system where there are short filenames, or a shortcut is desired to -refer to the OS name for that target. This is uses the value of -@code{$target_abbrev} rather than the @code{target_os}. - -The final file looked for is simply @file{default.exp}. If there is only -one operating system to support, then this file can be used. It's main -purpose is to offer some support for new operating systems, or for -unsupported cross targets. The last file looked for is -@file{unknown.exp}. This is usually limited to error handling for -unsupported targets. It's whole contents is typically. - -@cartouche -@smallexample -perror "Sorry, there is no support for this target" -exit 1 -@end smallexample -@end cartouche - -At the beginning of the init file, you must first determine the proper -executable name of the tool to execute, since the actual name of the -tool to be tested my vary from system to system. Here's an example -for the @sc{GNU} C compiler. - -@cartouche -@smallexample -global AR -# look for the archiver ar -if ![info exists AR] @{ - set AR [findfile $base_dir/../../binutils/ar $base_dir/../../binutils/ar [tr -ansform ar]] - verbose "AR defaulting to $AR" 2 -@} -@} - -global CFLAGS -if ![info exists CFLAGS] then @{ - set CFLAGS "" -@} -@end smallexample -@end cartouche - -It is always a good idea to first check the variable, and only set it if -it has not yet been defined. Often the proper value of @code{AR} is set -on the command line that invokes @file{runtest}. - -@kindex findfile -The @code{findfile} procedure takes as it's first argument a file name -to look for. The second argument is returned if the file is found, and -the third argument is returned if the file is not found. @code{base_dir} -is set internally by DejaGnu to the top level directory of the object -tree. - -@kindex transform -The @code{transform} procedure takes as its argument the native name of -a tool (such as @samp{gcc} for the compiler), and returns the name as -configured for that tool in the current installation. (For example, a -cross-compiling version of @sc{gnu} CC that generates MIPS code may be -installed with a name like @code{mips-idt-ecoff-gcc}.) - -In a test running native, writing the Tcl code for initialization is -usually quite simple. For cross configurations, however, more elaborate -instructions are usually needed to describe how to talk to a remote -target. - -Each initialization module defines up to four procedures with standard -names and purposes. The names of these procedures begin with -@samp{$tool}, the string that identifies tests for a particular tool: -@code{$tool_start}, @code{$tool_load}, @code{$tool_exit}, and -@code{$tool_version}. For example, the start procedure for @sc{gdb} is -called @code{gdb_start}. (Since start procedures are used differently -for batch and interactive tools, however, @code{runtest} itself never -calls the start procedure. Init files for interactive tools are -expected to end by running the start procedure.) - -@cindex utilities, loading from init file -@cindex defaults, setting in init file -The initialization module is also a good place to call @code{load_lib} -to get any collections of utility procedures meant for a family of test -cases, and to set up default values for any additional Tcl variables -needed for a specific set of tests. - -@xref{Target Dependent,,Target dependent procedures}, for full -descriptions of these procedures. - -@node DejaGnu Builtins -@section DejaGnu procedures -@cindex built in procedures, DejaGnu - -DejaGnu provides these Tcl procedures for use in test scripts. -You can also use any standard @code{expect} or Tcl function. These -procedures are stored in libraries, which DejaGnu loads at -runtime. Here's explanation of the library procedures that get loaded at -runtime. All other librarys are optional, and need to be loaded by the -testsuite. - -@menu -* framework.exp:: Core Internal Procedures. -* remote.exp:: Procedures for remote communication. -* utils.exp:: Utility procedures. -* target.exp:: Cross target procedures. -* debugger.exp:: Procedures for debugging your Tcl code. -@end menu - -@node framework.exp, remote.exp, ,DejaGnu Builtins -@subsection Core Internal Procedures -@cindex Core Internal Procedures - -@xref{Posix,,A POSIX conforming test framework}, for more detailed -explanations of the test outcomes (@samp{FAIL}, @samp{PASS}, -@samp{UNTESTED}, @samp{UNRESOLVED}, @samp{UNSUPPORTED}). - -@ftable @code -@item perror "@var{string} @var{number}" -@cindex test case, ERROR in -@kindex ERROR -Declares a severe error in the testing framework itself. -@code{perror} writes in the log files a message beginning with -@samp{ERROR}, appending the argument @var{string}. If the optional -@var{number} is supplied, then this is used to set the internal count of -errors to that value. - -As a side effect, @code{perror} also changes the effect of the next -@code{pass} or @code{fail} command: the test outcome becomes -@samp{UNRESOLVED}, since an automatic @samp{PASS} or @samp{FAIL} cannot -be trusted after a severe error in the test framework. If the optional -numeric value is @samp{0}, then there are no further side effects to -calling this function, and the following test outcome doesn't become -@samp{UNRESOLVED}. This can be used for errors with no known side -effects. - -@item warning "@var{string} @var{number}" -@cindex test case, WARNING in -@kindex WARNING -Declares detection of a minor error in the test case itself. -@code{warning} writes in the log files a message beginning with -@samp{WARNING}, appending the argument @var{string}. Use @code{warning} -rather than @code{error} for cases (such as communication failure -to be followed by a retry) where the test case can recover from the -error. If the optional @var{number} is supplied, then this is used to -set the internal count of warnings to that value. - -As a side effect, @code{warning_threshold} or more calls to -@code{warning} in a single test case also changes the effect of the next -@code{pass} or @code{fail} command: the test outcome becomes -@samp{UNRESOLVED} since an automatic @samp{PASS} or @samp{FAIL} may not -be trustworthy after many warnings. If the optional numeric value is -@samp{0}, then there are no further side effects to calling this -function, and the following test outcome doesn't become -@samp{UNRESOLVED}. This can be used for errors with no known side -effects. - -@item note "@var{string}" -@cindex test case, informational messages -@kindex NOTE -Appends an informational message to the log file. -@code{note} writes in the log files a message beginning with -@samp{NOTE}, appending the argument @var{string}. Use @code{note} -sparingly. @code{verbose} should be used for most such messages, -but in cases where a message is needed in the log file regardless of -the verbosity level use @code{note}. - -@item pass "@var{string}" -@cindex test case, declaring success -Declares a test to have passed. @code{pass} writes in the -log files a message beginning with @samp{PASS} (or @code{XPASS}, if -failure was expected), appending the argument @var{string}. - -@item fail "@var{string}" -@cindex test case, declaring failure -Declares a test to have failed. @code{fail} writes in the -log files a message beginning with @samp{FAIL} (or @code{XFAIL}, if -failure was expected), appending the argument @var{string}. - -@item unresolved "@var{string}" -@cindex test case, declaring ambiguity -Declares a test to have an unresolved outcome. @code{unresolved} writes -in the log file a message beginning with @samp{UNRESOLVED}, appending -the argument @var{string}. This usually means the test did not execute -as expected, and a human being must go over results to determine if it -passed or failed (and to improve the test case). - -@item untested "@var{string}" -@cindex test case, declaring no test -Declares a test was not run. @code{untested} writes in the log file a -message beginning with @samp{UNTESTED}, appending the argument -@var{string}. For example, you might use this in a dummy test whose -only role is to record that a test does not yet exist for some feature. - -@item unsupported "@var{string}" -@cindex test case, declaring no support -Declares that a test case depends on some facility that does not exist -in the testing environment. @code{unsupported} writes in the log file a -message beginning with @samp{UNSUPPORTED}, appending the argument -@var{string}. - -@item get_warning_threshold -@cindex test case, WARNING threshold -Returns the current value of @code{warning_threshold}. -The default value is 3. - -@item set_warning_threshold @var{threshold} -@cindex test case, WARNING threshold -Sets the value of @code{warning_threshold}. -A value of @code{0} disables it: calls to @code{warning} will not turn -a @samp{PASS} or @samp{FAIL} into an @samp{UNRESOLVED}. - -@item transform "@var{toolname}" -@cindex transform tool name -@cindex installed tool name -@cindex tool name, as installed -@cindex name transformations -Generates a string for the name of a tool as it was configured and -installed, given its native name (as the argument @var{toolname}). -This makes the assumption that all tools are installed using the same -naming conventions: it extrapolates from the invocation name for -@file{runtest}. For example, if you call @code{runtest} as -@file{m68k-vxworks-runtest}, the result of @w{@samp{ transform "gcc" }} -is @samp{m68k-vxworks-gcc}. - -@item ishost "@var{host}" -@cindex host configuration test -Tests for a particular @emph{host} environment. If the currently -configured host matches the argument string, the result is @code{1}; -otherwise the result is @code{0}. @var{host} must be a full three-part -@code{configure} host name; in particular, you may not use the shorter -nicknames supported by @code{configure} (but you can use wildcard -characters, using shell syntax, to specify sets of names). - -@item istarget "@var{target}" -@cindex target configuration test -Tests for a particular @emph{target} environment. If the currently -configured target matches the argument string, the result is @code{1}; -otherwise the result is @code{0}. @var{target} must be a full -three-part @code{configure} target name; in particular, you may not use -the shorter nicknames supported by @code{configure} (but you can use -wildcard characters, using shell syntax, to specify sets of names). If it is -passed a @code{NULL} string, then it returns the name of the build -canonical configuration. - -@item isbuild "@var{host}" -@cindex build host configuration test -Tests for a particular @emph{build host} environment. If the currently -configured host matches the argument string, the result is @code{1}; -otherwise the result is @code{0}. @var{host} must be a full three-part -@code{configure} host name; in particular, you may not use the shorter -nicknames supported by @code{configure} (but you can use wildcard -characters, using shell syntax, to specify sets of names). If it is -passed a @code{NULL} string, then it returns the name of the build -canonical configuration. - -item is3way "@var{host}" -@cindex canadian cross configuration test -Tests for a canadian cross. This is when the tests will be run on a -remotly hosted cross compiler. If it is a canadian cross, then the -result is @code{1}; otherwise the result is @code{0}. - -@item isnative -@cindex native configuration test -Tests whether the current configuration has the same host and target. -When it runs in a @emph{native} configuration this procedure returns a -@code{1}; otherwise it returns a @code{0}. - -@item load_lib "@var{library-file}" -@cindex load library file -Loads the file @var{library-file} by searching a fixed path built into -@code{runtest}. If DejaGnu has been installed, it looks in a path -starting with the installed library directory. If you are running -DejaGnu directly from a source directory, without first running -@samp{make install}, this path defaults to the current directory. In -either case, it then looks in the current directory for a directory -called @code{lib}. If there are duplicate definitions, the last one -loaded takes precedence over the earlier ones. - -@item setup_xfail "@var{config} @r{[}@var{bugid}@r{]}" -@c two spaces above to make it absolutely clear there's whitespace---a -@c crude sort of italic correction! -@cindex test case, expecting failure -@cindex failure, expected -@cindex expected failure -Declares that the test is expected to fail on a particular set of -configurations. The @var{config} argument must be a list of full -three-part @code{configure} target name; in particular, you may not use -the shorter nicknames supported by @code{configure} (but you can use the -common shell wildcard characters to specify sets of names). The -@var{bugid} argument is optional, and used only in the logging file -output; use it as a link to a bug-tracking system such as @sc{gnats} -(@pxref{Overview,, Overview, gnats.info, Tracking Bugs With GNATS}). - -@cindex @code{XFAIL}, producing -@cindex @code{XPASS}, producing -Once you use @code{setup_xfail}, the @code{fail} and @code{pass} -procedures produce the messages @samp{XFAIL} and @samp{XPASS} -respectively, allowing you to distinguish expected failures (and -unexpected success!) from other test outcomes. - -@emph{Warning:} you must clear the expected failure after using -@code{setup_xfail} in a test case. Any call to @code{pass} or -@code{fail} clears the expected failure implicitly; if the test has some -other outcome, e.g. an error, you can call @code{clear_xfail} to clear -the expected failure explicitly. Otherwise, the expected-failure -declaration applies to whatever test runs next, leading to surprising -results. - -@item clear_xfail @var{config} -@cindex cancelling expected failure -@cindex expected failure, cancelling -Cancel an expected failure (previously declared with @code{setup_xfail}) -for a particular set of configurations. The @var{config} argument is a -list of configuration target names. It is only necessary to call -@code{clear_xfail} if a test case ends without calling either -@code{pass} or @code{fail}, after calling @code{setup_xfail}. - -@item verbose @r{[}-log@r{]} @r{[}-n@r{]} @r{[}--@r{]} "@var{string}" @var{number} -@cindex @code{verbose} builtin function -Test cases can use this function to issue helpful messages depending on -the number of @samp{--verbose} options on the @code{runtest} command -line. It prints @var{string} if the value of the variable -@code{verbose} is higher than or equal to the optional @var{number}. The -default value for @var{number} is 1. Use the optional @samp{-log} argument -to cause @var{string} to always be added to the log file, even if it won't -be printed. Use the optional @samp{-n} argument to print @var{string} -without a trailing newline. Use the optional @samp{--} argument if -@var{string} begins with "-". - -@end ftable - -@noindent -@node remote.exp, utils.exp, framework.exp, DejaGnu Builtins -@subsection Remote Communication Procedures - -@kindex remote.exp -@kindex lib/remote.exp -@cindex remote connection procedures -@cindex communications procedures -@file{lib/remote.exp} defines these functions, for establishing and -managing communications: - -@emph{Procedures to establish a connection:} Each of these procedures -tries to establish the connection up to three times before returning. -Warnings (if retries will continue) or errors (if the attempt is -abandoned) report on communication failures. The result for any of -these procedures is either @code{-1}, when the connection cannot be -established, or the spawn ID returned by the @code{expect} command -@code{spawn}. - -It use the value of the @code{connect} field in the @code{target_info} -array (was @code{connectmode} as the type of connection to make. Current -supported connection types are tip, kermit, telnet, rsh, rlogin, and -netdata. If the @code{--reboot} option was used on the runtest command -line, then the target is rebooted before the connection is made. - -@ftable @code - -@item remote_open @var{type} -@cindex Opening a remote connection -@emph{Remote Connection Procedure.} This is passed @emph{host} or -@emph{target}. Host or target refers to whether it is a connection to a -remote target, or a remote host. This opens the connection to the -desired target or host using the default values in the configuration -system. It returns that @code{spawn_id} of the process that manages the -connection. This value can be used in @code{expect} or @code{exp_send} -statements, or passed to other procedures that need the connection -process's id. This also sets the @code{fileid} field in the -@code{target_info} array. - - -@item remote_close @var{shellid} -@cindex Closing a remote connection -@emph{shellid} is value returned by a call to @code{remote_open}. This -closes the connection to the target so resources can be used by -others. This parameter can be left off if the @code{fileid} field in the -@code{target_info} array is set. - -@item telnet @var{hostname} @var{port} -@itemx rlogin @var{hostname} -@itemx rsh @var{hostname} -@cindex IP network procedures -@cindex network (IP) procedures -@emph{IP network procedures.} @var{hostname} refers to the IP address or -name (for example, an entry in @file{/etc/hosts}) for this target. The -procedure names reflect the Unix utility used to establish a -connection. The optional @var{port} is used to specify the IP port -number. The value of the @code{netport} field in the @code{target_info} -array is used. (was @code{$netport}) This value has two parts, the -hostname and the port number, seperated by a @emph{:}. If @code{host} or -@code{target} is used in the @code{hostname} field, than the config -array is used for all information. - -@item tip @var{port} -@cindex serial line connection, @code{tip} -@emph{Serial line procedure.} Connect using the Unix utility @code{tip}. -@var{port} must be a name from the @code{tip} configuration file -@file{/etc/remote}. Often, this is called @samp{hardwire}, or something -like @samp{ttya}. This file holds all the configuration data for -the serial port. The value of the @code{serial} field in the -@code{target_info} array is used. (was @code{$serialport}) If -@code{host} or @code{target} is used in the @code{port} field, than -the config array is used for all information. - -@item kermit @var{port} @var{bps} -@cindex serial line connection, @code{kermit} -@emph{Serial line procedure.} Connect using the program @code{kermit}. -@var{port} is the device name, e.g. @file{/dev/ttyb}. @var{bps} is -the line speed to use (in bits per second) for the connection. The value -of the @code{serial} field in the @code{target_info} array is used. (was -@code{$serialport}) If @code{host} or @code{target} is used in the -@code{port} field, than the config array is used for all information. - -@end ftable - -@noindent -@emph{Procedures to manage a connection:} - -@ftable @code -@item tip_download @var{spawnid} @var{file} -@cindex download, @code{tip} -@cindex serial download, @code{tip} -Download @file{@var{file}} to the process @var{spawnid} (the value -returned when the connection was established), using the @code{~put} -command under @code{tip}. Most often used for single board computers -that require downloading programs in @sc{ascii} S-records. Returns -@code{1} if an error occurs, @code{0} otherwise. - -@item exit_remote_shell @var{spawnid} -@cindex terminating remote connection -@cindex remote connection, ending -Exits a remote process started by any of the connection procedures. -@var{spawnid} is the result of the connection procedure that started the -remote process. - -@item download @var{file} @r{[} @var{spawnid} @r{]} -@cindex download a file -After you establish a connection to a target, you can download programs -using this command. @code{download} reads in @var{file} (object code in -S-record format) and writes it to the device controlling this -@var{spawnid}. (From the point of view of the target, the S-record file -comes in via standard input.) - -If you have more than one target active, you can use the optional argument -@var{spawnid} to specify an alternative target (the default is the most -recently established @var{spawnid}.) -@end ftable - -@noindent -@node utils.exp, target.exp, remote.exp, DejaGnu Builtins -@subsection Utility Procedures - -@kindex utils.exp -@kindex lib/utils.exp -@file{lib/utils.exp} defines these utility procedures: - -@ftable @code -@item getdirs @var{dir} -@itemx getdirs @var{dir} @var{pattern} -@cindex directories matching a pattern -@cindex pattern match, directory -Returns a list of all the directories in the single directory @var{dir} -that match @var{pattern}. If you do not specify @var{pattern}, -@code{getdirs} assumes @samp{*}. You may use the common shell wildcard -characters in @var{pattern}. If no directories match the pattern, then a -@code{NULL} string is returned. - -@item find @var{dir} @var{pattern} -@cindex files matching a pattern -@cindex pattern match, filenames -Search for files whose names match @var{pattern} (using shell wildcard -characters for filename expansion). Search subdirectories recursively, -starting at @var{dir}. The result is the list of files whose names -match; if no files match, the result is empty. Filenames in the result -include all intervening subdirectory names. If no files match the -pattern, then a @code{NULL} string is returned. - -@item which @var{binary} -@cindex path lookup -Searches the execution path for an executable file @var{binary}, like -the the BSD @code{which} utility. This procedure uses the shell -environment variable @samp{PATH}. It returns @code{0} if the binary is -not in the path, or if there is no @samp{PATH} environment variable. If -@var{binary} is in the path, it returns the full path to @var{binary}. - -@item grep @var{filename} @var{regexp} -@item grep @var{filename} @var{regexp} line -@cindex regular expression, file contents -@cindex searching file contents -Search the file called @var{filename} (a fully specified path) for lines -that contain a match for regular expression @var{regexp}. The result is -a list of all the lines that match. If no lines match, the result is an -empty string. Specify @var{regexp} using the standard regular -expression style used by the Unix utility program @code{grep}. - -Use the optional third argument @samp{line} to start lines in the result -with the line number in @var{filename}. (This argument is simply an -option flag; type it just as shown---@samp{line}.) - -@item diff @var{filename} @var{filename} -@cindex finding file differences -@cindex comparing files -Compares the two files and returns a 1 if they match, or a 0 if they -don't. If @code{verbose} is set, then it'll print the differences to the -screen. - -@item slay @var{name} -@cindex slaying processes -This look in the process tabel for @var{name} and send it a unix -@code{SIGINT}, killing the process. - -@item absolute @var{path} -@cindex converting relative paths to absolute -This procedure takes the relative @var{path}, and converts it to an -absolute path. - -@item psource @var{filename} -@cindex sourcing Tcl files -This sources the file @var{filename}, and traps all errors. It also -ignores all extraneous output. If there was an error it returns a 1, -otherwise it returns a 0. - -@item prune @var{list} @var{pattern} -@cindex list, pruning -Remove elements of the Tcl list @var{list}. Elements are fields -delimited by spaces. The result is a copy of @var{list}, without any -elements that match @var{pattern}. You can use the common shell -wildcard characters to specify @var{pattern}. - -@item setenv @var{var} @var{val} -@cindex setting environment variables -Sets the variable @var{var} to the value @var{val}. - -@item unsetenv @var{var} -@cindex unsetting environment variables -Unsets the environment variable @var{var} - -@item getenv @var{var} -@cindex getting environment variables -returns the value of @var{var} in the environment if it exists, -otherwise it returns @code{NULL}. - -@item runtest_file_p @var{runtests} @var{testcase} -@cindex selecting a range of tests -@cindex tests, running specifically -Search @var{runtests} for @var{testcase} and return 1 if found, 0 if not. -@var{runtests} is a list of two elements. The first is the pathname of -the testsuite expect script running. The second is a copy of what was -on the right side of the @code{=} if @samp{foo.exp="@dots{}"} was specified, -or an empty string if no such argument is present. -This is used by tools like compilers where each testcase is a file. - -@item prune_system_crud @var{system} @var{text} -@cindex pruning system output, examining program output -For system @var{system}, delete text the host or target operating system might -issue that will interfere with pattern matching of program output in -@var{text}. An example is the message that is printed if a shared library -is out of date. - -@end ftable - -@noindent -@node target.exp, debugger.exp, utils.exp, DejaGnu Builtins -@subsection Cross target procedure - -@kindex target.exp -@kindex lib/target.exp -@file{lib/target.exp} defines these utility procedures: - -@ftable @code - -@item push_target @emph{name} -@cindex set current target -This makes the target named @emph{name} be the current target -connection. The value of @emph{name} is an index into the -@code{target_info} array and is set in the global config file. - -@item pop_target -@cindex unset current target -This unsets the current target connection. - -@item list_targets -@cindex lists supported targets -This lists all the supported targets for this architecture. - -@item push_host @emph{name} -@cindex set current host -This makes the host named @emph{name} be the current remote host -connection. The value of @emph{name} is an index into the -@code{target_info} array and is set in the global config file. - -@item pop_host -@cindex unset current host -This unsets the current host connection. - -@c @item compile @emph{file} -@cindex compile a file -This invokes the compiler as set by @code{CC} to compile the file -@emph{file}. The default options for many cross compilation targets are -@emph{guessed} by DejaGnu, and these options can be added to by passing -in more parameters as arguments to @code{compile}. Optionally, this will -also use the value of the @code{cflags} field in the target config -array. If the host is not the same as the build machines, then then -compiler is run on the remote host using @code{execute_anywhere}. - -@c @item archive @emph{file} -@cindex archive object files -This produces an archive file. Any parameters passed to @code{archive} -are used in addition to the default flags. Optionally, this will -also use the value of the @code{arflags} field in the target config -array. If the host is not the same as the build machines, then then -archiver is run on the remote host using @code{execute_anywhere}. - -@c @item ranlib @emph{file} -@cindex ranlib a file -This generates an index for the archive file for systems that aren't -POSIX yet. Any parameters passed to @code{ranlib} are used in for the -flags. - -@item execute_anywhere @emph{cmdline} -@cindex executing commands remotely -This executes the @emph{cmdline} on the proper host. This should be used -as a replacement for the Tcl command @code{exec} as this version -utilizes the target config info to execute this command on the build -machine or a remote host. All config information for the remote host -must be setup to have this command work. If this is a canadian cross, -(where we test a cross compiler that runs on a different host then where -DejaGnu is running) then a connection is made to the remote host and -the command is executed there. It returns either @emph{REMOTERROR} (for -an error) or the output produced when the command was executed. This is -used for running the tool to be tested, not a test case. - -@end ftable - -@node debugger.exp, , target.exp, DejaGnu Builtins -@subsection Debugging Procedures - -@kindex debugger.exp -@kindex lib/debugger.exp -@file{lib/debugger.exp} defines these utility procedures: - -@ftable @code - -@item dumpvars @emph{expr} -@cindex Print global variable values -This takes a csh style regular expression (glob rules) and prints the -values of the global variable names that match. It is abbreviated as -@code{dv} - -@item dumplocals @emph{expr} -@cindex Print local variable value -This takes a csh style regular expression (glob rules) and prints the -values of the local variable names that match. It is abbreviated as -@code{dl}. - -@item dumprocs @emph{expr} -@cindex Print procedure bodies -This takes a csh style regular expression (glob rules) and prints the -body of all procs that match. It is abbreviated as @code{dp} - -@item dumpwatch @emph{expr} -@cindex Print watchpoints -This takes a csh style regular expression (glob rules) and prints all -the watchpoints. It is abbreviated as @code{dw}. - -@c FIXME: finish these when the code is fixed. -@c @item watcharray @emph{element} @emph{type} -@c @cindex Set a watchpoint on an array -@c This sets an watchpoint of the @emph{element-type} on the -@c @item watchvar v null type -@c @cindex Set a watchpoint on a variable - -@item watchunset @emph{var} -@cindex Watch when a variable is unset -This breaks program execution when the variable @emph{var} is unset. It -is abbreviated as @code{wu}. - -@item watchwrite @emph{var} -@cindex Watch when a variable is written -This breaks program execution when the variable @emph{var} is -written. It is abbreviated as @code{ww}. - -@item watchread @emph{var} -@cindex Watch when a variable is read -This breaks program execution when the variable @emph{var} is read. It -is abbreviated as @code{wr}. - -@item watchdel @emph{watch} -@cindex Delete a watchpoint. -This deletes a the watchpoint for @emph{watch}. It is abbreviated as -@code{wd}. - -@item print @emph{var} -@cindex Printing variable values -This prints the value of the variable @emph{var}. It is abbreviated as -@code{p}. - -@item quit -@cindex Quiting DejaGnu -This makes runtest exit. It is abbreviated as @code{q}. - -@item bt -@cindex Print a backtrace -This prints a backtrace of the executed Tcl commands. - -@end ftable - -@node Target Dependent -@section Target dependent procedures -@cindex target dependent procedures - -@c FIXME? These may be renamed to just "start", "load", "exit", and -@c "version" eventually. - -Each combination of target and tool requires some target-dependent -procedures. The names of these procedures have a common form: the tool -name, followed by an underbar @samp{_}, and finally a suffix describing -the procedure's purpose. For example, a procedure to extract the -version from @sc{gdb} is called @samp{gdb_version}. @xref{Init Module,, -Initialization Module}, for a discussion of how DejaGnu arranges to find -the right procedures for each target. - -@code{runtest} itself calls only two of these procedures, -@code{@var{tool}_exit} and @code{@var{tool}_version}; these procedures use -no arguments. - -The other two procedures, @code{@var{tool}_start} and -@code{@var{tool}_load}, are only called by the test suites themselves -(or by testsuite-specific initialization code); they may take arguments -or not, depending on the conventions used within each test suite. - -@ftable @code -@item @var{tool}_start -@cindex start procedure, tested tools -Starts a particular tool. For an interactive tool, -@code{@var{tool}_start} starts and initializes the tool, leaving the -tool up and running for the test cases; an example is @code{gdb_start}, -the start function for @sc{gdb}. For a batch oriented tool, -@code{@var{tool}_start} is optional; the recommended convention is to -let @code{@var{tool}_start} run the tool, leaving the output in a -variable called @code{comp_output}. Test scripts can then analyze -@samp{$comp_output} to determine the test results. An example of this -second kind of start function is @code{gcc_start}, the start function -for @sc{gcc}. - -@code{runtest} itself @emph{does not call} @code{@var{tool}_start}. The -initialization module @file{@var{tool}_init.exp} must call -@code{@var{tool}_start} for interactive tools; for batch-oriented tools, -each individual test script calls @code{@var{tool}_start} (or makes -other arrangements to run the tool). - -@item @var{tool}_load -@cindex load procedure, tested tools -Loads something into a tool. For an interactive tool, this conditions -the tool for a particular test case; for example, @code{gdb_load} loads -a new executable file into the debugger. For batch oriented tools, -@code{@var{tool}_load} may do nothing---though, for example, the -@sc{gcc} support uses @code{gcc_load} to load and run a binary on the -target environment. Conventionally, @code{@var{tool}_load} leaves the -output of any program it runs in a variable called @samp{exec_output}. -Writing @code{@var{tool}_load} can be the most complex part of extending -DejaGnu to a new tool or a new target, if it requires much communication -coding or file downloading. - -Test scripts call @code{@var{tool}_load}. - -@item @var{tool}_exit -@cindex exit procedure, tested tools -Cleans up (if necessary) before @code{runtest} exits. For interactive -tools, this usually ends the interactive session. You can also use -@code{@var{tool}_exit} to remove any temporary files left over from the -tests. - -@code{runtest} calls @code{@var{tool}_exit}. - -@item @var{tool}_version -@cindex version procedure, tested tools -Prints the version label and number for @var{tool}. This is called by -the DejaGnu procedure that prints the final summary report. The output -should consist of the full path name used for the tested tool, and its -version number. - -@code{runtest} calls @code{@var{tool}_version}. -@end ftable - -The usual convention for return codes from any of these procedures -(although it is not required by @code{runtest}) is to return @code{0} if -the procedure succeeded, @code{1} if it failed, and @code{-1} if there -was a communication error. - -@node Cross Targets -@section Remote targets supported - -@cindex targets -@cindex remote testing -The DejaGnu distribution includes support for the following remote -targets. You can set the target name and the connect mode in the -@file{site.exp} file (using the Tcl variables @samp{targetname} and -@samp{connectmode}, respectively), or on the @code{runtest} command line -(using @samp{--name} and @samp{--connect}). - -@table @strong -@item @sc{amd} 29000, with UDI protocol -Configure DejaGnu for target @samp{a29k-amd-udi}. (Cygnus -@code{configure} also recognizes the abbreviation @samp{udi29k}.) Then, -to run tests, use the @code{runtest} target name to specify whether you -want to use a simulator, or a particular hardware board. The particular -string to use with @samp{--name} will depend on your UDI setup file, -@file{udi_soc} (if @file{udi_soc} is not in your working directory, the -environment variable @samp{UDICONF} should contain a path to this file). -For example, if your UDI setup file includes these lines: -@end table -@c table "ends" *only* to allow wider example below - -@cartouche -@smallexample -iss AF_UNIX * isstip -r /home/gnu/29k/src/osboot/sim/osboot -mon AF_UNIX * montip -t serial -baud 9600 -com /dev/ttyb -@end smallexample -@end cartouche - -@table @strong -@item @w{ } -@c fake out table/item into continuing w/same margin as before -You can use @samp{--name iss} to run tests on the simulator, and -@samp{--name mon} to run tests on the 29K hardware. See the -manufacturer's manuals for more information on UDI and @file{udi_soc}. -@c FIXME! Is there a better ref than "the manufacturer's manuals"? - -@kindex mondfe -The default connect protocol is @samp{mondfe} with either back end. -@code{mondfe} is the only shell DejaGnu supports for UDI targets. -@code{mondfe} is an @sc{amd} specific monitor program freely available -from @sc{amd}. - -@emph{Warning:} This target requires @sc{gdb} version 4.7.2 (or -greater). Earlier versions of @sc{gdb} do not fully support the -@code{load} command on this target, so DejaGnu has no way to load -executable files from the debugger. - -@item Motorola 680x0 boards, a.out or @sc{coff} object format -Configure DejaGnu for any remote target matching @samp{m68k-*}. - -@emph{Warning:} Most @samp{m68k-*} configurations run all tests only for -native testing (when the target is the same as the host). When you -specify most of these targets for a cross configuration, you will only be -able to use tests that run completely within the host (for example, -tests of the binary utilities such as the archiver; or compiler tests -that only generate code rather than running it). - -To run a.out or @sc{coff} binaries on a remote M68K, you must configure -DejaGnu for a particular target board. @samp{m68k-abug} is an example. -(In general for an embedded environment, because it does not have absolute -addresses, a.out is not a good choice for output format in any case; most -often S-records or Hex-32 are used instead.) - -@item Motorola 68K MVME 135 board running ABug boot monitor -Configure for @samp{m68k-abug-aout} or @samp{m68k-abug-coff} (as a -target). This boot monitor can only download S-records; therefore, the -DejaGnu tests for this environment require a linker command script to -convert either output format to S-records, setting the default addresses -for @code{.text}, @code{.bss}, and @code{.data}. - -With this configuration, the default for @samp{--connect} is @samp{tip}. -@samp{tip} is the only communications protocol supported for connecting -to @samp{m68k-abug-*} targets. @samp{tip} uses an @sc{ascii} downloader -(the @code{~put} command) to load S-records into the target board. The -@samp{--name} string must be a machine name that @code{tip} -understands (for example, on some @code{tip} implementations it must be -an entry from the initialization file for @code{tip}; this file is -sometimes called @file{/etc/remote}). - -See your system documentation for information on how to create new -entries in @file{/etc/remote}. (Some @sc{unix} systems are distributed -with at least one default entry with a name resembling @samp{hardwire}; -if your system has one, you can edit it, or make a modified copy with a -new name.) When you have a working @file{/etc/remote} entry -@var{abugtarget}, you should be able to type @samp{tip -@var{abugtarget}}, and get the prompt @samp{135ABUG>} from the board. -Use the same @var{abugtarget} string with @samp{runtest --name}. - -@item Motorola IDP board running the rom68k boot monitor -@c FIXME 1: this doesn't really say anything! OK, so functionality is -@c the same. Is object code the same (srecords)? Do we configure with -@c the same triplets? What is the default for --connect? Is -@c any comms method other than tip supported? What prompt to expect -@c when tip connected? -@c FIXME 2: should @code{BUG} below be @code{ABUG}? -This is the same in functionality as the MVME board running the -@code{BUG} boot monitor. Only the monitor commands and the addresses are -different. - -@item VxWorks (Motorola 68K or Intel 960) -Configure DejaGnu for either @samp{m68k-wrs-vxworks} (abbreviated -@samp{vxworks68}) or @samp{i960-wrs-vxworks} (abbreviated -@samp{vxworks960}). Since both targets support IP addressing, specify -the network address (for example, a host name from @file{/etc/hosts}) -with @samp{--name}. - -The default connect protocol is @samp{rlogin}, but you can use any of -@samp{--connect rlogin}, @samp{--connect telnet}, or @samp{--connect -rsh}. - -Test scripts need no special code to load programs into these targets; -since VxWorks supports NFS, all you must do is ensure test programs are -on an exported filesystem. - -@cindex VxWorks, link with @samp{-r} -When you compile for VxWorks, use the linker @samp{-r} option to make -the linker output relocatable---at least if you want to use library -routines. Many standard C routines are included in VxWorks; often no -additional libraries are needed. See your VxWorks system documentation -for additional details. -@end table - -@node Input Files -@section The files DejaGnu reads -@cindex input files - -The @code{runtest} program used to invoke DejaGnu is a short shell -script generated by @code{make} during the configuration process. Its -main task is to read the main test framework driver, @file{runtest.exp}. - -@file{runtest.exp}, in turn, reads @code{expect} code from certain other -files, in this order: - -@enumerate -@item -Each of the @file{site.exp} local definition files available. -@xref{Customizing,,Setting @code{runtest} defaults}, for details. - -@item -@file{lib/utils.exp}, a collection of utility procedures. @xref{DejaGnu -Builtins,,DejaGnu Builtins}, for descriptions of these procedures. - -@item -@file{lib/framework.exp}, a file of subroutines meant for @code{runtest} -itself rather than for general-purpose use in both @code{runtest} and -test suites. - -@item -@file{debugger.exp}, Don Libes' Tcl Debugger. (See @cite{A Debugger for -Tcl Applications} by Don Libes. This paper is distributed with -@code{expect} in PostScript form as the file -@file{expect/tcl-debug.ps}.) - -@item -@file{lib/remote.exp}, a collection of subroutines meant for connecting -to remote machines. - -@item -@file{lib/target.exp}, a collection of subroutines used for the -configuration systems in DejaGnu. These procedures typically manipulate -or utilize the configuration system. - -@item -@c FIXME! A comment in runtest.exp claims a system default is used if -@c no tool-specific init file is not available; I couldn't see where -@c the program flow actually does this, though---pesch 30jul1993. -An initialization file @code{@var{tool}_init.exp}. @xref{Init -Module,,Initialization module}, for more discussion of init files. -@end enumerate - -@c This hard page break is mainly intended for smallbook formatting; -@c some examples in this section come out better if this starts at a -@c page boundary. -@page -@node Output Files -@section The files DejaGnu writes -@cindex output files - -@code{runtest} always writes two kinds of output files: summary logs and -detailed logs. The contents of both of these are determined by your -tests. - -For troubleshooting, a third kind of output file is useful: use -@samp{--debug} to request an output file showing details of what -@code{expect} is doing internally. - -@menu -* Summary:: Files that summarize tests -* Detail:: Files that contain complete test results -* Debug:: Logging expect internal actions -@end menu - -@node Summary -@subsection Summary log -@cindex summary log - -@code{runtest} always produces a summary output file -@file{@var{tool}.sum}. This summary shows the names of all test files -run; for each test file, one line of output from each @code{pass} -command (showing status @samp{PASS} or @samp{XPASS}) or @code{fail} -command (status @samp{FAIL} or @samp{XFAIL}); trailing summary -statistics that count passing and failing tests (expected and -unexpected); and the full pathname and version number of the tool -tested. (All possible outcomes, and all errors, are always reflected in -the summary output file, regardless of whether or not you specify -@samp{--all}.) - -If any of your tests use the procedures @code{unresolved}, -@code{unsupported}, or @code{untested}, the summary output also -tabulates the corresponding outcomes. - -For example, after @samp{runtest --tool binutils}, look for a summary -log in @file{binutils.sum}. Normally, @code{runtest} writes this file -in your current working directory; use the @samp{--outdir} option to -select a different directory. - -@need 3500 -@noindent -Here is a short sample summary log: - -@cartouche -@smallexample -Test Run By rob on Mon May 25 21:40:57 PDT 1992 - === gdb tests === -Running ./gdb.t00/echo.exp ... -PASS: Echo test -Running ./gdb.all/help.exp ... -PASS: help add-symbol-file -PASS: help aliases -PASS: help breakpoint "bre" abbreviation -FAIL: help run "r" abbreviation -Running ./gdb.t10/crossload.exp ... -PASS: m68k-elf (elf-big) explicit format; loaded -XFAIL: mips-ecoff (ecoff-bigmips) "ptype v_signed_char" signed -C types - === gdb Summary === -# of expected passes 5 -# of expected failures 1 -# of unexpected failures 1 -/usr/latest/bin/gdb version 4.6.5 -q -@end smallexample -@end cartouche - -@node Detail -@subsection Detailed log -@cindex detailed log - -@code{runtest} also saves a detailed log file @file{@var{tool}.log}, -showing any output generated by tests as well as the summary output. -For example, after @samp{runtest --tool binutils}, look for a detailed -log in @file{binutils.log}. Normally, @code{runtest} writes this file -in your current working directory; use the @samp{--outdir} option to -select a different directory. - -@need 4000 -@noindent -Here is a brief example showing a detailed log for @sc{g++} tests: - -@cartouche -@smallexample -Test Run By rob on Mon May 25 21:40:43 PDT 1992 - - === g++ tests === - ---- Running ./g++.other/t01-1.exp --- - PASS: operate delete - ---- Running ./g++.other/t01-2.exp --- - FAIL: i960 bug EOF -p0000646.C: In function `int warn_return_1 ()': -p0000646.C:109: warning: control reaches end of non-void function -p0000646.C: In function `int warn_return_arg (int)': -p0000646.C:117: warning: control reaches end of non-void function -p0000646.C: In function `int warn_return_sum (int, int)': -p0000646.C:125: warning: control reaches end of non-void function -p0000646.C: In function `struct foo warn_return_foo ()': -p0000646.C:132: warning: control reaches end of non-void function - ---- Running ./g++.other/t01-4.exp --- - FAIL: abort -900403_04.C:8: zero width for bit-field `foo' ---- Running ./g++.other/t01-3.exp --- - FAIL: segment violation -900519_12.C:9: parse error before `;' -900519_12.C:12: Segmentation violation -/usr/latest/bin/gcc: Internal compiler error: program cc1plus got -fatal signal - - === g++ Summary === - -# of expected passes 1 -# of expected failures 3 -/usr/ps/bin/g++ version cygnus-2.0.1 -@end smallexample -@end cartouche - -@node Debug -@subsection Logging @code{expect} internal actions -@cindex debug log - -With the @samp{--debug} option, you can request a log file showing the -output from @code{expect} itself, running in debugging mode. This file -(@file{dbg.log}, in the directory where you start @code{runtest}) shows -each pattern @code{expect} considers in analyzing test output. - -This file reflects each @code{send} command, showing the string sent as -input to the tool under test; and each @code{expect} command, showing -each pattern it compares with the tool output. - -The log messages for @code{expect} begin with a message of the form - -@smallexample -expect: does @{@var{tool output}@} (spawn_id @var{n}) match pattern -@{@var{expected pattern}@}? -@end smallexample - -@noindent -For every unsuccessful match, @code{expect} issues a @samp{no} after -this message; if other patterns are specified for the same -@code{expect} command, they are reflected also, but without the first -part of the message (@samp{expect@dots{}match pattern}). - -When @code{expect} finds a match, the log for the successful match ends -with @samp{yes}, followed by a record of the @code{expect} variables set -to describe a successful match. Here is an excerpt from the debugging -log for a @sc{gdb} test: - -@c FIXME! Why is the second spawn_id shown 0 rather than 6? -@cartouche -@smallexample -send: sent @{break gdbme.c:34\n@} to spawn id 6 -expect: does @{@} (spawn_id 6) match pattern @{Breakpoint.*at.* file - gdbme.c, line 34.*\(gdb\) $@}? no -@{.*\(gdb\) $@}? no -expect: does @{@} (spawn_id 0) match pattern @{<return>@}? no -@{\(y or n\) @}? no -@{buffer_full@}? no -@{virtual@}? no -@{memory@}? no -@{exhausted@}? no -@{Undefined@}? no -@{command@}? no -break gdbme.c:34 -Breakpoint 8 at 0x23d8: file gdbme.c, line 34. -(gdb) expect: does @{break gdbme.c:34\r\nBreakpoint 8 at 0x23d8: -file gdbme.c, line 34.\r\n(gdb) @} (spawn_id 6) match pattern -@{Breakpoint.*at.* file gdbme.c, line 34.*\(gdb\) $@}? yes -expect: set expect_out(0,start) @{18@} -expect: set expect_out(0,end) @{71@} -expect: set expect_out(0,string) @{Breakpoint 8 at 0x23d8: file -gdbme.c, line 34.\r\n(gdb) @} -expect: set expect_out(spawn_id) @{6@} -expect: set expect_out(buffer) @{break gdbme.c:34\r\nBreakpoint 8 -at 0x23d8: file gdbme.c, line 34.\r\n(gdb) @} - PASS: 70 0 breakpoint line number in file -@end smallexample -@end cartouche - -@noindent -This example exhibits three properties of @code{expect} and DejaGnu that -might be surprising at first glance: - -@itemize @bullet -@item -Empty output for the first attempted match. The first set of attempted -matches shown ran against the output @samp{@{@}}---that is, no output. -@code{expect} begins attempting to match the patterns supplied -immediately; often, the first pass is against incomplete output (or -completely before all output, as in this case). - -@item -Interspersed tool output. The beginning of the log entry for the second -attempted match may be hard to spot: this is because the prompt -@samp{(gdb) } appears on the same line, just before the @samp{expect:} -that marks the beginning of the log entry. - -@item -Fail-safe patterns. Many of the patterns tested are fail-safe patterns -provided by @sc{gdb} testing utilities, to reduce possible -indeterminacy. It is useful to anticipate potential variations -caused by extreme system conditions (@sc{gdb} might issue the message -@samp{virtual memory exhausted} in rare circumstances), or by changes in -the tested program (@samp{Undefined command} is the likeliest outcome if -the name of a tested command changes). - -The pattern @samp{@{<return>@}} is a particularly interesting fail-safe -to notice; it checks for an unexpected @key{RET} prompt. This may -happen, for example, if the tested tool can filter output through a -pager. - -These fail-safe patterns (like the debugging log itself) are primarily -useful while developing test scripts. Use the @code{error} procedure to -make the actions for fail-safe patterns produce messages starting with -@samp{ERROR} on the @code{runtest} standard output, and in the detailed -log file. -@end itemize - -@node Tests -@chapter How To Write a Test Case -@cindex writing a test case -@cindex test case, writing - -@menu -* Writing:: Writing a test case -* Debugging:: Debugging a test case -* Adding:: Adding a test case to a test suite -* Hints:: Hints on writing a test case -* Variables:: Special variables used by test cases -@end menu - -@node Writing -@section Writing a test case - -The easiest way to prepare a new test case is to base it on an existing -one for a similar situation. There are two major categories of tests: -batch or interactive. Batch oriented tests are usually easier to write. - -The @sc{gcc} tests are a good example of batch oriented tests. All -@sc{gcc} tests consist primarily of a call to a single common procedure, -since all the tests either have no output, or only have a few warning -messages when successfully compiled. Any non-warning output is a test -failure. All the C code needed is kept in the test directory. The test -driver, written in @code{expect}, need only get a listing of all the C -files in the directory, and compile them all using a generic procedure. -This procedure and a few others supporting for these tests are kept in -the library module @file{lib/c-torture.exp} in the @sc{gcc} test suite. -Most tests of this kind use very few @code{expect} features, and are -coded almost purely in Tcl. - -@noindent -Writing the complete suite of C tests, then, consisted of these steps: - -@enumerate -@item -@cindex Granlund, Torbjorn -@cindex C torture test -Copying all the C code into the test directory. These tests were based on -the C-torture test created by Torbjorn Granlund (on behalf of the Free -Software Foundation) for @sc{gcc} development. - -@item -Writing (and debugging) the generic @code{expect} procedures for -compilation. - -@item -Writing the simple test driver: its main task is to search the directory -(using the Tcl procedure @code{glob} for filename expansion with -wildcards) and call a Tcl procedure with each filename. It also checks -for a few errors from the testing procedure. -@end enumerate - -Testing interactive programs is intrinsically more complex. Tests for most -interactive programs require some trial and error before they are complete. - -However, some interactive programs can be tested in a simple fashion -reminiscent of batch tests. For example, prior to the creation of -DejaGnu, the @sc{gdb} distribution already included a wide-ranging -testing procedure. This procedure was very robust, and had already -undergone much more debugging and error checking than many recent -DejaGnu test cases. Accordingly, the best approach was simply to -encapsulate the existing @sc{gdb} tests, for reporting purposes. -Thereafter, new @sc{gdb} tests built up a family of @code{expect} -procedures specialized for @sc{gdb} testing. - -@file{gdb.t10/crossload.exp} is a good example of an interactive test. -@c FIXME! Check what *kind* of example it is---work-intensive, or generic... - -@node Debugging -@section Debugging a test case -@cindex debugging a test case -@cindex test case, debugging - -@noindent -These are the kinds of debugging information available from DejaGnu: - -@enumerate -@item -Output controlled by test scripts themselves, explicitly allowed for by -the test author. This kind of debugging output appears in the detailed -output recorded in the @file{@var{tool}.log} file. To do the same for -new tests, use the @code{verbose} procedure (which in turn uses the -variable also called @code{verbose}) to control how much output to -generate. This will make it easier for other people running the test to -debug it if necessary. Whenever possible, if @samp{$verbose} is -@code{0}, there should be no output other than the output from -@code{pass}, @code{fail}, @code{error}, and @code{warning}. Then, to -whatever extent is appropriate for the particular test, allow -successively higher values of @samp{$verbose} to generate more -information. Be kind to other programmers who use your tests: provide -for a lot of debugging information. - -@item -Output from the internal debugging functions of Tcl and @code{expect}. -There is a command line options for each; both forms of debugging output -are recorded in the file @code{dbg.log} in the current directory. - -Use @samp{--debug} for information from the @code{expect} level; it -generates displays of the @code{expect} attempts to match the tool -output with the patterns specified (@pxref{Debug,,Debug Log}). This -output can be very helpful while developing test scripts, since it shows -precisely the characters received. Iterating between the latest attempt -at a new test script and the corresponding @file{dbg.log} can allow you -to create the final patterns by ``cut and paste''. This is sometimes -the best way to write a test case. - -Use @samp{--strace} to see more detail at the Tcl level; this shows how Tcl -procedure definitions expand, as they execute. The associated number -controls the depth of definitions expanded; see the discussion of -@samp{--strace} in @ref{Invoking runtest,,Running the Tests}. - -@item -Finally, if the value of @samp{verbose} is 3 or greater, @code{runtest} -turns on the @code{expect} command @code{log_user}. This command prints -all @code{expect} actions to the @code{expect} standard output, to the -detailed log file, and (if @samp{--debug} is on) to @file{dbg.log}. -@end enumerate - -@node Adding -@section Adding a test case to a test suite -@cindex adding a test case - -There are two slightly different ways to add a test case. One is to add -the test case to an existing directory. The other is to create a new -directory to hold your test. The existing test directories represent -several styles of testing, all of which are slightly different; examine -the directories for the tool of interest to see which (if any) is most -suitable. - -Adding a @sc{gcc} test can be very simple: just add the C code to any -directory beginning with @samp{gcc.} and it runs on the next -@samp{runtest --tool gcc}. - -To add a test to @sc{gdb}, first add any source code you will need to -the test directory. Then you can either create a new @code{expect} file, -or add your test to an existing one (any file with a @samp{.exp} -suffix). Creating a new @samp{.exp} file is probably a better idea if -the test is significantly different from existing tests. Adding it as a -separate file also makes upgrading easier. If the C code has to be -already compiled before the test will run, then you'll have to add it to -the @file{Makefile.in} file for that test directory, then run -@code{configure} and @code{make}. - -Adding a test by creating a new directory is very similar: - -@enumerate -@item -Create the new directory. All subdirectory names begin with the name of -the tool to test; e.g. @sc{g++} tests might be in a directory called -@file{g++.other}. There can be multiple test directories that start with -the same tool name (such as @samp{g++}). - -@item -Add the new directory name to the @samp{configdirs} definition in the -@file{configure.in} file for the test suite directory. This way when -@code{make} and @code{configure} next run, they include the new directory. - -@item -Add the new test case to the directory, as above. - -@item -To add support in the new directory for configure and make, you must -also create a @code{Makefile.in} and a @code{configure.in}. @xref{What -Configure Does,,What Configure Does, configure.info, Cygnus Configure}. -@end enumerate - -@c FIXME! Expand this sentence to at least a section, maybe a chapter... -@c The @file{admin} directory contains templates for a few common forms -@c of test. - -@node Hints -@section Hints on writing a test case -@cindex hints on test case writing - -There may be useful existing procedures already written for your test in -the @file{lib} directory of the DejaGnu distribution. @xref{DejaGnu -Builtins,,DejaGnu Builtins}. - -It is safest to write patterns that match @emph{all} the output -generated by the tested program; this is called @dfn{closure}. If a -pattern does not match the entire output, any output that remains will -be examined by the @emph{next} @code{expect} command. In this -situation, the precise boundary that determines which @code{expect} -command sees what is very sensitive to timing between the @code{expect} -task and the task running the tested tool. As a result, the test may -sometimes appear to work, but is likely to have unpredictable results. -(This problem is particularly likely for interactive tools, but can also -affect batch tools---especially for tests that take a long time to finish.) -The best way to ensure closure is to use the @samp{-re} option for the -@code{expect} command to write the pattern as a full regular -expressions; then you can match the end of output using a @samp{$}. It -is also a good idea to write patterns that match all available output by -using @samp{.*\} after the text of interest; this will also match any -intervening blank lines. Sometimes an alternative is to match end of -line using @samp{\r} or @samp{\n}, but this is usually too dependent on -terminal settings. -@c FIXME!! explain what "end of output" means for interactive task. -@c (Timeout or EOF, right?) - -Always escape punctuation, such as @samp{(} or @samp{"}, in your -patterns; for example, write @samp{\(}. If you forget to escape -punctuation, you will usually see an error message like @samp{extra -characters after close-quote}. - -If you have trouble understanding why a pattern does not match the -program output, try using the @samp{--debug} option to @code{runtest}, -and examine the debug log carefully. @xref{Debug,,Debug Log}. - -Be careful not to neglect output generated by setup rather than by the -interesting parts of a test case. For example, while testing @sc{gdb}, -I issue a send @samp{set height 0\n} command. The purpose is simply to -make sure @sc{gdb} never calls a paging program. The @samp{set height} -command in @sc{gdb} does not generate any output; but running @emph{any} -command makes @sc{gdb} issue a new @samp{(gdb) } prompt. If there were -no @code{expect} command to match this prompt, the output @samp{(gdb) } -begins the text seen by the next @code{expect} command---which might -make @emph{that} pattern fail to match. - -To preserve basic sanity, I also recommended that no test ever pass if -there was any kind of problem in the test case. To take an extreme -case, tests that pass even when the tool will not spawn are misleading. -Ideally, a test in this sort of situation should not fail either. -Instead, print an error message by calling one of the DejaGnu procedures -@code{error} or @code{warning}. - -@node Variables -@section Special variables used by test cases -@cindex special variables - -@cindex variables for all tests -Your test cases can use these variables, with conventional meanings (as -well as the variables saved in @file{site.exp} -@pxref{Customizing,,Setting @code{runtest} defaults}): - -@quotation -@emph{These variables are available to all test cases.} -@end quotation - -@ftable @code -@item prms_id -@cindex PRMS bug number -@cindex GNATS bug number -@cindex bug number -The tracking system (e.g. @sc{gnats}) number identifying a corresponding -bugreport. (@samp{0} if you do not specify it in the test script.) - -@item bug_id -@cindex bug number, extra -An optional bug id; may reflect a bug identification from another -organization. (@samp{0} if you do not specify it.) - -@item subdir -@cindex current test subdirectory -The subdirectory for the current test case. -@end ftable - -@quotation -@emph{These variables should never be changed. They appear in most -tests.} -@end quotation - -@ftable @code -@item expect_out(buffer) -@cindex last command output -The output from the last command. This is an internal variable set by -@code{expect}. - -@item exec_output -This is the output from a @code{@var{tool}_load} command. This only -applies to tools like @sc{gcc} and @sc{gas} which produce an object -file that must in turn be executed to complete a test. - -@item comp_output -This is the output from a @code{@var{tool}_start} command. This is -conventionally used for batch oriented programs, like @sc{gcc} and -@sc{gas}, that may produce interesting output (warnings, errors) without -further interaction. -@end ftable - -@node Extending -@chapter New Tools, Targets, or Hosts - -The most common ways to extend the DejaGnu framework are: adding a suite -of tests for a new tool to be tested; adding support for testing on a -new target; and porting @code{runtest} to a new host. - -@menu -* Adding Tools:: How to add tests for a new tool -* Adding Targets:: How to add a new target -* Porting:: Porting DejaGnu to a new host -@end menu - -@node Adding Tools -@section Writing tests for a new tool - -In general, the best way to learn how to write (code or even prose) is -to read something similar. This principle applies to test cases and to -test suites. Unfortunately, well-established test suites have a way of -developing their own conventions: as test writers become more -experienced with DejaGnu and with Tcl, they accumulate more utilities, -and take advantage of more and more features of @code{expect} and Tcl in -general. - -Inspecting such established test suites may make the prospect of -creating an entirely new test suite appear overwhelming. Nevertheless, -it is quite straightforward to get a new test suite going. - -@cindex Lupton, Robert -There is one test suite that is guaranteed not to grow more elaborate -over time: both it and the tool it tests were created expressly to -illustrate what it takes to get started with DejaGnu. The -@file{example/} directory of the DejaGnu distribution contains both an -interactive tool called @code{calc}, and a test suite for it. Reading -this test suite, and experimenting with it, is a good way to supplement -the information in this section. (Thanks to Robert Lupton for creating -@code{calc} and its test suite---and also the first version of this -section of the manual!) - -To help orient you further in this task, here is an outline of the steps -to begin building a test suite for a program @var{example}. - -@enumerate -@item -Create or select a directory to contain your new collection of tests. -Change to that directory (shown here as @code{testsuite}): - -@example -eg$ cd testsuite/ -@end example - -@item -Create a @file{configure.in} file in this directory, to control -configuration-dependent choices for your tests. So far as DejaGnu is -concerned, the important thing is to set a value for the variable -@code{target_abbrev}; this value is the link to the init file you will -write soon. (For simplicity, we assume the environment is Unix, and use -@samp{unix} as the value.) - -What else is needed in @file{configure.in} depends on the requirements -of your tool, your intended test environments, and which -@code{configure} system you use. This example is a minimal -@code{configure.in} for use with Cygnus Configure. (For an alternative -based on the FSF @code{autoconf} system, see the @code{calc} example -distributed with DejaGnu.) Replace @var{example} with the name of your -program: - -@cartouche -@smallexample -# This file is a shell script fragment -# for use with Cygnus configure. - -srctrigger="@var{example}.0" -srcname="The DejaGnu @var{example} tests" - -# per-host: - -# per-target: - -# everything defaults to unix for a target -target_abbrev=unix - -# post-target: - -@end smallexample -@end cartouche - -@item -Create @file{Makefile.in}, the source file used by @code{configure} to -build your @file{Makefile}. Its leading section should as usual contain -the values that @code{configure} may override: - -@cartouche -@smallexample -srcdir = . -prefix = /usr/local - -exec_prefix = $(prefix) -bindir = $(exec_prefix)/bin -libdir = $(exec_prefix)/lib -tooldir = $(libdir)/$(target_alias) - -datadir = $(exec_prefix)/lib/dejagnu - -RUNTEST = runtest -RUNTESTFLAGS = -FLAGS_TO_PASS = - -#### host, target, site specific Makefile frags come in here. -@end smallexample -@end cartouche - -This should be followed by the standard targets at your site. To begin -with, they need not do anything---for example, these definitions will -do: - -@cartouche -@smallexample - -all: - -info: - -install-info: - -install: -uninstall: - -clean: - -rm -f *~ core *.info* - -@end smallexample -@end cartouche - -It is also a good idea to make sure your @file{Makefile} can rebuild -itself if @file{Makefile.in} changes, with a target like this (which -works for either Cygnus or FSF Configure): - -@cartouche -@smallexample -Makefile : $(srcdir)/Makefile.in $(host_makefile_frag) \ - $(target_makefile_frag) - $(SHELL) ./config.status -@end smallexample -@end cartouche - -You also need to include two targets important to DejaGnu: @code{check}, -to run the tests, and @code{site.exp}, to set up the Tcl copies of -configuration-dependent values. The @code{check} target must run -@samp{runtest --tool @var{example}}: - -@cartouche -@smallexample -check: site.exp all - $(RUNTEST) $(RUNTESTFLAGS) $(FLAGS_TO_PASS) \ - --tool @var{example} --srcdir $(srcdir) -@end smallexample -@end cartouche - -The @code{site.exp} target should usually set up (among other things!) a -Tcl variable for the name of your program: - -@cartouche -@smallexample -site.exp: ./config.status Makefile - @@echo "Making a new config file..." - -@@rm -f ./tmp? - @@touch site.exp - - -@@mv site.exp site.bak - @@echo "## these variables are automatically\ - generated by make ##" > ./tmp0 - @@echo "# Do not edit here. If you wish to\ - override these values" >> ./tmp0 - @@echo "# add them to the last section" >> ./tmp0 - @@echo "set host_os $@{host_os@}" >> ./tmp0 - @@echo "set host_alias $@{host_alias@}" >> ./tmp0 - @@echo "set host_cpu $@{host_cpu@}" >> ./tmp0 - @@echo "set host_vendor $@{host_vendor@}" >> ./tmp0 - @@echo "set target_os $@{target_os@}" >> ./tmp0 - @@echo "set target_alias $@{target_alias@}" >> ./tmp0 - @@echo "set target_cpu $@{target_cpu@}" >> ./tmp0 - @@echo "set target_vendor $@{target_vendor@}" >> ./tmp0 - @@echo "set host_triplet $@{host_canonical@}" >> ./tmp0 - @@echo "set target_triplet $@{target_canonical@}">>./tmp0 - @@echo "set tool binutils" >> ./tmp0 - @@echo "set srcdir $@{srcdir@}" >> ./tmp0 - @@echo "set objdir `pwd`" >> ./tmp0 - @@echo "set @var{examplename} @var{example}" >> ./tmp0 - @@echo "## All variables above are generated by\ - configure. Do Not Edit ##" >> ./tmp0 - @@cat ./tmp0 > site.exp - @@sed < site.bak \ - -e '1,/^## All variables above are.*##/ d' \ - >> site.exp - -@@rm -f ./tmp? -@end smallexample -@end cartouche - -@item -Create a directory (in @file{testsuite/}) called @file{config/}: - -@example -eg$ mkdir config -@end example - -@item -Make an init file in this directory; its name must start with the -@code{target_abbrev} value, so call it @file{config/unix.exp}. -This is the file that contains the target-dependent procedures; -fortunately, most of them do not have to do very much in order for -@code{runtest} to run. - -If @var{example} is not interactive, you can get away with this minimal -@file{unix.exp} to begin with: - -@cartouche -@smallexample -proc foo_exit @{@} @{@} -proc foo_version @{@} @{@} -@end smallexample -@end cartouche - -If @var{example} is interactive, however, you might as well define a -start routine @emph{and invoke it} by using an init file like this: - -@cartouche -@smallexample -proc foo_exit @{@} @{@} -proc foo_version @{@} @{@} - -proc foo_start @{@} @{ - global @var{examplename} - spawn $@var{examplename} - expect @{ - -re "" @{@} - @} -@} -foo_start -@end smallexample -@end cartouche - -@item -Create a directory whose name begins with your tool's name, to contain -tests: - -@example -eg$ mkdir @var{example}.0 -@end example - -@item -Create a sample test file in @file{@var{example}.0}. Its name must end -with @samp{.exp}; you can use @samp{first-try.exp} To begin with, just -write there a line of Tcl code to issue a message: - -@cartouche -@smallexample -send_user "Testing: one, two...\n" -@end smallexample -@end cartouche - -@item -Back in the @file{testsuite/} (top level) directory, run - -@example -eg$ configure -@end example - -(You may have to specify more of a path, if a suitable @code{configure} -is not available in your execution path.) - -@item -You are now ready to triumphantly type @samp{make check} or -@samp{runtest --tool @var{example}}. You should see something like this: - -@cartouche -@smallexample -Test Run By rhl on Fri Jan 29 16:25:44 EST 1993 - - === @var{example} tests === - -Running ./@var{example}.0/first-try.exp ... -Testing: one, two... - - === @var{example} Summary === - -@end smallexample -@end cartouche - -There is no output in the summary, because so far the example does not -call any of the procedures that establish a test outcome. - -@item -Begin writing some real tests. For an interactive tool, you should -probably write a real exit routine in fairly short order; in any case, -you should also write a real version routine soon. -@end enumerate - -@node Adding Targets -@section Adding a target -@cindex adding a target - -DejaGnu has some additional requirements for target support, beyond the -general-purpose provisions of Cygnus @code{configure}. @code{runtest} -must actively communicate with the target, rather than simply generating -or managing code for the target architecture. Therefore, each tool -requires an initialization module for each target. For new targets, you -must supply a few Tcl procedures to adapt DejaGnu to the target. This -permits DejaGnu itself to remain target independent. @xref{Init -Module,,Initialization module}, for a discussion of the naming -conventions that enable DejaGnu to locate and use init files. - -Usually the best way to write a new initialization module is to edit an -existing initialization module; some trial and error will be required. -If necessary, you can use the @samp{--debug} option to see what -is really going on. - -When you code an initialization module, be generous in printing -information controlled by the @code{verbose} procedure (@pxref{DejaGnu -Builtins, DejaGnu procedures}). - -Most of the work is in getting the communications right. Communications -code (for several situations involving IP networks or serial lines) is -available in a DejaGnu library file, @file{lib/remote.exp}. -@xref{DejaGnu Builtins,,DejaGnu Builtins}. - -@c FIXME! Say something about Tcl debugger here. -If you suspect a communication problem, try running the connection -interactively from @code{expect}. (There are three ways of running -@code{expect} as an interactive interpreter. You can run @code{expect} -with no arguments, and control it completely interactively; or you can -use @samp{expect -i} together with other command-line options and -arguments; or you can run the command @code{interpreter} from any -@code{expect} procedure. Use @code{return} to get back to the calling -procedure (if any), or @code{return -tcl} to make the calling procedure -itself return to its caller; use @code{exit} or end-of-file to leave -@code{expect} altogether.) Run the program whose name is recorded in -@samp{$connectmode}, with the arguments in @samp{$targetname}, to -establish a connection. You should at least be able to get a prompt -from any target that is physically connected. - -@node Porting -@section Porting to a new host -@cindex porting to a new host - -The task of porting DejaGnu is basically that of porting Tcl and -@code{expect}. Tcl and @code{expect}, as distributed with DejaGnu, both -use @code{autoconf}; they should port automatically to most Unix -systems. - -Once Tcl and @code{expect} are ported, DejaGnu should run. Most system -dependencies are taken care of by using @code{expect} as the main -command shell. - -@node Installation -@appendix Installing DejaGnu - -@cindex host, explained -@cindex target, explained -@cindex DejaGnu configuration -@cindex configuring DejaGnu -Once you have the DejaGnu source unpacked and available, you must first -configure the software to specify where it is to run (and the associated -defaults); then you can proceed to installing it. - -@menu -* Configuring DejaGnu:: -* Installing DejaGnu:: -@end menu - -@node Configuring DejaGnu -@section Configuring the DejaGnu test driver - -It is usually best to configure in a directory separate -from the source tree, specifying where to find the source with the -optional @samp{--srcdir} option to @code{configure}. DejaGnu uses the -GNU @code{autoconf} to configure itself. For more info on using -autoconf, read the GNU autoconf manual. To configure, execute the -@file{configure} program, no other options are required. For an example, -to configure in a seperate tree for objects, execute the configure -script from the source tree like this: - -@smallexample -../dejagnu-1.3/configure -@end smallexample - -DejaGnu doesn't care at config time if it's for testing a native system -or a cross system. That is determined at runtime by using the config -files. - -@cindex @code{prefix}, configure options -@cindex @code{exec_prefix}, configure options. -You may also want to use the @code{configure} option @samp{--prefix} to -specify where you want DejaGnu and its supporting code installed. By -default, installation is in subdirectories of @file{/usr/local}, but you -can select any alternate directory @var{altdir} by including -@samp{--prefix=@var{altdir}} on the @code{configure} command line. -(This value is captured in the Makefile variables @code{prefix} -and @code{exec_prefix}.) - -@cindex auxiliary programs -@cindex test suite distributions -@cindex @code{make} builds part of tests -Save for a small number of example tests, the DejaGnu distribution -itself does not include any test suites; these are available separately. -Test suites for the @sc{gnu} compiler (testing both GCC and G++) and for -the @sc{gnu} binary utilities are distributed in parallel with the -DejaGnu distribution (but packaged as separate files). The test suite -for the @sc{gnu} debugger is distributed in parallel with each release -of GDB itself, starting with GDB 4.9. After configuring the top-level -DejaGnu directory, unpack and configure the test directories for the -tools you want to test; then, in each test directory, run @code{make} to -build auxiliary programs required by some of the tests. - -@node Installing DejaGnu -@section Installing DejaGnu - -@cindex installing DejaGnu -To install DejaGnu in your filesystem (either in @file{/usr/local}, or -as specified by your @samp{--prefix} option to @code{configure}), execute - -@example -eg$ make install -@end example - -@noindent -@samp{make install} does these things for DejaGnu: - -@enumerate -@item -Look in the path specified for executables (@file{$exec_prefix}) for -directories called @file{lib} and @file{bin}. If these directories do -not exist, @samp{make install} creates them. - -@item -Create another directory in the @file{lib} directory, called -@file{dejagnu}. - -@item -Copy the @code{runtest} shell script into @file{$exec_prefix/bin}. - -@item -Copy all the library files (used to support the framework) into -@file{$exec_prefix/lib/dejagnu}. - -@item -Copy @file{runtest.exp} into @file{$exec_prefix/lib/dejagnu}. This is -the main Tcl code implementing DejaGnu. - -@end enumerate - -Each test suite collection comes with simple installation instructions -in a @file{README} file; in general, the test suites are designed to be -unpacked in the source directory for the corresponding tool, and extract -into a directory called @file{testsuite}. - -@node Index -@unnumbered Index - -@printindex cp - -@contents - -@bye |