diff options
author | Ben Elliston <bje@gnu.org> | 2003-08-16 06:31:50 +0000 |
---|---|---|
committer | Ben Elliston <bje@gnu.org> | 2003-08-16 06:31:50 +0000 |
commit | 730d3ca912ec9d19d42285799beaf56a1c0d2252 (patch) | |
tree | 6b07272f8b88b29573bead20054d5d4c77e9a6df | |
parent | ba34cc87f7949afbd0150353fca324261a477574 (diff) | |
download | dejagnu-730d3ca912ec9d19d42285799beaf56a1c0d2252.zip dejagnu-730d3ca912ec9d19d42285799beaf56a1c0d2252.tar.gz dejagnu-730d3ca912ec9d19d42285799beaf56a1c0d2252.tar.bz2 |
* doc/overview.sgml (overview): Start overhauling chapter.
* doc/README.Writers: New document on style conventions.
* doc/ref.sgml: Replace "test suite" with "testsuite" throughout.
* doc/user.sgml: Likewise.
* doc/overview.sgml: Likewise.
* doc/dejagnu.texi (Design Goals): Change "Deja Gnu" to "DejaGnu".
-rw-r--r-- | ChangeLog | 10 | ||||
-rw-r--r-- | doc/README.Writers | 5 | ||||
-rw-r--r-- | doc/dejagnu.texi | 2 | ||||
-rw-r--r-- | doc/overview.sgml | 157 | ||||
-rw-r--r-- | doc/ref.sgml | 6 | ||||
-rw-r--r-- | doc/user.sgml | 46 |
6 files changed, 120 insertions, 106 deletions
@@ -1,3 +1,13 @@ +2003-08-16 Ben Elliston <bje@wasabisystems.com> + + * doc/overview.sgml (overview): Start overhauling chapter. + + * doc/README.Writers: New document on style conventions. + * doc/ref.sgml: Replace "test suite" with "testsuite" throughout. + * doc/user.sgml: Likewise. + * doc/overview.sgml: Likewise. + * doc/dejagnu.texi (Design Goals): Change "Deja Gnu" to "DejaGnu". + 2003-08-12 Ben Elliston <bje@wasabisystems.com> * Makefile.am (tarball): Update path to dejagnu.spec. diff --git a/doc/README.Writers b/doc/README.Writers new file mode 100644 index 0000000..4f49612 --- /dev/null +++ b/doc/README.Writers @@ -0,0 +1,5 @@ +Conventions: + + * Use DejaGnu, not "Dejagnu" or "Deja Gnu". + * Use "testsuite", not "test suite". + * Refer to the Tcl, Tk and Expect packages using proper capitalisation. diff --git a/doc/dejagnu.texi b/doc/dejagnu.texi index 47443a1..ef433b9 100644 --- a/doc/dejagnu.texi +++ b/doc/dejagnu.texi @@ -407,7 +407,7 @@ DejaGnu can use @code{rsh}, @code{rlogin}, @code{telnet}, @code{tip}, @cindex name ``DejaGnu'' @cindex DejaGnu, the name @cindex Menapace, Julia -Julia Menapace first coined the term ``Deja Gnu'' to describe an earlier +Julia Menapace first coined the term ``DejaGnu'' to describe an earlier testing framework at Cygnus Support. When we replaced it with the Expect-based framework, it was like DejaGnu all over again@dots{} diff --git a/doc/overview.sgml b/doc/overview.sgml index df5bfc1..e1bc35c 100644 --- a/doc/overview.sgml +++ b/doc/overview.sgml @@ -118,22 +118,22 @@ into another language, under the above conditions for modified versions. <preface id=preface> <title>Abstract</title> - <para>This document attempts to describe the functionality of - DejaGnu, the GNU Testing Framework. DejaGnu is entirely written in + <para>This document describes the functionality of DejaGnu, the + testing framework of the GNU project. DejaGnu is written in <productname>Expect</productname>, which uses <productname>Tcl</productname> as a command - language. <productname>Expect</productname> serves as a very - programmable shell; you can run any program, as with the usual - Unix command shells---but once the program is started, your - test script has fully programmable control of - its input and output. This does not just apply to the programs - under test; <command>expect</command> can also run any auxiliary - program, such as <command>diff</command> or <command>sh</command>, - with full control over its input and output.</para> - - <para>DejaGnu itself is merely a framework for creation of a test - suites. Test suites are distributed separately for each GNU - tool.</para> + language. <productname>Expect</productname> acts as a very + programmable shell. As with other Unix command shells, you can + run any program, but once the program is started, your test script + has programmable control over its input and output. This does not + just apply to the programs under test; <command>expect</command> + can also run any auxiliary program, such as + <command>diff</command> or <command>sh</command>, with full + control over its input and output.</para> + + <para>DejaGnu itself is merely a framework for the creation of + testsuites. Testsuites are distributed with each + application.</para> </preface> @@ -149,16 +149,25 @@ into another language, under the above conditions for modified versions. Tcl procedures crafted to support writing a test harness. A <emphasis>Test Harness</emphasis> is the testing infrastructure that is created to support a specific program - or tool. Each program can have multiple test suites, all + or tool. Each program can have multiple testsuites, all supported by a single test harness. DejaGnu is written in <productname>Expect</productname>, which in turn uses <productname>Tcl</productname> -- Tool command language. There is more information on Tcl at the <ulink - URL="http://www.scriptics.com">Scriptics</ulink> web site, and the + URL="http://www.scriptics.com">Scriptics</ulink> web site and the Expect web site is at <ulink URL="http://expect.nist.gov">NIST</ulink>.</para> - - <para>DejaGnu offers several advantages for testing:</para> + + <para>Julia Menapace first coined the term ``DejaGnu'' to describe + an earlier testing framework at Cygnus Support she had written + for <command>GDB</command>. When we replaced it with the + Expect-based framework, it was like DejaGnu all over again. + More importantly, it was also named after my daughter, <ulink + URL="mailto:deja@welcomehome.org">Deja Snow Savoye</ulink> + (now 13 years old as of September 2003), who was a toddler + during DejaGnu's beginnings.</para> + + <para>DejaGnu offers several advantages for testing:</para> <itemizedlist mark="bullet" spacing="compact"> @@ -170,47 +179,37 @@ into another language, under the above conditions for modified versions. <listitem><para>DejaGnu provides a layer of abstraction which allows you to write tests that are portable to any host or target where a program must be tested. For instance, a test - for <command>GDB</command> can run (from any Unix - based host) on any target architecture that DejaGnu - supports. Currently DejaGnu runs tests on many single board - computers, whose operating software ranges from just a boot - monitor to a full-fledged, Unix-like realtime OS.</para> - </listitem> + for <command>GDB</command> can run from any supported host + system on any supported target system. DejaGnu runs tests on + many single board computers, whose operating software ranges + from a simple boot monitor to a real-time OS.</para> + </listitem> <listitem><para>All tests have the same output format. This makes it easy to integrate testing into other software development processes. DejaGnu's output is designed to be - parsed by other filtering script, and it is also human + parsed by other filtering script and it is also human readable.</para> </listitem> - <listitem><para>Using Tcl and expect, it's easy to create wrappers - for existing test suites. By incorporating existing tests under + <listitem><para>Using Tcl and Expect, it's easy to create wrappers + for existing testsuites. By incorporating existing tests under DejaGnu, it's easier to have a single set of report analyse programs..</para> </listitem> </itemizedlist> - <para>Running tests requires two things: the testing framework, and - the test suites themselves. Tests are usually written in - <productname>Expect</productname> using Tcl, but you can also use a - Tcl script to run a test suite that is not based on - <productname>Expect</productname>. - (<productname>expect</productname> script filenames conventionally - use <emphasis>.exp</emphasis> as a suffix; for example, the main - implementation of the DejaGnu test driver is in the file + <para>Running tests requires two things: the testing framework and + the testsuites themselves. Tests are usually written in + <productname>Expect</productname> using Tcl, but you can also use + a Tcl script to run a testsuite that is not based on + <productname>Expect</productname>. <productname>Expect</productname> + script filenames conventionally use <emphasis>.exp</emphasis> as a + suffix; for example, the main implementation of the DejaGnu test + driver is in the file <productname>runtest.exp</productname>.)</para> - <para>Julia Menapace first coined the term ``Deja Gnu'' to describe an - earlier testing framework at Cygnus Support she had written for - <command>GDB</command>. When we replaced it with the Expect-based - framework, it was like DejaGnu all over again. More importantly, it - was also named after my daughter,<ulink - URL="mailto:deja@welcomehome.org">Deja Snow Savoye</ulink> (now 12 - years old in Sept of 2002), who was a toddler during DejaGnu's - creation.</para> - </sect1> <sect1 id=new xreflabel="Release Notes"> @@ -227,7 +226,7 @@ into another language, under the above conditions for modified versions. are:</para> <itemizedlist> - <listitem><para>More builtin support for building target binaries + <listitem><para>More built-in support for building target binaries with the correct linker flags. Currently this only works with <productname>GCC</productname> as the cross compiler, preferably with a target supported by @@ -258,7 +257,7 @@ into another language, under the above conditions for modified versions. <para>To use DejaGnu on Windows, you need to first install the <ulink URL="http://www.cygwin.com/">Cygwin</ulink> release. This works as of the B20.1 release. Cygwin is a POSIX - system for Windows. This covers both utility programs, and a libray + system for Windows. This covers both utility programs and a library that adds POSIX system calls to Windows. Among them is pseudo tty support for Windows that emulates the POSIX pty standard. The latest Cygwin is always available from <ulink @@ -275,20 +274,20 @@ into another language, under the above conditions for modified versions. <sect1 id=designgoals xreflabel="Design Goals"> <title>Design Goals</title> - <para>DejaGnu grew out of the internal needs of Cygnus Solutions. (then - Cygnus Support). Cygnus maintained and enhanced a variety of free - programs in many different environments, and we needed a testing - tool that:</para> + <para>DejaGnu grew out of the internal needs of Cygnus Solutions, + the company formerly known as Cygnus Support. Cygnus maintained + and enhanced a variety of free programs in many different + environments and we needed a testing tool that:</para> <itemizedlist mark="bullet"> <listitem><para>was useful to developers while fixing - bugs.</para></listitem> + bugs;</para></listitem> <listitem><para>automated running many tests during a software - release process.</para></listitem> + release process;</para></listitem> <listitem><para>was portable among a variety of host - computers.</para></listitem> + computers;</para></listitem> <listitem><para>supported cross-development - testing.</para></listitem> + testing;</para></listitem> <listitem><para>permitted testing interactive programs, like <command>GDB</command>; and </para></listitem> <listitem><para>permitted testing batch oriented programs, like @@ -301,18 +300,18 @@ into another language, under the above conditions for modified versions. make sure that <command>GDB</command> works as well when cross-debugging as it does in a native configuration. </para> - <para>Probably the greatest challenge was testing in a cross-development - environment (which can be a real nightmare). Most cross-development - environments are customized by each developer. Even when buying packaged - boards from vendors there are many differences. The communication - interfaces vary from a serial line to ethernet. DejaGnu was designed with - a modular communication setup, so that each kind of communication can be - added as required, and supported thereafter. Once a communication - procedure is coded, any test can use it. Currently DejaGnu can use - <command>rsh</command>, <command>rlogin</command>, - <command>telnet</command>, <command>tip</command>, - <command>kermit</command>, and <command>mondfe</command> for remote - communications.</para> + <para>Probably the greatest challenge was testing in a + cross-development environment. Most cross-development + environments are customized by each developer. Even when buying + packaged boards from vendors there are many differences. The + communication interfaces vary from a serial line to Ethernet. + DejaGnu was designed with a modular communication setup, so that + each kind of communication can be added as required and supported + thereafter. Once a communication procedure is coded, any test can + use it. Currently DejaGnu can use <command>rsh</command>, + <command>rlogin</command>, <command>telnet</command>, + <command>tip</command>, <command>kermit</command> and + <command>mondfe</command> for remote communications.</para> </sect1> @@ -320,7 +319,7 @@ into another language, under the above conditions for modified versions. <title>A POSIX conforming test framework</title> <para>DejaGnu conforms to the POSIX 1003.3 standard for test - frameworks. I was also a member of that committe.</para> + frameworks. Rob Savoye was a member of that committee.</para> <para>The POSIX standard 1003.3 defines what a testing framework needs to provide, in order to permit the creation of POSIX conformance test @@ -329,23 +328,23 @@ into another language, under the above conditions for modified versions. to POSIX conformance. POSIX 1003.3 does not specify a particular testing framework, but at this time there is only one other POSIX conforming test framework: TET. TET was created by Unisoft for a consortium comprised of - X/Open, Unix International, and the Open Software Foundation.</para> + X/Open, Unix International and the Open Software Foundation.</para> <para>The POSIX documentation refers to <firstterm>assertions</firstterm>. An assertion is a description of behavior. For example, if a standard says ``The sun shall shine'', a corresponding assertion might be ``The sun is shining.'' A test based on this assertion would pass or fail - depending on whether it is daytime or nighttime. It is important to note + depending on whether it is day or night. It is important to note that the standard being tested is never 1003.3; the standard being tested is some other standard, for which the assertions were written.</para> - <para>As there is no test suite to test testing frameworks for POSIX + <para>As there is no testsuite to test testing frameworks for POSIX 1003.3 conformance, verifying conformance to this standard is done by repeatedly reading the standard and experimenting. One of the main - things 1003.3 does specify is the set of allowed output messages, and + things 1003.3 does specify is the set of allowed output messages and their definitions. Four messages are supported for a required feature of - POSIX conforming systems, and a fifth for a conditional feature. DejaGnu - supports the use of all five output messages; in this sense a test suite + POSIX conforming systems and a fifth for a conditional feature. DejaGnu + supports the use of all five output messages. In this sense a testsuite that uses exactly these messages can be considered POSIX conforming. These definitions specify the output of a test case:</para> @@ -393,9 +392,9 @@ into another language, under the above conditions for modified versions. code. If the test isn't actually run, it must produce <emphasis>UNRESOLVED</emphasis> rather than just leaving that test out of the output. This means that you have to be careful when - writing tests, to not carelessly use tcl statements like + writing tests to not carelessly use Tcl commands like <emphasis>return</emphasis>---if you alter the flow of control of the - tcl code you must insure that every test still produces some result + Tcl code you must insure that every test still produces some result code.</para> <para>Here are some of the ways a test may wind up @@ -429,7 +428,7 @@ into another language, under the above conditions for modified versions. <variablelist> <varlistentry> <term>UNTESTED</term> - <listitem><para>A test was not run. This is a placeholder, used + <listitem><para>A test was not run. This is a place-holder, used when there is no real test case yet.</para></listitem> </varlistentry> </variablelist> @@ -453,10 +452,10 @@ into another language, under the above conditions for modified versions. </variablelist> <para>DejaGnu uses the same output procedures to produce these messages - for all test suites, and these procedures are already known to conform - to POSIX 1003.3. For a DejaGnu test suite to conform to POSIX 1003.3, + for all testsuites and these procedures are already known to conform + to POSIX 1003.3. For a DejaGnu testsuite to conform to POSIX 1003.3, you must avoid the <emphasis>setup</emphasis>xfail} procedure as - described in the <emphasis>PASS</emphasis> section above, and you must + described in the <emphasis>PASS</emphasis> section above and you must be careful to return <emphasis>UNRESOLVED</emphasis> where appropriate, as described in the <emphasis>UNRESOLVED</emphasis> section above.</para> diff --git a/doc/ref.sgml b/doc/ref.sgml index ed642b0..dc9686d 100644 --- a/doc/ref.sgml +++ b/doc/ref.sgml @@ -51,8 +51,8 @@ <emphasis>exec</emphasis>prefix}.)</para> <para>Save for a small number of example tests, the DejaGnu distribution - itself does not include any test suites; these are available - separately. Test suites for the GNU development tools are included in + itself does not include any testsuites; these are available + separately. Testsuites for the GNU development tools are included in those releases. After configuring the top-level DejaGnu directory, unpack and configure the test directories for the tools you want to test; then, in each test directory, run <emphasis>make check</emphasis> to build @@ -3161,7 +3161,7 @@ and <symbol>${tool}_load</symbol>}, are only called by the test suites themselves (or by testsuite-specific initialization code); they may take arguments or not, depending on the - conventions used within each test suite.</para> + conventions used within each testsuite.</para> <para>The usual convention for return codes from any of these procedures (although it is not required by diff --git a/doc/user.sgml b/doc/user.sgml index cfafca6..1a4c7f2 100644 --- a/doc/user.sgml +++ b/doc/user.sgml @@ -711,7 +711,7 @@ powerpc-linux-gcc -g -O2 -o calc calc.o <chapter id=runningtests> <title>Running Tests</title> - <para>There are two ways to execute a test suite. The most + <para>There are two ways to execute a testsuite. The most common way is when there is existing support in the <filename>Makefile</filename>. This support consists of a <emphasis>check</emphasis> target. The other way is to execute the @@ -812,7 +812,7 @@ powerpc-linux-gcc -g -O2 -o calc calc.o <varlistentry> <term>UNRESOLVED</term> <listitem><para>Output from a test requires manual inspection; the - test suite could not automatically determine the outcome. For + testsuite could not automatically determine the outcome. For example, your tests can report this outcome is when a test does not complete as expected.</para></listitem> </varlistentry> @@ -844,7 +844,7 @@ powerpc-linux-gcc -g -O2 -o calc calc.o <listitem><para>Indicates a major problem (detected by the test case itself) in running the test. This is usually an unrecoverable error, such as a missing file or loss of communication to the target. (POSIX - test suites should not emit this message; use + testsuites should not emit this message; use <emphasis>UNSUPPORTED</emphasis>, <emphasis>UNTESTED</emphasis>, or <emphasis>UNRESOLVED</emphasis> instead, as appropriate.)</para></listitem> @@ -1054,7 +1054,7 @@ powerpc-linux-gcc -g -O2 -o calc calc.o <varlistentry id=tool-opt> <term><option>--tool[name(s)]</option></term> - <listitem><para>Specifies which test suite to run, and what + <listitem><para>Specifies which testsuite to run, and what initialization module to use. <option>--tool</option> is used <emphasis>only</emphasis> for these two purposes. It is <emphasis>not</emphasis> used to name the executable program to @@ -1163,7 +1163,7 @@ powerpc-linux-gcc -g -O2 -o calc calc.o <para>Typically, you don't need must to use any command-line options. <option>--tool</option> used is only required when there are more than - one test suite in the same directory. The default options are in the + one testsuite in the same directory. The default options are in the local site.exp file, created by "make site.exp".</para> <para>For example, if the directory <filename>gdb/testsuite</filename> @@ -1465,7 +1465,7 @@ powerpc-linux-gcc -g -O2 -o calc calc.o DejaGnu test environment using Tcl variables. This ties the DejaGnu test scripts into the <command>configure</command> and <command>make</command> programs. If this file is setup correctly, - it is possible to execute a test suite merely by typing + it is possible to execute a testsuite merely by typing <command>runtest</command>.</para> <para>DejaGnu supports two <filename>site.exp</filename> @@ -1487,7 +1487,7 @@ powerpc-linux-gcc -g -O2 -o calc calc.o <symbol>DEJAGNU</symbol> to the name of the file. This is also refered to as the ``global'' config file.</para> - <para>Any directory containing a configured test suite also has a + <para>Any directory containing a configured testsuite also has a local <filename>site.exp</filename>, capturing configuration values specific to the tool under test. Since <command>runtest</command> loads these values last, the individual test configuration can @@ -1496,7 +1496,7 @@ powerpc-linux-gcc -g -O2 -o calc calc.o <para>You can usually generate or update the testsuite's local <filename>site.exp</filename> by typing <command>make - site.exp</command> in the test suite directory, after the test + site.exp</command> in the testsuite directory, after the test suite is configured.</para> <para>You can also have a file in your home directory called @@ -1519,7 +1519,7 @@ powerpc-linux-gcc -g -O2 -o calc calc.o local to each test directory, rather than in the global <filename>site.exp</filename> in the installed DejaGnu library. This file is mostly for supplying tool specific info - that is required by the test suite.</para> + that is required by the testsuite.</para> <para>All local <filename>site.exp</filename> files have two sections, separated by comment text. The first section is @@ -1601,7 +1601,7 @@ powerpc-linux-gcc -g -O2 -o calc calc.o <para>This file defines the required fields for a local config file, namely the three config triplets, and the srcdir. It also defines several other Tcl variables that are used exclusivly by - the GCC test suite. For most test cases, the CXXFLAGS and LDFLAGS + the GCC testsuite. For most test cases, the CXXFLAGS and LDFLAGS are supplied by DejaGnu itself for cross testing, but to test a compiler, GCC needs to manipulate these itself.</para> @@ -2067,8 +2067,8 @@ powerpc-linux-gcc -g -O2 -o calc calc.o <chapter id=Extending xreflabel="Extending DejaGnu"> <title>Extending DejaGnu</title> - <sect1 id=addsuite xreflabel="Adding a new Test Suite"> - <title>Adding A New Test Suite</title> + <sect1 id=addsuite xreflabel="Adding a new Testsuite"> + <title>Adding A New Testsuite</title> <para>The testsuite for a new tool should always be located in that tools source directory. DejaGnu require the directory be named @@ -2084,29 +2084,29 @@ powerpc-linux-gcc -g -O2 -o calc calc.o <para>In general, the best way to learn how to write (code or even prose) is to read something similar. This principle applies to test cases and - to test suites. Unfortunately, well-established test suites have a way + to testsuites. Unfortunately, well-established testsuites have a way of developing their own conventions: as test writers become more experienced with DejaGnu and with Tcl, they accumulate more utilities, and take advantage of more and more features of <productname>Expect</productname> and <productname>Tcl</productname> in general.</para> - <para>Inspecting such established test suites may make the prospect of - creating an entirely new test suite appear overwhelming. Nevertheless, - it is quite straightforward to get a new test suite going.</para> + <para>Inspecting such established testsuites may make the prospect of + creating an entirely new testsuite appear overwhelming. Nevertheless, + it is quite straightforward to get a new testsuite going.</para> - <para>There is one test suite that is guaranteed not to grow more + <para>There is one testsuite that is guaranteed not to grow more elaborate over time: both it and the tool it tests were created expressly to illustrate what it takes to get started with DejaGnu. The <filename>example/</filename> directory of the DejaGnu distribution contains both an interactive tool called <command>calc</command>, and a - test suite for it. Reading this test suite, and experimenting with it, + testsuite for it. Reading this testsuite, and experimenting with it, is a good way to supplement the information in this section. (Thanks to - Robert Lupton for creating calc and its test suite---and also the first + Robert Lupton for creating calc and its testsuite---and also the first version of this section of the manual!)</para> <para>To help orient you further in this task, here is an outline of the - steps to begin building a test suite for a program example.</para> + steps to begin building a testsuite for a program example.</para> <itemizedlist mark=bullet> @@ -2897,8 +2897,8 @@ powerpc-linux-gcc -g -O2 -o calc calc.o </sect1> - <sect1 id=adding xreflabel="Adding A Test Case To A Test Suite"> - <title>Adding A Test Case To A Test Suite.</title> + <sect1 id=adding xreflabel="Adding A Test Case To A Testsuite"> + <title>Adding A Test Case To A Testsuite.</title> <para>There are two slightly different ways to add a test case. One is to add the test case to an existing directory. The @@ -2937,7 +2937,7 @@ powerpc-linux-gcc -g -O2 -o calc calc.o <listitem><para>Add the new directory name to the <symbol>configdirs</symbol> definition in the - <filename>configure.in</filename> file for the test suite + <filename>configure.in</filename> file for the testsuite directory. This way when <command>make</command> and <command>configure</command> next run, they include the new directory.</para></listitem> |