path: root/doc/overview.sgml

<!DOCTYPE book PUBLIC "-//OASIS//DTD DocBook V3.1//EN" [

<!-- Begin Document Specific Declarations -->

<?Fm: Validation Off>

<!ENTITY version "1.4.4">
<!ENTITY dj "DejaGnu">

<!ENTITY dejagnu-copyright "
	<YEAR>2004</YEAR>
	<HOLDER>Free Software Foundation, Inc.</HOLDER>">

<!ENTITY dejagnu-code-copyright "
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
This file documents the GNU Testing Framework ``DejaGnu''

Copyright 1992, 1993, 1994, 1995, 1996, 1997, 1998, 1999, 2000, 2001,
	  2002, 2003, 2004 Free Software Foundation, Inc.

This text may be freely distributed under the terms of the GNU
General Public License.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
">

<!ENTITY dejagnu-copyright "
Copyright 1992, 1993, 1994, 1995, 1996, 1997, 1998, 1999, 2000, 2001,
	  2002, 2003, 2004 Free Software Foundation, Inc.

Permission is granted to make and distribute verbatim copies of
this manual provided the copyright notice and this permission notice
are preserved on all copies.

Permission is granted to copy and distribute modified versions of this
manual under the conditions for verbatim copying, provided also that
the entire resulting derived work is distributed under the terms of a
permission notice identical to this one.

Permission is granted to copy and distribute translations of this manual
into another language, under the above conditions for modified versions.
">

<!-- The reference material -->
<!entity ref SYSTEM "ref.sgml">

<!-- The user manual -->
<!entity user SYSTEM "user.sgml">

<!-- End Document Specific Declarations -->

]>

<book>
  <bookinfo>
    <title>&dj;</title>
    <subtitle>The GNU Testing Framework</subtitle>
    <date>3 February 2004</date>
    <edition> &version</edition>
    <releaseinfo> New release</releaseinfo>
    <authorgroup>
      <author>
        <firstname>Rob Savoye</firstname>
        <affiliation>
          <orgname>Free Software Foundation</orgname></affiliation>
        <!-- <authorblurb>
          <title>Rob Savoye</title>
          <para>
            His home page is at <ulink>
            URL="http://www.welcomehome.org/rob.html">this
            location</ulink>
            </para>
        </authorblurb>
        -->
      </author>
      <author>
        <firstname>Ben Elliston</firstname>
        <affiliation><orgname>Free Software Foundation</orgname></affiliation>
      </author>
    </authorgroup>
    <address>
      <email>rob@welcomehome.org</email>
    </address>
    <!-- &cygnus-street-address; -->
    <copyright>&dejagnu-copyright;</copyright>
    <!-- <legalnotice>
      <para> -->
        <!-- [FIXME: must put legal notice here] -->
      <!-- </para> -->
      <!-- &cygnus-legal-notice; -->
    <!-- </legalnotice> -->
    <revhistory>
      <revision>
        <revnumber>0.7</revnumber>
	<date>2004-02-02</date>
	<authorinitials>bje</authorinitials>
	<revremark>So much restructuring it's not funny ..</revremark>
      </revision>
      <revision>
        <revnumber>0.6.2</revnumber>
        <date>2002-07-16</date>
        <authorinitials>rob</authorinitials>
        <revremark>Add new tutorial as a new chapter.</revremark>
      </revision>
      <revision>
        <revnumber>0.6.1</revnumber>
        <date>2001-02-16</date>
        <authorinitials>rob</authorinitials>
        <revremark>Add info on the new dejagnu.h file.</revremark>
      </revision>
      <revision>
        <revnumber>0.6</revnumber>
        <date>2001-02-16</date>
        <authorinitials>rob</authorinitials>
        <revremark>Updated for new release.</revremark>
      </revision>
      <revision>
        <revnumber>0.5</revnumber>
        <date>2000-01-24</date>
        <authorinitials>rob</authorinitials>
        <revremark>Initial version after conversion to DocBook.</revremark>
      </revision>
    </revhistory>

  </bookinfo>

  <toc></toc>

  <preface id=preface>
    <title>Abstract</title>

    <para>This document describes the functionality of DejaGnu, the
    testing framework of the GNU Project.  DejaGnu is a framework for
    testing other programs and many widely used GNU packages such as
    the GNU Compiler Collection (GCC) and the GNU Debugger (GDB) are
    tested using DejaGnu.  Its purpose is to provide a uniform testing
    environment. Programs can be just as easily tested locally or on a
    remote computer of a different type than the computer running
    DejaGnu.  Each program can have multiple testsuites, all supported
    by a single test harness. &dj; is written in
    <productname>Expect</productname> which in turn relies upon
    <productname>Tcl</productname>.</para> </preface>

  <chapter id=overview xreflabel=Introduction>
    <title>Introduction</title>
    <sect1 id=whatis xreflabel="What is &dj;?">
      <title>What is &dj;?</title>

    <para><productname>&dj;</productname> is a framework for
	testing other programs.  Its purpose is to provide a uniform
	testing environment. Think of it as a library of
	<productname>Tcl</productname> procedures to support writing a
	test harness. A <emphasis>test harness</emphasis> is the
	testing infrastructure created to support a specific
	program. Each program can have multiple testsuites, all
	supported by a single test harness. &dj; is written in
	<productname>Expect</productname>, which in turn relies upon
	<productname>Tcl</productname>. More information on
	<productname>Tcl</productname> and
	<productname>Expect</productname> can be found at the <ulink
	URL="http://www.tcltk.com">Tcl/Tk web site</ulink> and the
	<ulink URL="http://expect.nist.gov">Expect home page</ulink>,
	respectively.</para>
	
   <para>Julia Menapace first coined the term ``&dj;'' to describe an
        earlier testing framework she had written at Cygnus Support
        for testing <command>GDB</command>. When it was replaced it
        with the Expect-based framework, it was like &dj; all over
        again.  More importantly, it was also named after Deja Snow
        Savoye (13 years old as of September 2003), who was a toddler
        during &dj;'s beginnings.</para>

    <para>There are several advantages to testing with &dj;:</para>

    <itemizedlist mark="bullet" spacing="compact">

      <listitem><para>The flexibility and consistency of the &dj;
	framework make it easy to write tests for any program, including 
	batch-oriented and interactive programs.</para>
      </listitem>

      <listitem><para>&dj; provides a layer of abstraction which
	allows you to write tests that are portable to any host or
	target where a program must be tested. For instance, a test
	for <productname>GDB</productname> can run from any supported
	host system on any supported target system. &dj; runs tests
	on many single board computers, whose operating software
	ranges from a simple boot monitor to a real-time OS.</para>
	</listitem>

      <listitem><para>All tests have the same output format. This
	makes it easy to integrate testing into other software
	development processes. &dj;'s output is human readable, but
	can be easily parsed by other software.</para>
      </listitem>

      <listitem><para>Using <productname>Tcl</productname> and
      <productname>Expect</productname>, it is easy to create wrappers
      for existing testsuites. By incorporating existing tests under
      &dj;, it is easier to have a single set of reporting
      programs.</para>

      </listitem>
    </itemizedlist>

    <para>Running tests requires two things: the testing framework and
    the testsuites themselves. Tests are usually written in
    <productname>Expect</productname> using
    <productname>Tcl</productname>, but you can also use a
    <productname>Tcl</productname> script to run a testsuite that is
    not based on
    <productname>Expect</productname>. <productname>Expect</productname>
    scripts use <emphasis>.exp</emphasis> as a filename extension as a
    convention.</para>
	
  </sect1>
  <sect1 id=new xreflabel="Release Notes">
    <title>What's new in version &version;</title>

    <para>Version &version; has a number of substantial changes over
    version 1.3. The most visible change is that the version of
    <productname>Expect</productname> and
    <productname>Tcl</productname> included in the release are
    up-to-date with the current stable net releases. The biggest
    change is to the target configuration system.  While this greatly
    improved cross-testing, is has made that subsystem very
    complicated. Other changes include:</para>

    <itemizedlist>
      <listitem><para>More built-in support for building target
        binaries with the correct linker flags. Currently this only
        works with <productname>GCC</productname> as the
        cross-compiler.</para></listitem>

      <listitem><para>Lots of bug fixes from years of heavy use at
        Cygnus Solutions.</para></listitem>

      <listitem><para>The use of <productname>Autoconf</productname>
      and <productname>Automake</productname> for
      building.</para></listitem>

      <listitem><para>Updated documentation in SGML format using the
	<ulink
	URL="http://nis-www.lanl.gov/~rosalia/mydocs/docbook-intro.html">GNU
	DocBook tools</ulink>.</para></listitem>
	
      <listitem><para>Beta level support for Windows, however this is
      still work in progress. This requires the <ulink
      URL="http://www.cygwin.com/">Cygwin</ulink> POSIX subsystem for
      Windows.</para></listitem>

    </itemizedlist>

    <sect2 id=cygwin xreflabel="Windows Support">
      <title>Windows Support</title>

      <para>To use &dj; on Windows, you need to first install the
	<ulink URL="http://www.cygwin.com/">Cygwin</ulink>
	system. Cygwin is a POSIX system for Windows. This covers both
	utility programs and a library that adds POSIX system calls to
	Windows. Among them is pseudo tty support for Windows that
	emulates the POSIX pty standard. The latest Cygwin is always
	available from <ulink URL="http://www.cygwin.com/">the Cygwin
	home page</ulink>. This works well enough to run
	<emphasis>"make check"</emphasis> on the GNU development tools
	after building them on a native Cygwin system.</para>
    </sect2>
  </sect1>

  <sect1 id=designgoals xreflabel="Design Goals">
    <title>Design Goals</title>

    <para>&dj; grew out of the internal needs of Cygnus Solutions,
    the company formerly known as Cygnus Support. Cygnus maintained
    and enhanced a variety of free programs in many different
    environments.  A testing tool was needed that:</para>

    <itemizedlist mark="bullet">
      <listitem><para>was useful to developers while fixing
      bugs;</para></listitem>
      <listitem><para>automated running many tests during a software
      release process;</para></listitem>
      <listitem><para>was portable among a variety of host
      computers;</para></listitem>
      <listitem><para>supported cross-development
      testing;</para></listitem>
      <listitem><para>permitted testing interactive programs, like
      <command>GDB</command>; and </para></listitem>
      <listitem><para>permitted testing batch oriented programs, like
      <command>GCC</command>.</para></listitem>
    </itemizedlist>

    <para>Some of the requirements proved challenging.  For example,
    interactive programs do not lend themselves very well to automated
    testing, but this capability is important.  For instance, it is
    imperative to make sure that <productname>GDB</productname> works
    as well when cross-debugging as it does in a native
    configuration.</para>

    <para>Probably the greatest challenge was testing in a
    cross-development environment.  Most cross-development
    environments are customized by each developer.  Even when buying
    packaged boards from vendors there are many differences.  The
    communication interfaces vary from a serial line to Ethernet.
    &dj; was designed with a modular communication setup, so that
    each mode of communication can be added as required and supported
    thereafter.  Once a communication procedure is implemented, any
    testsuite can use it.  Currently &dj; can use
    <command>rsh</command>, <command>rlogin</command>,
    <command>telnet</command>, <command>tip</command>,
    <command>kermit</command> and <command>mondfe</command> for remote
    communications.</para>

    </sect1>

    <sect1 id=posix xreflabel="A POSIX Conforming Test Framework">
      <title>A POSIX conforming test framework</title>

      <para>&dj; conforms to the POSIX 1003.3 standard for test
      frameworks.  The POSIX standard 1003.3 defines what a testing
      framework needs to provide, in order to permit the creation of
      POSIX conformance test suites. This standard is primarily
      oriented to running POSIX conformance tests, but its
      requirements also support testing of features not related to
      POSIX conformance.  POSIX 1003.3 does not specify a particular
      testing framework, but at the time of writing there is only one
      other POSIX conforming test framework: TET. TET was created by
      Unisoft for a consortium comprised of X/Open, Unix International
      and the Open Software Foundation.</para>

      <para>The POSIX documentation refers to <firstterm>assertions</firstterm>.
      An assertion is a description of behavior.  For example, if a standard
      says ``The sun shall shine'', a corresponding assertion might be ``The
      sun is shining.''  A test based on this assertion would pass or fail
      depending on whether it is day or night.  It is important to note
      that the standard being tested is never 1003.3; the standard being tested
      is some other standard, for which the assertions were written.</para>

      <para>As there is no testsuite to test testing frameworks for POSIX
      1003.3 conformance, verifying conformance to this standard is done by
      repeatedly reading the standard and experimenting.  One of the main
      things 1003.3 does specify is the set of allowed output messages and
      their definitions.  Four messages are supported for a required feature of
      POSIX conforming systems and a fifth for a conditional feature. &dj;
      supports the use of all five output messages.  In this sense a testsuite
      that uses exactly these messages can be considered POSIX conforming.
      These definitions specify the output of a test
      case:</para>

      <variablelist>
	<varlistentry>
	  <term>PASS</term>
	  <listitem><para>A test has succeeded.  That is, it demonstrated that
	  the assertion is true.</para></listitem>
	</varlistentry>

        <varlistentry>
	  <term>XFAIL</term>
	  <listitem><para>POSIX 1003.3 does not incorporate the notion of
	  expected failures, so <emphasis>PASS</emphasis>, instead of
	  <emphasis>XPASS</emphasis>, must also be returned for test cases
	  which were expected to fail and did not.  This means that
	  <emphasis>PASS</emphasis> is in some sense more ambiguous than if
	  <emphasis>XPASS</emphasis> is also used.</para></listitem>
	</varlistentry>

        <varlistentry>
	  <term>FAIL</term>
	  <listitem><para>A test has produced the bug it was intended to
	  capture.  That is, it has demonstrated that the assertion is false.
	  The <emphasis>FAIL</emphasis> message is based on the test case only.
	  Other messages are used to indicate a failure of the framework. As
	  with <emphasis>PASS</emphasis>, POSIX tests must return
	  <emphasis>FAIL</emphasis> rather than <emphasis>XFAIL</emphasis> even
	  if a failure was expected.</para></listitem>
	</varlistentry>

        <varlistentry>
	  <term>UNRESOLVED</term> <listitem><para>A test produced
	  indeterminate results.  Usually, this means the test
	  executed in an unexpected fashion; this outcome requires a
	  human being to go over the results and determine if the test
	  should have passed or failed.  This message is also used for
	  any test that requires human intervention because it is
	  beyond the abilities of the testing framework.  Any
	  unresolved test should be resolved to
	  <emphasis>PASS</emphasis> or <emphasis>FAIL</emphasis>
	  before a test run can be considered finished.</para>

	  <para>Note that for POSIX compliance, each assertion must
	  produce a test result code.  If the test isn't actually run,
	  it must produce <emphasis>UNRESOLVED</emphasis> rather than
	  just leaving that test out of the output.  This means that
	  you have to be careful when writing tests to not carelessly
	  use <productname>Tcl</productname> commands like
	  <emphasis>return</emphasis>---if you alter the flow of
	  control of the <productname>Tcl</productname> code you must
	  insure that every test still produces some result
	  code.</para>

	  <para>Here are some of the ways a test may wind up
	  <emphasis>UNRESOLVED</emphasis>:</para></listitem>

	</varlistentry>
	</variablelist>

          <itemizedlist mark=bullet>
	    <listitem><para>The execution of a test is
	    interrupted.</para></listitem>

	    <listitem><para>A test does not produce a clear
	    result. This is usually because there was an
	    <emphasis>ERROR</emphasis> from &dj; while processing
	    the test, or because there were three or more
	    <emphasis>WARNING</emphasis> messages. Any
	    <emphasis>WARNING</emphasis> or <emphasis>ERROR</emphasis>
	    messages can invalidate the output of the test.  This
	    usually requires a human being to examine the output to
	    determine what really happened---and to improve the test
	    case.</para></listitem>

	    <listitem><para>A test depends on a previous test, which
	    fails.</para></listitem>

	    <listitem><para>The test was set up
	    incorrectly.</para></listitem>
	  </itemizedlist>

	<variablelist>
	  <varlistentry>
	    <term>UNTESTED</term>
	    <listitem><para>A test was not run.  This is a place-holder, used
	    when there is no real test case yet.</para></listitem>
	  </varlistentry>
	</variablelist>

	<variablelist>
	  <varlistentry>
  	    <term>UNSUPPORTED</term>
	    <listitem><para>There is no support for the tested case.  This may
  	    mean that a conditional feature of an operating system, or of a
  	    compiler, is not implemented.  &dj; also uses this message when
  	    a testing environment (often a ``bare board'' target) lacks basic
  	    support for compiling or running the test case.  For example, a
  	    test for the system subroutine <emphasis>gethostname</emphasis>
  	    would never work on a target board running only a boot
  	    monitor.</para></listitem>
	  </varlistentry>
	</variablelist>

        <para>&dj; uses the same output procedures to produce these messages
	for all testsuites and these procedures are already known to conform
	to POSIX 1003.3.  For a &dj; testsuite to conform to POSIX 1003.3,
	you must avoid the <emphasis>setup</emphasis>xfail} procedure as
	described in the <emphasis>PASS</emphasis> section above and you must
	be careful to return <emphasis>UNRESOLVED</emphasis> where appropriate,
	as described in the <emphasis>UNRESOLVED</emphasis> section
	above.</para>
     </sect1>

  </chapter>

<chapter id=install>
    <title>Installation</title>

  <sect1 id=obtaining xreflabel="Obtaining DejaGnu">
    <title>Obtaining DejaGnu</title>

    <para>You can obtain DejaGnu from the DejaGnu web site at the
    <ulink URL="http://www.gnu.org">Free Software Foundation</ulink>,
    which is at <ulink
    URL="http://www.gnu.org/software/dejagnu/">www.gnu.org/software/dejagnu/
    </ulink></para>
  </sect1>

  <sect1 id=installation xreflabel="Installation">
    <title>Installation</title>

    <para>Once you have the DejaGnu source unpacked and available, you must
    first configure the software to specify where it is to run (and the
    associated defaults); then you can proceed to installing it.</para>

    <sect2 id=configuring xreflabel="Configuring DejaGnu">
      <title>Configuring DejaGnu</title>

      <para>It is usually best to configure in a directory separate from the
      source tree, specifying where to find the source with the optional
      <emphasis>--srcdir</emphasis> option to
      <emphasis>configure</emphasis>. DejaGnu uses the GNU
      <emphasis>autoconf</emphasis> to configure itself. For more info on using
      autoconf, read the GNU autoconf manual. To configure, execute the
      <filename>configure</filename> program, no other options are
      required. For an example, to configure in a seperate tree for objects,
      execute the configure script from the source tree like this:</para>

      <screen>
        ../dejagnu-&version/configure
      </screen>

      <para>DejaGnu doesn't care at config time if it's for testing a native
      system or a cross system. That is determined at runtime by using the
      config files.</para>

      <para>You may also want to use the <command>configure</command> option
      <emphasis>--prefix</emphasis> to specify where you want DejaGnu and its
      supporting code installed.  By default, installation is in subdirectories
      of <filename>/usr/local</filename>, but you can select any alternate
      directory <symbol>altdir</symbol> by including
      <option>--prefix</option>{altdir}} on the
      <command>configure</command> command line. (This value is captured in
      the Makefile variables <emphasis>prefix</emphasis> and
      <emphasis>exec</emphasis>prefix}.)</para>

      <para>Save for a small number of example tests, the DejaGnu distribution
      itself does not include any testsuites; these are available
      separately. Testsuites for the GNU development tools are included in
      those releases. After configuring the top-level DejaGnu directory, unpack
      and configure the test directories for the tools you want to test; then,
      in each test directory, run <emphasis>make check</emphasis> to build
      auxiliary programs required by some of the tests, and run the test
      suites.</para>

    </sect2>

    <sect2 id=installing  xreflabel="Installing DejaGnu">
      <title>Installing DejaGnu</title>

      <para>To install DejaGnu in your filesystem (either in
      <filename>/usr/local</filename>, or as specified by your
      <emphasis>--prefix</emphasis> option to <emphasis>configure</emphasis>),
      execute.</para>

      <screen>
        eg$ make install
      </screen>

      <para><emphasis>make install</emphasis>does thes things for
      DejaGnu:</para>

      <itemizedlist mark=bullet>
        <listitem><para>Look in the path specified for executables
          <symbol>$exec_prefix</symbol>) for directories called
          <filename>lib</filename> and <filename>bin</filename>. If these
          directories do not exist, <emphasis>make install</emphasis> creates
          them.</para></listitem>

        <listitem><para>Create another directory in the
        <filename>share</filename> directory, called
        <filename>dejagnu</filename>, and copy all the library files into
        it.</para></listitem>

        <listitem><para>Create a directory in the
        <filename>dejagnu/share</filename> directory, called
        <filename>config</filename>, and copy all the configuration files into
        it.</para></listitem>

	<listitem><para>Copy the <emphasis>runtest</emphasis> shell script into
	<filename>$exec_prefix/bin</filename>.</para></listitem>

	<listitem><para>Copy <filename>runtest.exp</filename> into
	<filename>$exec_prefix/lib/dejagnu</filename>. This is the main Tcl
	code implementing DejaGnu.</para></listitem>

      </itemizedlist>
  </sect2>
  </sect1>
</chapter>

  <!-- include the user manual -->
  &user;

  <!-- include the reference manual -->
  &ref;

</book>

<!-- Keep this comment at the end of the file
Local variables:
mode: sgml
sgml-omittag:t
sgml-shorttag:t
sgml-namecase-general:t
sgml-general-insert-case:lower
sgml-minimize-attributes:nil
sgml-always-quote-attributes:t
sgml-indent-step:1
sgml-indent-data:nil
sgml-parent-document:nil
sgml-exposed-tags:nil
sgml-local-catalogs:nil
sgml-local-ecat-files:nil
End:
-->