Understanding the runtest script in DejaGnu Philip A. Wilsey Clifton Labs, Inc philip.wilsey@cliftonlabs.com September 26, 2001 DejaGnu is a tcl program to help run test scripts for batch and interactive programs. If you have not yet read the DejaGnu documentation don't, read this document first and play with this example. If you have, you have my sympathies. Hopefully this document and example will provide a high level overview of what goes on in runtest. You will still have to read the DejaGnu documentation. It contains many details of the low level functions and operation that I do not address in this document. FIRST: I AM NOT A DEJAGNU OR TCL EXPERT. I have simply spent the past week studying and trying to understand the execution behavior of the runtest script. As part of my studies I built the accompanying example that you may (or may not) find helpful in understanding just what is going on with the runtest script. Fortunately tcl is a pretty simply language and a quick scan of Ousterhout's text on tcl has been more than sufficient for me to conduct these experiments. Again, this document only records my observations. SECOND: I am not interested in or concerned with testing cross-compilation systems or anything exotic like remote execution. I was/am simply interested in using a test harness in a general purpose conformance suite. Consequently I have not examined all the switches and variable related to -host_board, -target, or remote execution. I was simply trying to gain an overall understanding of what and when the various tcl/expect files and procedures are executed. THIRD: There are really two parts to my studies, specifically: (i) the study of runtest, and (ii) the use of the auto- tools to realize a "make check" target that will invoke the runtest script for you. As a result, this document is organized into three parts: Part I: An overview of runtest Part II: Building a "make check" target with automake Part III: Running the example For small projects, you may not be interested in the autoconf/automake tools and can safely skip Part II of this document. FOURTH: This example is setup only for recording the files and procedures that runtest uses. Technically it will compile a simple c++ hello world program, however the test system does not verify it's operation. If you run make check, it will simply dump out a bunch of DEBUG statements and exit. Please don't expect anything more. FINALLY: For many years now we have been managing our regression suite using hand rolled perl scripts that require continual maintenance. We turned to DejaGnu in hopes to reduce some of our efforts in this direction. Unfortunately, I found understanding DejaGnu to be very painful and at several times during my study I was ready to chuck it all. Now that I have a better understanding of the system, I believe that in the long run I will be much happier that I stuck with it. Hopefully you will think so as well. NOTE: I have only used/studied runtest on a Debian Linux box. I do not know how well this knowledge will generalize or be useful to you. Please use at your own risk. PART I: AN OVERVIEW OF RUNTEST ------------------------------ I.1 The essence of Runtest -------------------------- runtest is designed to be run from a test subdirectory where all tests are stored in subdirectories with a naming convention of TOOL.TESTNAME. Subdirectories not conforming to the TOOL.TESTNAME naming convention are not examined. runtest examines recursively all the subdirectories under each TOOL.TESTNAME subdirectories for expect scripts to be executed (it assumes that everything with the ".exp" suffix is an expect script). It will then attempt to execute every expect script it finds. For example, let's take a look at the contents of the helloworld.* subdirectories in the testsuite subdirectory of this example: peabody>ls -R testsuite/helloworld.test* ~/test/dejagnu/helloDemo testsuite/helloworld.test1: total 8 4 helloworld.test1-1/ 4 test1.exp testsuite/helloworld.test1/helloworld.test1-1: total 4 4 test1-1.exp testsuite/helloworld.test2: total 8 4 test2.exp 4 test3.exp testsuite/helloworld.test3: total 12 4 config/ 4 lib/ 4 non_compliant_dir_name/ testsuite/helloworld.test3/config: total 4 4 config.exp testsuite/helloworld.test3/lib: total 4 4 lib.exp testsuite/helloworld.test3/non_compliant_dir_name: total 4 4 non_compliant.exp peabody> Once everything is setup, issuing the command "runtest" from the testsuite subdirectory on my machine causes all of the expect scripts to be executed in this order: Running ./helloworld.test1/helloworld.test1-1/test1-1.exp ... Running ./helloworld.test1/test1.exp ... Running ./helloworld.test2/test2.exp ... Running ./helloworld.test2/test3.exp ... Running ./helloworld.test3/non_compliant_dir_name/non_compliant.exp ... Looks like a depth first search; however, I can find nothing in the documentation that indicates that a specific order will be followed. So you should not count on it. Also notice that the TOOL.TESTNAME naming convention is only enforced at the root directory from where the runtest command is issued -- beneath the first level subdirectories all subdirectories except those named "lib" and "config" are examined for expect scripts. I.2 The nuts and bolts of Runtest --------------------------------- A few more points. runtest is setup to look for scripts to load for configuration information and runtime procedures. In many cases if runtest cannot find a script it will continue. Four important tcl procedures that you should define (in testsuite/lib/TOOL.exp) are: TOOL_init TOOL_finish TOOL_exit TOOL_version TOOL_init is called prior to running each expect script it locates under the testsuite subdirectory. runtest invokes TOOL_finish after executing each expect script. TOOL_exit is invoked after TOOL_finish for the last expect script executes. Finally TOOL_version is invoked to allow the testsute to report the version of the tool that was just tested. If you want to add tool specific arguments on the on the runtest command line you can also define: TOOL_option_proc TOOL_option_help TOOL_option_proc is invoked as runtest parses the command line arguments. TOOL_option_help is invoked when runtest is invoked with the "-help" command line argument (this is actually pretty cool). In general runtest will take the following steps (the expect scripts named in these steps are named relative to the directory where the runtest command is issued): 1. It will look for and load the file ~/.dejagnurc for command line options. I have not experimented with this and cannot comment on what can and cannot be placed in this file. 2. It will try to load load the file ./site.exp (if you use automake, this file will be automagically created for you). Technically this file is organized into two parts, one that is set when the .configure script is executed; the second part can be edited and changed by the user. 3. It will try to load the file ./lib/TOOL.exp (this is probably a good place to locate definitions for the TOOL_init/TOOL_exit/TOOL_version procedures. 4. It will try to load some configuration files. Specifically 4a. It will try to load a base-config.exp file. On my system it searches, in order, the following subdirectories: ./config ./../config ./../../config ./../../../config 4b. It will try to load a system specific configuration file. I believe you can/should use this to setup different configuration information for various operating systems that the test suite is to run in. On my system it searches, in order, for (the search is terminated after the first success): ./config/unix.exp ./config/gnu.exp ./config/default.exp ./config/unknown.exp ./../config/unix.exp ./../config/gnu.exp ./../config/default.exp ./../config/unknown.exp ./../../config/unix.exp ./../../config/gnu.exp ./../../config/default.exp ./../../config/unknown.exp ./../../../config/unix.exp ./../../../config/gnu.exp ./../../../config/default.exp ./../../../config/unknown.exp 5. Now it will recursively search the ./TOOL.* subdirectories and for each expect script it locates it will: 5a. Invoke the procedure TOOL_init giving the relative pathname where the expect script is found as an argument. 5b. Run the expect script. 5c. Invoke the proecture TOOL_finish with no arguments. 6. After running all of the located expect script it will invoke the TOOL_exit procedure. 7. Next the TOOL_version procedure is invoked. 8. Finally, runtest exits. That's it. Sadly it took me almost a week to learn this.... I.3 Some notes I made during my week of study --------------------------------------------- 1. site.exp is loaded twice. Why? I have no clue. 2. As far as I can tell the "testsuite" name is completely artificial. runtest does not appear to depend on it. The autoconf/automake family also do not seem to depend on it. 3. Searching for files to load: There is a method to this madness. However, it's not very regular: for example why does 4a not look into the testsuite subdirectory and 4b does. 3a. Look into srcdir (where the runtest command was issued) for lib/tool.exp; I guess the principle concept is to put tool specific and platform independent expect scripts here. 3b. Next look for for platform scripts in --srcdir/config (on my linux system it looks for one of unix.exp, gnu.exp, and default.exp (in this order and stops with the first one it finds). My confusion on the search patterns of runtest is magnified by the fact that I am trying to issue the runtest command from outside the testsuite subdirectory....don't do this. 4. Executing expect scripts. 4a. If an expect script is named on the command line it (and only it) is run. otherwise.... 4b. runtest looks in the testsuite directory and does a glob on TOOL.*/*.exp (ok, it's a recursive glob TOOL.*/.../*.exp) and executes all expect script it sees (ok, it appears to ignore all subdirectories named "lib" or "config"). Before execution of each script, runtest invokes TOOL_init with the relative path (relative to where the directory where the runtest command was issued) name of that script as it's only argument. After execution of each script runtest invokes TOOL_finish with no arguments. 5. load_lib: load_lib is a tcl script in the DejaGnu system that you can use to load tcl/expect files. It has an odd search path. In particular, on my machine, the search path is: 5a. the dejagnu install directory (/usr/share/dejagnu) and its accompanying library directory (/usr/share/dejagnu/lib) 5b. in a dejagnu/lib directory one above the current directory (where runtest was run). that is in ../dejagnu/lib 5c. in the --srcdir lib 5d. in the directory where runtest was issued 5e and this is the funny one, in a dejagnu/lib directory two directories above (../../dejagnu/lib). 6. hmmmm how does DEJAGNULIBS alter the library search path?? I did not investigate this. 7. COOL: I can name an expect script in the --ignore command line option to prune it from the set of scripts that are executed. I.4 Some gotchas ---------------- 1. As far as I can tell the "testsuite" name is completely artificial and you can store your tests in whatever subdirectory name you like. 2. Don't try to issue the runtest command from outside the testsuite subdirectory using the "--srcdir target" command line argument. Unless you really know what's going on, it will cause you grief. For some reason the runtest will look for some things in the $target location and for others it will look in the directory where the runtest command was issued. 3. The search paths used to look for things varies all over the places. The system has tremendous configurability for building and controlling the expect scripts. However the flexibility for controlling what and where things are loaded by runtest is really quite limited. For example, site.exp must be located in the directory where the runtest command is issued. I can discover no way to alter its location. PART II: BUILDING a "make check" TARGET WITH AUTOMAKE ----------------------------------------------------- NOTE: before I began this study I did not know anything about autoconf/automake and friends (I still don't). 1. If you understand these tools it's really quite simple. In the directory containing the testsuite you must have a Makefile.am to build the "make check" target. You only need to define the dejagnu option for AUTOMAKE_OPTIONS. If you want you can also add runtest command line arguments to the RUNTESTDEFAULTFLAGS variable. The only problem I had with this was realizing that the Makefile.am to locate these definitions in is the one located *in* the testsuite subdirectory. DO NOT make these definitions in the root Makefile.am file (unless you intend to place you TOOL.* subdirectories there). 2. I cannot figure out a good way to add alternate check targets (e.g., make check-test1) to run alternate tests on the command line. Since I really don't know automake very well, this could very likely be very easy to do, but my ignorance prevents me from discovering how. Presently I have simply defined the alternate target by hand (see make check-demo in the testsuite/Makefile.am file). PART III: RUNNING THE EXAMPLE ----------------------------- The example contains a bunch of expect/tcl files, some of them are unused (actually I don't believe I ever used anything outside of standard tcl in any of my scripts). I created this example to explore what was going on so you will see, for example, unix.exp in the lib, testsuite/lib, and testsuite/config subdirectories. I did this to learn where things were loaded from. Consequently you should *not* use my setup to build a testing framework from. This system should only be used to discover the runtime behaviors of runtest. To build this example, you will need a system configured with a c++ compiler and the autoconf and automake tools installed. With these tools in place, the build procedure is simply: autoconf aclocal automake ./configure make check Pretty much that's it (you may or may not need the aclocal step). After the ./configure, the makefile in the testsuite subdirectory has two relevant targets, namely: make check make check-demo Good Luck.