1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
|
<?xml version="1.0"?>
<!DOCTYPE article PUBLIC "-//OASIS//DTD DocBook XML V4.1.2//EN"
"http://www.oasis-open.org/docbook/xml/4.1.2/docbookx.dtd" [
<!ENTITY legal SYSTEM "legal.xml">
<!ENTITY appversion "1.5.1">
<!ENTITY version "1.5.1">
<!ENTITY manrevision "1.0">
<!ENTITY date "Feb 2004">
<!ENTITY app "<application>DejaGnu</application>">
<!ENTITY appname "DejaGnu">
<!ENTITY dj "DejaGnu">
<!-- The reference material -->
<!ENTITY ref SYSTEM "ref.xml">
<!-- The user manual -->
<!ENTITY user SYSTEM "user.xml">
]>
<!-- Begin Document Specific Declarations -->
<article>
<articleinfo>
<title>&dj;</title>
<subtitle>The GNU Testing Framework</subtitle>
<date>January 2006</date>
<authorgroup>
<author>
<firstname>Rob</firstname>
<surname>Savoye</surname>
</author>
<author>
<firstname>Ben</firstname>
<surname>Elliston</surname>
</author>
</authorgroup>
<copyright>
<year>2006</year>
<holder>Free Software Foundation, Inc.</holder>
</copyright>
<!-- <legalnotice>
<para> -->
<!-- [FIXME: must put legal notice here] -->
<!-- </para> -->
<!-- </legalnotice> -->
<revhistory>
<revision>
<revnumber>0.6.2</revnumber>
<date>2002-07-16</date>
<authorinitials>rob</authorinitials>
<revremark>Add new tutorial as a new sect1.</revremark>
</revision>
<revision>
<revnumber>0.6.1</revnumber>
<date>2001-02-16</date>
<authorinitials>rob</authorinitials>
<revremark>Add info on the new dejagnu.h file.</revremark>
</revision>
<revision>
<revnumber>0.6</revnumber>
<date>2001-02-16</date>
<authorinitials>rob</authorinitials>
<revremark>Updated for new release.</revremark>
</revision>
<revision>
<revnumber>0.5</revnumber>
<date>2000-01-24</date>
<authorinitials>rob</authorinitials>
<revremark>Initial version after conversion to DocBook.</revremark>
</revision>
</revhistory>
</articleinfo>
<sect1 id="preface">
<title>Abstract</title>
<para>This document describes the functionality of &dj;, the
testing framework of the GNU project. &dj; is written in
<productname>Expect</productname>, which uses
<productname>Tcl</productname> as a command
language. <productname>Expect</productname> acts as a very
programmable shell. As with other Unix command shells, you can
run any program, but once the program is started, your test script
has programmable control over its input and output. This does not
just apply to the programs under test; <command>expect</command>
can also run any auxiliary program, such as
<command>diff</command> or <command>sh</command>, with full
control over its input and output.</para>
<para>&dj; itself is merely a framework for the creation of
testsuites. Testsuites are distributed with each
application.</para>
</sect1>
<sect1 id="overview" xreflabel="Overview">
<title>Overview</title>
<sect2 id="whatis" xreflabel="What is &dj; ?">
<title>What is &dj; ?</title>
<para><productname>&dj;</productname> is a framework for
testing other programs. Its purpose is to provide a single
front end for all tests. Think of it as a custom library of
Tcl procedures crafted to support writing a test harness. A
<emphasis>Test Harness</emphasis> is the testing
infrastructure that is created to support a specific program
or tool. Each program can have multiple testsuites, all
supported by a single test harness. &dj; is written in
<productname>Expect</productname>, which in turn uses
<productname>Tcl</productname> -- Tool command
language. There is more information on Tcl at the <ulink
url="http://www.scriptics.com">Scriptics</ulink> web site and the
Expect web site is at <ulink
url="http://expect.nist.gov">NIST</ulink>.</para>
<para>Julia Menapace first coined the term ``&dj;'' to describe
an earlier testing framework at Cygnus Support she had written
for <command>GDB</command>. When we replaced it with the
Expect-based framework, it was like &dj; all over again.
More importantly, it was also named after my daughter, <ulink
url="http://www.welcomehome.org/deja/">Deja Snow Savoye</ulink>
(now 14 years old as of Feb 2004), who was a toddler
during &dj;'s beginnings.</para>
<para>&dj; offers several advantages for testing:</para>
<itemizedlist mark="bullet" spacing="compact">
<listitem><para>The flexibility and consistency of the &dj;
framework make it easy to write tests for any program, with
either batch oriented, or interactive programs.</para>
</listitem>
<listitem><para>&dj; provides a layer of abstraction which
allows you to write tests that are portable to any host or
target where a program must be tested. For instance, a test
for <command>GDB</command> can run from any supported host
system on any supported target system. &dj; runs tests on
many single board computers, whose operating software ranges
from a simple boot monitor to a real-time OS.</para>
</listitem>
<listitem><para>All tests have the same output format. This
makes it easy to integrate testing into other software
development processes. &dj;'s output is designed to be
parsed by other filtering script and it is also human
readable.</para>
</listitem>
<listitem><para>Using Tcl and Expect, it's easy to create wrappers
for existing testsuites. By incorporating existing tests under
&dj;, it's easier to have a single set of report analyse
programs..</para>
</listitem>
</itemizedlist>
<para>Running tests requires two things: the testing framework and
the testsuites themselves. Tests are usually written in
<productname>Expect</productname> using Tcl, but you can also use
a Tcl script to run a testsuite that is not based on
<productname>Expect</productname>. <productname>Expect</productname>
script filenames conventionally use <emphasis>.exp</emphasis> as a
suffix; for example, the main implementation of the &dj; test
driver is in the file
<productname>runtest.exp</productname>.)</para>
</sect2>
<sect2 id="new" xreflabel="Release Notes">
<title>New In This Release</title>
<para>This release has a number of substantial changes over version
1.3. The most visible change is that the version of Expect and Tcl
included in the release are up-to-date with the current stable net
releases. The biggest change is years of modifications to the
target configuration system, used for cross testing. While this
greatly improved cross testing, it has made that subsystem very
complicated. The goal is to have this entirely rewritten using
<productname>iTcl</productname> by the next release. Other changes
are:</para>
<itemizedlist>
<listitem>
<para>More built-in support for building target binaries
with the correct linker flags. Currently this only works with
<productname>GCC</productname> as the cross compiler,
preferably with a target supported by
<xref linkend="libgloss"/>.
</para>
</listitem>
<listitem><para>Lots of little bug fixes from years of heavy
use at Cygnus Solutions.</para></listitem>
<listitem><para>&dj; now uses
<productname>Automake</productname> for Makefile
configuration.</para></listitem>
<listitem><para>Updated documentation, now in DocBook XML.
</para></listitem>
<listitem><para>Windows support. There is beta level support for
Windows that is still a work in progress. This requires the
<ulink url="http://www.cygwin.com/">Cygwin</ulink> POSIX
subsystem for Windows.</para></listitem>
</itemizedlist>
<sect3 id="cygwin" xreflabel="Windows Support">
<title>Windows Support</title>
<para>To use &dj; on Windows, you need to first install the
<ulink url="http://www.cygwin.com/">Cygwin</ulink>
release. This works as of the B20.1 release. Cygwin is a POSIX
system for Windows. This covers both utility programs and a library
that adds POSIX system calls to Windows. Among them is pseudo tty
support for Windows that emulates the POSIX pty standard. The
latest Cygwin is always available from <ulink
url="http://www.cygwin.com/">this location</ulink>. This
works well enough to run <emphasis>"make check"</emphasis> of
the GNU development tree on Windows after a native build. But the
nature of ptys on Windows is still evolving. Your mileage may
vary.</para>
</sect3>
</sect2>
<sect2 id="designgoals" xreflabel="Design Goals">
<title>Design Goals</title>
<para>&dj; grew out of the internal needs of Cygnus Solutions,
the company formerly known as Cygnus Support. Cygnus maintained
and enhanced a variety of free programs in many different
environments and we needed a testing tool that:</para>
<itemizedlist mark="bullet">
<listitem><para>was useful to developers while fixing
bugs;</para></listitem>
<listitem><para>automated running many tests during a software
release process;</para></listitem>
<listitem><para>was portable among a variety of host
computers;</para></listitem>
<listitem><para>supported cross-development
testing;</para></listitem>
<listitem><para>permitted testing interactive programs, like
<command>GDB</command>; and </para></listitem>
<listitem><para>permitted testing batch oriented programs, like
<command>GCC</command>.</para></listitem>
</itemizedlist>
<para>Some of the requirements proved challenging. For example,
interactive programs do not lend themselves very well to automated testing.
But all the requirements are important: for instance, it is imperative to
make sure that <command>GDB</command> works as well when cross-debugging
as it does in a native configuration. </para>
<para>Probably the greatest challenge was testing in a
cross-development environment. Most cross-development
environments are customized by each developer. Even when buying
packaged boards from vendors there are many differences. The
communication interfaces vary from a serial line to Ethernet.
&dj; was designed with a modular communication setup, so that
each kind of communication can be added as required and supported
thereafter. Once a communication procedure is coded, any test can
use it. Currently &dj; can use <command>rsh</command>,
<command>rlogin</command>, <command>telnet</command>,
<command>tip</command>, <command>kermit</command> and
<command>mondfe</command> for remote communications.</para>
</sect2>
<sect2 id="posix" xreflabel="A POSIX Conforming Test Framework">
<title>A POSIX conforming test framework</title>
<para>&dj; conforms to the POSIX 1003.3 standard for test
frameworks. Rob Savoye was a member of that committee.</para>
<para>The POSIX standard 1003.3 defines what a testing framework needs to
provide, in order to permit the creation of POSIX conformance test
suites. This standard is primarily oriented to running POSIX conformance
tests, but its requirements also support testing of features not related
to POSIX conformance. POSIX 1003.3 does not specify a particular testing
framework, but at this time there is only one other POSIX conforming test
framework: TET. TET was created by Unisoft for a consortium comprised of
X/Open, Unix International and the Open Software Foundation.</para>
<para>The POSIX documentation refers to <firstterm>assertions</firstterm>.
An assertion is a description of behavior. For example, if a standard
says ``The sun shall shine'', a corresponding assertion might be ``The
sun is shining.'' A test based on this assertion would pass or fail
depending on whether it is day or night. It is important to note
that the standard being tested is never 1003.3; the standard being tested
is some other standard, for which the assertions were written.</para>
<para>As there is no testsuite to test testing frameworks for POSIX
1003.3 conformance, verifying conformance to this standard is done by
repeatedly reading the standard and experimenting. One of the main
things 1003.3 does specify is the set of allowed output messages and
their definitions. Four messages are supported for a required feature of
POSIX conforming systems and a fifth for a conditional feature. &dj;
supports the use of all five output messages. In this sense a testsuite
that uses exactly these messages can be considered POSIX conforming.
These definitions specify the output of a test
case:</para>
<variablelist>
<varlistentry>
<term>PASS</term>
<listitem><para>A test has succeeded. That is, it demonstrated that
the assertion is true.</para></listitem>
</varlistentry>
<varlistentry>
<term>XFAIL</term>
<listitem><para>POSIX 1003.3 does not incorporate the notion of
expected failures, so <emphasis>PASS</emphasis>, instead of
<emphasis>XPASS</emphasis>, must also be returned for test cases
which were expected to fail and did not. This means that
<emphasis>PASS</emphasis> is in some sense more ambiguous than if
<emphasis>XPASS</emphasis> is also used.</para></listitem>
</varlistentry>
<varlistentry>
<term>FAIL</term>
<listitem><para>A test has produced the bug it was intended to
capture. That is, it has demonstrated that the assertion is false.
The <emphasis>FAIL</emphasis> message is based on the test case only.
Other messages are used to indicate a failure of the framework. As
with <emphasis>PASS</emphasis>, POSIX tests must return
<emphasis>FAIL</emphasis> rather than <emphasis>XFAIL</emphasis> even
if a failure was expected.</para></listitem>
</varlistentry>
<varlistentry>
<term>UNRESOLVED</term>
<listitem><para>A test produced indeterminate results. Usually, this
means the test executed in an unexpected fashion; this outcome
requires that a human being go over results, to determine if the test
should have passed or failed. This message is also used for any test
that requires human intervention because it is beyond the abilities
of the testing framework. Any unresolved test should resolved to
<emphasis>PASS</emphasis> or <emphasis>FAIL</emphasis> before a test
run can be considered finished.</para>
<para>Note that for POSIX, each assertion must produce a test result
code. If the test isn't actually run, it must produce
<emphasis>UNRESOLVED</emphasis> rather than just leaving that test
out of the output. This means that you have to be careful when
writing tests to not carelessly use Tcl commands like
<emphasis>return</emphasis>---if you alter the flow of control of the
Tcl code you must insure that every test still produces some result
code.</para>
<para>Here are some of the ways a test may wind up
<emphasis>UNRESOLVED</emphasis>:</para></listitem>
</varlistentry>
</variablelist>
<itemizedlist>
<listitem><para>A test's execution is
interrupted.</para></listitem>
<listitem><para>A test does not produce a clear
result. This is usually because there was an
<emphasis>ERROR</emphasis> from &dj; while processing
the test, or because there were three or more
<emphasis>WARNING</emphasis> messages. Any
<emphasis>WARNING</emphasis> or <emphasis>ERROR</emphasis>
messages can invalidate the output of the test. This
usually requires a human being to examine the output to
determine what really happened---and to improve the test
case.</para></listitem>
<listitem><para>A test depends on a previous test, which
fails.</para></listitem>
<listitem><para>The test was set up
incorrectly.</para></listitem>
</itemizedlist>
<variablelist>
<varlistentry>
<term>UNTESTED</term>
<listitem><para>A test was not run. This is a place-holder, used
when there is no real test case yet.</para></listitem>
</varlistentry>
</variablelist>
<para>The only remaining output message left is intended to test
features that are specified by the applicable POSIX standard as
conditional:</para>
<variablelist>
<varlistentry>
<term>UNSUPPORTED</term>
<listitem><para>There is no support for the tested case. This may
mean that a conditional feature of an operating system, or of a
compiler, is not implemented. &dj; also uses this message when
a testing environment (often a ``bare board'' target) lacks basic
support for compiling or running the test case. For example, a
test for the system subroutine <emphasis>gethostname</emphasis>
would never work on a target board running only a boot
monitor.</para></listitem>
</varlistentry>
</variablelist>
<para>&dj; uses the same output procedures to produce these messages
for all testsuites and these procedures are already known to conform
to POSIX 1003.3. For a &dj; testsuite to conform to POSIX 1003.3,
you must avoid the <emphasis>setup_xfail</emphasis> procedure as
described in the <emphasis>PASS</emphasis> section above and you must
be careful to return <emphasis>UNRESOLVED</emphasis> where appropriate,
as described in the <emphasis>UNRESOLVED</emphasis> section
above.</para>
</sect2>
</sect1>
<!-- include the user manual -->
&user;
<!-- include the reference manual -->
&ref;
</article>
<!-- Keep this comment at the end of the file
Local variables:
mode: sgml
sgml-omittag:t
sgml-shorttag:t
sgml-namecase-general:t
sgml-general-insert-case:lower
sgml-minimize-attributes:nil
sgml-always-quote-attributes:t
sgml-indent-step:1
sgml-indent-data:nil
sgml-parent-document:nil
sgml-exposed-tags:nil
sgml-local-catalogs:nil
sgml-local-ecat-files:nil
End:
-->
|