1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
|
Testing with QMTest
===================
You can use QMTest to test G++. (In the future, it may be possible to
test other parts of GCC with QMTest as well, but it is not possible
yet.)
The use of QMTest to run the G++ tests has not been approved as an
officially supported testing procedure. Therefore, you must run the
tests using DejaGNU (with "make check-g++") before committing changes
that affect G++.
QMTest emulates DejaGNU behavior very closely when running the tests.
QMTest has two output modes: a DejaGNU emulation mode and a native
QMTest mode.
In the DejaGNU mode, you should receive output that is almost exactly
the same as the DejaGNU output; in particular, you should see the same
number of passes, failures, etc. When using the DejaGNU-style output,
QMTest uses the "xfail" indications in the test cases to determine
which tests are expected to pass and which are expected to fail, and
presents that information in the same way as DejaGNU.
In the QMTest mode, the number of passes and failures will be
different from that obtained when using DejaGNU. The reason is that a
single source file may contain multiple DejaGNU tests. In DejaGNU,
each line where a diagnostic is expected is considered a separate
test. Testing for successful compilation and testing for successful
execution of the generated program are considered separate tests. So,
a single source file "test.C" could contain, say, seven tests; some of
which might pass and some of which might fail.
In the QMTest mode, each source file is considered a single test. If
any of the seven sub-tests fail, the entire test is considered to
fail. However, QMTest does present information about *why* the test
failed, so the same information is effectively available.
In the QMTest mode, whether or not a test is expected to fail is
determined not by an indication in the test, but rather by comparing
the new results to the results of a previous run. Testing for whether
a change caused a regression is very simple: run the tests before
making the change, run them again after making the change, and let
QMTest compare the results.
The mode chosen only affects the output from QMTest, not how it runs
the tests or how it stores the data. Therefore, if you choose to run
in the QMTest mode and later want to get the DejaGNU style output, or
vice versa, you can do that as described below.
Setting Up
==========
You must download and install the following software:
- Python 2.2 (or greater)
See http://www.python.org.
You may already have Python on your system; in particular, many
GNU/Linux systems ship with Python installed.
Installation instructions are available on the web-site.
- A current version of QMTest. No released version provides all of
the functionality required, so you must obtain QMTest from CVS.
To do that, follow the instructions at:
http://www.codesourcery.com/qmtest
Installation instructions are available in the file called README
after you check out QMTest.
- The "qmtc" and "qmtest_gcc" QMTest support packages. These are
available from the same CVS repository as QMTest. For example, to
check out "qmtc", do:
cvs -d :pserver:anoncvs@cvs.codesourcery.com:/home/qm/Repository \
co qmtc
You do not have to install these packages; you need only check them
out.
Running the Tests
=================
First, you must set QMTEST_CLASS_PATH so that it can find the qmtc and
qmtest_gcc support packages:
export QMTEST_CLASS_PATH=/path/to/qmtc:/path/to/qmtest_gcc
The, run "make qmtest-g++" in the gcc directory of your build tree.
Here are some more advanced usage instructions:
1. To run a particular set of tests (rather than all of the tests),
use the make variable "QMTEST_GPP_TESTS". For example,
make QMTEST_GPP_TESTS="g++.dg" qmtest-g++
will run only the tests in the g++.dg subdirectory, and:
make QMTEST_GPP_TESTS="g++.dg/special/conpr-1.C \
g++.old-deja/g++.other/access2.C"
qmtest-g++
will run only the two tests indicated.
2. To run qmtest with particular flags, use the make variables
"QMTESTFLAGS" and "QMTESTRUNFLAGS". For example:
make QMTESTFLAGS="-v" QMTESTRUNFLAGS="-f full" qmtest-g++
will run qmtest like this:
qmtest -v run -f full ...
(The "-f full" mode will provide detailed information about each
test as it runs.)
3. To run the compiler with particular flags, use QMTESTRUNFLAGS to
set the QMTest context variable "CompilerTable.cplusplus_options",
like this:
make \
QMTESTRUNFLAGS='-c CompilerTable.cplusplus_options="-funroll-loops"' \
qmtest-g++
The compiler will then use the "-funroll-loops" switch when
compiling.
4. If qmtest is not in your path, you can indicate the full path to
QMTest by using the make variable "QMTEST_PATH", like this:
make QMTEST_PATH=/path/to/qmtest qmtest-g++
5. To start the QMTest GUI, use:
make qmtest-gui
(Note that this will run the program called "mozilla" in your path.
If you want to use another browser, you must configure qmtest as
described in its manual.)
Bear in mind that the QMTest GUI is insecure; malicious users with
access to your machine may be able to run commands as if they were
you. The QMTest GUI only binds to the loopback IP addresss, which
provides a measure of security, but not enough for use in untrusted
environments.
6. If you have a multiprocessor, you can run the tests in parallel by
passing the "-j" option to qmtest:
make QMTESTRUNFLAGS="-j 4" qmtest-g++
will run tests in four threads. (It is also possible to run tests
across multiple machines; for more information see the QMTest
manual.)
7. If a test (say "g++.dg/abi/bitfield1.C") fails, and you want to get
more detailed information, you can do:
cd qmtestsuite
qmtest summarize g++.qmr g++.dg/abi/bitfield1.C
to get more information about the commands that were run and the
output produced.
|
