diff options
Diffstat (limited to 'libstdc++-v3/doc/html/test.html')
| -rw-r--r-- | libstdc++-v3/doc/html/test.html | 722 |
1 files changed, 722 insertions, 0 deletions
diff --git a/libstdc++-v3/doc/html/test.html b/libstdc++-v3/doc/html/test.html new file mode 100644 index 0000000..8a8694c --- /dev/null +++ b/libstdc++-v3/doc/html/test.html @@ -0,0 +1,722 @@ +<?xml version="1.0" encoding="ISO-8859-1"?> +<!DOCTYPE html + PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" + "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> + +<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en"> +<head> + <meta name="AUTHOR" content="bkoz@gcc.gnu.org (Benjamin Kosnik)" /> + <meta name="KEYWORDS" content="c++, libstdc++, test, regression, g++" /> + <meta name="DESCRIPTION" content="README for the GNU libstdc++ effort." /> + <meta name="GENERATOR" content="vi and eight fingers" /> + <title>libstdc++ Testing Instructions</title> +<link rel="StyleSheet" href="lib3styles.css" /> +</head> +<body> + +<h1 class="centered"><a name="top">Testing Details</a></h1> + +<p class="fineprint"><em> + The latest version of this document is always available at + <a href="http://gcc.gnu.org/onlinedocs/libstdc++/test.html"> + http://gcc.gnu.org/onlinedocs/libstdc++/test.html</a>. +</em></p> + +<p><em> + To the <a href="http://gcc.gnu.org/libstdc++/">libstdc++ homepage</a>. +</em></p> + +<!-- ####################################################### --> +<hr /> +<h2>Contents</h2> +<ul> + <li><a href="#org">Testsuite organization and naming conventions</a></li> + <li><a href="#util">Utilities: abicheck and libtestc++</a></li> + <li><a href="#new">How to write a new test case</a></li> + <li><a href="#check">Options for running the tests</a></li> + <li><a href="#debug">Running debug-mode tests</a></li> + <li><a href="#future">Future</a></li> + <li><a href="#internals">DejaGNU internals</a></li> +</ul> + +<hr /> + +<!-- ####################################################### --> + +<h2><a name="org">Testsuite organization and naming conventions</a></h2> + <p> + The directory <em>libsrcdir/testsuite</em> contains the + individual test cases organized in sub-directories corresponding + to chapters of the C++ standard (detailed below), the dejagnu + test harness support files, and sources to various testsuite + utilities that are packaged in a separate testing library. + </p> + + <p> All test cases for functionality required by the runtime + components of the C++ standard (ISO 14882) are files within the + following directories. + </p> + + <pre> +17_intro +18_support +19_diagnostics +20_util +21_strings +22_locale +23_containers +25_algorithms +26_numerics +27_io + </pre> + + <p> + In addition, the following directories include test files: + </p> + + <pre> +tr1 Tests for components as described by the Technical Report on Standard Library Extensions (TR1). +backward Tests for backwards compatibility and deprecated features. +demangle Tests for __cxa_demangle, the IA 64 C++ ABI demangler +ext Tests for extensions. +performance Tests for performance analysis, and performance regressions. +thread Tests for threads. + </pre> + + <p> + Some directories don't have test files, but instead contain + auxiliary information (<a href="#internals">more information</a>): + </p> + + <pre> +config Files for the dejagnu test harness. +lib Files for the dejagnu test harness. +libstdc++* Files for the dejagnu test harness. +data Sample text files for testing input and output. +util Files for libtestc++, utilities and testing routines. + </pre> + + <p> + Within a directory that includes test files, there may be + additional subdirectories, or files. Originally, test cases + were appended to one file that represented a particular section + of the chapter under test, and was named accordingly. For + instance, to test items related to <code> 21.3.6.1 - + basic_string::find [lib.string::find]</code> in the standard, + the following was used: + </p> + <pre> +21_strings/find.cc + </pre> + <p> + However, that practice soon became a liability as the test cases + became huge and unwieldy, and testing new or extended + functionality (like wide characters or named locales) became + frustrating, leading to aggressive pruning of test cases on some + platforms that covered up implementation errors. Now, the test + suite has a policy of one file, one test case, which solves the + above issues and gives finer grained results and more manageable + error debugging. As an example, the test case quoted above + becomes: + </p> + <pre> +21_strings/basic_string/find/char/1.cc +21_strings/basic_string/find/char/2.cc +21_strings/basic_string/find/char/3.cc +21_strings/basic_string/find/wchar_t/1.cc +21_strings/basic_string/find/wchar_t/2.cc +21_strings/basic_string/find/wchar_t/3.cc + </pre> + + <p> + All new tests should be written with the policy of one test + case, one file in mind. + </p> + + <p> + In addition, there are some special names and suffixes that are + used within the testsuite to designate particular kinds of + tests. + </p> + +<ul> +<li> + <em>_xin.cc</em> + <p> + This test case expects some kind of interactive input in order + to finish or pass. At the moment, the interactive tests are not + run by default. Instead, they are run by hand, like: + </p> + <pre> +g++ 27_io/objects/char/3_xin.cc +cat 27_io/objects/char/3_xin.in | a.out + </pre> +</li> +<li> + <em>.in</em> + <p> + This file contains the expected input for the corresponding <em> + _xin.cc</em> test case. + </p> +</li> +<li> + <em>_neg.cc</em> + <p> + This test case is expected to fail: it's a negative test. At the + moment, these are almost always compile time errors. + </p> +</li> +<li> + <em>char</em> + <p> + This can either be a directory name or part of a longer file + name, and indicates that this file, or the files within this + directory are testing the <code>char</code> instantiation of a + template. + </p> +</li> +<li> + <em>wchar_t</em> + <p> + This can either be a directory name or part of a longer file + name, and indicates that this file, or the files within this + directory are testing the <code>wchar_t</code> instantiation of + a template. Some hosts do not support <code>wchar_t</code> + functionality, so for these targets, all of these tests will not + be run. + </p> +</li> +<li> + <em>thread</em> + <p> + This can either be a directory name or part of a longer file + name, and indicates that this file, or the files within this + directory are testing situations where multiple threads are + being used. + </p> +</li> +<li> + <em>performance</em> + <p> + This can either be an enclosing directory name or part of a + specific file name. This indicates a test that is used to + analyze runtime performance, for performance regression testing, + or for other optimization related analysis. At the moment, these + test cases are not run by default. + </p> +</li> +</ul> + +<hr /> +<h2><a name="util">Utilities: abi_check and libtestc++</a></h2> + <p> + The testsuite directory also contains some files that implement + functionality that is intended to make writing test cases easier, + or to avoid duplication, or to provide error checking in a way that + is consistent across platforms and test harnesses. A stand-alone + executable, called <em>abi_check</em>, and a static library called + <em>libtestc++</em> are constructed. Both of these items are not + installed, and only used during testing. + </p> + + <p> + These files include the following functionality: + </p> + + <ul> + <li> + <em>testsuite_abi.h</em>, + <em>testsuite_abi.cc</em>, + <em>testsuite_abi_check.cc</em> + <p> + Creates the executable <em>abi_check</em>. + Used to check correctness of symbol versioning, visibility of + exported symbols, and compatibility on symbols in the shared + library, for hosts that support this feature. More information + can be found in the ABI documentation <a href="abi.html"> here</a> + </p> + </li> + <li> + <em>testsuite_allocator.h</em>, + <em>testsuite_allocator.cc</em> + <p> + Contains specialized allocators that keep track of construction + and destruction. Also, support for overriding global new and + delete operators, including verification that new and delete + are called during execution, and that allocation over max_size + fails. + </p> + </li> + <li> + <em>testsuite_character.h</em> + <p> + Contains <code>std::char_traits</code> and + <code>std::codecvt</code> specializations for a user-defined + POD. + </p> + </li> + <li> + <em>testsuite_hooks.h</em>, + <em>testsuite_hooks.cc</em> + <p> + A large number of utilities, including: + </p> + <ul> + <li>VERIFY</li> + <li>set_memory_limits</li> + <li>verify_demangle</li> + <li>run_tests_wrapped_locale</li> + <li>run_tests_wrapped_env</li> + <li>try_named_locale</li> + <li>try_mkfifo</li> + <li>func_callback</li> + <li>counter</li> + <li>copy_tracker</li> + <li>copy_constructor</li> + <li>assignment_operator</li> + <li>destructor</li> + <li>pod_char, pod_int and associated char_traits specializations</li> + </ul> + <p></p> + </li> + <li> + <em>testsuite_io.h</em> + <p> + Error, exception, and constraint checking for + <code>std::streambuf, std::basic_stringbuf, std::basic_filebuf</code>. + </p> + </li> + <li> + <em>testsuite_iterators.h</em> + <p> + Wrappers for various iterators. + </p> + </li> + <li> + <em>testsuite_performance.h</em> + <p> + A number of class abstractions for performance counters, and + reporting functions including: + </p> + <ul> + <li>time_counter</li> + <li>resource_counter</li> + <li>report_performance</li> + </ul> + <p></p> + </li> + </ul> + +<hr /> +<h2><a name="new">How to write a new test case</a></h2> + + <p> + The first step in making a new test case is to choose the correct + directory and file name, given the organization as previously + described. + </p> + + <p> + All files are copyright the FSF, and GPL'd: this is very + important. The first copyright year should correspond to the date + the file was checked in to SVN. + </p> + + <p> + As per the dejagnu instructions, always return 0 from main to + indicate success. + </p> + + <p> + A bunch of utility functions and classes have already been + abstracted out into the testsuite utility library, <code> + libtestc++</code>. To use this functionality, just include the + appropriate header file: the library or specific object files will + automatically be linked in as part of the testsuite run. + </p> + + <p> + For a test that needs to take advantage of the dejagnu test + harness, what follows below is a list of special keyword that + harness uses. Basically, a test case contains dg-keywords (see + dg.exp) indicating what to do and what kinds of behavior are to be + expected. New test cases should be written with the new style + DejaGnu framework in mind. + </p> + + <p> + To ease transition, here is the list of dg-keyword documentation + lifted from dg.exp. + </p> + +<pre> +# The currently supported options are: +# +# dg-prms-id N +# set prms_id to N +# +# dg-options "options ..." [{ target selector }] +# specify special options to pass to the tool (eg: compiler) +# +# dg-do do-what-keyword [{ target/xfail selector }] +# `do-what-keyword' is tool specific and is passed unchanged to +# ${tool}-dg-test. An example is gcc where `keyword' can be any of: +# preprocess|compile|assemble|link|run +# and will do one of: produce a .i, produce a .s, produce a .o, +# produce an a.out, or produce an a.out and run it (the default is +# compile). +# +# dg-error regexp comment [{ target/xfail selector } [{.|0|linenum}]] +# indicate an error message <regexp> is expected on this line +# (the test fails if it doesn't occur) +# Linenum=0 for general tool messages (eg: -V arg missing). +# "." means the current line. +# +# dg-warning regexp comment [{ target/xfail selector } [{.|0|linenum}]] +# indicate a warning message <regexp> is expected on this line +# (the test fails if it doesn't occur) +# +# dg-bogus regexp comment [{ target/xfail selector } [{.|0|linenum}]] +# indicate a bogus error message <regexp> use to occur here +# (the test fails if it does occur) +# +# dg-build regexp comment [{ target/xfail selector }] +# indicate the build use to fail for some reason +# (errors covered here include bad assembler generated, tool crashes, +# and link failures) +# (the test fails if it does occur) +# +# dg-excess-errors comment [{ target/xfail selector }] +# indicate excess errors are expected (any line) +# (this should only be used sparingly and temporarily) +# +# dg-output regexp [{ target selector }] +# indicate the expected output of the program is <regexp> +# (there may be multiple occurrences of this, they are concatenated) +# +# dg-final { tcl code } +# add some tcl code to be run at the end +# (there may be multiple occurrences of this, they are concatenated) +# (unbalanced braces must be \-escaped) +# +# "{ target selector }" is a list of expressions that determine whether the +# test succeeds or fails for a particular target, or in some cases whether the +# option applies for a particular target. If the case of `dg-do' it specifies +# whether the test case is even attempted on the specified target. +# +# The target selector is always optional. The format is one of: +# +# { xfail *-*-* ... } - the test is expected to fail for the given targets +# { target *-*-* ... } - the option only applies to the given targets +# +# At least one target must be specified, use *-*-* for "all targets". +# At present it is not possible to specify both `xfail' and `target'. +# "native" may be used in place of "*-*-*". + +Example 1: Testing compilation only +// { dg-do compile } + +Example 2: Testing for expected warnings on line 36, which all targets fail +// { dg-warning "string literals" "" { xfail *-*-* } 36 + +Example 3: Testing for expected warnings on line 36 +// { dg-warning "string literals" "" { target *-*-* } 36 + +Example 4: Testing for compilation errors on line 41 +// { dg-do compile } +// { dg-error "no match for" "" { target *-*-* } 41 } + +Example 5: Testing with special command line settings, or without the +use of pre-compiled headers, in particular the stdc++.h.gch file. Any +options here will override the DEFAULT_CXXFLAGS and PCH_CXXFLAGS set +up in the normal.exp file. +// { dg-options "-O0" { target *-*-* } } +</pre> + + <p> + More examples can be found in the libstdc++-v3/testsuite/*/*.cc files. + </p> + +<hr /> +<h2><a name="check">Options for running the tests</a></h2> + + <p> There are several options for running tests, including testing + the regression tests, testing a subset of the regression tests, + testing the performance tests, testing just compilation, testing + installed tools, etc. In addition, there is a special rule for + checking the exported symbols of the shared library. + </p> + + <p>You can check the status of the build without installing it + using the dejagnu harness, much like the rest of the gcc tools.</p> + <pre> make check</pre> + <p>in the <em>libbuilddir</em> directory.</p> + <p>or</p> + <pre> make check-target-libstdc++-v3</pre> + <p>in the <em>gccbuilddir</em> directory.</p> + + <p> + These commands are functionally equivalent and will create a + 'testsuite' directory underneath <em>libbuilddir</em> containing + the results of the tests. Two results files will be generated: + <em> libstdc++.sum</em>, which is a PASS/FAIL summary for each + test, and <em>libstdc++.log</em> which is a log of the exact + command line passed to the compiler, the compiler output, and + the executable output (if any). + </p> + + +<p> +To debug the dejagnu test harness during runs, try invoking with a +specific argument to the variable RUNTESTFLAGS, as below. +</p> + +<pre> +make check-target-libstdc++-v3 RUNTESTFLAGS="-v" +</pre> +or +<pre> +make check-target-libstdc++-v3 RUNTESTFLAGS="-v -v" +</pre> + +<p> +To run a subset of the library tests, you will need to generate the +<em>testsuite_files</em> file by running <tt>make testsuite_files</tt> +in the <em>libbuilddir/testsuite</em> directory, described below. +Edit the file to remove the tests you don't want and then run the +testsuite as normal. +</p> + + +<p> +There are two ways to run on a simulator: set up DEJAGNU to point to a +specially crafted site.exp, or pass down --target_board flags. +</p> +Example flags to pass down for various embedded builds are as follows: +<pre> +--target=powerpc-eabism (libgloss/sim) +make check-target-libstdc++-v3 RUNTESTFLAGS="--target_board=powerpc-sim" + +--target=calmrisc32 (libgloss/sid) +make check-target-libstdc++-v3 RUNTESTFLAGS="--target_board=calmrisc32-sid" + +--target=xscale-elf (newlib/sim) +make check-target-libstdc++-v3 RUNTESTFLAGS="--target_board=arm-sim" +</pre> + +<p> Also, here is an example of how to run the libstdc++ testsuite for a +multilibed build directory with different ABI settings: +</p> +<pre> +make check-target-libstdc++-v3 RUNTESTFLAGS='--target_board \"unix{-mabi=32,,-mabi=64}\"' +</pre> + +<p> +You can run the tests with a compiler and library that have already +been installed. Make sure that the compiler (e.g., <code>g++</code>) +is in your <code>PATH</code>. If you are using shared libraries, then +you must also ensure that the directory containing the shared version +of libstdc++ is in your <code>LD_LIBRARY_PATH</code>, or equivalent. +If your GCC source tree is at <code>/path/to/gcc</code>, then you can +run the tests as follows: +</p> +<pre> +runtest --tool libstdc++ --srcdir=/path/to/gcc/libstdc++-v3/testsuite +</pre> +<p> +The testsuite will create a number of files in the directory in which you +run this command,. Some of those files might use the same name as +files created by other testsuites (like the ones for GCC and G++), so +you should not try to run all the testsuites in parallel from the same +directory. +</p> + + <p> In addition, there are some testing options that are mostly of + interest to library maintainers and system integrators. As such, + these tests may not work on all cpu and host combinations, and may need to + be executed in the <em>libbuilddir/testsuite</em> directory. These options + include, but are not necessarily limited to, the following: + </p> + + <pre> + make testsuite_files</pre> + <p> + Five files are generated that determine what test files + are run. These files are: + </p> + <ul> + <li> + <em>testsuite_files </em> + <p> This is a list of all the test cases that will be run. Each + test case is on a separate line, given with an absolute path + from the <em>libsrcdir/testsuite</em> directory. + </p> + </li> + + <li> + <em>testsuite_files_interactive </em> + <p> This is a list of all the interactive test cases, using the + same format as the file list above. These tests are not run by default. + </p> + </li> + + <li> + <em>testsuite_files_performance</em> + <p> This is a list of all the performance test cases, using the + same format as the file list above. These tests are not run by default. + </p> + </li> + + <li> + <em>testsuite_thread</em> + <p> This file indicates that the host system can run tests which + incolved multiple threads. + </p> + </li> + + <li> + <em>testsuite_wchar_t</em> + <p> This file indicates that the host system can run the wchar_t + tests, and corresponds to the macro definition <code> + _GLIBCXX_USE_WCHAR_T</code> in the file c++config.h. + </p> + </li> + </ul> + + <pre> + make check-abi</pre> + <p>The library ABI can be tested. This involves testing the shared + library against an ABI-defining previous version of symbol exports. </p> + + <pre> + make check-compile</pre> + <p>This rule compiles, but does not link or execute, the + <em>testsuite_files</em> test cases and displays the output on stdout.</p> + + <pre> + make check-performance</pre> + <p>This rule runs through the <em>testsuite_files_performance</em> + test cases and collects information for performance analysis and + can be used to spot performance regressions. Various timing + information is collected, as well as number of hard page faults, + and memory used. This is not run by default, and the implementation + is in flux. +</p> + + <p> + We are interested in any strange failures of the + testsuite; please see <a href="faq/index.html#2_4">FAQ 2.4</a> + for which files to examine. + </p> + +<hr/> +<h2><a name="debug">Running debug-mode tests</a></h2> +<p>To run the libstdc++ test suite under the <a + href="debug.html#safe">debug mode</a>, + edit <code>libstdc++-v3/scripts/testsuite_flags</code> to add the + compile-time flag <code>-D_GLIBCXX_DEBUG</code> to the result + printed by the <code>--build-cxx</code> option. Additionally, add + the <code>-D_GLIBCXX_DEBUG_PEDANTIC</code> flag to turn on pedantic + checking. The libstdc++ test suite should produce precisely the same + results under debug mode that it does under release mode: any + deviation indicates an error in either the library or the test + suite.</p> + +<hr /> +<h2><a name="future">Future</a></h2> + +<p> +Shared runs need to be implemented, for targets that support shared libraries. +</p> + +<p> +Diffing of expected output to standard streams needs to be finished off. +</p> + +<p> +The V3 testing framework supports, or will eventually support, +additional keywords for the purpose of easing the job of writing +test cases. All V3-keywords are of the form <code>@xxx@</code>. +Currently plans for supported keywords include: +</p> + +<dl> +<dt> <code> @require@ <files> </code> </dt> +<dd> + <p> + The existence of <files> is essential for the test to complete + successfully. For example, a test case foo.C using bar.baz as + input file could say + </p> + <pre> + // @require@ bar.baz</pre> + <p> + The special variable % stands for the rootname, e.g. the + file-name without its `.C' extension. Example of use (taken + verbatim from 27_io/filebuf.cc) + </p> + <pre> + // @require@ %-*.tst %-*.txt</pre> +</dd> +<dt> <code> @diff@ <first-list> <second-list> </code> </dt> +<dd> + <p> + After the test case compiles and ran successfully, diff + <first-list> against <second-list>, these lists should + have the same length. The test fails if diff returns non-zero a + pair of files. + </p> +</dd> +</dl> + +<hr /> +<h2><a name="internals">DejaGNU internals</a></h2> + +<p>This is information for those looking at making changes to the testsuite +structure, and/or needing to trace dejagnu's actions with --verbose. This +will not be useful to people who are "merely" adding new tests to the existing +structure. +</p> + +<p>The first key point when working with dejagnu is the idea of a "tool". +Files, directories, and functions are all implicitly used when they are +named after the tool in use. Here, the tool will always be "libstdc++". +</p> + +<p>The <code>lib</code> subdir contains support routines. The +<code>lib/libstdc++.exp</code> file ("support library") is loaded +automagically, and must explicitly load the others. For example, files can +be copied from the core compiler's support directory into <code>lib</code>. +</p> + +<p>Some routines in <code>lib/libstdc++.exp</code> are callbacks, some are +our own. Callbacks must be prefixed with the name of the tool. To easily +distinguish the others, by convention our own routines are named "v3-*". +</p> + +<p>The next key point when working with dejagnu is "test files". Any +directory whose name starts with the tool name will be searched for test files. +(We have only one.) In those directories, any <code>.exp</code> file is +considered a test file, and will be run in turn. Our main test file is called +<code>normal.exp</code>; it runs all the tests in testsuite_files using the +callbacks loaded from the support library. +</p> + +<p>The <code>config</code> directory is searched for any particular "target +board" information unique to this library. This is currently unused and sets +only default variables. +</p> + + +<!-- ####################################################### --> + +<hr /> +<p class="fineprint"><em> +See <a href="17_intro/license.html">license.html</a> for copying conditions. +Comments and suggestions are welcome, and may be sent to +<a href="mailto:libstdc++@gcc.gnu.org">the libstdc++ mailing list</a>. +</em></p> + + +</body> +</html> |