From ca5e1be83df4629faf3bccabcc8e712447ac390e Mon Sep 17 00:00:00 2001 From: Benjamin Kosnik Date: Tue, 20 May 2003 22:25:09 +0000 Subject: README: Move to... 2003-05-20 Benjamin Kosnik * testsuite/README: Move to... * docs/html/test.html: ...here. Add documentation. * docs/html/install.html: Move testing bits out.. * docs/html/documentation.html: Add separate testing link. * testsuite/performance: Add. * testsuite/performance/allocator.cc: New. * testsuite/performance/complex_norm.cc: New. * testsuite/performance/cout_insert_int.cc: New. * testsuite/performance/fstream_seek_write.cc: New. * testsuite/performance/ifstream_getline.cc: New. * testsuite/performance/map_create_fill.cc: New. * testsuite/performance/ofstream_insert_float.cc: New. * testsuite/performance/ofstream_insert_int.cc: New. * testsuite/performance/string_append.cc: New. * testsuite/lib/libstdc++-v3-dg.exp (v3-compute-tests): Filter performance tests. From-SVN: r67040 --- libstdc++-v3/docs/html/documentation.html | 3 +- libstdc++-v3/docs/html/install.html | 32 +- libstdc++-v3/docs/html/test.html | 564 ++++++++++++++++++++++++++++++ 3 files changed, 567 insertions(+), 32 deletions(-) create mode 100644 libstdc++-v3/docs/html/test.html (limited to 'libstdc++-v3/docs') diff --git a/libstdc++-v3/docs/html/documentation.html b/libstdc++-v3/docs/html/documentation.html index f1e2cd7..827ed22 100644 --- a/libstdc++-v3/docs/html/documentation.html +++ b/libstdc++-v3/docs/html/documentation.html @@ -45,11 +45,12 @@

Configuring, Building, Installing

Configuring, Building, Testing, Installing

diff --git a/libstdc++-v3/docs/html/install.html b/libstdc++-v3/docs/html/install.html index e7e495a..038291b8 100644 --- a/libstdc++-v3/docs/html/install.html +++ b/libstdc++-v3/docs/html/install.html @@ -14,7 +14,7 @@ -

libstdc++-v3 INSTALL

Getting started: configure, build, install

The latest version of this document is always available at @@ -312,36 +312,6 @@ se_NO.UTF-8 UTF-8 the headers and library files will be moved under lib/gcc-lib/ instead.

-
You can check the status of the build without installing it using
-
- make check
-
or you can check the status of the installed library using
-
- make check-install
-
in the libbuilddir directory. - These commands will create a 'testsuite' directory underneath - libbuilddir containing the results of the tests. We are - interested in any strange failures of the testsuite; please see - FAQ 2.4 for which files to examine. -
- -
In addition, there are some testing options that are mostly of - interest to library maintainers and system integrators. As such, - these tests may not work on all cpu and host combinations. These - options include, but are not necessarily limited to, the following: -
- -
The library ABI can be tested using
-
- make check-abi
- -
The library can also be tested using a bash script, instead of - the default dejagnu test harness
-
- make check-script
-
or
-
- make check-script-install

Using the library
diff --git a/libstdc++-v3/docs/html/test.html b/libstdc++-v3/docs/html/test.html new file mode 100644 index 0000000..61e3725 --- /dev/null +++ b/libstdc++-v3/docs/html/test.html @@ -0,0 +1,564 @@ + + + + + + + + + + libstdc++-v3 Testing Instructions + + + + +
Testing Details
+ +
+ The latest version of this document is always available at + + http://gcc.gnu.org/onlinedocs/libstdc++/test.html. +
+ +
+ To the libstdc++-v3 homepage. +
+ + +
+
Contents
+
+
Testsuite organization and naming conventions
+
Utilities: abicheck and libv3test
+
How to write a new test case
+
Options for running the tests
+
Future
+
+ +
+ + + +
Testsuite organization and naming conventions
+
+ The directory libsrcdir/testsuite contains the test + files, test harness, and utility information for verifying the + correctness of C++ library on a given host. It includes the + following directories, each named after a specific chapter of + the C++ standard, and each containing test files or + subdirectories of test files that test for that particular part + of the standard. +
+ +
+17_intro +18_support +19_diagnostics +20_util +21_strings +22_locale +23_containers +25_algorithms +26_numerics +27_io +
+ +
+ In addition, the following directories include test files: +
+ +
+backward Tests for backwards compatibility and deprecated features. +demangle Tests for __cxa_demangle, the IA 64 C++ ABI demangler +ext Tests for extensions. +performance Tests for performance analysis, and performance regressions. +thread Tests for threads. +
+ +
+ Some directories don't have test files, but instead contain + auxiliary information: +
+ +
+config Files for the dejagnu test harness. +lib Files for the dejagnu test harness. +libstdc++-v3.dg Files for the dejagnu test harness. +data Sample text files for testing input and output. +
+ +
+ Within a directory that includes test files, there may be + additional subdirectories, or files: this particular point is in + flux. Originally, test cases were appended to one file that + represented a particular section of the chapter under test, and + was named accordingly. For instance, to test items related to + 21.3.6.1 - basic_string::find [lib.string::find] + in the standard, the following was used: +
+
+21_strings/find.cc +
+
+ However, that practice soon became a liability as the test cases + became huge and unwieldy, and testing new or extended + functionality (like wide characters or named locales) became + frustrating, leading to aggressive pruning of test cases on some + platforms that covered up implementation errors. Now, the test + suite is converging on a policy of one file, one test case, + which solves the above issues and gives finer grained results + and more manageable error debugging. As an example, the test case + quoted above becomes: +
+
+21_strings/basic_string/find/char/1.cc +21_strings/basic_string/find/char/2.cc +21_strings/basic_string/find/char/3.cc +21_strings/basic_string/find/wchar_t/1.cc +21_strings/basic_string/find/wchar_t/2.cc +21_strings/basic_string/find/wchar_t/3.cc +
+ +
+ All new tests should be written with the policy of one test + case, one file in mind. At some point the entire testsuite will + be converted: the current status is that the 21_string, + 22_locale, 27_io, and demangle directories have all been + transitioned. +
+ +
+ In addition, there are some special names and suffixes that are + used within the testsuite to designate particular kinds of + tests. +
+ +
+
+ _xin.cc +
+ This test case expects some kind of interactive input in order + to finish or pass. At the moment, the interactive tests are not + run by default. Instead, they are run by hand, like: +
+g++ 27_io/objects/char/3_xin.cc +cat 27_io/objects/char/3_xin.in | a.out +
+
+
+
+ .in +
+ This file contains the expected input for the corresponding + _xin.cc test case. +
+
+
+ _neg.cc +
+ This test case is expected to fail: it's a negative test. At the + moment, these are almost always compile time errors. +
+
+
+ char +
+ This can either be a directory name or part of a longer file + name, and indicates that this file, or the files within this + directory are testing the char instantiation of a + template. +
+
+
+ wchar_t +
+ This can either be a directory name or part of a longer file + name, and indicates that this file, or the files within this + directory are testing the wchar_t instantiation of + a template. Some hosts do not support wchar_t + functionality, so for these targets, all of these tests will not + be run. +
+
+
+ performance +
+ This can either be an enclosing directory name or part of a + specific file name. This indicates a test that is used to + analyze runtime performance, for performance regression testing, + or for other optimization related analysis. At the moment, these + test cases are not run by default, and instead assumed to be run + manually. +
+
+
+ +
+
Utilities: abicheck and libv3test
+
+ The testsuite directory also contains some files that implement + functionality that is intended to make writing test cases easier, + or to avoid duplication, or to provide error checking in a way that + is consistent across platforms and test harnesses. A stand-alone + executable, called abi_check, and a static library called + libv3test are constructed during the build. Both of these + items are not installed, and only used during testing. +
+ +
+ These files include the following functionality: +
+ +
+
+ abi_check.cc +
+ Creates the executable abi_check. + Used to check correctness of symbol versioning, visibility of + exported symbols, and compatibility on symbols in the shared + library, for hosts that support this feature. More information + can be found in the ABI documentation here +
+
+
+ testsuite_allocator.h and + testsuite_allocator.cc +
+ Specialized allocators that keep track of construction and destruction +
+
+
+ testsuite_hooks.h and + testsuite_hooks.cc +
+ A large number of utilities, including: +
+
+
VERIFY
+
set_memory_limits
+
verify_demangle
+
run_tests_wrapped_locale
+
run_tests_wrapped_env
+
try_named_locale
+
counter
+
copy_constructor
+
assignment_operator
+
destructor
+
copy_tracker
+
pod_char, pod_int and associated char_traits specializations
+
+
+
+
+ printnow.c +
+ A cross-platform timer for use in one of the older harnesses + to determine compilation and link time. +
+
+
+ +
+
How to write a new test case
+ +
+ The first step in making a new test case is to choose the correct + directory and file name, given the organization as previously + described. +
+ +
+ All files are copyright the FSF, and GPL'd: this is very + important. The first copyright year should correspond to the date + the file was checked in to CVS. +
+ +
+ As per the dejagnu instructions, always return 0 from main to + indicate success. +
+ +
+ A bunch of utility functions and classes have already been + abstracted out into the testsuite utility library, + libv3test. To use this functionality, just include the + appropriate header file: the library will automatically be linked + in as part of the testsuite run. +
+ +
+ For a test that needs to take advantage of the dejagnu test + harness, what follows below is a list of special keyword that + harness uses. Basically, a test case contains dg-keywords (see + dg.exp) indicating what to do and what kinds of behavior are to be + expected. New test cases should be written with the new style + DejaGnu framework in mind. +
+ +
+ To ease transition, here is the list of dg-keyword documentation + lifted from dg.exp. +
+ +
+# The currently supported options are: +# +# dg-prms-id N +# set prms_id to N +# +# dg-options "options ..." [{ target selector }] +# specify special options to pass to the tool (eg: compiler) +# +# dg-do do-what-keyword [{ target/xfail selector }] +# `do-what-keyword' is tool specific and is passed unchanged to +# ${tool}-dg-test. An example is gcc where `keyword' can be any of: +# preprocess|compile|assemble|link|run +# and will do one of: produce a .i, produce a .s, produce a .o, +# produce an a.out, or produce an a.out and run it (the default is +# compile). +# +# dg-error regexp comment [{ target/xfail selector } [{.|0|linenum}]] +# indicate an error message is expected on this line +# (the test fails if it doesn't occur) +# Linenum=0 for general tool messages (eg: -V arg missing). +# "." means the current line. +# +# dg-warning regexp comment [{ target/xfail selector } [{.|0|linenum}]] +# indicate a warning message is expected on this line +# (the test fails if it doesn't occur) +# +# dg-bogus regexp comment [{ target/xfail selector } [{.|0|linenum}]] +# indicate a bogus error message use to occur here +# (the test fails if it does occur) +# +# dg-build regexp comment [{ target/xfail selector }] +# indicate the build use to fail for some reason +# (errors covered here include bad assembler generated, tool crashes, +# and link failures) +# (the test fails if it does occur) +# +# dg-excess-errors comment [{ target/xfail selector }] +# indicate excess errors are expected (any line) +# (this should only be used sparingly and temporarily) +# +# dg-output regexp [{ target selector }] +# indicate the expected output of the program is +# (there may be multiple occurrences of this, they are concatenated) +# +# dg-final { tcl code } +# add some tcl code to be run at the end +# (there may be multiple occurrences of this, they are concatenated) +# (unbalanced braces must be \-escaped) +# +# "{ target selector }" is a list of expressions that determine whether the +# test succeeds or fails for a particular target, or in some cases whether the +# option applies for a particular target. If the case of `dg-do' it specifies +# whether the test case is even attempted on the specified target. +# +# The target selector is always optional. The format is one of: +# +# { xfail *-*-* ... } - the test is expected to fail for the given targets +# { target *-*-* ... } - the option only applies to the given targets +# +# At least one target must be specified, use *-*-* for "all targets". +# At present it is not possible to specify both `xfail' and `target'. +# "native" may be used in place of "*-*-*". + +Example 1: Testing compilation only +// { dg-do compile } + +Example 2: Testing for expected warnings on line 36, which all targets fail +// { dg-warning "string literals" "" { xfail *-*-* } 36 + +Example 3: Testing for expected warnings on line 36 +// { dg-warning "string literals" "" { target *-*-* } 36 + +Example 4: Testing for compilation errors on line 41 +// { dg-do compile } +// { dg-error "no match for" "" { target *-*-* } 41 } +
+ +
+ More examples can be found in the libstdc++-v3/testsuite/*/*.cc files. +
+ +
+
Options for running the tests
+ +
There are several ways to run the testsuite. There are two + harnesses, one using dejagnu and one using bash. In addition, there + is a special rule for checking the ABI of the shared library. +
+ +
You can check the status of the build without installing it + using the dejagnu harness, much like the rest of the gcc tools.
+
+
make check
in the libbuilddir directory.
+
or
+
make check-target-libstdc++-v3
in the + gccbuilddir directory.
+ +
+ These commands are equivalent and will create a 'testsuite' + directory underneath libbuilddir containing the results + of the tests. Two results files will be generated: + libstdc++-v3.sum, which is a PASS/FAIL summary for each + test, and libstdc++.log which is a log of the exact + command line passed to the compiler, the compiler output, and + the executable output (if any). In addition, four files are + generated that determine what test files are run. These files + are: +
+ +
+
+ testsuite_files +
This is a list of all the test cases that will be run. Each + test case is on a separate line, given with an absolute path + from the libsrcdir/testsuite directory. +
+
+ +
+ testsuite_files_interactive +
This is a list of all the interactive test cases, using the + same format as the file list above. These tests are not run by default. +
+
+ +
+ testsuite_files_performance +
This is a list of all the performance test cases, using the + same format as the file list above. These tests are not run by default. +
+
+ +
+ testsuite_wchar_t +
This file indicates that the host system can run the wchar_t + tests, and corresponds to the macro definition + _GLIBCPP_USE_WCHAR_T in the file c++config.h. +
+
+
+ +
+To debug the dejagnu test harness during runs, try invoking with a +specific argument to the variable RUNTESTFLAGS, as below. +
+ +
+make check-target-libstdc++-v3 RUNTESTFLAGS="-v" +
+or +
+make check-target-libstdc++-v3 RUNTESTFLAGS="-v -v" +
+ +There are two ways to run on a simulator: set up DEJAGNU to point to a +specially crafted site.exp, or pass down --target_board flags. + +Example flags to pass down for various embedded builds are as follows: + +
+--target=powerpc-eabism (libgloss/sim) +make check-target-libstdc++-v3 RUNTESTFLAGS="--target_board=powerpc-sim" + +--target=calmrisc32 (libgloss/sid) +make check-target-libstdc++-v3 RUNTESTFLAGS="--target_board=calmrisc32-sid" + +--target=xscale-elf (newlib/sim) +make check-target-libstdc++-v3 RUNTESTFLAGS="--target_board=arm-sim" +
+ +
In addition, there are some testing options that are mostly of + interest to library maintainers and system integrators. As such, + these tests may not work on all cpu and host combinations, and must + be executed in the libbuilddir directory.
These options + include, but are not necessarily limited to, the following: +
+ +
+ The library can also be tested using a bash script, instead of + the default dejagnu test harness. +
+
+ make check-script
+
+ These commands use the generated test_file lists as above, but + run all the tests using both shared and static linking, and in + addition provide some additional diffing of expected output + files for the input/output tests. (This added diff may or may + not be useful or necessary at the moment.) In addition, these + tests provide size information for all the generated test cases, + so that size data for new compiler or linker features can be + collected. At one time timing information was attempted, so that + compile speeds, link speeds, etc. could be measured, however at + the moment all timing information is currently disabled. +
+ +
+ make check-script-install
+
As directly above, but tests an installed library, not the + library and compiler in the build tree. +
+ +
+ make check-abi
+
The library ABI can be tested. This involves testing the shared + library against an ABI-defining previous version.
+ +
+ We are interested in any strange failures of the + testsuite; please see FAQ 2.4 + for which files to examine. +
+ +
+
Future
+ +
+Shared runs need to be implemented, for targets that support shared libraries. +
+ +
+Diffing of expected output to standard streams needs to be finished off. +
+ +
+The V3 testing framework supports, or will eventually support, +additional keywords for the purpose of easing the job of writing +test cases. All V3-keywords are of the form @xxx@. Currently plans +for supported keywords include: +
+ + @require@ + The existence of is essential for the test to complete + successfully. For example, a test case foo.C using bar.baz as + input file could say + // @require@ bar.baz + The special variable % stands for the rootname, e.g. the + file-name without its `.C' extension. Example of use (taken + verbatim from 27_io/filebuf.cc) + // @require@ %-*.tst %-*.txt + + @diff@ + After the test case compiles and ran successfully, diff + against , these lists should have the + same length. The test fails if diff returns non-zero a pair of + files. + + + +
+
+See license.html for copying conditions. +Comments and suggestions are welcome, and may be sent to +the libstdc++ mailing list. +
+ + + + -- cgit v1.1