Table Of Contents

Previous topic

Build Variants

Next topic

Building individual libraries with cmake

This Page

Testing

Boost’s CMake-based build system provides regression testing via CTest, which comes as part of CMake. This regression testing system can be used by Boost developers to test their libraries locally and also by testers and users to submit regression tests to a CDash server, which collects and reports regression-testing results from different sites. This document assumes that the reader has already learned how to build and configure Boost using CMake.

Re-run the CMake configuration for Boost, after setting BUILD_TESTING to ON. You may notice that configuration takes significantly longer when we are building all of the regression tests. You may enable regression testing for only a subset of Boost libraries (this speeds configuration and build time) by setting the variable BOOST_TEST_LIBRARIES to a semicolon separated list of library names, e.g. (from the CMakeCache.txt)

spirit;serialization;variant

Be sure to re-configure CMake once you are done tweaking these options, and generate makefiles or project files, by clicking OK (on Microsoft Windows) or pressing g (on Unix).

To test all, libraries, set BOOST_TEST_LIBRARIES to ALL

If you know what you’re going to be testing, you can simplify the whole process with e.g.

cmake ../src -DBUILD_TESTING=ON -DBOOST_TEST_LIBRARIES=mpi;graph_parallel

The Dashboard

Donated by kitware, it is here:

http://www.cdash.org/CDashPublic/index.php?project=Boost

Build

Following the same building process described in Quickstart. For Unix users, don’t forget the -i option to make, and also possibly -j 2 (or more) to run the build process in parallel. Building all of the regression tests for the Boost libraries can take a long time.

make -i -j 2

Note

If you change Boost source files in a way that affects your tests, you will need to re-run the build process to update the libraries and tests before moving on to the next step.

Test

Once regression tests have finished building, go into the command line or command prompt and enter the Boost binary directory. Then, run the command:

ctest

to execute all of the regression tests. The ctest executable will be stored in the binary directory for CMake. On Unix platforms, this is the same place where ccmake resides. On Windows platforms, it will be in C:\Program Files\CMake\bin. The ctest program should produce output like the following:

Start processing tests
Test project /Users/dgregor/Projects/boost-darwin
  1/ 22 Testing any-any_test                    Passed
  2/ 22 Testing any-any_to_ref_test             Passed
  3/ 22 Testing function-lib_function_test      Passed
  4/ 22 Testing function-function_n_test        Passed
  5/ 22 Testing function-allocator_test         Passed
  6/ 22 Testing function-stateless_test         Passed
  7/ 22 Testing function-lambda_test            Passed
  8/ 22 Testing function-function_test_fail1 ***Failed - supposed to fail
  9/ 22 Testing function-function_test_fail2 ***Failed - supposed to fail
 10/ 22 Testing function-function_30            Passed
 11/ 22 Testing function-function_arith_cxx98   Passed
 12/ 22 Testing function-function_arith_porta   Passed
 13/ 22 Testing function-sum_avg_cxx98          Passed
 14/ 22 Testing function-sum_avg_portable       Passed
 15/ 22 Testing function-mem_fun_cxx98          Passed
 16/ 22 Testing function-mem_fun_portable       Passed
 17/ 22 Testing function-std_bind_cxx98         Passed
 18/ 22 Testing function-std_bind_portable      Passed
 19/ 22 Testing function-function_ref_cxx98     Passed
 20/ 22 Testing function-function_ref_portabl   Passed
 21/ 22 Testing function-contains_test          Passed
 22/ 22 Testing function-contains2_test         Passed

100% tests passed, 0 tests failed out of 22

Here, we have only enabled testing of the Boost.Any and Boost.Function libraries, by setting TEST_BOOST_ANY and TEST_BOOST_FUNCTION to ON while all of the other TEST_BOOST_‘’LIBNAME’’ options are set to OFF.

Warning

Again, This ctest step runs the tests without first running a build. If you change a source file and run the ctest step you will see that no build is invoked.

To run just a subset of the tests, pass -R and a regular expression to ctest (see the output of ctest --help-full). For example, to run all of the Python tests, use:

ctest -R python

There is also a -E (exclude) option which does the inverse of -R. ctest --help shows the full list of options.

Targets

The testing subsystem adds a toplevel target to the build. In the case of e.g. mpi, (again when BUILD_TESTING is on, and BOOST_TEST_LIBRARIES is either empty or contains mpi), the target mpi-test will first build the test drivers, then run ctest -R mpi-. Example: for intrusive,

% make intrusive-test
[  0%] Built target intrusive-multiset_test
[  0%] Built target intrusive-splay_multiset_test
...
[100%] Built target intrusive-avl_set_test
[100%] Built target intrusive-sg_multiset_test
Start processing tests
Test project /home/troy/Projects/boost/patches/build/libs/intrusive
 1/ 21 Testing intrusive-splay_multiset_test    Passed
 2/ 21 Testing intrusive-slist_test .........   Passed
 3/ 21 Testing intrusive-stateful_value_trait   Passed
 4/ 21 Testing intrusive-sg_set_test ........   Passed
 5/ 21 Testing intrusive-treap_multiset_test ***Failed
...
20/ 21 Testing intrusive-sg_multiset_test ...   Passed
21/ 21 Testing intrusive-multiset_test ......   Passed

95% tests passed, 1 tests failed out of 21

The following tests FAILED:
          5 - intrusive-treap_multiset_test (Failed)
Errors while running CTest

Submit Results

The ctest command can be used by individual developers to test local changes to their libraries. The same program can also be used to build all of Boost, run its regression tests, and submit the results to a central server where others can view them. Currently, regression test results based on the CMake build system are displayed on the Dart server at http://www.cdash.org/CDashPublic/index.php?project=Boost

To submit “experimental” results to the Dart server, configure a Boost binary tree by following the configuration instructions in the section Quickstart, and then enable regression testing via the BOOST_TESTING option, as described above. At this point, don’t build anything! We’ll let CTest do that work. You may want to customize some of the advanced CMake options, such as SITE (to give your site name), and MAKECOMMAND (which, for makefile targets, governs the top-level make call when building Boost). Finally, go into the Boost binary directory and execute:

ctest -D Experimental

CTest will then reconfigure Boost, build all of the Boost libraries and regression tests, execute the regression tests, and submit the results to the Dart dashboard at http://www.cdash.org/CDashPublic/index.php?project=Boost. Results submitted to the dashboard are usually browsable momentarily within a minute or two.

Note

Although we are running regression tests on several flavors of Unix on a nightly basis, we have done very little work to ensure that regression testing runs smoothly on other platforms. We will remedy this problem in the near future.

Automatic testing

Continuous

Here is a recommended setup.

Create a directory ctest on your test drone containing subdirectories for the branches you’ll be testing, in this case release and trunk.

boost/
  ctest/
    branches/
      release/
        continuous/
          build/        <= run ctest here
          src/          <= checkout to here
        nightly/
          build/        <= run ctest here
          src/          <= checkout to here

and check out source to the directories listed above. We’ll do the release branch as an example:

% cd boost/ctest/branches/release
% svn co http://svn.boost.org/svn/boost/branches/release src
#
# lots of output
#
% mkdir continuous
% cd continuous

now configure your build, enabling testing. In this case I’ll also use an alternate compiler, from Intel:

% cmake ../src -DBUILD_TESTING=ON -DCMAKE_C_COMPILER=icc -DCMAKE_CXX_COMPILER=icpc
-- The C compiler identification is Intel
-- The CXX compiler identification is Intel
-- Check for working C compiler: /opt/intel/Compiler/11.0/083/bin/intel64/icc
-- Check for working C compiler: /opt/intel/Compiler/11.0/083/bin/intel64/icc -- works

   (etc)

-- Configuring done
-- Generating done
-- Build files have been written to: /home/troy/Projects/boost/ctest/release/continuous/build

Now run ctest -D Continuous in a loop:

% while true
while> do
while> ctest -D Continuous
while> sleep 600   # take it easy on the repository
while> done
   Site: zinc
   Build name: intel-11.0-linux
Create new tag: 20090514-2207 - Continuous
Start processing tests
Updating the repository
   Updating the repository: /home/troy/Projects/boost/ctest/release/nightly/src
   Use SVN repository type
   Old revision of repository is: 53002
   New revision of repository is: 53005
   Gathering version information (one . per revision):

   (etc)

If you add -V or -VV you’ll get a little more feedback about what is going on. On unix it is handy to do this via the utility screen.

Todo

Figure out how to do this on windows, encapsulate some of this scripting. Just use the ctest builtin scripting language.

Nightly

Nightly testing should run triggered by a cron job or by Windows Task Scheduler or what-have-you. You will need,

  • a directory to work in
  • installed cmake/ctest/svn

but not a checkout of boost. CTest will do the checkout, build, test and submit on its own.

Create a directory to run in. As in the previous section, we’ll use boost/ctest/branches/release/nightly, which I’ll call $DIR. The CTest script should look like the following (you can copy/paste this into $DIR/CTestNightly.cmake

execute_process(COMMAND /bin/pwd
  OUTPUT_VARIABLE PWD
  OUTPUT_STRIP_TRAILING_WHITESPACE)

message(STATUS "Running nightly build in ${PWD}")

set(CTEST_SOURCE_DIRECTORY ${PWD}/src)
set(CTEST_BINARY_DIRECTORY ${PWD}/build)

# what cmake command to use for configuring this dashboard
set(CTEST_CMAKE_COMMAND "/usr/local/bin/cmake")
set(CTEST_CTEST_COMMAND "/usr/local/bin/ctest")
set(CTEST_CVS_COMMAND "svn")

set(CTEST_CVS_CHECKOUT  "${CTEST_CVS_COMMAND} co https://svn.boost.org/svn/boost/branches/release ${CTEST_SOURCE_DIRECTORY}")

# which ctest command to use for running the dashboard
set(CTEST_COMMAND
  "${CTEST_CTEST_COMMAND} -VV -D Experimental -A ${PWD}/notes.txt -O ctest.log"
  )



####################################################################
# The values in this section are optional you can either
# have them or leave them commented out
####################################################################

# should ctest wipe the binary tree before running
set(CTEST_START_WITH_EMPTY_BINARY_DIRECTORY TRUE)

#
# this is the initial cache to use for the binary tree, be careful to escape
# any quotes inside of this string if you use it
#
# Yes you can pass cmake -DBUILD_WHATEVER=ON type options here.
#
set(CTEST_INITIAL_CACHE "

CMAKE_CXX_COMPILER:STRING=/opt/intel/Compiler/11.0/083/bin/intel64/icpc
CMAKE_C_COMPILER:STRING=/opt/intel/Compiler/11.0/083/bin/intel64/icc

SITE:STRING=zinc
MAKECOMMAND:STRING=make -i -j2
DART_TESTING_TIMEOUT:STRING=30
BUILD_TESTING:STRING=ON
CVSCOMMAND:FILEPATH=${CTEST_CVS_COMMAND}
")

You will need to customize several variables:

  • CTEST_CMAKE_COMMAND the path to your cmake binary
  • CTEST_CTEST_COMMAND the path to your ctest binary (should be in the same place as cmake)
  • CTEST_CVS_COMMAND The path to subversion.
  • CMAKE_CXX_COMPILER:STRING, CMAKE_C_COMPILER:STRING Note the syntax here, the trailing :STRING. If you are using a nonstandard compiler, set it here.
  • MAKECOMMAND:STRING The path to your make command. NOTE if you set this, be sure that the -i (ignore-errors) flag is included. If it isn’t, the first compile/link error will stop the build and testing will commence.
  • SITE:STRING This is what will appear as the ‘hostname’ in your posted dashboards. Customize as you like.

Now you’ll create a “notes” file, call it notes.txt. This will be visible from the dashboard. Add the output of, e.g:

gcc --version
uname -a

and the contents of the CTestNightly.cmake itself. Example:

**************** CMAKE DASHBOARD NOTES *****************

Notes file for CMake Nightly Dashboard.

This dashboard is being generated on an eight core, 64 bit
Ubuntu 9.04 machine with an intel c++ compiler.


Questions about this Dashboard should be directed to troy@resophonic.com

Linux zinc 2.6.27-11-generic #1 SMP Wed Apr 1 20:53:41 UTC 2009 x86_64 GNU/Linux

icpc (ICC) 11.0 20090318
Copyright (C) 1985-2009 Intel Corporation.  All rights reserved.

--------------- Script file ---------------

(contents of CTestNightly.cmake)

Now run (as a cronjob or at the command line):

ctest -S CTestNightly.cmake

if you want extra verbosity add a -VV flag. You’ll see something like the following:

+ /opt/local/bin/ctest -VV -S CMakeDashBoard.cmake
* Extra verbosity turned on
Reading Script: /Users/troy/ctest/boost/release/nightly/CMakeDashBoard.cmake
Run cmake command: /opt/i3/ports/bin/cmake "/Users/troy/ctest/boost/release/nightly/src"
-- The C compiler identification is GNU
-- The CXX compiler identification is GNU
-- Check for working C compiler: /usr/bin/gcc
(etc)
-- Configuring done
-- Generating done
-- Build files have been written to: /Users/troy/ctest/boost/release/nightly/build
Run ctest command: /opt/i3/ports/bin/ctest -VV -D Nightly -A /Users/troy/ctest/boost/release/nightly/notes.txt -O ctest.log
UpdateCTestConfiguration  from :/Users/troy/ctest/boost/release/nightly/build/DartConfiguration.tcl
Parse Config file:/Users/troy/ctest/boost/release/nightly/build/DartConfiguration.tcl
   Site: silver
   Build name: gcc-4.0.1-macos
(etc, etc)

You’ll see it configure again, run... and sooner or later you’ll see your results on The Dashboard. a dashboard posted.