APyCoT user manual
APyCoT user manual
Table des Matières
Chapitre I
Concepts
This section defines the vocabulary used and presents the global architecture of APyCoT.
1. Vocabulary
  • A repository is a usually version controled source database, like SVN or Hg.
  • A package is a generic term for some files related together as a project in a source repository.
  • A preprocessor allow a specific construction / installation step required to build a test environment.
  • A check is a single functional test which may be applied to test a package. A checker is the object applying this functional test.
  • A test suite (or shortly a test) is defined in a tests configuration file. A test defines:
    • the repository of the package to test,
    • the path of the package in the repository,
    • optionaly a revision tag,
    • the list of checkers to apply,
    • checkers specific options,
    • a list of dependencies which have to be installed in the test environment.
  • The result database is the set of gathered results from the daily execution of tests (currently a set of xml files).
  • A report is created by the following components:
    • a decorator that may be used to add some information to the tests data, such as checkins log messages,
    • a reporter which create the content of the report from data collected during tests execution,
    • a formatter is then used to transform the report object to a particular format (plain text, html...),
    • a transport is finally used to write the report on the file system, send it by mail and so on.
2. Global architecture
The following diagram describes the different parts of this testing automation framework.
Components interacting are :
  • runtest, which is responsible for a single test suite execution. A test will be run in a dedicated environment, created by extracting data and optionaly doing some preprocessing on it. It outputs the test execution results on stdout using a user friendly format or an easily parsable one, dedicated to communicate with runtests.
  • runtests coordinate all the tests execution. It will spawn simultaneously a predefined number of tests by launching the runtest command and parses their outputs back in order to write them to a result database.
  • runreports extract data from the result database and create
    different reports from that data.
You will have to write two configuration files to describe both your needs and your environment:
  1. a tests definition for runtests and runtest. This file describes your packages, how to fetch them, which checks should be applied to each of them and so on,
  2. a reports defintion for runreports. This file describesxs reports that should be generated from the gathered data.
Chapitre II
Executing tests
1. Presentation
The aim of Apycot is to run a battery of tests on your different projects. Usually, you'll define a test for each project, and this test will include some checks to run on the project. The following components are used to define a test:
  • the repository hosting the project source files
  • preprocessors which allow some specific construction / installation steps required to build the test environment where checks will be executed
  • checkers defining the functional tests which will be applied to test a particular aspect of the project
2. Tests configuration file
The tests configuration file looks like a INI file. There is two required sections : the Main section and the Data section. You'll need some additional sections to define groups and tests. Note that some options found in a configuration file may be overriden using the command line tools.
A sample configuration file is available in the examples directory of the distribution. Another one is available in sample/tester.ini.
2.1. The Main section
The main section defines how to run all tests together. It's a kind of runtests configuration, supporting the following options :
name
req
description
load
no
comma separated list of modules to preload, usually to register additional checkers, reporters...
threads
no
number of threads to use by runtests (default to 1)
verbosity
no
integer giving the execution verbosity (default to 0)
includes
no
A comma separated list of configuration file to include path can be either absolute or relative and containts environment variable or home directory resolution. Value defined in the including file overwrite those in the included. The Value defined in the first elements of the list are overwrite by those in the last. Missing file and recursive include are ignored
includes -test
no
Same as includes but all section define in the included file will be marked as text.
The main section can contain any other variables which will be used as a base for all test definitions. See below for more explanation.
Note that only test listed in the tests variable will be executed by default.
2.2. The Data section
The data section describes how to store execution data. It has the following options :
name
req
description
location
yes
output file or directory (depends on the mode)
mode
no
execution mode (daily, hourly, default to daily)
encoding
no
encoding to use in the data file (default to UTF-8)
There should not be anything else in the data section (actually it would be ignored).
2.3. Group definitions
A group defines some variables common to a group of packages. Exemple groups are Python_package and Zope_package for instance. A new group is defined by adding a section with the name of the group and by adding group's variables in this section.
2.4. Test definitions
A test is defined by the following options :
name
req
description
groups
no
comma separated list of groups this test belongs to
repository
yes
the repository type followed by the source repository identifier (eg hg:/home/src/pylint/ )
path
yes
the path of the package in the repository
is_test
no
Mark this section as a test. default to 0 except for section includes throught includes-tests. Can be set to 0 or 1 to force test execution.
tag
no
an optional tag used to retreive the package from the repository
checks
yes
comma separated list of checks to apply
dependencies
no
comma separated list of tests or package repository ids needed to be installed to run this test
max_cpu_time
no
The maximum CPU Time in second to be used by this test.
max_time
no
The maximum Real Time in second this will run test.
max_memory
no
The maximum Memory in bytes the test can allocate
max_retrieve
no
The delay in second while the test try to abort nicely (reporting the error and cleaning up the environement before it's killed. default to 60.
preprocessors
no
list to preprocessors to execute when this test is used as a dependancy
All the limit value can be express with unit (s, min, h, d) or (KB, MB, GB) This unit can be combined . eg : 3h 42min 56 second space and coma are ignored
additional option can be provide to pass usefull information to the report: +----------------------+-----+------------------------------------------------+ | name | req | description | +======================+=====+================================================+ | repository_view_url | no | The base URL to view repository file | +----------------------+-----+------------------------------------------------+ | repository_view_root | no | the repository root in the url view | | | | (if required) | +----------------------+-----+------------------------------------------------+ + A test is built from some global, command lines, group and specific variables. The best way to explain this is to explain the algorithm used :
  1. get extras main variables
  2. get groups variables, if the test belong to one ore more group
  3. get test specific variables
  4. extract test, checkers and preprocessors options and consider everything else as environment variables.
if the same variable is found in different places, the latest readen is kept.
2.5. Giving options to preprocessors and checkers
You can give options to preprocessors and checkers in the configuration file by preceding the option name by the preprocessor or checker name. For instance, you can give the treshold option to the python_lint checker by adding
python_lint_treshold = 4
or you can tell you want to use the setup_install preprocessor before the python_unittest check by adding
setup_install_match_checks=python_unittest
Options specifics to preprocessors and checkers are described in their respective section.
3. Repositories
A repository is used to store your projects source files. Each test have to define the repository and the location where the project files can be found in it (using at least the repository, repository_type and path options, plus optionaly the tag option).
Note only versioned repositories support the "log" command required by the activity decorator.
APyCot supports the following repository types. If you don't find the one you're looking for, you may still write your own repository wrapper, and contribute it ;). The best way to do so for the moment is to look at the existant ones...
3.1. cvs
This repository type is used to fetch sources from a CVS repository.
3.2. svn
This repository type is used to fetch sources from a Subversion repository.
3.3. hg
This repository type is used to fetch sources from a Mercurial repository.
3.4. fs
This repository type is used to fetch sources from a file system repository (non versioned). This may be useful to test projects in your working directory (that's actually whats the --fs option of runtest is using instead of the repository defined in the configuration file).
3.5. null
Used for internal and test purpose.
3.6. mock
Used for internal and test purpose.
4. Preprocessors
A preprocessor is used to initialize the test environment. It may activated by a dependency setup or before a check is done (in that last case, the preprocessor is called before the check, and called back when the check is done to clean the environment).
If an error is raised by a preprocessor, the test or the check will be skipped with an error status.
Each preprocessor support options below to configure on which occasions it will be launched :
name
req
description
match_checks
no
comma separated list of checker names (track checkers)
test can also define a preprocessors option listing the preprocessors to use when used as a dependancy
APyCot comes with the following preprocessors. If you don't find the feature you're looking for, you may still write your own preprocessor, and contribute it ;). The best way to do so for the moment is to look at the existant ones... Some preprocessors depends on a third-party program (usually a Python package or an external command), so they may be not available according to your system setup.
4.1. setup_install
depends on
distutils
description
Use a distutils'setup.py script to install a Python package. The setup.py should provide an "install" function which run the setup and return a "dist" object (i.e. the object return by the distutils.setup function). This preprocessor may modify the PATH and PYTHONPATH environment variables.
4.2. build_debian
depends on
description
Use the devtools buildeb module to build debian packages. The upstream package should have a "debian" directory containing the debian packages description. The generated packages will be in the package upstream directory, allowing to call the lintian checker on them.
4.3. zope_install
depends on
Zope's SOFTWARE_HOME and INSTANCE_HOME
description
Install a Zope product in a Zope test environment (i.e. INSTANCE_HOME), by creating a symbolic link of the extracted directory in the instance's Products directory. The user running tester must be able to create this link !
4.4. make
depends on
make
description
A whatever builder / installer using make. You can configure this preprocessor to run make in the desired package subdirectories (if no one provided, make will be run at the root of the package).
options
name
req
description
directories
no
comma separated list of package's directories where make should be run
target
no
the target to make
5. Checkers
A checker provides some basic test functionnalities. The aim of a checker is to "success" or to "fail" on the package, according to the test result. A check may also return an "error" status, indicating an unexpected error during the check execution or the test environment construction.
APyCot comes with the "checkers" described here. If you don't find the feature you're looking for, you may still write your own checker, and contribute it ;). The best way to do so for the moment is to look at the existant ones... Some checkers depends on a third-party program (usually a Python package or an external command), so they may be not available according to your system setup.
5.1. General Checkers options
All checkers support the following options :
name
req.
description
max_status
no
A maximum result this checker cannot exceed
status_cap_reason
no
The reason why this checker is capped
5.2. Files checkers
A files checker acts on files according to their extension. The check succeed if each file succeed the checker's tests independantly.
All the files checkers support the following options :
name
req.
description
ignore
no
comma separated list some files or directories to ignore
5.2.1. python_syntax
extensions
.py
description
Checks the syntax of Python files using the compile function coming with Python.
5.2.2. html_tidy
depends on
extensions
.html, .htm
description
Checks the syntax of HTML files using the mx Tidy module.
5.2.3. pt_syntax
depends on
extensions
.pt, .zpt
description
Checks the syntax of Page Template files using Zope's PageTemplates package.
5.2.4. xml_well_formed
depends on
extensions
.xml
description
Checks the well-formness of XML files using the lxml parser.
5.2.5. xml_valid
depends on
extensions
.xml
description
Checks the validity of XML files using the lxml validating parser.
5.2.6. rest_syntax
depends on
extensions
.rst, .txt
description
Checks the syntax of ReST files using the docutils package.
5.2.7. debian_lint
depends on
lintian
extensions
.deb, .dsc
description
Checks debian packages by parsing the output of lintian.
5.2.8. debian_piuparts
depends on
piuparts (and sudo configuration)
extensions
.deb
description
Checks debian packages by installing and uninstalling them with piuparts
5.3. Package checkers
A package checker acts more globally, to locate some data or to make interact results on each files to finally make the check succeed or failed.
5.3.1. python_lint
depends on
description
Use Pylint to check a score for python package. The check fails if the score is inferior to a given treshold.
options
name
req.
description
treshold
yes
the minimal note to obtain for the package from PyLint
show_categories
no
comma separated list of letter used to filter the message displayed default to Error and Fatal
pylintrc
no
The path to a pylint configuration file
5.3.2. python_unittest
depends on
pyunit
description
Execute tests found in the "test" or "tests" directory of the package. The check succeed if no test cases failed. Note each test module is executed by a spawed python interpreter and the output is parsed, so tests should use the default text output of the unittest framework, and avoid messages on stderr.
name
req.
description
coverage
no
Enable or disable coverage test (0 or 1, default to 1 when devtools is available)
test_dirs
no
List of comma separated candidates of tests directory. default to "test, tests"
ignored_python_versions
no
Comma separated version of python to ignore when running the test.
tested_python_versions
no
Comma separated version of python to test when running the test.
use_pkginfo_python_versions
no
0, or 1 (default to 1) run the tests with the python's versions defined in the pkginfo module.
  • First, versions defined in the pkginfo module are imported (if enable).
  • Then versions defined into tested_python_versions are added.
  • finally version in ignored_python_versions are removed.
5.3.3. python_test_coverage
depends on
description
When devtools is available, test will be launched in a coverage mode. This test will gather coverage information, and will succeed if the test coverage is superior to a given treshold. This checker must be executed after the python_unittest checker.
options
name
req.
description
treshold
yes
the minimal note to obtain for the test coverage
5.3.4. pkg_doc
description
Check some standard package documentation :
  • presence of some required files (README, INSTALL, ChangeLog)
  • plain text files in the "doc" directory are ReST files
  • xml files in the "doc" directory are well formed
  • html files in the "doc" directory are correct
The 3 last tests will be done according to the presence of the respective checkers (which depends on external packages).
options
name
req.
description
ignore
no
comma separated list of files or directories to ignore
5.3.5. python_pkg
depends on
description
Check a package is conform to the standard source tree as described in the devtools package for a Python package. It'll also check the content of some of the specified files, like __pkginfo__.py, MANIFEST.in...
5.3.6. zope_pkg
depends on
description
Check a package is conform to the standard source tree as described in the devtools package, for a Zope product. It'll also check the content of some of the specified files, like __pkginfo__.py, MANIFEST.in...
Chapitre III
Creating reports
1. Presentation
A report will present some information extracted from the database of execution results. A report may be plain text, html or image files, and is usually written on the file system or sent by email. The following components define a report:
  • a reporter, responsible to create the content from tests data, as a report object
  • a formatter, used by some reporters to convert the report object to a particular format (usually reports generating images don't use a formatter)
  • a transport, used to get the report on the right place (on the file system, in your mailbox...)
Some decorators may also be used to add some information to the tests data, such as checkins log messages extracted from the repository or some precomputed data. Unlike the other report's components, decorators are defined globally, not for each report.
2. Reports configuration file
The reports configuration file looks like a INI file. There is two required sections : the Main section and the Data section. You'll need some additional sections to define your reports. Note that some options found in a configuration file may be overriden using the command line tools.
A sample configuration file is available in the examples directory of the distribution (tester_reports.ini). Another one, used to generate some of the samples reports is available in samples/tester_reports.ini.
2.1. The Main section
The main section defines which reports should be created. It's a kind of runreports configuration, supporting the following options:
name
req
description
reports
yes
comma separated list of report'names where each report have its own section in the configuration file
decorators
no
comma separated list of decorators to run before the reports creation
verbosity
no
integer giving the execution verbosity (default to 0)
The main section can contain any other variables which will be used as a base for all report definitions.
2.2. The Data section
The data section describes how to fetch test execution data. It has the following options:
name
req
description
location
yes
output file or directory (depends on the mode)
mode
no
execution mode (daily or hourly, default to daily)
There should not be anything else in the data section (actually it would be ignored) and values entered here should be consistent with the ones found in the tests configuration file..
2.3. Transport / formatter definitions
Transports and formatters may be defined by a section in the configuration file (the section's title gives the transport's identifier). The following option indicates the transport type:
name
req
description
type
yes
name of the transport or formatter to use
The rest of the section gives options according to the desired transport. Available transport and their respective options are fully described in the Transports section. If you don't have to specify additional option to the transport / formatter, you can use its type as identifier instead of a section's name.
2.4. Report definitions
A report is defined by a section in the configuration file (the section's title gives the report's identifier). The following option indicates the reporter to use:
name
req
description
type
yes
name of the report to produce
The rest of the section gives options according to the desired report. Options are fully described in the Reporters section.
2.5. Giving options to decorators
You can give options to decorators in the configuration file by preceding the option name by the decorator's name. For instance, you can give the list of fields to consider by the contest decorator by adding
contest_fields = coverage,pylint_evaluation
Options specifics to each decorator are described in their respective section.
3. Decorators
A decorator is used to add some data on information found in the database of execution results. It's important to note that decoration is runned before the any reports, and once for all in a runreports execution. So every reports will get the same decorated data, there is no way for the moment to specify different decorators for different reports without creating them by reexecuting runreports.
APyCot comes with the following decorators. If you don't find the feature you're looking for, you may still write your own decorator, and contribute it ;). The best way to do so for the moment is to look at the existant ones... Some decorators depends on a third-party program (usually a Python package or an external command), so they may be not available according to your system setup.
3.1. activity
description
This decorator add a dummy check to each test, with a status sets to active or to sleep according to the activity detection result. Activity detection is done by extracting log messages from the source repositories for days until the plan execution date (see option below). Log messages extracted are added in the check.
options
name
req
description
time
no
number of time interval (day or hourly according to the mode to consider (default to 1)
3.2. contest
description
This decorator add a dummy check to each test, with a status sets to a number representing the rank of the test in the contest. Rank is evaluated by a customizable set of raw values to compare (the highest is the best). Used values are added to the check.
options
name
req
description
fields
yes
list of raw fields to consider
expression
yes
a python expression calculating the contest rating for a test (selected fields are available as variables)
4. Reporters
A reporter will present some information found in the database of execution results. A reporter may directly output a file (usually an image) or a report object that will be formatted using a formatters. There are two kinds of reporters: simple reporters which create their content from a single tests execution result (i.e. for a daily result), whereas evolution reporters build report by analysing changes between two or more tests execution results.
Most reporters support the following options :
name
req
description
link_to
no
link to another report internaly
filter
no
comma separated list of test names to consider (default to all)
title
no
report title
top_links
no
link to other reports on the report's top
summary
no
summary level (0 disable it, default to 2)
treshold
no
log treshold to include messages (default to ERROR) possible values are : INFO, WARNING, ERROR
APyCot comes with the following reporters. If you don't find the feature you're looking for, you may still write your own reporter, and contribute it ;). The best way to do so for the moment is to look at the existant ones... Some reporters depends on a third-party program (usually a Python package or an external command), so they may be not available according to your system setup.
4.1. Simple reporters
A simple reporter acts on a single tests execution result.
4.1.1. minimal_report
output
report object requiring a formatter
description
This report display most information but not log messages. Each test has its own section, with checks in subsections.
4.1.2. synthetized_report
output
report object requiring a formatter
description
A short table report with tests as rows and checks as columns, displaying the status matrix.
4.1.3. detailed_report
output
report object requiring a formatter
description
A full report including log messages, raw values for each check of each test in their respective subsection / section.
4.1.4. multi_files_detailed_report
output
a set of report objects requiring a formatter
description
The same content as the detailed_report but with each test data in a separated file.
4.1.5. contest_report
output
report object requiring a formatter
description
Ths report display information extracted by the contest decorator in an ordered table, using selected field as columns, and tests as row in an ordered manner. You must use the contest decorator to get this report.
sample
4.2. Evolution reporters
An evolution reporter acts on a set of tests execution results. Below are some usual options found on evolution reporters :
name
req
description
interval
no
number of time unit (day or hour depending of the mode to take in consideration (default to 1)
columns
no
field to take accounts (default to all)
An evolution reporter takes the latest test execution date or the one given to runreports using the --date option as reference, and consider test results <interval> time unit until this date.
4.2.1. status_evolution_report
output
report object requiring a formatter
Display the evolution of the total number of checks, with the number of succeed, failed, error checks (and optionaly other checks'status) in a table with date as row.
status
4.2.2. status_evolution_graph
output
image file
depends on
description
The same content as the status_evolution_report report but in graphical form.
name
req
description
graph_type
no
default to line
4.2.3. activity_evolution_graph
description
status evolution report as an image fil
4.2.4. test_activity_report
output
image file
require
activity decorator on
description
test activity report: include test status change and vcs activity
5. Formatters
A formatter is used by some reporters to convert the report object to a particular format (plain text, html, docbook...)/
APyCot comes with the following formatters. If you don't find the feature you're looking for, you may still write your own formatter, and contribute it ;). The best way to do so for the moment is to look at the existant ones...
5.1. text
Get the report as plain text (should be ReST compatible).
5.2. html
Get the report as HTML. Reports using this format support internal links while text reports don't. You may set the following option on this formatter:
name
req
description
css
no
URL of a CSS file
6. Transports
A transport is used to get a report on the right place (on the file system, in your mailbox...).
APyCot comes with the following transports. If you don't find the feature you're looking for, you may still write your own transport, and contribute it ;). The best way to do so for the moment is to look at the existant ones...
6.1. file
The file transport will put your reports in a specific directory on the file system. Existant reports will be overriden at each reports generation. It supports the following options:
name
req
description
destination
yes
the file system destination directory
url_base
no
URL that may be used as base to get reports from the web
6.2. datedfile
The datedfile transport will put your reports in a specific directory on the file system, in a separated directory for each run. It supports the following options:
name
req
description
destination
yes
the file system destination directory
url_base
no
URL that may be used as base to get reports from the web
6.3. datedzope
The datedzope transport will put your reports in a specific directory on a Zope CMF site, in a separated directory for each run. It supports the following options:
name
req
description
destination
yes
the file system destination directory
url_base
yes
URL that used as base to get reports from the web
user
no
the user name for the connection to the base_url
password
no
the user password for the connection to the base_url
folder_type
no
the CMF type used for folders (default to Folder)
document_type
no
the CMF type used for reports (default to RawDocument)
6.4. email
The email transport may be used to send a report by mail. It supports the following options:
name
req
description
dest
yes
list of destination addresses
from
yes
email from name and address
subject
no
email subject (default to APyCot Report)
smtp_server
no
SMTP server host name (default to localhost)
smtp_port
no
SMTP server port (default to 25)
smtp_debug
no
debug the smtp connexion (default to 0)