APyCoT user manual
|
APyCoT user manual |
|
Table des Matières This section defines the vocabulary used and presents the global architecture of
APyCoT.
The following diagram describes the different parts of this testing
automation framework.
 Components interacting are :
You will have to write two configuration files to describe both your
needs and your environment:
The aim of Apycot is to run a battery of tests on your different projects.
Usually, you'll define a test for each project, and this test will include
some checks to run on the project. The following components are used to
define a test:
The tests configuration file looks like a INI file. There is two required
sections : the Main section and the Data section. You'll need some additional
sections to define groups and tests. Note that some options found in a
configuration file may be overriden using the command line tools.
A sample configuration file is available in the examples directory of the
distribution. Another one is available in sample/tester.ini.
The main section defines how to run all tests together. It's a kind of
runtests configuration, supporting the following options :
The main section can contain any other variables which will be used as a base
for all test definitions. See below for more explanation.
Note that only test listed in the tests variable will be executed by
default.
The data section describes how to store execution data. It has the following
options :
There should not be anything else in the data section (actually it would be
ignored).
A group defines some variables common to a group of packages. Exemple groups are
Python_package and Zope_package for instance. A new group is defined by adding a
section with the name of the group and by adding group's variables in this
section.
A test is defined by the following options :
All the limit value can be express with unit (s, min, h, d) or (KB, MB, GB)
This unit can be combined . eg : 3h 42min 56 second
space and coma are ignored
additional option can be provide to pass usefull information to the report:
+----------------------+-----+------------------------------------------------+
| name | req | description |
+======================+=====+================================================+
| repository_view_url | no | The base URL to view repository file |
+----------------------+-----+------------------------------------------------+
| repository_view_root | no | the repository root in the url view |
| | | (if required) |
+----------------------+-----+------------------------------------------------+
+
A test is built from some global, command lines, group and specific
variables. The best way to explain this is to explain the algorithm used :
if the same variable is found in different places, the latest readen is kept.
You can give options to preprocessors and checkers in the configuration file by
preceding the option name by the preprocessor or checker name. For instance, you
can give the treshold option to the python_lint checker by adding
python_lint_treshold = 4
or you can tell you want to use the setup_install preprocessor before the
python_unittest check by adding
setup_install_match_checks=python_unittest
Options specifics to preprocessors and checkers are described in their
respective section.
A repository is used to store your projects source files. Each test have to
define the repository and the location where the project files can be found in
it (using at least the repository, repository_type and path
options, plus optionaly the tag option).
Note only versioned repositories support the "log" command required by the
activity decorator.
APyCot supports the following repository types. If you don't find the one you're
looking for, you may still write your own repository wrapper, and contribute it
;). The best way to do so for the moment is to look at the existant ones...
This repository type is used to fetch sources from a CVS repository.
This repository type is used to fetch sources from a Subversion repository.
This repository type is used to fetch sources from a Mercurial repository.
This repository type is used to fetch sources from a file system repository (non
versioned). This may be useful to test projects in your working directory
(that's actually whats the --fs option of runtest is using instead of the
repository defined in the configuration file).
Used for internal and test purpose.
Used for internal and test purpose.
A preprocessor is used to initialize the test environment. It may activated by a
dependency setup or before a check is done (in that last case, the preprocessor
is called before the check, and called back when the check is done to clean the
environment).
If an error is raised by a preprocessor, the test or the check will be skipped
with an error status.
Each preprocessor support options below to configure on which occasions it will
be launched :
test can also define a preprocessors option listing the preprocessors to use
when used as a dependancy
APyCot comes with the following preprocessors. If you don't find the feature
you're looking for, you may still write your own preprocessor, and contribute
it ;). The best way to do so for the moment is to look at the existant ones...
Some preprocessors depends on a third-party program (usually a Python package or
an external command), so they may be not available according to your system
setup.
depends on
distutils
description
Use a distutils'setup.py script to install a Python package. The setup.py should
provide an "install" function which run the setup and return a "dist" object
(i.e. the object return by the distutils.setup function). This preprocessor may
modify the PATH and PYTHONPATH environment variables.
depends on
description
Use the devtools buildeb module to build debian packages. The upstream package
should have a "debian" directory containing the debian packages description. The
generated packages will be in the package upstream directory, allowing to call
the lintian checker on them.
depends on
Zope's SOFTWARE_HOME and INSTANCE_HOME
description
Install a Zope product in a Zope test environment (i.e. INSTANCE_HOME), by
creating a symbolic link of the extracted directory in the instance's Products
directory. The user running tester must be able to create this link !
depends on
make
description
A whatever builder / installer using make. You can configure this preprocessor
to run make in the desired package subdirectories (if no one provided, make will
be run at the root of the package).
A checker provides some basic test functionnalities. The aim of a checker is to
"success" or to "fail" on the package, according to the test result. A
check may also return an "error" status, indicating an unexpected error during
the check execution or the test environment construction.
APyCot comes with the "checkers" described here. If you don't find the feature
you're looking for, you may still write your own checker, and contribute it
;). The best way to do so for the moment is to look at the existant ones...
Some checkers depends on a third-party program (usually a Python package or
an external command), so they may be not available according to your system
setup.
All checkers support the following options :
A files checker acts on files according to their extension. The check succeed
if each file succeed the checker's tests independantly.
All the files checkers support the following options :
extensions
.py
description
Checks the syntax of Python files using the compile function coming with
Python.
depends on
extensions
.html, .htm
description
Checks the syntax of HTML files using the mx Tidy module.
depends on
extensions
.pt, .zpt
description
Checks the syntax of Page Template files using Zope's PageTemplates package.
depends on
extensions
.xml
description
Checks the well-formness of XML files using the lxml parser.
depends on
extensions
.xml
description
Checks the validity of XML files using the lxml validating parser.
depends on
extensions
.rst, .txt
description
Checks the syntax of ReST files using the docutils package.
depends on
lintian
extensions
.deb, .dsc
description
Checks debian packages by parsing the output of lintian.
depends on
piuparts (and sudo configuration)
extensions
.deb
description
Checks debian packages by installing and uninstalling them with piuparts
A package checker acts more globally, to locate some data or to make interact
results on each files to finally make the check succeed or failed.
depends on
description
Use Pylint to check a score for python package. The check fails if the score is
inferior to a given treshold.
depends on
pyunit
description
Execute tests found in the "test" or "tests" directory of the package. The check
succeed if no test cases failed. Note each test module is executed by a spawed
python interpreter and the output is parsed, so tests should use the default
text output of the unittest framework, and avoid messages on stderr.
depends on
description
When devtools is available, test will be launched in a coverage mode. This test
will gather coverage information, and will succeed if the test coverage is
superior to a given treshold. This checker must be executed after the
python_unittest checker.
depends on
description
Check some standard package documentation :
The 3 last tests will be done according to the presence of the respective
checkers (which depends on external packages).
depends on
description
Check a package is conform to the standard source tree as described in the
devtools package for a Python package. It'll also check the content of some
of the specified files, like __pkginfo__.py, MANIFEST.in...
depends on
description
Check a package is conform to the standard source tree as described in the
devtools package, for a Zope product. It'll also check the content of some
of the specified files, like __pkginfo__.py, MANIFEST.in...
A report will present some information extracted from the database of execution
results. A report may be plain text, html or image files, and is usually written
on the file system or sent by email. The following components define a report:
Some decorators may also be used to add some information to the tests data,
such as checkins log messages extracted from the repository or some precomputed
data. Unlike the other report's components, decorators are defined globally, not
for each report.
The reports configuration file looks like a INI file. There is two required
sections : the Main section and the Data section. You'll need some additional
sections to define your reports. Note that some options found in a
configuration file may be overriden using the command line tools.
A sample configuration file is available in the examples directory of the
distribution (tester_reports.ini). Another one, used to generate some of the
samples reports is available in samples/tester_reports.ini.
The main section defines which reports should be created. It's a kind of
runreports configuration, supporting the following options:
The main section can contain any other variables which will be used as a base
for all report definitions.
The data section describes how to fetch test execution data. It has the following
options:
There should not be anything else in the data section (actually it would be
ignored) and values entered here should be consistent with the ones found in the
tests configuration file..
Transports and formatters may be defined by a section in the configuration
file (the section's title gives the transport's identifier). The following
option indicates the transport type:
The rest of the section gives options according to the desired transport.
Available transport and their respective options are fully described in the
Transports section.
If you don't have to specify additional option to the transport / formatter,
you can use its type as identifier instead of a section's name.
A report is defined by a section in the configuration file (the section's title
gives the report's identifier). The following option indicates the reporter to
use:
The rest of the section gives options according to the desired report. Options
are fully described in the Reporters section.
You can give options to decorators in the configuration file by preceding the
option name by the decorator's name. For instance, you can give the list of
fields to consider by the contest decorator by adding
contest_fields = coverage,pylint_evaluation
Options specifics to each decorator are described in their respective section.
A decorator is used to add some data on information found in the database of
execution results. It's important to note that decoration is runned before the
any reports, and once for all in a runreports execution. So every reports will
get the same decorated data, there is no way for the moment to specify different
decorators for different reports without creating them by reexecuting
runreports.
APyCot comes with the following decorators. If you don't find the feature
you're looking for, you may still write your own decorator, and contribute
it ;). The best way to do so for the moment is to look at the existant ones...
Some decorators depends on a third-party program (usually a Python package or
an external command), so they may be not available according to your system
setup.
description
This decorator add a dummy check to each test, with a status sets to active or
to sleep according to the activity detection result. Activity detection is
done by extracting log messages from the source repositories for days until the
plan execution date (see option below). Log messages extracted are added in the
check.
description
This decorator add a dummy check to each test, with a status sets to a number
representing the rank of the test in the contest. Rank is evaluated by a
customizable set of raw values to compare (the highest is the best). Used values
are added to the check.
A reporter will present some information found in the database of
execution results. A reporter may directly output a file (usually an image) or a
report object that will be formatted using a formatters. There are two kinds of
reporters: simple reporters which create their content from a single tests
execution result (i.e. for a daily result), whereas evolution reporters build
report by analysing changes between two or more tests execution results.
Most reporters support the following options :
APyCot comes with the following reporters. If you don't find the feature
you're looking for, you may still write your own reporter, and contribute
it ;). The best way to do so for the moment is to look at the existant ones...
Some reporters depends on a third-party program (usually a Python package or
an external command), so they may be not available according to your system
setup.
A simple reporter acts on a single tests execution result.
output
report object requiring a formatter
description
This report display most information but not log messages. Each test has its
own section, with checks in subsections.
sample
output
report object requiring a formatter
description
A short table report with tests as rows and checks as columns, displaying
the status matrix.
sample
output
report object requiring a formatter
description
A full report including log messages, raw values for each check of each test
in their respective subsection / section.
sample
output
a set of report objects requiring a formatter
description
The same content as the detailed_report but with each test data in a
separated file.
sample
output
report object requiring a formatter
description
Ths report display information extracted by the contest decorator in an
ordered table, using selected field as columns, and tests as row in an ordered
manner.
You must use the contest decorator to get this report.
sample
An evolution reporter acts on a set of tests execution results. Below are some
usual options found on evolution reporters :
An evolution reporter takes the latest test execution date or the one given to
runreports using the --date option as reference, and consider test results
<interval> time unit until this date.
output
report object requiring a formatter
Display the evolution of the total number of checks, with the number of
succeed, failed, error checks (and optionaly other checks'status) in a table
with date as row.
status
output
image file
depends on
description
The same content as the status_evolution_report report but in graphical form.
sample
description
status evolution report as an image fil
output
image file
require
activity decorator on
description
test activity report: include test status change and vcs activity
A formatter is used by some reporters to convert the report object to a
particular format (plain text, html, docbook...)/
APyCot comes with the following formatters. If you don't find the feature you're
looking for, you may still write your own formatter, and contribute it ;). The
best way to do so for the moment is to look at the existant ones...
Get the report as plain text (should be ReST compatible).
Get the report as HTML. Reports using this format support internal links while
text reports don't. You may set the following option on this formatter:
A transport is used to get a report on the right place (on the
file system, in your mailbox...).
APyCot comes with the following transports. If you don't find the feature
you're looking for, you may still write your own transport, and contribute
it ;). The best way to do so for the moment is to look at the existant ones...
The file transport will put your reports in a specific directory on the
file system. Existant reports will be overriden at each reports generation.
It supports the following options:
The datedfile transport will put your reports in a specific directory on the
file system, in a separated directory for each run.
It supports the following options:
The datedzope transport will put your reports in a specific directory on a
Zope CMF site, in a separated directory for each run.
It supports the following options:
The email transport may be used to send a report by mail.
It supports the following options:
|
