From 4b5cf3f7c549412e60baa925a42c80dbf17ffdd7 Mon Sep 17 00:00:00 2001 From: Paolo Bonzini Date: Thu, 21 Feb 2019 11:15:02 +0100 Subject: interpreter: add "protocol" kwarg to test This is the first step towards adding support for TAP. --- docs/markdown/Reference-manual.md | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) (limited to 'docs/markdown') diff --git a/docs/markdown/Reference-manual.md b/docs/markdown/Reference-manual.md index 6c3b3ad..412135f 100644 --- a/docs/markdown/Reference-manual.md +++ b/docs/markdown/Reference-manual.md @@ -1406,10 +1406,7 @@ executable to run. The executable can be an [executable build target object](#build-target-object) returned by [`executable()`](#executable) or an [external program object](#external-program-object) returned by -[`find_program()`](#find_program). The executable's exit code is used -by the test harness to record the outcome of the test, for example -exit code zero indicates success. For more on the Meson test harness -protocol read [Unit Tests](Unit-tests.md). +[`find_program()`](#find_program). Keyword arguments are the following: @@ -1446,6 +1443,12 @@ Keyword arguments are the following: before test is executed even if they have `build_by_default : false`. Since 0.46.0 +- `protocol` specifies how the test results are parsed. For now + it must be `exitcode`, that is the executable's exit code is used + by the test harness to record the outcome of the test. For example + an exit code of zero indicates success. For more on the Meson test harness + protocol read [Unit Tests](Unit-tests.md). Since 0.50.0 + Defined tests can be run in a backend-agnostic way by calling `meson test` inside the build dir, or by using backend-specific commands, such as `ninja test` or `msbuild RUN_TESTS.vcxproj`. -- cgit v1.1 From f2e513791e56886a145a8e72854841b9f9122ca6 Mon Sep 17 00:00:00 2001 From: Paolo Bonzini Date: Thu, 21 Feb 2019 17:51:10 +0100 Subject: mtest: add support for hard errors Hard errors also come from the GNU Automake test protocol. They happen when e.g., the set-up of a test case scenario fails, or when some other unexpected or highly undesirable condition is encountered. TAP will use them for parse errors too. Add them to the exitcode protocol first. --- docs/markdown/Unit-tests.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) (limited to 'docs/markdown') diff --git a/docs/markdown/Unit-tests.md b/docs/markdown/Unit-tests.md index a8e7273..9148bd5 100644 --- a/docs/markdown/Unit-tests.md +++ b/docs/markdown/Unit-tests.md @@ -51,10 +51,12 @@ By default Meson uses as many concurrent processes as there are cores on the tes $ MESON_TESTTHREADS=5 ninja test ``` -## Skipped tests +## Skipped tests and hard errors Sometimes a test can only determine at runtime that it can not be run. The GNU standard approach in this case is to exit the program with error code 77. Meson will detect this and report these tests as skipped rather than failed. This behavior was added in version 0.37.0. +In addition, sometimes a test fails set up so that it should fail even if it is marked as an expected failure. The GNU standard approach in this case is to exit the program with error code 99. Again, Meson will detect this and report these tests as `ERROR`, ignoring the setting of `should_fail`. This behavior was added in version 0.50.0. + ## Testing tool The goal of the meson test tool is to provide a simple way to run tests in a variety of different ways. The tool is designed to be run in the build directory. -- cgit v1.1 From 91f847d308b57adec89245308b60ae063026b456 Mon Sep 17 00:00:00 2001 From: Paolo Bonzini Date: Wed, 27 Feb 2019 07:25:33 +0100 Subject: mtest: implement TAP parsing This provides an initial support for parsing TAP output. It detects failures and skipped tests without relying on exit code, as well as early termination of the test due to an error or a crash. For now, subtests are not recorded in the TestRun object. However, because the TAP output goes on stdout, it is printed by --print-errorlogs when a test does not behave as expected. Handling subtests as TestRuns, and serializing them to JSON, can be added later. The parser was written specifically for Meson, and comes with its own test suite. Fixes #2923. --- docs/markdown/Reference-manual.md | 10 +++++----- docs/markdown/Unit-tests.md | 6 +++++- 2 files changed, 10 insertions(+), 6 deletions(-) (limited to 'docs/markdown') diff --git a/docs/markdown/Reference-manual.md b/docs/markdown/Reference-manual.md index 412135f..f2b0416 100644 --- a/docs/markdown/Reference-manual.md +++ b/docs/markdown/Reference-manual.md @@ -1443,11 +1443,11 @@ Keyword arguments are the following: before test is executed even if they have `build_by_default : false`. Since 0.46.0 -- `protocol` specifies how the test results are parsed. For now - it must be `exitcode`, that is the executable's exit code is used - by the test harness to record the outcome of the test. For example - an exit code of zero indicates success. For more on the Meson test harness - protocol read [Unit Tests](Unit-tests.md). Since 0.50.0 +- `protocol` specifies how the test results are parsed and can be one + of `exitcode` (the executable's exit code is used by the test harness + to record the outcome of the test) or `tap` ([Test Anything + Protocol](https://www.testanything.org/)). For more on the Meson test + harness protocol read [Unit Tests](Unit-tests.md). Since 0.50.0 Defined tests can be run in a backend-agnostic way by calling `meson test` inside the build dir, or by using backend-specific diff --git a/docs/markdown/Unit-tests.md b/docs/markdown/Unit-tests.md index 9148bd5..9e61739 100644 --- a/docs/markdown/Unit-tests.md +++ b/docs/markdown/Unit-tests.md @@ -53,7 +53,11 @@ $ MESON_TESTTHREADS=5 ninja test ## Skipped tests and hard errors -Sometimes a test can only determine at runtime that it can not be run. The GNU standard approach in this case is to exit the program with error code 77. Meson will detect this and report these tests as skipped rather than failed. This behavior was added in version 0.37.0. +Sometimes a test can only determine at runtime that it can not be run. + +For the default `exitcode` testing protocol, the GNU standard approach in this case is to exit the program with error code 77. Meson will detect this and report these tests as skipped rather than failed. This behavior was added in version 0.37.0. + +For TAP-based tests, skipped tests should print a single line starting with `1..0 # SKIP`. In addition, sometimes a test fails set up so that it should fail even if it is marked as an expected failure. The GNU standard approach in this case is to exit the program with error code 99. Again, Meson will detect this and report these tests as `ERROR`, ignoring the setting of `should_fail`. This behavior was added in version 0.50.0. -- cgit v1.1