From 9b45cc9931866cedbe46d4372329074595267bfa Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Alex=20Benn=C3=A9e?= Date: Fri, 1 Dec 2023 09:36:28 +0000 Subject: docs/devel: rationalise unstable gitlab tests under FLAKY_TESTS MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit It doesn't make sense to have two classes of flaky tests. While it may take the constrained environment of CI to trigger failures easily it doesn't mean they don't occasionally happen on developer machines. As CI is the gating factor to passing there is no point developers running the tests locally anyway unless they are trying to fix things. While we are at it update the language in the docs to discourage the QEMU_TEST_FLAKY_TESTS becoming a permanent solution. Reviewed-by: Philippe Mathieu-Daudé Reviewed-by: Richard Henderson Acked-by: Stefan Hajnoczi Signed-off-by: Alex Bennée Message-Id: <20231201093633.2551497-3-alex.bennee@linaro.org> --- docs/devel/testing.rst | 31 ++++++++++++++++++++----------- 1 file changed, 20 insertions(+), 11 deletions(-) (limited to 'docs/devel') diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst index 22218db..76465b8 100644 --- a/docs/devel/testing.rst +++ b/docs/devel/testing.rst @@ -1371,23 +1371,32 @@ conditions. For example, tests that take longer to execute when QEMU is compiled with debug flags. Therefore, the ``AVOCADO_TIMEOUT_EXPECTED`` variable has been used to determine whether those tests should run or not. -GITLAB_CI -^^^^^^^^^ -A number of tests are flagged to not run on the GitLab CI. Usually because -they proved to the flaky or there are constraints on the CI environment which -would make them fail. If you encounter a similar situation then use that -variable as shown on the code snippet below to skip the test: +QEMU_TEST_FLAKY_TESTS +^^^^^^^^^^^^^^^^^^^^^ +Some tests are not working reliably and thus are disabled by default. +This includes tests that don't run reliably on GitLab's CI which +usually expose real issues that are rarely seen on developer machines +due to the constraints of the CI environment. If you encounter a +similar situation then raise a bug and then mark the test as shown on +the code snippet below: .. code:: - @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab') + # See https://gitlab.com/qemu-project/qemu/-/issues/nnnn + @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab') def test(self): do_something() -QEMU_TEST_FLAKY_TESTS -^^^^^^^^^^^^^^^^^^^^^ -Some tests are not working reliably and thus are disabled by default. -Set this environment variable to enable them. +Tests should not live in this state forever and should either be fixed +or eventually removed. + +To run such tests locally you will need to set the environment +variable. For example: + +.. code:: + + env QEMU_TEST_FLAKY_TESTS=1 ./pyvenv/bin/avocado run \ + tests/avocado/boot_linux.py:BootLinuxPPC64.test_pseries_tcg Uninstalling Avocado ~~~~~~~~~~~~~~~~~~~~ -- cgit v1.1