aboutsummaryrefslogtreecommitdiff
path: root/docs/devel/testing.rst
AgeCommit message (Collapse)AuthorFilesLines
2024-09-04docs/devel: Split testing docs from the build docs and move to separate folderThomas Huth1-1557/+0
Building and testing are two separate topics, so let's split the testing into a separate category and move the related files into a separate folder. Message-ID: <20240830133841.142644-42-thuth@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
2024-07-30docs/devel: document how to run individual TCG testsAlex Bennée1-0/+13
Since 6f6ca067d2 (tests/tcg: add some help output for running individual tests) we made it easier to run individual tests for a given architecture. Lets reference that in the developer documentation. Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Message-Id: <20240729144414.830369-8-alex.bennee@linaro.org>
2024-07-30docs/devel: update the testing introductionAlex Bennée1-3/+18
Move the mention of "check-help" up to the intro text and also mention the meson test integration. Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Message-Id: <20240729144414.830369-7-alex.bennee@linaro.org>
2024-06-06docs/devel: update references to centos to non-versioned containerAlex Bennée1-4/+4
>From the website: "After May 31, 2024, CentOS Stream 8 will be archived and no further updates will be provided." We have updated a few bits but there are still references that need fixing. Rather than bump I've replaced them with references to the Debian image so we don't have to bump at the next update. Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Reviewed-by: Thomas Huth <thuth@redhat.com> Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Message-Id: <20240603175328.3823123-3-alex.bennee@linaro.org>
2024-02-23tests/avocado: mark boot_linux.py long runtime instead of flakyNicholas Piggin1-0/+11
The ppc64 and s390x tests were first marked skipIf GITLAB_CI by commit c0c8687ef0f ("tests/avocado: disable BootLinuxPPC64 test in CI"), and commit 0f26d94ec9e ("tests/acceptance: skip s390x_ccw_vrtio_tcg on GitLab") due to being very heavy-weight for gitlab CI. Commit 9b45cc99318 ("docs/devel: rationalise unstable gitlab tests under FLAKY_TESTS") changed this to being flaky but it isn't really, it just had a long runtime. So take the SPEED=slow variable from qtests and introduce it to avocado, and make these tests require it. Reviewed-by: Cédric Le Goater <clg@kaod.org> Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
2024-02-20docs: correct typosManos Pitsidianakis1-1/+1
Correct typos automatically found with the `typos` tool <https://crates.io/crates/typos> Signed-off-by: Manos Pitsidianakis <manos.pitsidianakis@linaro.org> Reviewed-by: Alex Bennée <alex.bennee@linaro.org> Acked-by: Michael S. Tsirkin <mst@redhat.com> Reviewed-by: Michael Tokarev <mjt@tls.msk.ru> (mjt: trivial fixup) Signed-off-by: Michael Tokarev <mjt@tls.msk.ru>
2023-12-01gitlab: add optional job to run flaky avocado testsAlex Bennée1-7/+8
One problem with flaky tests is they often only fail under CI conditions which makes it hard to debug. We add an optional allow_fail job so developers can trigger the only the flaky tests in the CI environment if they are debugging. Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Acked-by: Stefan Hajnoczi <stefanha@redhat.com> Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Message-Id: <20231201093633.2551497-8-alex.bennee@linaro.org>
2023-12-01docs/devel: rationalise unstable gitlab tests under FLAKY_TESTSAlex Bennée1-11/+20
It doesn't make sense to have two classes of flaky tests. While it may take the constrained environment of CI to trigger failures easily it doesn't mean they don't occasionally happen on developer machines. As CI is the gating factor to passing there is no point developers running the tests locally anyway unless they are trying to fix things. While we are at it update the language in the docs to discourage the QEMU_TEST_FLAKY_TESTS becoming a permanent solution. Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org> Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Acked-by: Stefan Hajnoczi <stefanha@redhat.com> Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Message-Id: <20231201093633.2551497-3-alex.bennee@linaro.org>
2023-11-16tests/avocado: Replace assertRegexpMatches() for Python 3.12 compatibilityPhilippe Mathieu-Daudé1-1/+1
assertRegexpMatches() has been removed in Python 3.12 and should be replaced by assertRegex(). See: https://docs.python.org/3.12/whatsnew/3.12.html#id3 Inspired-by: Thomas Huth <thuth@redhat.com> Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org> Message-ID: <20231114144832.71612-1-philmd@linaro.org> Reviewed-by: Thomas Huth <thuth@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
2023-11-16tests/avocado: Replace assertEquals() for Python 3.12 compatibilityThomas Huth1-1/+1
assertEquals() has been removed in Python 3.12 and should be replaced by assertEqual(). See: https://docs.python.org/3.12/whatsnew/3.12.html#id3 Message-ID: <20231114134326.287242-1-thuth@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org> Signed-off-by: Thomas Huth <thuth@redhat.com>
2023-11-13tests/tsan: Rename the file with the entries that should be ignoredThomas Huth1-2/+2
Let's use a better file name here. Message-ID: <20231109174720.375873-1-thuth@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org> Signed-off-by: Thomas Huth <thuth@redhat.com>
2023-10-12python/qemu: rename command() to cmd()Vladimir Sementsov-Ogievskiy1-5/+5
Use a shorter name. We are going to move in iotests from qmp() to command() where possible. But command() is longer than qmp() and don't look better. Let's rename. You can simply grep for '\.command(' and for 'def command(' to check that everything is updated (command() in tests/docker/docker.py is unrelated). Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> Reviewed-by: Eric Blake <eblake@redhat.com> Reviewed-by: Cédric Le Goater <clg@kaod.org> Reviewed-by: Juan Quintela <quintela@redhat.com> Message-id: 20231006154125.1068348-6-vsementsov@yandex-team.ru [vsementsov: also update three occurrences in tests/avocado/machine_aspeed.py and keep r-b] Signed-off-by: John Snow <jsnow@redhat.com>
2023-10-07tcg: Correct invalid mentions of 'softmmu' by 'system-mode'Philippe Mathieu-Daudé1-1/+1
Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org> Message-ID: <20231004090629.37473-6-philmd@linaro.org> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2023-08-28Revert "tests: Use separate virtual environment for avocado"Paolo Bonzini1-7/+7
This reverts commit e8e4298feadae7924cf7600bb3bcc5b0a8d7cbe9. ensuregroup allows to specify both the acceptable versions of avocado, and a locked version to be used when avocado is not installed as a system pacakge. This lets us install avocado in pyvenv/ using "mkvenv.py" and reuse the distro package on Fedora and CentOS Stream (the only distros where it's available). ensuregroup's usage of "(>=..., <=...)" constraints when evaluating the distro package, and "==" constraints when installing it from PyPI, makes it possible to avoid conflicts between the known-good version and a package plugins included in the distro. This is because package plugins have "==" constraints on the version that is included in the distro, and, using "pip install avocado==88.1" on a venv that includes system packages will result in an error: avocado-framework-plugin-varianter-yaml-to-mux 98.0 requires avocado-framework==98.0, but you have avocado-framework 88.1 which is incompatible. avocado-framework-plugin-result-html 98.0 requires avocado-framework==98.0, but you have avocado-framework 88.1 which is incompatible. But at the same time, if the venv does not include a system distribution of avocado then we can install a known-good version and stick to LTS releases. Resolves: https://gitlab.com/qemu-project/qemu/-/issues/1663 Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2023-07-18tests/avocado: Disable the test_sbsaref_edk2_firmware by defaultThomas Huth1-0/+5
The test fails occasionally, see e.g.: https://gitlab.com/thuth/qemu/-/jobs/4196177756#L489 https://gitlab.com/thuth/qemu/-/jobs/4623296271#L290 It also fails on my laptop in ca. 50% of all runs. Thus disable it by default by using the QEMU_TEST_FLAKY_TESTS environment variable to fence it (which we also already use in flaky qtests). While we're at it, also document this variable in docs/devel/testing.rst. Message-Id: <20230710170155.7192-1-thuth@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
2023-07-03docs/devel: remind developers to run CI container pipeline when updating imagesAni Sinha1-0/+6
When new dependencies and packages are added to containers, its important to run CI container generation pipelines on gitlab to make sure that there are no obvious conflicts between packages that are being added and those that are already present. Running CI container pipelines will make sure that there are no such breakages before we commit the change updating the containers. Add a line in the documentation reminding developers to run the pipeline before submitting the change. It will also ease the life of the maintainers. Signed-off-by: Ani Sinha <anisinha@redhat.com> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> Message-Id: <20230506072012.10350-1-anisinha@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org> Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Message-Id: <20230630180423.558337-5-alex.bennee@linaro.org>
2023-06-28docs/devel/testing: Update the 'Docker Debugging' sectionPhilippe Mathieu-Daudé1-1/+1
Since commit 93cc0506f6 ("tests/docker: Use Fedora containers for MinGW cross-builds in the gitlab-CI") the MinGW toolchain is packaged inside the fedora-win[32/64]-cross images. Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> Message-Id: <20230624142211.8888-2-philmd@linaro.org>
2023-06-06tests: Use separate virtual environment for avocadoPaolo Bonzini1-7/+7
This reverts commits eea2d141179 ("Makefile: remove $(TESTS_PYTHON)", 2023-05-26) and 9c6692db550 ("tests: Use configure-provided pyvenv for tests", 2023-05-18). Right now, there is a conflict between wanting a ">=" constraint when using a distro-provided package and wanting a "==" constraint when installing Avocado from PyPI; this would provide the best of both worlds in terms of resiliency for both distros that have required packages and distros that don't. The conflict is visible also for meson, where we would like to install the latest 0.63.x version but also accept a distro 1.1.x version. But it is worse for avocado, for two reasons: 1) we cannot use an "==" constraint to install avocado if the venv includes a system avocado. The distro will package plugins that have "==" constraints on the version that is included in the distro, and, using "pip install avocado==88.1" on a venv that includes system packages will result in this error: ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. avocado-framework-plugin-varianter-yaml-to-mux 98.0 requires avocado-framework==98.0, but you have avocado-framework 88.1 which is incompatible. avocado-framework-plugin-result-html 98.0 requires avocado-framework==98.0, but you have avocado-framework 88.1 which is incompatible. make[1]: Leaving directory '/home/berrange/src/virt/qemu/build' 2) we cannot use ">=" either if the venv does _not_ include a system avocado, because that would result in the installation of v101.0 which is the one we've just reverted. So the idea is to encode the dependencies as an (acceptable, locked) tuple, like this hypothetical TOML that would be committed inside python/ and used by mkvenv.py: [meson] meson = { minimum = "0.63.0", install = "0.63.3", canary = "meson" } [docs] # 6.0 drops support for Python 3.7 sphinx = { minimum = "1.6", install = "<6.0", canary = "sphinx-build" } sphinx_rtd_theme = { minimum = "0.5" } [avocado] avocado-framework = { minimum = "88.1", install = "88.1", canary = "avocado" } Once this is implemented, it would also be possible to install avocado in pyvenv/ using "mkvenv.py ensure", thus using the distro package on Fedora and CentOS Stream (the only distros where it's available). But until this is implemented, keep avocado in a separate venv. There is still the benefit of using a single python for meson custom_targets and for sphinx. Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2023-05-18tests: Use configure-provided pyvenv for testsJohn Snow1-7/+7
This patch changes how the avocado tests are provided, ever so slightly. Instead of creating a new testing venv, use the configure-provided 'pyvenv' instead, and install optional packages into that. Signed-off-by: John Snow <jsnow@redhat.com> Message-Id: <20230511035435.734312-20-jsnow@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2023-05-16docs/devel: remind developers to run CI container pipeline when updating imagesAni Sinha1-0/+6
When new dependencies and packages are added to containers, its important to run CI container generation pipelines on gitlab to make sure that there are no obvious conflicts between packages that are being added and those that are already present. Running CI container pipelines will make sure that there are no such breakages before we commit the change updating the containers. Add a line in the documentation reminding developers to run the pipeline before submitting the change. It will also ease the life of the maintainers. Signed-off-by: Ani Sinha <anisinha@redhat.com> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> Message-Id: <20230506072012.10350-1-anisinha@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
2023-03-01testing: update ubuntu2004 to ubuntu2204Alex Bennée1-2/+2
The 22.04 LTS release has been out for almost a year now so its time to update all the remaining images to the current LTS. We can also drop some hacks we need for older clang TSAN support. We will keep the ubuntu2004 container around for those who wish to test builds on the currently still supported baseline. Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Reviewed-by: John Snow <jsnow@redhat.com> Reviewed-by: Thomas Huth <thuth@redhat.com> Message-Id: <20230228190653.1602033-9-alex.bennee@linaro.org>
2023-02-27ci, docker: update CentOS and OpenSUSE Python to non-EOL versionsPaolo Bonzini1-0/+6
Python 3.6 is at end-of-life. Update the libvirt-ci module to a version that supports overrides for targets and package mappings; this way, QEMU can use the newer versions provided by CentOS 8 (Python 3.8) and OpenSUSE 15.3 (Python 3.9). Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2023-02-27docs/devel: update and clarify lcitool instructionsPaolo Bonzini1-31/+39
Shorten a bit the description of what libvirt-ci does, the name of the data files is not relevant at that point. However, the procedures to add new build prerequisites are lacking some information, particularly with respect to regenerating the output test files for lcitool's unit tests. While at it, also update the paths in the libvirt-ci repository. Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2022-11-05Fix some typos in documentation and commentsStefan Weil1-1/+1
Most of them were found and fixed using codespell. Signed-off-by: Stefan Weil <sw@weilnetz.de> Reviewed-by: Peter Maydell <peter.maydell@linaro.org> Reviewed-by: Thomas Huth <thuth@redhat.com> Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com> Message-Id: <20221030105944.311940-1-sw@weilnetz.de> Signed-off-by: Laurent Vivier <laurent@vivier.eu>
2022-09-27docs/devel: testing: Document writing portable test casesBin Meng1-0/+30
Update the best practices of how to write portable test cases that can be built and run successfully on both Linux and Windows hosts. Signed-off-by: Bin Meng <bin.meng@windriver.com> Reviewed-by: Marc-André Lureau <marcandre.lureau@redhat.com> Message-Id: <20220927110632.1973965-55-bmeng.cn@gmail.com> Reviewed-by: Thomas Huth <thuth@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
2022-09-20tests/docker: remove the Debian base imagesAlex Bennée1-1/+1
We no longer use these in any of our images. Clean-up the remaining comments and documentation that reference them and remove from the build. Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Reviewed-by: Thomas Huth <thuth@redhat.com> Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Message-Id: <20220914155950.804707-31-alex.bennee@linaro.org>
2022-08-29docs/devel/testing: fix minor typoMatheus Tavares Bernardino1-1/+1
Signed-off-by: Matheus Tavares Bernardino <quic_mathbern@quicinc.com> Message-Id: <421fb28678d1077c0b14978e359b4e1469cc0168.1661262376.git.quic_mathbern@quicinc.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
2022-06-01docs/devel: clean-up the CI links in the docsAlex Bennée1-0/+2
There where some broken links so fix those up with proper references to the devel docs. I also did a little light copy-editing to reflect the current state and broke up a paragraph to reduce the "wall of text" effect. Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> Message-Id: <20220527153603.887929-34-alex.bennee@linaro.org>
2022-04-20docs/devel: add some notes on the binfmt-image-debian targetsAlex Bennée1-0/+38
We document some of this on the wiki but lets move it into our official developer notes documentation. Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Cc: Paolo Bonzini <pbonzini@redhat.com> Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Message-Id: <20220419091020.3008144-3-alex.bennee@linaro.org>
2022-01-28configure: fix parameter expansion of --cross-cc-cflags optionsMatheus Ferst1-1/+1
Without this fix, any use of --cross-cc-cflags-* causes a message like: $ ../configure --cross-cc-ppc64le=clang --cross-cc-cflags-ppc64le="-target powerpc64le-unknown-linux-gnu -sysroot ..." ../configure: 1: eval: cross_cc_cflags_--cross-cc-cflags-ppc64le=-target: not found ../configure: 3816: export: cross_cc_cflags_--cross-cc-cflags-ppc64le: bad variable name Signed-off-by: Matheus Ferst <matheus.ferst@eldorado.org.br> Message-Id: <20220120173142.2755077-1-matheus.ferst@eldorado.org.br> [Fix other occurrences too, noted by Philippe Mathieu-Daudé. - Paolo] Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2022-01-18tests: integrate lcitool for generating build env manifestsDaniel P. Berrangé1-3/+101
This introduces https://gitlab.com/libvirt/libvirt-ci as a git submodule at tests/lcitool/libvirt-ci The 'lcitool' program within this submodule will be used to automatically generate build environment manifests from a definition of requirements in tests/lcitool/projects/qemu.yml It will ultimately be capable of generating - Dockerfiles - Package lists for installation in VMs - Variables for configuring Cirrus CI environments When a new build pre-requisite is needed for QEMU, if this package is not currently known to libvirt-ci, it must first be added to the 'mappings.yml' file in the above git repo. Then the submodule can be updated and the build pre-requisite added to the tests/lcitool/projects/qemu.yml file. Now all the build env manifests can be re-generated using 'make lcitool-refresh' This ensures that when a new build pre-requisite is introduced, it is added to all the different OS containers, VMs and Cirrus CI environments consistently. It also facilitates the addition of containers targetting new distros or updating existing containers to new versions of the same distro, where packages might have been renamed. Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Message-Id: <20211215141949.3512719-8-berrange@redhat.com> Message-Id: <20220105135009.1584676-8-alex.bennee@linaro.org>
2021-11-22docs: Render binary names as monospaced textPhilippe Mathieu-Daudé1-4/+4
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com> Message-Id: <20211118192744.64325-1-philmd@redhat.com> Reviewed-by: Darren Kenny <darren.kenny@oracle.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
2021-11-08tests/avocado: Rename avocado_qemu.Test -> QemuSystemTestPhilippe Mathieu-Daudé1-4/+4
To run user-mode emulation tests, we introduced the avocado_qemu.QemuUserTest which inherits from avocado_qemu.QemuBaseTest. System-mode emulation tests are based on the avocado_qemu.Test class, which also inherits avocado_qemu.QemuBaseTest. To avoid confusion, rename it as avocado_qemu.QemuSystemTest. Suggested-by: Wainer dos Santos Moschetta <wainersm@redhat.com> Reviewed-by: Willian Rampazzo <willianr@redhat.com> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org> Message-Id: <20211105143416.148332-7-f4bug@amsat.org>
2021-11-08tests/acceptance: rename tests acceptance to tests avocadoWillian Rampazzo1-25/+24
In the discussion about renaming the `tests/acceptance` [1], the conclusion was that the folders inside `tests` are related to the framework running the tests and not directly related to the type of the tests. This changes the folder to `tests/avocado` and adjusts the MAKEFILE, the CI related files and the documentation. [1] https://lists.gnu.org/archive/html/qemu-devel/2021-05/msg06553.html Reviewed-by: Niek Linnenbank <nieklinnenbank@gmail.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com> Tested-by: Philippe Mathieu-Daudé <philmd@redhat.com> Signed-off-by: Willian Rampazzo <willianr@redhat.com> Message-Id: <20211105155354.154864-3-willianr@redhat.com> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
2021-10-01docs: reorganize testing.rstPaolo Bonzini1-70/+76
Clean up the heading levels to use === --- ~~~ ^^^ '''. Reorganize the outline for the Avocado part, and always include headings for the class names. Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2021-10-01docs: move gcov section at the end of testing.rstPaolo Bonzini1-19/+19
gcov testing applies to all tests, not just make check. Move it out of the make check section. Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2021-09-27tests/Makefile: add AVOCADO_TESTS option to make check-acceptanceWillian Rampazzo1-0/+27
Add the possibility of running all the tests from a single file, or multiple files, running a single test within a file or multiple tests within multiple files using `make check-acceptance` and the AVOCADO_TESTS environment variable. Suggested-by: Daniel P. Berrangé <berrange@redhat.com> Signed-off-by: Willian Rampazzo <willianr@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com> Message-Id: <20210923161141.232208-4-willianr@redhat.com>
2021-09-27docs/devel/testing: add instruction to run a single acceptance testWillian Rampazzo1-0/+28
Add instructions to the Acceptance tests section about running a single test file or a test within the test file. Signed-off-by: Willian Rampazzo <willianr@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com> Message-Id: <20210923161141.232208-3-willianr@redhat.com>
2021-09-27tests/Makefile: allow control over tags during check-acceptanceWillian Rampazzo1-0/+14
Although it is possible to run a specific test using the avocado command-line, a user may want to use a specific tag while running the ``make check-acceptance`` during the development or debugging. This allows using the AVOCADO_TAGS environment variable where the user takes total control of which tests should run based on the tags defined. This also makes the check-acceptance command flexible to restrict tests based on tags while running on CI. e.g.: AVOCADO_TAGS="foo bar baz" make check-acceptance Signed-off-by: Willian Rampazzo <willianr@redhat.com> Tested-by: Wainer dos Santos Moschetta <wainersm@redhat.com> Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com> Reviewed-by: Thomas Huth <thuth@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com> Message-Id: <20210923161141.232208-2-willianr@redhat.com>
2021-09-01docs/devel/testing: add -p option to the debug section of QEMU iotestsEmanuele Giuseppe Esposito1-0/+4
Signed-off-by: Emanuele Giuseppe Esposito <eesposit@redhat.com> Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Reviewed-by: Max Reitz <mreitz@redhat.com> Message-Id: <20210809090114.64834-17-eesposit@redhat.com> Signed-off-by: Hanna Reitz <hreitz@redhat.com>
2021-09-01docs/devel/testing: add -valgrind option to the debug section of QEMU iotestsEmanuele Giuseppe Esposito1-0/+6
Signed-off-by: Emanuele Giuseppe Esposito <eesposit@redhat.com> Reviewed-by: Max Reitz <mreitz@redhat.com> Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Message-Id: <20210809090114.64834-15-eesposit@redhat.com> Signed-off-by: Hanna Reitz <hreitz@redhat.com>
2021-09-01docs/devel/testing: add -gdb option to the debugging section of QEMU iotestsEmanuele Giuseppe Esposito1-0/+11
Signed-off-by: Emanuele Giuseppe Esposito <eesposit@redhat.com> Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Reviewed-by: Max Reitz <mreitz@redhat.com> Message-Id: <20210809090114.64834-10-eesposit@redhat.com> Signed-off-by: Hanna Reitz <hreitz@redhat.com>
2021-09-01docs/devel/testing: add debug section to the QEMU iotests chapterEmanuele Giuseppe Esposito1-0/+8
Introduce the "Debugging a test case" section, in preparation to the additional flags that will be added in the next patches. Signed-off-by: Emanuele Giuseppe Esposito <eesposit@redhat.com> Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Reviewed-by: Max Reitz <mreitz@redhat.com> Message-Id: <20210809090114.64834-5-eesposit@redhat.com> Signed-off-by: Hanna Reitz <hreitz@redhat.com>
2021-08-02docs/devel: Format literals correctlyPeter Maydell1-4/+4
In rST markup, single backticks `like this` represent "interpreted text", which can be handled as a bunch of different things if tagged with a specific "role": https://docutils.sourceforge.io/docs/ref/rst/restructuredtext.html#interpreted-text (the most common one for us is "reference to a URL, which gets hyperlinked"). The default "role" if none is specified is "title_reference", intended for references to book or article titles, and it renders into the HTML as <cite>...</cite> (usually comes out as italics). Fix various places in the devel section of the manual which were using single backticks when double backticks (for literal text) were intended. Signed-off-by: Peter Maydell <peter.maydell@linaro.org> Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com> Message-id: 20210726142338.31872-6-peter.maydell@linaro.org
2021-07-13tests/acceptance: Automatic set -cpu to the test vmWainer dos Santos Moschetta1-0/+17
This introduces a new feature to the functional tests: automatic setting of the '-cpu VALUE' option to the created vm if the test is tagged with 'cpu:VALUE'. The 'cpu' property is made available to the test object as well. For example, for a simple test as: def test(self): """ :avocado: tags=cpu:host """ self.assertEqual(self.cpu, "host") self.vm.launch() The resulting QEMU evocation will be like: qemu-system-x86_64 -display none -vga none \ -chardev socket,id=mon,path=/var/tmp/avo_qemu_sock_pdgzbgd_/qemu-1135557-monitor.sock \ -mon chardev=mon,mode=control -cpu host Reviewed-by: Cleber Rosa <crosa@redhat.com> Tested-by: Cleber Rosa <crosa@redhat.com> Reviewed-by: Willian Rampazzo <willianr@redhat.com> Signed-off-by: Wainer dos Santos Moschetta <wainersm@redhat.com> Message-Id: <20210430133414.39905-2-wainersm@redhat.com> Signed-off-by: Cleber Rosa <crosa@redhat.com>
2021-07-13Acceptance Tests: support choosing specific distro and versionCleber Rosa1-0/+65
The tests based on the LinuxTest class give the test writer a ready to use guest operating system, currently pinned to Fedora 31. With this change, it's now possible to choose different distros and versions, similar to how other tags and parameter can be set for the target arch, accelerator, etc. One of the reasons for this work, is that some development features depend on updates on the guest side. For instance the tests on virtiofs_submounts.py, require newer kernels, and may benefit from running, say on Fedora 34, without the need for a custom kernel. Please notice that the pre-caching of the Fedora 31 images done during the early stages of `make check-acceptance` (before the tests are actually executed) are not expanded here to cover every new image added. But, the tests will download other needed images (and cache them) during the first execution. Signed-off-by: Cleber Rosa <crosa@redhat.com> Message-Id: <20210414221457.1653745-4-crosa@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com> Reviewed-by: Willian Rampazzo <willianr@redhat.com> Signed-off-by: Cleber Rosa <crosa@redhat.com>
2021-06-01Acceptance Tests: add basic documentation on LinuxTest base classCleber Rosa1-0/+26
Signed-off-by: Cleber Rosa <crosa@redhat.com> Reviewed-by: Marc-André Lureau <marcandre.lureau@redhat.com> Reviewed-by: Willian Rampazzo <willianr@redhat.com> Reviewed-by: Eric Auger <eric.auger@redhat.com> Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com> Message-Id: <20210412044644.55083-10-crosa@redhat.com> Signed-off-by: John Snow <jsnow@redhat.com>
2021-03-20docs/devel/testing.rst: Fix references to unit testsWainer dos Santos Moschetta1-3/+3
With the recent move of the unit tests to tests/unit directory some instructions under the "Unit tests" section became imprecise, which are fixed by this change. Fixes: da668aa15b99 ("tests: Move unit tests into a separate directory") Signed-off-by: Wainer dos Santos Moschetta <wainersm@redhat.com> Reviewed-by: Thomas Huth <thuth@redhat.com> Reviewed-by: Willian Rampazzo <willianr@redhat.com> Message-Id: <20210318174407.2299930-1-wainersm@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
2021-02-24docs/devel: add forward reference to check-tcgAlex Bennée1-0/+5
For completeness reference the check-tcg tests in the container preamble text. Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org> Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Reviewed-by: Thomas Huth <thuth@redhat.com> Cc: Thomas Huth <thuth@redhat.com> Message-Id: <20210222101455.12640-7-alex.bennee@linaro.org>
2021-02-24docs/devel: update the container based testsAlex Bennée1-15/+46
This section has grown a little stale so clean-up the language and examples for current usage: - refer to containers at the top - mention podman can also be used - add podman prerequisites section - move to using "docker-help" for online help - mention the registry and it's purpose - don't refer to out-of-date min-glib image Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Reviewed-by: Thomas Huth <thuth@redhat.com> Message-Id: <20210222101455.12640-6-alex.bennee@linaro.org>