aboutsummaryrefslogtreecommitdiff
path: root/tests/Makefile.include
AgeCommit message (Collapse)AuthorFilesLines
2023-11-24tests: respect --enable/--disable-download for AvocadoPaolo Bonzini1-1/+1
Pass the content of $mkvenv_flags (which is either "--online" or empty) down to tests/Makefile.include. Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2023-10-02Makefile: build plugins before running TCG testsPaolo Bonzini1-1/+1
Add back test-plugins and, after making sure it is always defined, do so unconditionally. Reported-by: Alex Bennée <alex.bennee@linaro.org> Fixes: 2c13c574418 ("configure, meson: move --enable-plugins to meson", 2023-09-07) Reviewed-by: Alex Bennée <alex.bennee@linaro.org> Tested-by: Alex Bennée <alex.bennee@linaro.org> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2023-09-07configure, meson: move --enable-plugins to mesonPaolo Bonzini1-1/+1
While the option still needs to be parsed in the configure script (it's needed by tests/tcg, and also to decide about recursing into contrib/plugins), passing it to Meson can be done with -D instead of using config-host.mak. Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2023-08-28Revert "tests: Use separate virtual environment for avocado"Paolo Bonzini1-11/+8
This reverts commit e8e4298feadae7924cf7600bb3bcc5b0a8d7cbe9. ensuregroup allows to specify both the acceptable versions of avocado, and a locked version to be used when avocado is not installed as a system pacakge. This lets us install avocado in pyvenv/ using "mkvenv.py" and reuse the distro package on Fedora and CentOS Stream (the only distros where it's available). ensuregroup's usage of "(>=..., <=...)" constraints when evaluating the distro package, and "==" constraints when installing it from PyPI, makes it possible to avoid conflicts between the known-good version and a package plugins included in the distro. This is because package plugins have "==" constraints on the version that is included in the distro, and, using "pip install avocado==88.1" on a venv that includes system packages will result in an error: avocado-framework-plugin-varianter-yaml-to-mux 98.0 requires avocado-framework==98.0, but you have avocado-framework 88.1 which is incompatible. avocado-framework-plugin-result-html 98.0 requires avocado-framework==98.0, but you have avocado-framework 88.1 which is incompatible. But at the same time, if the venv does not include a system distribution of avocado then we can install a known-good version and stick to LTS releases. Resolves: https://gitlab.com/qemu-project/qemu/-/issues/1663 Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2023-06-06tests: Use separate virtual environment for avocadoPaolo Bonzini1-7/+9
This reverts commits eea2d141179 ("Makefile: remove $(TESTS_PYTHON)", 2023-05-26) and 9c6692db550 ("tests: Use configure-provided pyvenv for tests", 2023-05-18). Right now, there is a conflict between wanting a ">=" constraint when using a distro-provided package and wanting a "==" constraint when installing Avocado from PyPI; this would provide the best of both worlds in terms of resiliency for both distros that have required packages and distros that don't. The conflict is visible also for meson, where we would like to install the latest 0.63.x version but also accept a distro 1.1.x version. But it is worse for avocado, for two reasons: 1) we cannot use an "==" constraint to install avocado if the venv includes a system avocado. The distro will package plugins that have "==" constraints on the version that is included in the distro, and, using "pip install avocado==88.1" on a venv that includes system packages will result in this error: ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. avocado-framework-plugin-varianter-yaml-to-mux 98.0 requires avocado-framework==98.0, but you have avocado-framework 88.1 which is incompatible. avocado-framework-plugin-result-html 98.0 requires avocado-framework==98.0, but you have avocado-framework 88.1 which is incompatible. make[1]: Leaving directory '/home/berrange/src/virt/qemu/build' 2) we cannot use ">=" either if the venv does _not_ include a system avocado, because that would result in the installation of v101.0 which is the one we've just reverted. So the idea is to encode the dependencies as an (acceptable, locked) tuple, like this hypothetical TOML that would be committed inside python/ and used by mkvenv.py: [meson] meson = { minimum = "0.63.0", install = "0.63.3", canary = "meson" } [docs] # 6.0 drops support for Python 3.7 sphinx = { minimum = "1.6", install = "<6.0", canary = "sphinx-build" } sphinx_rtd_theme = { minimum = "0.5" } [avocado] avocado-framework = { minimum = "88.1", install = "88.1", canary = "avocado" } Once this is implemented, it would also be possible to install avocado in pyvenv/ using "mkvenv.py ensure", thus using the distro package on Fedora and CentOS Stream (the only distros where it's available). But until this is implemented, keep avocado in a separate venv. There is still the benefit of using a single python for meson custom_targets and for sphinx. Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2023-06-06Revert "tests/requirements.txt: bump up avocado-framework version to 101.0"Paolo Bonzini1-11/+7
This reverts commit ec5ffa0056389c3c10ea2de1e78366f66f4e5abc. Bumping avocado to version 101 has two issues. First, there are problems where Avocado is not logging of command lines or terminal output, and not collecting Python logs outside the avocado namespace. Second, the recent changes to Python handling mean that there is a single virtual environment for all the build, instead of a separate one for testing. Requiring a too-new version of avocado causes conflicts with any avocado plugins installed on the host: $ make check-venv make[1]: Entering directory '/home/berrange/src/virt/qemu/build' GIT ui/keycodemapdb tests/fp/berkeley-testfloat-3 tests/fp/berkeley-softfloat-3 dtc VENVPIP install -e /home/berrange/src/virt/qemu/python/ VENVPIP install -r /home/berrange/src/virt/qemu/tests/requirements.txt ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. avocado-framework-plugin-varianter-yaml-to-mux 98.0 requires avocado-framework==98.0, but you have avocado-framework 101.0 which is incompatible. avocado-framework-plugin-result-html 98.0 requires avocado-framework==98.0, but you have avocado-framework 101.0 which is incompatible. make[1]: Leaving directory '/home/berrange/src/virt/qemu/build' To avoid this issue, tests/requirements.txt should use a ">=" constraint and the version of Avocado should be limited to what distros provide in the system packages. Only Fedora has Avocado, and more specifically version 92.0 (though 98.0 is also available as a module). As a first step, this patch reverts the introduction of a too-new Avocado. Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2023-05-26Makefile: remove $(TESTS_PYTHON)Paolo Bonzini1-5/+3
It is now the same as $(PYTHON), since the latter always points at pyvenv/bin/python3. Reviewed-by: Alex Bennée <alex.bennee@linaro.org> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2023-05-18tests: Use configure-provided pyvenv for testsJohn Snow1-5/+5
This patch changes how the avocado tests are provided, ever so slightly. Instead of creating a new testing venv, use the configure-provided 'pyvenv' instead, and install optional packages into that. Signed-off-by: John Snow <jsnow@redhat.com> Message-Id: <20230511035435.734312-20-jsnow@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2023-04-27tests/requirements.txt: bump up avocado-framework version to 101.0Kautuk Consul1-7/+11
Avocado version 101.0 has a fix to re-compute the checksum of an asset file if the algorithm used in the *-CHECKSUM file isn't the same as the one being passed to it by the avocado user (i.e. the avocado_qemu python module). In the earlier avocado versions this fix wasn't there due to which if the checksum wouldn't match the earlier checksum (calculated by a different algorithm), the avocado code would start downloading a fresh image from the internet URL thus making the test-cases take longer to execute. Bump up the avocado-framework version to 101.0. Signed-off-by: Kautuk Consul <kconsul@linux.vnet.ibm.com> Tested-by: Hariharan T S <hariharan.ts@linux.vnet.ibm.com> Reviewed-by: Thomas Huth <thuth@redhat.com> Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Message-Id: <20230327115030.3418323-2-kconsul@linux.vnet.ibm.com> Message-Id: <20230424092249.58552-2-alex.bennee@linaro.org>
2022-10-06configure: move tests/tcg/Makefile.prereqs to root build directoryPaolo Bonzini1-3/+0
It will not be specific to tests/tcg anymore, since it will be possible to build firmware using container-based cross compilers too. Signed-off-by: Paolo Bonzini <pbonzini@redhat.com> Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Message-Id: <20220929114231.583801-26-alex.bennee@linaro.org>
2022-10-06tests/tcg: add distclean rulePaolo Bonzini1-0/+11
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com> Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Message-Id: <20220929114231.583801-22-alex.bennee@linaro.org>
2022-10-06tests/tcg: remove -f from Makefile invocationPaolo Bonzini1-4/+3
Instead of linking tests/tcg/Makefile.target into the build tree, name the symbolic link "Makefile" and create it in every target subdirectory. This makes it possible to just invoke "make" in tests/tcg subdirectories. Signed-off-by: Paolo Bonzini <pbonzini@redhat.com> Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Message-Id: <20220929114231.583801-21-alex.bennee@linaro.org>
2022-10-06tests: simplify Makefile invocation for tests/tcgPaolo Bonzini1-6/+3
Remove the DOCKER_SCRIPT and TARGET variable from the Makefile invocation for tests/tcg. For DOCKER_SCRIPT, resolve the path to docker.py in configure; for TARGET, move it to config-$(TARGET).mak and use a symbolic link to break the cycle. The symbolic link is still needed because tests/tcg includes dummy config files for targets that are not buildable. Once that is cleaned up, the symbolic link will go away too. Signed-off-by: Paolo Bonzini <pbonzini@redhat.com> Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Message-Id: <20220929114231.583801-20-alex.bennee@linaro.org>
2022-06-06tests: install "qemu" namespace package into venvJohn Snow1-0/+1
This patch adds the "qemu" namespace package to the $build/tests/venv directory. It does so in "editable" mode, which means that changes to the source python directory will actively be reflected by the venv. This patch also then removes any sys.path hacking from the avocado test scripts directly. By doing this, the environment of where to find these packages is managed entirely by the virtual environment and not by the scripts themselves. Signed-off-by: John Snow <jsnow@redhat.com> Reviewed-by: Paolo Bonzini <pbonzini@redhat.com> Message-Id: <20220526000921.1581503-7-jsnow@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2022-06-06tests: add quiet-venv-pip macroJohn Snow1-6/+6
Factor out the "test venv pip" macro; rewrite the "check-venv" rule to be a little more compact. Replace the "PIP" pseudo-command output with "VENVPIP" to make it 1% more clear that we are talking about using pip to install something into a venv. Signed-off-by: John Snow <jsnow@redhat.com> Reviewed-by: Paolo Bonzini <pbonzini@redhat.com> Message-Id: <20220526000921.1581503-6-jsnow@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2022-06-06tests: silence pip upgrade warnings during venv creationJohn Snow1-2/+2
Turn off the nag warning coaxing us to upgrade pip. It's not really that interesting to see in CI logs, and as long as nothing is broken -- nothing is broken. Signed-off-by: John Snow <jsnow@redhat.com> Reviewed-by: Paolo Bonzini <pbonzini@redhat.com> Message-Id: <20220526000921.1581503-5-jsnow@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2022-06-06tests: use python3 as the python executable nameJohn Snow1-1/+1
Use "python3" instead of "python" as per PEP0394: https://peps.python.org/pep-0394/ This should always be defined (in a venv, at least!), matching the preferred python shebang of "#!/usr/bin/env python3". Signed-off-by: John Snow <jsnow@redhat.com> Reviewed-by: Paolo Bonzini <pbonzini@redhat.com> Message-Id: <20220526000921.1581503-4-jsnow@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2022-06-06tests: add "TESTS_PYTHON" variable to MakefileJohn Snow1-3/+4
This is a convenience feature: $(PYTHON) points to the Python executable we were instructed to use by the configure script. We use that Python to create a virtual environment with the "check-venv" target in tests/Makefile.include. $(TESTS_PYTHON) points to the Python executable belonging to the virtual environment tied to the build. This Python executable is a symlink to the binary used to create the venv, which will be the version provided at configure time. Using $(TESTS_PYTHON) therefore uses the $(PYTHON) executable, but with paths modified to use packages installed to the venv. Signed-off-by: John Snow <jsnow@redhat.com> Reviewed-by: Paolo Bonzini <pbonzini@redhat.com> Message-Id: <20220526000921.1581503-3-jsnow@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2022-06-06tests/Makefile.include: Fix 'make check-help' outputDario Faggioli1-15/+15
Since commit 3d2f73ef75e ("build: use "meson test" as the test harness"), check-report.tap is no more, and we have check-report.junit.xml. Update the output of 'make check-help', which was still listing 'check-report.tap', accordingly. Fixes: 3d2f73ef75e Signed-off-by: Dario Faggioli <dfaggioli@suse.com> Message-Id: <165366545439.6869.11633009118019728798.stgit@work> Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2022-06-01tests/tcg: merge configure.sh back into main configure scriptPaolo Bonzini1-1/+0
tests/tcg/configure.sh has a complicated story. In the beginning its code ran as part of the creation of config-target.mak files, and that is where it placed the information on the target compiler. However, probing for the buildability of TCG tests required multiple inclusions of config-target.mak in the _main_ Makefile (not in Makefile.target, which took care of building the QEMU executables in the pre-Meson era), which polluted the namespace. Thus, it was moved to a separate directory. It created small config-*.mak files in $(BUILD_DIR)/tests/tcg. Those were also included multiple times, but at least they were small and manageable; this was also an important step in disentangling the TCG tests from Makefile.target. Since then, Meson has allowed the configure script to go on a diet. A few compilation tests survive (mostly for sanitizers) but these days it mostly takes care of command line parsing, looking for tools, and setting up the environment for Meson to do its stuff. It's time to extend configure with the capability to build for more than just one target: not just tests, but also firmware. As a first step, integrate all the logic to find cross compilers in the configure script, and move tests/tcg/configure.sh back there (though as a separate loop, not integrated in the one that generates target configurations for Meson). tests/tcg is actually very close to being buildable as a standalone project, so I actually expect the compiler tests to move back to tests/tcg, as a "configure" script of sorts which would run at Make time after the docker images are built. The GCC tree has a similar idea of doing only bare-bones tree-wide configuration and leaving the rest for Make time. Signed-off-by: Paolo Bonzini <pbonzini@redhat.com> Acked-by: Richard Henderson <richard.henderson@linaro.org> Message-Id: <20220517092616.1272238-8-pbonzini@redhat.com> Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Message-Id: <20220527153603.887929-19-alex.bennee@linaro.org>
2022-06-01build: do a full build before running TCG testsPaolo Bonzini1-1/+2
TCG tests need both QEMU and firmware to be built, so do "ninja all" before trying to run them. Signed-off-by: Paolo Bonzini <pbonzini@redhat.com> Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Message-Id: <20220517092616.1272238-5-pbonzini@redhat.com> Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Message-Id: <20220527153603.887929-16-alex.bennee@linaro.org>
2022-04-20tests/tcg: invoke Makefile.target directly from QEMU's makefilePaolo Bonzini1-19/+21
Build the "docker.py cc" invocation directly in tests/tcg/configure.sh, and remove the Makefile.qemu wrapper around Makefile.target. The config-*.mak files now include the actual variables used when building the tests, rather than the CROSS_* variables that Makefile.qemu used to "translate". This is a first step towards generalizing the cross-compilation infrastructure so that it can be used for firmware as well. Signed-off-by: Paolo Bonzini <pbonzini@redhat.com> Message-Id: <20220401141326.1244422-15-pbonzini@redhat.com> Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Message-Id: <20220419091020.3008144-18-alex.bennee@linaro.org>
2022-04-20tests/tcg: list test targets in Makefile.prereqsPaolo Bonzini1-7/+7
Omit the rules altogether for targets that do not have a compiler. Makefile.qemu now is only invoked if the tests are actually built/run. Signed-off-by: Paolo Bonzini <pbonzini@redhat.com> Message-Id: <20220401141326.1244422-14-pbonzini@redhat.com> Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Message-Id: <20220419091020.3008144-17-alex.bennee@linaro.org>
2022-04-20tests/tcg: prepare Makefile.prereqs at configure timePaolo Bonzini1-3/+6
List the dependencies of the build-tcg-tests-* and run-tcg-tests-* targets in a Makefile fragment, without going through Makefile.prereqs's "parsing" of config-*.mak. Signed-off-by: Paolo Bonzini <pbonzini@redhat.com> Message-Id: <20220401141326.1244422-13-pbonzini@redhat.com> Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Message-Id: <20220419091020.3008144-16-alex.bennee@linaro.org>
2022-03-23tests/Makefile.include: Let "make clean" remove the TCG tests, tooThomas Huth1-1/+1
"make clean" should clear all binaries that have been built, but so far it left the TCG tests still in place. Let's make sure that they are now removed, too. Signed-off-by: Thomas Huth <thuth@redhat.com> Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Message-Id: <20220301085900.1443232-1-thuth@redhat.com> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
2022-03-22tests: Do not treat the iotests as separate meson test target anymoreThomas Huth1-8/+1
If there is a failing iotest, the output is currently not logged to the console anymore. To get this working again, we need to run the meson test runner with "--print-errorlogs" (and without "--verbose" due to a current meson bug that will be fixed here: https://github.com/mesonbuild/meson/commit/c3f145ca2b9f5.patch ). We could update the "meson test" call in tests/Makefile.include, but actually it's nicer and easier if we simply do not treat the iotests as separate test target anymore and integrate them along with the other test suites. This has the disadvantage of not getting the detailed progress indication there anymore, but since that was only working right in single-threaded "make -j1" mode anyway, it's not a huge loss right now. Signed-off-by: Thomas Huth <thuth@redhat.com> Message-Id: <20220310075048.2303495-1-thuth@redhat.com> Tested-by: Hanna Reitz <hreitz@redhat.com> Signed-off-by: Hanna Reitz <hreitz@redhat.com>
2022-02-21configure, meson: move guest-agent, tools to mesonPaolo Bonzini1-1/+1
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2022-02-09tests/Makefile.include: clean-up old codeAlex Bennée1-4/+0
This is no longer needed since a2ce7dbd91 ("meson: convert tests/qtest to meson", 2020-08-21) Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org> Message-Id: <20220204204335.1689602-2-alex.bennee@linaro.org>
2022-01-28build: make check-block a meson testPaolo Bonzini1-13/+3
"meson test" can be asked to run tests verbosely; this makes it usable also for qemu-iotests's own harness, and it lets "make check-block" reuse mtest2make.py's infrastructure to find and build test dependencies. Adjust check-block.sh to use the standard exit code that reports a test as skipped. Alternatively, in the future we could make it produce TAP output, which is consistent with all other "make check" tests. Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2022-01-18tests: Fix typo in check-help outputPhilippe Mathieu-Daudé1-1/+1
Fix typo in 'make check-help' output. Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org> Message-Id: <20220111175528.22294-1-f4bug@amsat.org> Signed-off-by: Laurent Vivier <laurent@vivier.eu>
2021-11-08tests/acceptance: rename tests acceptance to tests avocadoWillian Rampazzo1-2/+2
In the discussion about renaming the `tests/acceptance` [1], the conclusion was that the folders inside `tests` are related to the framework running the tests and not directly related to the type of the tests. This changes the folder to `tests/avocado` and adjusts the MAKEFILE, the CI related files and the documentation. [1] https://lists.gnu.org/archive/html/qemu-devel/2021-05/msg06553.html Reviewed-by: Niek Linnenbank <nieklinnenbank@gmail.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com> Tested-by: Philippe Mathieu-Daudé <philmd@redhat.com> Signed-off-by: Willian Rampazzo <willianr@redhat.com> Message-Id: <20211105155354.154864-3-willianr@redhat.com> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
2021-11-08tests/acceptance: introduce new check-avocado targetWillian Rampazzo1-5/+12
This introduces a new `make` target, `check-avocado`, and adds a deprecation message about the `check-acceptance` target. This is a preparation for renaming the `tests/acceptance` folder to `tests/avocado`. The plan is to remove the call to the `check-avocado` target one or two months after the release and leave the warning to force people to move to the new `check-avocado` target. Later, the `check-acceptance` target can be removed. The intent is to avoid a direct impact during the current soft freeze. Suggested-by: Philippe Mathieu-Daudé <philmd@redhat.com> Signed-off-by: Willian Rampazzo <willianr@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com> Tested-by: Philippe Mathieu-Daudé <philmd@redhat.com> Acked-by: Thomas Huth <thuth@redhat.com> Message-Id: <20211105155354.154864-2-willianr@redhat.com> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
2021-10-13build: fix "make check" without earlier "make"Paolo Bonzini1-5/+12
"make check", if not preceded by "make", will not build the tools needed by qemu-iotests. This happens because qemu-iotests, aka "make check-block", is not yet part of meson.build. While at it, remove the reference to the now-dead QEMU_IOTESTS_HELPERS-y variable. Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2021-10-13tests: add missing dependency for check-blockPaolo Bonzini1-0/+1
qemu-iotests run qemu-storage-daemon, make sure it is up to date. Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2021-10-12python, iotests: remove socket_scm_helperJohn Snow1-1/+0
It's not used anymore, now. Signed-off-by: John Snow <jsnow@redhat.com> Reviewed-by: Hanna Reitz <hreitz@redhat.com> Reviewed-by: Paolo Bonzini <pbonzini@redhat.com> Message-id: 20210923004938.3999963-11-jsnow@redhat.com Signed-off-by: John Snow <jsnow@redhat.com>
2021-09-27Acceptance Tests: improve check-acceptance descriptionCleber Rosa1-1/+1
The "check-acceptance" make rule won't necessarily run *all* available tests, because it employs a filter based on the currently configured targets. This change in the description of the rule makes that behavior extra clear. Signed-off-by: Cleber Rosa <crosa@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com> Message-Id: <20210924185506.2542588-3-crosa@redhat.com>
2021-09-27tests/Makefile: add AVOCADO_TESTS option to make check-acceptanceWillian Rampazzo1-1/+4
Add the possibility of running all the tests from a single file, or multiple files, running a single test within a file or multiple tests within multiple files using `make check-acceptance` and the AVOCADO_TESTS environment variable. Suggested-by: Daniel P. Berrangé <berrange@redhat.com> Signed-off-by: Willian Rampazzo <willianr@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com> Message-Id: <20210923161141.232208-4-willianr@redhat.com>
2021-09-27tests/Makefile: allow control over tags during check-acceptanceWillian Rampazzo1-3/+9
Although it is possible to run a specific test using the avocado command-line, a user may want to use a specific tag while running the ``make check-acceptance`` during the development or debugging. This allows using the AVOCADO_TAGS environment variable where the user takes total control of which tests should run based on the tags defined. This also makes the check-acceptance command flexible to restrict tests based on tags while running on CI. e.g.: AVOCADO_TAGS="foo bar baz" make check-acceptance Signed-off-by: Willian Rampazzo <willianr@redhat.com> Tested-by: Wainer dos Santos Moschetta <wainersm@redhat.com> Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com> Reviewed-by: Thomas Huth <thuth@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com> Message-Id: <20210923161141.232208-2-willianr@redhat.com>
2021-07-13Acceptance tests: do not try to reuse packages from the systemCleber Rosa1-1/+1
The premise behind the original behavior is that it would save people from downloading Avocado (and other dependencies) if already installed on the system. To be honest, I think it's extremely rare that the same versions described as dependencies will be available on most systems. But, the biggest motivations here are that: 1) Hacking on QEMU in the same system used to develop Avocado leads to confusion with regards to the exact bits that are being used; 2) Not reusing Python packages from system wide installations gives extra assurance that the same behavior will be seen from tests run on different machines; With regards to downloads, pip already caches the downloaded wheels and tarballs under ~/.cache/pip, so there should not be more than one download even if the venv is destroyed and recreated. Signed-off-by: Cleber Rosa <crosa@redhat.com> Message-Id: <20210415215141.1865467-3-crosa@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com> Reviewed-by: Willian Rampazzo <willianr@redhat.com> Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com> Signed-off-by: Cleber Rosa <crosa@redhat.com>
2021-07-09configs: rename default-configs to configs and reorganiseAlex Bennée1-1/+1
In preparation for offering variation to our build configurations lets move everything and rename it to default. Common included base configs are also renamed. During the cleanup the stale usb.mak and pci.mak references were removed from MAINTAINERS. Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Cc: Paolo Bonzini <pbonzini@redhat.com> Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Message-Id: <20210707131744.26027-5-alex.bennee@linaro.org> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2021-03-12tests: remove "make check-speed" in favor of "make bench"Paolo Bonzini1-6/+2
"make check-speed" has been broken since the removal of ninja2make last October. It was just a backwards-compatibility alias for "make bench-speed", which in turn is in principle a subset of "make bench". Advertise the latter and drop "make check-speed" completely since no one has noticed. Reported-by: Thomas Huth <thuth@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com> Message-Id: <20210310164612.285362-1-pbonzini@redhat.com> Reviewed-by: Thomas Huth <thuth@redhat.com> Reviewed-by: Willian Rampazzo <willianr@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
2021-03-09Acceptance Tests: restore filtering of tests by target archCleber Rosa1-1/+1
Previously, tests were being filtered by the matching target architectures to be built. The benefit, compared to the current situation, is a more concise test job that won't show tests canceled because a matching QEMU binary was not found (those tests won't even be attempted). Signed-off-by: Cleber Rosa <crosa@redhat.com> Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com> Reviewed-by: Willian Rampazzo <willianr@redhat.com> Message-Id: <20210225232122.1254879-3-crosa@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
2021-03-09Acceptance Tests: restore downloading of VM imagesCleber Rosa1-1/+2
The "get-vm-images" target defined in tests/Makefile.include is a prerequisite for "check-acceptance", so that those files get downloaded before the Avocado job even starts. It looks like on c401c058a1c a TARGETS variable was introduced with a different content than it was previously coming from the main Makefile. From that point on, the "get-vm-images" succeed without doing anything because there was no matching architecture to download. This restores the download of images (that match targets to be built) before the job starts, eliminating downloads and their associated failures during the tests. Signed-off-by: Cleber Rosa <crosa@redhat.com> Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com> Reviewed-by: Willian Rampazzo <willianr@redhat.com> Message-Id: <20210225232122.1254879-2-crosa@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
2021-02-08tests/Makefile.include: don't use TARGET_DIRS for check-tcgAlex Bennée1-4/+6
TARGET_DIRS reflects what we wanted to configure which in the normal case is all our targets. However once meson has pared-down our target list due to missing features we need to check the final list of ninja-targets. This prevents check-tcg barfing on a --disable-tcg build. Suggested-by: Paolo Bonzini <pbonzini@redhat.com> Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Acked-by: Paolo Bonzini <pbonzini@redhat.com> Message-Id: <20210202134001.25738-14-alex.bennee@linaro.org>
2021-02-08scripts/mtest2make.py: export all-%s-targets variable and use itAlex Bennée1-1/+1
There are some places where the conditional makefile support is the simplest solution. Now we don't expose CONFIG_TCG as a variable create a new one that can be checked for the check-help output. As check-tcg is a PHONY target we re-use check-softfloat to gate that as well. Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Acked-by: Paolo Bonzini <pbonzini@redhat.com> Message-Id: <20210202134001.25738-13-alex.bennee@linaro.org>
2021-01-29tests/Makefile.include: export PYTHON for check-block.shVladimir Sementsov-Ogievskiy1-0/+1
check-block.sh called by make check-block rely on PYTHON variable being set. Fixes: f203080bbd9f9e5b31041b1f2afcd6040c5aaec5 Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Message-Id: <20210129051346.56109-1-vsementsov@virtuozzo.com> Tested-by: Thomas Huth <thuth@redhat.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2020-10-17build: replace ninjatool with ninjaPaolo Bonzini1-1/+1
Now that the build is done entirely by Meson, there is no need to keep the Makefile conversion. Instead, we can ask Ninja about the targets it exposes and forward them. The main advantages are, from smallest to largest: - reducing the possible namespace pollution within the Makefile - removal of a relatively large Python program - faster build because parsing Makefile.ninja is slower than parsing build.ninja; and faster build after Meson runs because we do not have to generate Makefile.ninja. - tracking of command lines, which provides more accurate rebuilds In addition the change removes the requirement for GNU make 3.82, which was annoying on Mac, and avoids bugs on Windows due to ninjatool not knowing how to convert Windows escapes to POSIX escapes. Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2020-10-17tests/Makefile.include: unbreak non-tcg buildsPaolo Bonzini1-1/+1
Remove from check-block the requirement that all TARGET_DIRS are built. Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2020-10-04tests: tcg: do not use implicit rulesPaolo Bonzini1-3/+3
Use pattern rules to clarify which targets are going to match the rule and to provide clearer error messages. Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2020-09-10plugins: move the more involved plugins to contribAlex Bennée1-1/+1
We have an exploding complexity problem in the testing so lets just move the more involved plugins into contrib. tests/plugins still exist for the basic plugins that exercise the API. We restore the old pre-meson style Makefile for contrib as it also doubles as a guide for out-of-tree plugin builds. While we are at it add some examples to the documentation and a specific plugins build target. Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Message-Id: <20200909112742.25730-11-alex.bennee@linaro.org>
generated by cgit v1.1 at 2024-12-28 14:34:10 +0000