aboutsummaryrefslogtreecommitdiff
path: root/docs/devel/testing
AgeCommit message (Collapse)AuthorFilesLines
4 daysdocs/devel: fix over-quoting of QEMU_TEST_KEEP_SCRATCHAlex Bennée1-1/+1
Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> Message-ID: <20250717104105.2656786-1-alex.bennee@linaro.org> Signed-off-by: Thomas Huth <thuth@redhat.com>
11 daysdocs: use :kbd: role in sphinx docsManos Pitsidianakis1-2/+2
Sphinx supports the :kbd: role for notating keyboard input. They get formatted as <kbd> HTML elements in the readthedocs theme we currently use for Sphinx. Besides the better visual formatting, it also helps with accessibility as screen readers can announce the semantics of the <kbd> element to the user. Signed-off-by: Manos Pitsidianakis <manos.pitsidianakis@linaro.org> Message-ID: <20250709-docs_rst_improvements-v2-1-cb5096ad0022@linaro.org> Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Message-ID: <20250710104531.3099313-7-alex.bennee@linaro.org>
2025-05-09docs/devel/testing/functional: Fix typoGustavo Romero1-1/+1
Fix the duplication of the word 'run'. Signed-off-by: Gustavo Romero <gustavo.romero@linaro.org> Reviewed-by: Peter Maydell <peter.maydell@linaro.org> Reviewed-by: Michael Tokarev <mjt@tls.msk.ru> Signed-off-by: Michael Tokarev <mjt@tls.msk.ru>
2025-04-23Remove the remainders of the Avocado testsThomas Huth4-600/+0
Now that all Avocado tests have been converted to or been replaced by other functional tests, we can delete the remainders of the Avocado tests from the QEMU source tree. Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> Message-ID: <20250414113031.151105-16-thuth@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
2025-04-23docs/devel/testing: Dissolve the ci-definitions.rst.inc fileThomas Huth3-133/+81
This file was meant for defining the vocabulary for our testing efforts, but it did not age well: First, the definitions are not only about the CI part, but also about testing in general, so most of the information should rather reside in main.rst instead. Second, some vocabulary has never been really adopted by the QEMU project, for example we never really use the word "system testing" since "system" rather means the system emulator binaries in the QEMU project (and we also don't do any testing with other components like libvirt and virt-managers here). It also defines that the qtests are the "functional" tests in QEMU, which is not really up to date anymore after the "tests/functional" framework has been introduced a couple of months ago (FWIW, the qtests could rather be seen as a mix between unit testing and functional testing). To solve this problem, move the useful parts of this file into main.rst and directly into ci.rst, and drop the ones (like "system testing") that we don't really need anymore. Message-ID: <20250314085959.1585568-1-thuth@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
2025-04-23gitlab-ci: Update QEMU_JOB_AVOCADO and QEMU_CI_AVOCADO_TESTINGThomas Huth1-10/+9
Since we don't run the Avocado jobs in the CI anymore, rename these variables to QEMU_JOB_FUNCTIONAL and QEMU_CI_FUNCTIONAL. Also, there was a mismatch between the documentation and the implementation of QEMU_CI_AVOCADO_TESTING: While the documentation said that you had to "Set this variable to have the tests using the Avocado framework run automatically", you indeed needed to set it to make the pipelines appear in your dashboard - but they were never run automatically in forks and had to be triggered manually. Let's improve this now: No need to hide these pipelines from the users by default anymore (the functional tests should be stable enough nowadays), and rather allow the users to run the pipelines auto- matically with this switch now instead, as was documented. Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> Message-ID: <20250414113031.151105-15-thuth@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
2025-03-19docs/devel/testing/functional: Add a section about loggingThomas Huth1-2/+20
We should tell the users where to find the log file. While we're at it, also rename the "Overview" heading to a more accurate "Introduction to writing tests" instead. Reported-by: Aditya Gupta <adityag@linux.ibm.com> Message-ID: <20250318092021.53719-1-thuth@redhat.com> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
2025-03-07doc: add missing 'Asset' type in function test docAditya Gupta1-2/+4
Seems 'Asset' got missed in the documentation by mistake. Also fix the one spellcheck issue pointed by spellcheck Signed-off-by: Aditya Gupta <adityag@linux.ibm.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org> Message-ID: <20250306060706.1982992-1-adityag@linux.ibm.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
2025-03-06tests/functional: set 'qemu_bin' as an object level fieldDaniel P. Berrangé1-1/+1
The 'qemu_bin' field is currently set on the class, despite being accessed as if it were an object instance field with 'self.qemu_bin'. This is no obvious need to have it as a class field, so move it into the object instance. Reviewed-by: Thomas Huth <thuth@redhat.com> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Message-ID: <20250228102738.3064045-3-berrange@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
2025-01-30tests/functional: Add a decorator for skipping long running testsThomas Huth1-0/+8
Some tests have a very long runtime and might run into timeout issues e.g. when QEMU has been compiled with --enable-debug. Add a decorator for marking them more easily. Rename the corresponding environment variable to be more in sync with the other QEMU_TEST_ALLOW_* switches that we already have, and add a paragraph about it in the documentation. Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> Message-ID: <20250128152839.184599-2-thuth@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
2025-01-17docs/devel: add a codebase sectionPierrick Bouvier2-1/+10
Present the various parts of QEMU and organization of codebase. Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Signed-off-by: Pierrick Bouvier <pierrick.bouvier@linaro.org> Message-Id: <20241209183104.365796-6-pierrick.bouvier@linaro.org> [AJB: tweak commit summary, update MAINTAINERS] Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Message-Id: <20250116160306.1709518-36-alex.bennee@linaro.org>
2024-11-27docs/devel/testing/functional: Clarify that we have to use the build folderThomas Huth1-2/+3
Make it clear that the commands have to be run from the folder with the build, and use the python3 from our pyvenv to make sure that the pycotap module is available. Message-ID: <20241112115302.470527-1-thuth@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
2024-11-25tests/functional: add a QMP backdoor for debugging stalled testsDaniel P. Berrangé1-0/+10
Support the QEMU_TEST_QMP_BACKDOOR=backdoor.sock env variable as a way to get a QMP backdoor for debugging a stalled QEMU test. Most typically this would be used if running the tests directly: $ QEMU_TEST_QMP_BACKDOOR=backdoor.sock \ QEMU_TEST_QEMU_BINARY=./build/qemu-system-arm \ PYTHONPATH=./python \ ./tests/functional/test_arm_tuxrun.py And then, when the test stalls, in a second shell run: $ ./scripts/qmp/qmp-shell backdoor.sock Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> Message-Id: <20241121154218.1423005-18-berrange@redhat.com> Reviewed-by: Thomas Huth <thuth@redhat.com> Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Message-Id: <20241121165806.476008-18-alex.bennee@linaro.org>
2024-11-25tests/functional: automatically clean up scratch files after testsDaniel P. Berrangé1-0/+6
The build/tests/functional subdirectories are consuming huge amounts of disk space. Split the location for scratch files into a 'scratch' sub-directory, separate from log files, and delete it upon completion of each test. The new env variable QEMU_TEST_KEEP_SCRATCH can be set to preserve this scratch dir for debugging access if required. Reviewed-by: Alex Bennée <alex.bennee@linaro.org> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> Message-Id: <20241121154218.1423005-3-berrange@redhat.com> Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Message-Id: <20241121165806.476008-3-alex.bennee@linaro.org>
2024-10-24docs/devel: update tsan build documentationPierrick Bouvier1-4/+22
Mention it's now possible to build with gcc, instead of clang, and explain how to build a sanitized glib version. Signed-off-by: Pierrick Bouvier <pierrick.bouvier@linaro.org> Reviewed-by: Thomas Huth <thuth@redhat.com> Message-Id: <20240910174013.1433331-4-pierrick.bouvier@linaro.org> Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Message-Id: <20241023113406.1284676-6-alex.bennee@linaro.org>
2024-10-15docs/devel/blkverify: Convert to rST formatPeter Maydell2-0/+74
Convert blkverify.txt to rST format. Signed-off-by: Peter Maydell <peter.maydell@linaro.org> Reviewed-by: Thomas Huth <thuth@redhat.com> Message-id: 20240816132212.3602106-3-peter.maydell@linaro.org
2024-10-15docs/devel/blkdebug: Convert to rST formatPeter Maydell2-0/+178
Convert blkdebug.txt to rST format. We put it into index-build.rst because it falls under the "test" part of "QEMU Build and Test System". Signed-off-by: Peter Maydell <peter.maydell@linaro.org> Reviewed-by: Thomas Huth <thuth@redhat.com> Message-id: 20240816132212.3602106-2-peter.maydell@linaro.org
2024-10-11docs: avoid footnotes consisting of just URLsPaolo Bonzini1-13/+13
Replace the footnotes with inline links whenever the footnote text consists of nothing but the URL. While at it, make the link texts consistent in the surrounding areas, for example avoiding usage of "here" for the link's text. In the case of acpi-bits.rst this fixes a build failure with Sphinx 8.1.0, because the FOSDEM link was duplicated in the paragraph and the new version is a lot stricter about unreferenced footnotes. Reviewed-by: Peter Maydell <peter.maydell@linaro.org> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2024-09-17docs/fuzz: fix outdated mention to enable-sanitizersMatheus Tavares Bernardino1-2/+3
This options has been removed at cb771ac1f5 (meson: Split --enable-sanitizers to --enable-{asan, ubsan}, 2024-08-13), so let's update its last standing mention in the docs. Signed-off-by: Matheus Tavares Bernardino <quic_mathbern@quicinc.com> Reviewed-by: Brian Cain <bcain@quicinc.com> Message-ID: <0ecf4e1ab26771009d74a2ce61e7c17ddc586ef7.1726226316.git.quic_mathbern@quicinc.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
2024-09-11Merge tag 'pull-request-2024-09-11' of https://gitlab.com/thuth/qemu into ↵Peter Maydell1-2/+2
staging * Split --enable-sanitizers to --enable-{asan, ubsan} * Build MSYS2 job using multiple CPUs * Fix "make distclean" wrt contrib/plugins/ * Convert more Avocado tests to plain standalone functional tests * Fix bug that breaks "make check-functional" when tesseract is missing * Use builtin hashlib of Python in the functional tests * Update the FreeBSD CI jobs to 14.1 # -----BEGIN PGP SIGNATURE----- # # iQJFBAABCAAvFiEEJ7iIR+7gJQEY8+q5LtnXdP5wLbUFAmbhY4YRHHRodXRoQHJl # ZGhhdC5jb20ACgkQLtnXdP5wLbU/aw/9HXl9H8BUDn8lnoEmxuuQSk8F19n/l5pt # en3L8pMBt4dGFe/9KaGes2GFfid+cp2zlx+qQhA4HW35ntMJorF/qinOH/JGDtoM # 3O6RGZrQPn60zD9P2EbFVCrVYysVYCEu0U3Uglj6tf33bE0L7SJsQxqcbIciyIj5 # aq3Te0yMM2lqzCdMqNpWHGn3VMZRvbRaGBPDU4RLP8V2Bpz1iiRE+6HCH9Kg7HzS # OmleeXtvcyInG+54onjfTcn4/XA27pl1UU04KFv5PrRPB3M2FspHn7oOT2yyQ+ls # 79mqIcd8PvycCT+3ch9p8KhVtbVBgZGmeemALLvk5FxysaWnl4KtSqmQNdqSvvpV # waDDKlLaSnjEHDUse3bCJX0m4d7/vTBY5fOYxqZ4z5dl63csDtgPY4/VF4XR08sP # tR1mW+2qEH9eygsxuKcBjx/j7Etpy+jL9pX2ii1V3ElhjjYuEnpEiURa+TaqPjpZ # jmPtBEszzUdPbrD707tDkW3/ezT7VAnASQeYneJXB/JQG6K6Z//05iX6oCzCbRm3 # ceW/fem3UaeGYpzbMdoZToTuNlXEyS7NDcr39xJjH4LyRTPJAX4zeqUEdzces9g/ # u4Dw6rJ0Yhj4rscKxRvGl3/BH6CTI+8IAsbju2B/CnVLTqaABB0q/MDB90aB44xX # bAVsl4P03Uk= # =5TR0 # -----END PGP SIGNATURE----- # gpg: Signature made Wed 11 Sep 2024 10:31:50 BST # gpg: using RSA key 27B88847EEE0250118F3EAB92ED9D774FE702DB5 # gpg: issuer "thuth@redhat.com" # gpg: Good signature from "Thomas Huth <th.huth@gmx.de>" [full] # gpg: aka "Thomas Huth <thuth@redhat.com>" [full] # gpg: aka "Thomas Huth <huth@tuxfamily.org>" [full] # gpg: aka "Thomas Huth <th.huth@posteo.de>" [unknown] # Primary key fingerprint: 27B8 8847 EEE0 2501 18F3 EAB9 2ED9 D774 FE70 2DB5 * tag 'pull-request-2024-09-11' of https://gitlab.com/thuth/qemu: (24 commits) Update FreeBSD CI jobs FreeBSD 14.1 tests/functional/qemu_test: Use Python hashlib instead of external programs tests/functional: Fix bad usage of has_cmd tests/functional: Convert the multiprocess avocado test into a standalone test tests/functional: Convert the or1k-sim Avocado test tests/functional: Convert the m68k MCF5208EVB Avocado test tests/functional: Convert the Alpha Clipper Avocado test tests/functional: Convert Aarch64 Raspi4 avocado tests tests/functional: Convert Aarch64 Raspi3 avocado tests tests/functional: Convert ARM Raspi2 avocado tests tests/functional: Convert mips32eb 4Kc Malta avocado tests tests/functional: Convert nanomips Malta avocado tests tests/functional: Convert mips32el Malta YAMON avocado test tests/functional: Convert mips64el 5KEc Malta avocado tests tests/functional: Convert mips64el I6400 Malta avocado tests tests/functional: Convert mips64el Fuloong2e avocado test (2/2) tests/functional: Convert the m68k Q800 Avocado test into a functional test tests/functional: Add the LinuxKernelTest for testing the Linux boot process MAINTAINERS: Remove myself from the Meson section MAINTAINERS: Remove myself as reviewer ... Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
2024-09-11meson: Split --enable-sanitizers to --enable-{asan, ubsan}Richard Henderson1-2/+2
We do not always want both address and undefined behavior sanitizers running at the same time. For the gitlab custom-runners, drop to only --enable-ubsan. These jobs are not run by default, but as will be obvious in the next patch, we don't run ASan on x86 either, and it seems wrong to hold aarch64 and s390x to a different standard. Signed-off-by: Richard Henderson <richard.henderson@linaro.org> Reviewed-by: Thomas Huth <thuth@redhat.com> Message-ID: <20240813095216.306555-2-richard.henderson@linaro.org> Signed-off-by: Thomas Huth <thuth@redhat.com>
2024-09-10docs/devel: fix duplicate lineAlex Bennée1-6/+0
I guess the same change came in via two patch series. Remove the repetition. Fixes: 2a851fca9f (docs/devel: remind developers to run CI container pipeline when updating images) Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org> Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Message-Id: <20240910173900.4154726-5-alex.bennee@linaro.org>
2024-09-04docs/devel/testing: Add documentation for functional testsThomas Huth3-0/+351
Document the new functional testing framework. The text is originally based on the Avocado documentation, but heavily modified to match the new framework. Message-ID: <20240830133841.142644-45-thuth@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
2024-09-04docs/devel/testing: Rename avocado_qemu.Test classThomas Huth1-16/+16
The avocado_qemu.Test class has been renamed a while back in commit 2283b627bc ("tests/avocado: Rename avocado_qemu.Test -> QemuSystemTest"), so we should reflect this now in the documentation, too. Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> Message-ID: <20240830133841.142644-44-thuth@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
2024-09-04docs/devel/testing: Split the Avocado documentation into a separate fileThomas Huth3-564/+583
The main testing documentation file got very overloaded already. Thus let's split the Avocado information into a separate file. Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> Message-ID: <20240830133841.142644-43-thuth@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
2024-09-04docs/devel: Split testing docs from the build docs and move to separate folderThomas Huth10-0/+3190
Building and testing are two separate topics, so let's split the testing into a separate category and move the related files into a separate folder. Message-ID: <20240830133841.142644-42-thuth@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com>