diff options
author | Richard Henderson <richard.henderson@linaro.org> | 2021-11-08 18:50:09 +0100 |
---|---|---|
committer | Richard Henderson <richard.henderson@linaro.org> | 2021-11-08 18:50:09 +0100 |
commit | 114f3c8cc427333dbae331dfd2ecae64676b087e (patch) | |
tree | 6b609b516b20585ab8ad70cd86055c37132fd169 /docs | |
parent | 5e10ccc2706768bcba608f5d174d4079944c270a (diff) | |
parent | b94d00898a9ee708b4d4b79432c2272905a56b06 (diff) | |
download | qemu-114f3c8cc427333dbae331dfd2ecae64676b087e.zip qemu-114f3c8cc427333dbae331dfd2ecae64676b087e.tar.gz qemu-114f3c8cc427333dbae331dfd2ecae64676b087e.tar.bz2 |
Merge remote-tracking branch 'remotes/philmd/tags/avocado-20211108' into staging
Integration testing patches
- Rename tests/acceptance/ -> tests/avocado/
- Rename avocado_qemu.Test -> avocado_qemu.QemuSystemTest
- Introduce QemuUserTest class
- Add the first linux-user test, covering the bFLT loader
# gpg: Signature made Mon 08 Nov 2021 05:16:46 PM CET
# gpg: using RSA key FAABE75E12917221DCFD6BB2E3E32C2CDEADC0DE
# gpg: Good signature from "Philippe Mathieu-Daudé (F4BUG) <f4bug@amsat.org>" [full]
* remotes/philmd/tags/avocado-20211108:
tests/avocado: Remove p7zip binary availability check
tests/avocado: Rename avocado_qemu.Test -> QemuSystemTest
tests/avocado: Add bFLT loader linux-user test
tests/avocado: Share useful helpers from virtiofs_submounts test
tests/avocado: Introduce QemuUserTest base class
tests/avocado: Make pick_default_qemu_bin() more generic
tests/avocado: Extract QemuBaseTest from Test
tests/acceptance: rename tests acceptance to tests avocado
tests/acceptance: introduce new check-avocado target
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
Diffstat (limited to 'docs')
-rw-r--r-- | docs/devel/build-system.rst | 2 | ||||
-rw-r--r-- | docs/devel/ci-definitions.rst.inc | 2 | ||||
-rw-r--r-- | docs/devel/testing.rst | 57 | ||||
-rw-r--r-- | docs/system/arm/orangepi.rst | 8 |
4 files changed, 34 insertions, 35 deletions
diff --git a/docs/devel/build-system.rst b/docs/devel/build-system.rst index 84282ca..7a83f5f 100644 --- a/docs/devel/build-system.rst +++ b/docs/devel/build-system.rst @@ -416,7 +416,7 @@ number of dynamically created files listed later. ``tests/Makefile.include`` Rules for external test harnesses. These include the TCG tests, - ``qemu-iotests`` and the Avocado-based acceptance tests. + ``qemu-iotests`` and the Avocado-based integration tests. ``tests/docker/Makefile.include`` Rules for Docker tests. Like tests/Makefile, this file is included diff --git a/docs/devel/ci-definitions.rst.inc b/docs/devel/ci-definitions.rst.inc index 32e22ff..6d5c6fd 100644 --- a/docs/devel/ci-definitions.rst.inc +++ b/docs/devel/ci-definitions.rst.inc @@ -59,7 +59,7 @@ to system testing [5]_. Note that, in some cases, system testing may require interaction with third-party software, like operating system images, databases, networks, and so on. -On QEMU, system testing is represented by the 'check-acceptance' target from +On QEMU, system testing is represented by the 'check-avocado' target from 'make'. Flaky tests diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst index 7500f07..60c5902 100644 --- a/docs/devel/testing.rst +++ b/docs/devel/testing.rst @@ -653,17 +653,16 @@ supported. To start the fuzzer, run Alternatively, some command different from "qemu-img info" can be tested, by changing the ``-c`` option. -Acceptance tests using the Avocado Framework --------------------------------------------- +Integration tests using the Avocado Framework +--------------------------------------------- -The ``tests/acceptance`` directory hosts functional tests, also known -as acceptance level tests. They're usually higher level tests, and -may interact with external resources and with various guest operating -systems. +The ``tests/avocado`` directory hosts integration tests. They're usually +higher level tests, and may interact with external resources and with +various guest operating systems. These tests are written using the Avocado Testing Framework (which must be installed separately) in conjunction with a the ``avocado_qemu.Test`` -class, implemented at ``tests/acceptance/avocado_qemu``. +class, implemented at ``tests/avocado/avocado_qemu``. Tests based on ``avocado_qemu.Test`` can easily: @@ -695,11 +694,11 @@ Tests based on ``avocado_qemu.Test`` can easily: Running tests ~~~~~~~~~~~~~ -You can run the acceptance tests simply by executing: +You can run the avocado tests simply by executing: .. code:: - make check-acceptance + make check-avocado This involves the automatic creation of Python virtual environment within the build tree (at ``tests/venv``) which will have all the @@ -714,12 +713,12 @@ specific version, they may be on packages named ``python3-venv`` and ``python3-pip``. It is also possible to run tests based on tags using the -``make check-acceptance`` command and the ``AVOCADO_TAGS`` environment +``make check-avocado`` command and the ``AVOCADO_TAGS`` environment variable: .. code:: - make check-acceptance AVOCADO_TAGS=quick + make check-avocado AVOCADO_TAGS=quick Note that tags separated with commas have an AND behavior, while tags separated by spaces have an OR behavior. For more information on Avocado @@ -728,31 +727,31 @@ tags, see: https://avocado-framework.readthedocs.io/en/latest/guides/user/chapters/tags.html To run a single test file, a couple of them, or a test within a file -using the ``make check-acceptance`` command, set the ``AVOCADO_TESTS`` +using the ``make check-avocado`` command, set the ``AVOCADO_TESTS`` environment variable with the test files or test names. To run all tests from a single file, use: .. code:: - make check-acceptance AVOCADO_TESTS=$FILEPATH + make check-avocado AVOCADO_TESTS=$FILEPATH The same is valid to run tests from multiple test files: .. code:: - make check-acceptance AVOCADO_TESTS='$FILEPATH1 $FILEPATH2' + make check-avocado AVOCADO_TESTS='$FILEPATH1 $FILEPATH2' To run a single test within a file, use: .. code:: - make check-acceptance AVOCADO_TESTS=$FILEPATH:$TESTCLASS.$TESTNAME + make check-avocado AVOCADO_TESTS=$FILEPATH:$TESTCLASS.$TESTNAME The same is valid to run single tests from multiple test files: .. code:: - make check-acceptance AVOCADO_TESTS='$FILEPATH1:$TESTCLASS1.$TESTNAME1 $FILEPATH2:$TESTCLASS2.$TESTNAME2' + make check-avocado AVOCADO_TESTS='$FILEPATH1:$TESTCLASS1.$TESTNAME1 $FILEPATH2:$TESTCLASS2.$TESTNAME2' The scripts installed inside the virtual environment may be used without an "activation". For instance, the Avocado test runner @@ -760,9 +759,9 @@ may be invoked by running: .. code:: - tests/venv/bin/avocado run $OPTION1 $OPTION2 tests/acceptance/ + tests/venv/bin/avocado run $OPTION1 $OPTION2 tests/avocado/ -Note that if ``make check-acceptance`` was not executed before, it is +Note that if ``make check-avocado`` was not executed before, it is possible to create the Python virtual environment with the dependencies needed running: @@ -775,20 +774,20 @@ a test file. To run tests from a single file within the build tree, use: .. code:: - tests/venv/bin/avocado run tests/acceptance/$TESTFILE + tests/venv/bin/avocado run tests/avocado/$TESTFILE To run a single test within a test file, use: .. code:: - tests/venv/bin/avocado run tests/acceptance/$TESTFILE:$TESTCLASS.$TESTNAME + tests/venv/bin/avocado run tests/avocado/$TESTFILE:$TESTCLASS.$TESTNAME Valid test names are visible in the output from any previous execution -of Avocado or ``make check-acceptance``, and can also be queried using: +of Avocado or ``make check-avocado``, and can also be queried using: .. code:: - tests/venv/bin/avocado list tests/acceptance + tests/venv/bin/avocado list tests/avocado Manual Installation ~~~~~~~~~~~~~~~~~~~ @@ -806,16 +805,16 @@ Alternatively, follow the instructions on this link: Overview ~~~~~~~~ -The ``tests/acceptance/avocado_qemu`` directory provides the +The ``tests/avocado/avocado_qemu`` directory provides the ``avocado_qemu`` Python module, containing the ``avocado_qemu.Test`` class. Here's a simple usage example: .. code:: - from avocado_qemu import Test + from avocado_qemu import QemuSystemTest - class Version(Test): + class Version(QemuSystemTest): """ :avocado: tags=quick """ @@ -860,10 +859,10 @@ and hypothetical example follows: .. code:: - from avocado_qemu import Test + from avocado_qemu import QemuSystemTest - class MultipleMachines(Test): + class MultipleMachines(QemuSystemTest): def test_multiple_machines(self): first_machine = self.get_vm() second_machine = self.get_vm() @@ -913,7 +912,7 @@ like this: self.ssh_command('some_command_to_be_run_in_the_guest') Please refer to tests that use ``avocado_qemu.LinuxTest`` under -``tests/acceptance`` for more examples. +``tests/avocado`` for more examples. QEMUMachine ~~~~~~~~~~~ @@ -1204,7 +1203,7 @@ And remove any package you want with:: pip uninstall <package_name> -If you've used ``make check-acceptance``, the Python virtual environment where +If you've used ``make check-avocado``, the Python virtual environment where Avocado is installed will be cleaned up as part of ``make check-clean``. .. _checktcg-ref: diff --git a/docs/system/arm/orangepi.rst b/docs/system/arm/orangepi.rst index 6f23907..c55694d 100644 --- a/docs/system/arm/orangepi.rst +++ b/docs/system/arm/orangepi.rst @@ -250,14 +250,14 @@ and set the following environment variables before booting: Optionally you may save the environment variables to SD card with 'saveenv'. To continue booting simply give the 'boot' command and NetBSD boots. -Orange Pi PC acceptance tests -""""""""""""""""""""""""""""" +Orange Pi PC integration tests +"""""""""""""""""""""""""""""" -The Orange Pi PC machine has several acceptance tests included. +The Orange Pi PC machine has several integration tests included. To run the whole set of tests, build QEMU from source and simply provide the following command: .. code-block:: bash $ AVOCADO_ALLOW_LARGE_STORAGE=yes avocado --show=app,console run \ - -t machine:orangepi-pc tests/acceptance/boot_linux_console.py + -t machine:orangepi-pc tests/avocado/boot_linux_console.py |