From c2511b1632e109130df524121dfb7d2413216d3c Mon Sep 17 00:00:00 2001 From: Yanan Wang Date: Wed, 29 Sep 2021 10:58:02 +0800 Subject: machine: Deprecate "parameter=0" SMP configurations MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit In the SMP configuration, we should either provide a topology parameter with a reasonable value (greater than zero) or just omit it and QEMU will compute the missing value. The users shouldn't provide a configuration with any parameter of it specified as zero (e.g. -smp 8,sockets=0) which could possibly cause unexpected results in the -smp parsing. So we deprecate this kind of configurations since 6.2 by adding the explicit sanity check. Signed-off-by: Yanan Wang Reviewed-by: Cornelia Huck Reviewed-by: Daniel P. Berrangé Message-Id: <20210929025816.21076-3-wangyanan55@huawei.com> Signed-off-by: Paolo Bonzini --- docs/about/deprecated.rst | 15 +++++++++++++++ 1 file changed, 15 insertions(+) (limited to 'docs') diff --git a/docs/about/deprecated.rst b/docs/about/deprecated.rst index 2f7db9a..0bed6ec 100644 --- a/docs/about/deprecated.rst +++ b/docs/about/deprecated.rst @@ -160,6 +160,21 @@ Use ``-display sdl`` instead. Use ``-display curses`` instead. +``-smp`` ("parameter=0" SMP configurations) (since 6.2) +''''''''''''''''''''''''''''''''''''''''''''''''''''''' + +Specified CPU topology parameters must be greater than zero. + +In the SMP configuration, users should either provide a CPU topology +parameter with a reasonable value (greater than zero) or just omit it +and QEMU will compute the missing value. + +However, historically it was implicitly allowed for users to provide +a parameter with zero value, which is meaningless and could also possibly +cause unexpected results in the -smp parsing. So support for this kind of +configurations (e.g. -smp 8,sockets=0) is deprecated since 6.2 and will +be removed in the near future, users have to ensure that all the topology +members described with -smp are greater than zero. Plugin argument passing through ``arg=`` (since 6.1) '''''''''''''''''''''''''''''''''''''''''''''''''''''''''''' -- cgit v1.1 From 70367f091777419f42e5f68f4206deb641335877 Mon Sep 17 00:00:00 2001 From: Vitaly Kuznetsov Date: Thu, 2 Sep 2021 11:35:26 +0200 Subject: i386: Support KVM_CAP_HYPERV_ENFORCE_CPUID By default, KVM allows the guest to use all currently supported Hyper-V enlightenments when Hyper-V CPUID interface was exposed, regardless of if some features were not announced in guest visible CPUIDs. hv-enforce-cpuid feature alters this behavior and only allows the guest to use exposed Hyper-V enlightenments. The feature is supported by Linux >= 5.14 and is not enabled by default in QEMU. Signed-off-by: Vitaly Kuznetsov Message-Id: <20210902093530.345756-5-vkuznets@redhat.com> Signed-off-by: Paolo Bonzini --- docs/hyperv.txt | 17 ++++++++++++++--- 1 file changed, 14 insertions(+), 3 deletions(-) (limited to 'docs') diff --git a/docs/hyperv.txt b/docs/hyperv.txt index 000638a..072709a 100644 --- a/docs/hyperv.txt +++ b/docs/hyperv.txt @@ -203,8 +203,11 @@ When the option is set to 'on' QEMU will always enable the feature, regardless of host setup. To keep guests secure, this can only be used in conjunction with exposing correct vCPU topology and vCPU pinning. -4. Development features -======================== +4. Supplementary features +========================= + +4.1. hv-passthrough +=================== In some cases (e.g. during development) it may make sense to use QEMU in 'pass-through' mode and give Windows guests all enlightenments currently supported by KVM. This pass-through mode is enabled by "hv-passthrough" CPU @@ -215,8 +218,16 @@ values from KVM to QEMU. "hv-passthrough" overrides all other "hv-*" settings on the command line. Also, enabling this flag effectively prevents migration as the list of enabled enlightenments may differ between target and destination hosts. +4.2. hv-enforce-cpuid +===================== +By default, KVM allows the guest to use all currently supported Hyper-V +enlightenments when Hyper-V CPUID interface was exposed, regardless of if +some features were not announced in guest visible CPUIDs. 'hv-enforce-cpuid' +feature alters this behavior and only allows the guest to use exposed Hyper-V +enlightenments. + -4. Useful links +5. Useful links ================ Hyper-V Top Level Functional specification and other information: https://github.com/MicrosoftDocs/Virtualization-Documentation -- cgit v1.1 From e1f9a8e8c90ae54387922e33e5ac4fd759747d01 Mon Sep 17 00:00:00 2001 From: Vitaly Kuznetsov Date: Thu, 2 Sep 2021 11:35:28 +0200 Subject: i386: Implement pseudo 'hv-avic' ('hv-apicv') enlightenment The enlightenment allows to use Hyper-V SynIC with hardware APICv/AVIC enabled. Normally, Hyper-V SynIC disables these hardware features and suggests the guest to use paravirtualized AutoEOI feature. Linux-4.15 gains support for conditional APICv/AVIC disablement, the feature stays on until the guest tries to use AutoEOI feature with SynIC. With 'HV_DEPRECATING_AEOI_RECOMMENDED' bit exposed, modern enough Windows/ Hyper-V versions should follow the recommendation and not use the (unwanted) feature. Signed-off-by: Vitaly Kuznetsov Message-Id: <20210902093530.345756-7-vkuznets@redhat.com> Signed-off-by: Paolo Bonzini --- docs/hyperv.txt | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) (limited to 'docs') diff --git a/docs/hyperv.txt b/docs/hyperv.txt index 072709a..cd1ea3b 100644 --- a/docs/hyperv.txt +++ b/docs/hyperv.txt @@ -189,7 +189,15 @@ enabled. Requires: hv-vpindex, hv-synic, hv-time, hv-stimer -3.17. hv-no-nonarch-coresharing=on/off/auto +3.18. hv-avic (hv-apicv) +======================= +The enlightenment allows to use Hyper-V SynIC with hardware APICv/AVIC enabled. +Normally, Hyper-V SynIC disables these hardware feature and suggests the guest +to use paravirtualized AutoEOI feature. +Note: enabling this feature on old hardware (without APICv/AVIC support) may +have negative effect on guest's performace. + +3.19. hv-no-nonarch-coresharing=on/off/auto =========================================== This enlightenment tells guest OS that virtual processors will never share a physical core unless they are reported as sibling SMT threads. This information -- cgit v1.1 From af7228b88dbe80ed5d5258b49be8b48ab351a476 Mon Sep 17 00:00:00 2001 From: Vitaly Kuznetsov Date: Thu, 2 Sep 2021 11:35:29 +0200 Subject: i386: Make Hyper-V version id configurable Currently, we hardcode Hyper-V version id (CPUID 0x40000002) to WS2008R2 and it is known that certain tools in Windows check this. It seems useful to provide some flexibility by making it possible to change this info at will. CPUID information is defined in TLFS as: EAX: Build Number EBX Bits 31-16: Major Version Bits 15-0: Minor Version ECX Service Pack EDX Bits 31-24: Service Branch Bits 23-0: Service Number Signed-off-by: Vitaly Kuznetsov Message-Id: <20210902093530.345756-8-vkuznets@redhat.com> Signed-off-by: Paolo Bonzini --- docs/hyperv.txt | 14 ++++++++++++++ 1 file changed, 14 insertions(+) (limited to 'docs') diff --git a/docs/hyperv.txt b/docs/hyperv.txt index cd1ea3b..7803495 100644 --- a/docs/hyperv.txt +++ b/docs/hyperv.txt @@ -211,6 +211,20 @@ When the option is set to 'on' QEMU will always enable the feature, regardless of host setup. To keep guests secure, this can only be used in conjunction with exposing correct vCPU topology and vCPU pinning. +3.20. hv-version-id-{build,major,minor,spack,sbranch,snumber} +============================================================= +This changes Hyper-V version identification in CPUID 0x40000002.EAX-EDX from the +default (WS2008R2). +- hv-version-id-build sets 'Build Number' (32 bits) +- hv-version-id-major sets 'Major Version' (16 bits) +- hv-version-id-minor sets 'Minor Version' (16 bits) +- hv-version-id-spack sets 'Service Pack' (32 bits) +- hv-version-id-sbranch sets 'Service Branch' (8 bits) +- hv-version-id-snumber sets 'Service Number' (24 bits) + +Note: hv-version-id-* are not enlightenments and thus don't enable Hyper-V +identification when specified without any other enlightenments. + 4. Supplementary features ========================= -- cgit v1.1 From f701ecec2bbaae2d04985eba87924a7329534e9a Mon Sep 17 00:00:00 2001 From: Vitaly Kuznetsov Date: Thu, 2 Sep 2021 11:35:30 +0200 Subject: i386: Change the default Hyper-V version to match WS2016 KVM implements some Hyper-V 2016 functions so providing WS2008R2 version is somewhat incorrect. While generally guests shouldn't care about it and always check feature bits, it is known that some tools in Windows actually check version info. For compatibility reasons make the change for 6.2 machine types only. Signed-off-by: Vitaly Kuznetsov Message-Id: <20210902093530.345756-9-vkuznets@redhat.com> Signed-off-by: Paolo Bonzini --- docs/hyperv.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) (limited to 'docs') diff --git a/docs/hyperv.txt b/docs/hyperv.txt index 7803495..5d99fd9 100644 --- a/docs/hyperv.txt +++ b/docs/hyperv.txt @@ -214,7 +214,7 @@ exposing correct vCPU topology and vCPU pinning. 3.20. hv-version-id-{build,major,minor,spack,sbranch,snumber} ============================================================= This changes Hyper-V version identification in CPUID 0x40000002.EAX-EDX from the -default (WS2008R2). +default (WS2016). - hv-version-id-build sets 'Build Number' (32 bits) - hv-version-id-major sets 'Major Version' (16 bits) - hv-version-id-minor sets 'Minor Version' (16 bits) -- cgit v1.1 From bcfdfae78f111fa3c0f81b2708098a545201bb68 Mon Sep 17 00:00:00 2001 From: Paolo Bonzini Date: Tue, 7 Sep 2021 16:27:08 +0200 Subject: docs: name included files ".rst.inc" Signed-off-by: Paolo Bonzini --- docs/devel/ci-definitions.rst | 121 -------------------------------------- docs/devel/ci-definitions.rst.inc | 121 ++++++++++++++++++++++++++++++++++++++ docs/devel/ci-jobs.rst | 51 ---------------- docs/devel/ci-jobs.rst.inc | 51 ++++++++++++++++ docs/devel/ci-runners.rst | 117 ------------------------------------ docs/devel/ci-runners.rst.inc | 117 ++++++++++++++++++++++++++++++++++++ docs/devel/ci.rst | 6 +- 7 files changed, 292 insertions(+), 292 deletions(-) delete mode 100644 docs/devel/ci-definitions.rst create mode 100644 docs/devel/ci-definitions.rst.inc delete mode 100644 docs/devel/ci-jobs.rst create mode 100644 docs/devel/ci-jobs.rst.inc delete mode 100644 docs/devel/ci-runners.rst create mode 100644 docs/devel/ci-runners.rst.inc (limited to 'docs') diff --git a/docs/devel/ci-definitions.rst b/docs/devel/ci-definitions.rst deleted file mode 100644 index 32e22ff..0000000 --- a/docs/devel/ci-definitions.rst +++ /dev/null @@ -1,121 +0,0 @@ -Definition of terms -=================== - -This section defines the terms used in this document and correlates them with -what is currently used on QEMU. - -Automated tests ---------------- - -An automated test is written on a test framework using its generic test -functions/classes. The test framework can run the tests and report their -success or failure [1]_. - -An automated test has essentially three parts: - -1. The test initialization of the parameters, where the expected parameters, - like inputs and expected results, are set up; -2. The call to the code that should be tested; -3. An assertion, comparing the result from the previous call with the expected - result set during the initialization of the parameters. If the result - matches the expected result, the test has been successful; otherwise, it has - failed. - -Unit testing ------------- - -A unit test is responsible for exercising individual software components as a -unit, like interfaces, data structures, and functionality, uncovering errors -within the boundaries of a component. The verification effort is in the -smallest software unit and focuses on the internal processing logic and data -structures. A test case of unit tests should be designed to uncover errors due -to erroneous computations, incorrect comparisons, or improper control flow [2]_. - -On QEMU, unit testing is represented by the 'check-unit' target from 'make'. - -Functional testing ------------------- - -A functional test focuses on the functional requirement of the software. -Deriving sets of input conditions, the functional tests should fully exercise -all the functional requirements for a program. Functional testing is -complementary to other testing techniques, attempting to find errors like -incorrect or missing functions, interface errors, behavior errors, and -initialization and termination errors [3]_. - -On QEMU, functional testing is represented by the 'check-qtest' target from -'make'. - -System testing --------------- - -System tests ensure all application elements mesh properly while the overall -functionality and performance are achieved [4]_. Some or all system components -are integrated to create a complete system to be tested as a whole. System -testing ensures that components are compatible, interact correctly, and -transfer the right data at the right time across their interfaces. As system -testing focuses on interactions, use case-based testing is a practical approach -to system testing [5]_. Note that, in some cases, system testing may require -interaction with third-party software, like operating system images, databases, -networks, and so on. - -On QEMU, system testing is represented by the 'check-acceptance' target from -'make'. - -Flaky tests ------------ - -A flaky test is defined as a test that exhibits both a passing and a failing -result with the same code on different runs. Some usual reasons for an -intermittent/flaky test are async wait, concurrency, and test order dependency -[6]_. - -Gating ------- - -A gate restricts the move of code from one stage to another on a -test/deployment pipeline. The step move is granted with approval. The approval -can be a manual intervention or a set of tests succeeding [7]_. - -On QEMU, the gating process happens during the pull request. The approval is -done by the project leader running its own set of tests. The pull request gets -merged when the tests succeed. - -Continuous Integration (CI) ---------------------------- - -Continuous integration (CI) requires the builds of the entire application and -the execution of a comprehensive set of automated tests every time there is a -need to commit any set of changes [8]_. The automated tests can be composed of -the unit, functional, system, and other tests. - -Keynotes about continuous integration (CI) [9]_: - -1. System tests may depend on external software (operating system images, - firmware, database, network). -2. It may take a long time to build and test. It may be impractical to build - the system being developed several times per day. -3. If the development platform is different from the target platform, it may - not be possible to run system tests in the developer’s private workspace. - There may be differences in hardware, operating system, or installed - software. Therefore, more time is required for testing the system. - -References ----------- - -.. [1] Sommerville, Ian (2016). Software Engineering. p. 233. -.. [2] Pressman, Roger S. & Maxim, Bruce R. (2020). Software Engineering, - A Practitioner’s Approach. p. 48, 376, 378, 381. -.. [3] Pressman, Roger S. & Maxim, Bruce R. (2020). Software Engineering, - A Practitioner’s Approach. p. 388. -.. [4] Pressman, Roger S. & Maxim, Bruce R. (2020). Software Engineering, - A Practitioner’s Approach. Software Engineering, p. 377. -.. [5] Sommerville, Ian (2016). Software Engineering. p. 59, 232, 240. -.. [6] Luo, Qingzhou, et al. An empirical analysis of flaky tests. - Proceedings of the 22nd ACM SIGSOFT International Symposium on - Foundations of Software Engineering. 2014. -.. [7] Humble, Jez & Farley, David (2010). Continuous Delivery: - Reliable Software Releases Through Build, Test, and Deployment, p. 122. -.. [8] Humble, Jez & Farley, David (2010). Continuous Delivery: - Reliable Software Releases Through Build, Test, and Deployment, p. 55. -.. [9] Sommerville, Ian (2016). Software Engineering. p. 743. diff --git a/docs/devel/ci-definitions.rst.inc b/docs/devel/ci-definitions.rst.inc new file mode 100644 index 0000000..32e22ff --- /dev/null +++ b/docs/devel/ci-definitions.rst.inc @@ -0,0 +1,121 @@ +Definition of terms +=================== + +This section defines the terms used in this document and correlates them with +what is currently used on QEMU. + +Automated tests +--------------- + +An automated test is written on a test framework using its generic test +functions/classes. The test framework can run the tests and report their +success or failure [1]_. + +An automated test has essentially three parts: + +1. The test initialization of the parameters, where the expected parameters, + like inputs and expected results, are set up; +2. The call to the code that should be tested; +3. An assertion, comparing the result from the previous call with the expected + result set during the initialization of the parameters. If the result + matches the expected result, the test has been successful; otherwise, it has + failed. + +Unit testing +------------ + +A unit test is responsible for exercising individual software components as a +unit, like interfaces, data structures, and functionality, uncovering errors +within the boundaries of a component. The verification effort is in the +smallest software unit and focuses on the internal processing logic and data +structures. A test case of unit tests should be designed to uncover errors due +to erroneous computations, incorrect comparisons, or improper control flow [2]_. + +On QEMU, unit testing is represented by the 'check-unit' target from 'make'. + +Functional testing +------------------ + +A functional test focuses on the functional requirement of the software. +Deriving sets of input conditions, the functional tests should fully exercise +all the functional requirements for a program. Functional testing is +complementary to other testing techniques, attempting to find errors like +incorrect or missing functions, interface errors, behavior errors, and +initialization and termination errors [3]_. + +On QEMU, functional testing is represented by the 'check-qtest' target from +'make'. + +System testing +-------------- + +System tests ensure all application elements mesh properly while the overall +functionality and performance are achieved [4]_. Some or all system components +are integrated to create a complete system to be tested as a whole. System +testing ensures that components are compatible, interact correctly, and +transfer the right data at the right time across their interfaces. As system +testing focuses on interactions, use case-based testing is a practical approach +to system testing [5]_. Note that, in some cases, system testing may require +interaction with third-party software, like operating system images, databases, +networks, and so on. + +On QEMU, system testing is represented by the 'check-acceptance' target from +'make'. + +Flaky tests +----------- + +A flaky test is defined as a test that exhibits both a passing and a failing +result with the same code on different runs. Some usual reasons for an +intermittent/flaky test are async wait, concurrency, and test order dependency +[6]_. + +Gating +------ + +A gate restricts the move of code from one stage to another on a +test/deployment pipeline. The step move is granted with approval. The approval +can be a manual intervention or a set of tests succeeding [7]_. + +On QEMU, the gating process happens during the pull request. The approval is +done by the project leader running its own set of tests. The pull request gets +merged when the tests succeed. + +Continuous Integration (CI) +--------------------------- + +Continuous integration (CI) requires the builds of the entire application and +the execution of a comprehensive set of automated tests every time there is a +need to commit any set of changes [8]_. The automated tests can be composed of +the unit, functional, system, and other tests. + +Keynotes about continuous integration (CI) [9]_: + +1. System tests may depend on external software (operating system images, + firmware, database, network). +2. It may take a long time to build and test. It may be impractical to build + the system being developed several times per day. +3. If the development platform is different from the target platform, it may + not be possible to run system tests in the developer’s private workspace. + There may be differences in hardware, operating system, or installed + software. Therefore, more time is required for testing the system. + +References +---------- + +.. [1] Sommerville, Ian (2016). Software Engineering. p. 233. +.. [2] Pressman, Roger S. & Maxim, Bruce R. (2020). Software Engineering, + A Practitioner’s Approach. p. 48, 376, 378, 381. +.. [3] Pressman, Roger S. & Maxim, Bruce R. (2020). Software Engineering, + A Practitioner’s Approach. p. 388. +.. [4] Pressman, Roger S. & Maxim, Bruce R. (2020). Software Engineering, + A Practitioner’s Approach. Software Engineering, p. 377. +.. [5] Sommerville, Ian (2016). Software Engineering. p. 59, 232, 240. +.. [6] Luo, Qingzhou, et al. An empirical analysis of flaky tests. + Proceedings of the 22nd ACM SIGSOFT International Symposium on + Foundations of Software Engineering. 2014. +.. [7] Humble, Jez & Farley, David (2010). Continuous Delivery: + Reliable Software Releases Through Build, Test, and Deployment, p. 122. +.. [8] Humble, Jez & Farley, David (2010). Continuous Delivery: + Reliable Software Releases Through Build, Test, and Deployment, p. 55. +.. [9] Sommerville, Ian (2016). Software Engineering. p. 743. diff --git a/docs/devel/ci-jobs.rst b/docs/devel/ci-jobs.rst deleted file mode 100644 index 277975e..0000000 --- a/docs/devel/ci-jobs.rst +++ /dev/null @@ -1,51 +0,0 @@ -Custom CI/CD variables -====================== - -QEMU CI pipelines can be tuned by setting some CI environment variables. - -Set variable globally in the user's CI namespace ------------------------------------------------- - -Variables can be set globally in the user's CI namespace setting. - -For further information about how to set these variables, please refer to:: - - https://docs.gitlab.com/ee/ci/variables/#add-a-cicd-variable-to-a-project - -Set variable manually when pushing a branch or tag to the user's repository ---------------------------------------------------------------------------- - -Variables can be set manually when pushing a branch or tag, using -git-push command line arguments. - -Example setting the QEMU_CI_EXAMPLE_VAR variable: - -.. code:: - - git push -o ci.variable="QEMU_CI_EXAMPLE_VAR=value" myrepo mybranch - -For further information about how to set these variables, please refer to:: - - https://docs.gitlab.com/ee/user/project/push_options.html#push-options-for-gitlab-cicd - -Here is a list of the most used variables: - -QEMU_CI_AVOCADO_TESTING -~~~~~~~~~~~~~~~~~~~~~~~ -By default, tests using the Avocado framework are not run automatically in -the pipelines (because multiple artifacts have to be downloaded, and if -these artifacts are not already cached, downloading them make the jobs -reach the timeout limit). Set this variable to have the tests using the -Avocado framework run automatically. - -AARCH64_RUNNER_AVAILABLE -~~~~~~~~~~~~~~~~~~~~~~~~ -If you've got access to an aarch64 host that can be used as a gitlab-CI -runner, you can set this variable to enable the tests that require this -kind of host. The runner should be tagged with "aarch64". - -S390X_RUNNER_AVAILABLE -~~~~~~~~~~~~~~~~~~~~~~ -If you've got access to an IBM Z host that can be used as a gitlab-CI -runner, you can set this variable to enable the tests that require this -kind of host. The runner should be tagged with "s390x". diff --git a/docs/devel/ci-jobs.rst.inc b/docs/devel/ci-jobs.rst.inc new file mode 100644 index 0000000..277975e --- /dev/null +++ b/docs/devel/ci-jobs.rst.inc @@ -0,0 +1,51 @@ +Custom CI/CD variables +====================== + +QEMU CI pipelines can be tuned by setting some CI environment variables. + +Set variable globally in the user's CI namespace +------------------------------------------------ + +Variables can be set globally in the user's CI namespace setting. + +For further information about how to set these variables, please refer to:: + + https://docs.gitlab.com/ee/ci/variables/#add-a-cicd-variable-to-a-project + +Set variable manually when pushing a branch or tag to the user's repository +--------------------------------------------------------------------------- + +Variables can be set manually when pushing a branch or tag, using +git-push command line arguments. + +Example setting the QEMU_CI_EXAMPLE_VAR variable: + +.. code:: + + git push -o ci.variable="QEMU_CI_EXAMPLE_VAR=value" myrepo mybranch + +For further information about how to set these variables, please refer to:: + + https://docs.gitlab.com/ee/user/project/push_options.html#push-options-for-gitlab-cicd + +Here is a list of the most used variables: + +QEMU_CI_AVOCADO_TESTING +~~~~~~~~~~~~~~~~~~~~~~~ +By default, tests using the Avocado framework are not run automatically in +the pipelines (because multiple artifacts have to be downloaded, and if +these artifacts are not already cached, downloading them make the jobs +reach the timeout limit). Set this variable to have the tests using the +Avocado framework run automatically. + +AARCH64_RUNNER_AVAILABLE +~~~~~~~~~~~~~~~~~~~~~~~~ +If you've got access to an aarch64 host that can be used as a gitlab-CI +runner, you can set this variable to enable the tests that require this +kind of host. The runner should be tagged with "aarch64". + +S390X_RUNNER_AVAILABLE +~~~~~~~~~~~~~~~~~~~~~~ +If you've got access to an IBM Z host that can be used as a gitlab-CI +runner, you can set this variable to enable the tests that require this +kind of host. The runner should be tagged with "s390x". diff --git a/docs/devel/ci-runners.rst b/docs/devel/ci-runners.rst deleted file mode 100644 index 7817001..0000000 --- a/docs/devel/ci-runners.rst +++ /dev/null @@ -1,117 +0,0 @@ -Jobs on Custom Runners -====================== - -Besides the jobs run under the various CI systems listed before, there -are a number additional jobs that will run before an actual merge. -These use the same GitLab CI's service/framework already used for all -other GitLab based CI jobs, but rely on additional systems, not the -ones provided by GitLab as "shared runners". - -The architecture of GitLab's CI service allows different machines to -be set up with GitLab's "agent", called gitlab-runner, which will take -care of running jobs created by events such as a push to a branch. -Here, the combination of a machine, properly configured with GitLab's -gitlab-runner, is called a "custom runner". - -The GitLab CI jobs definition for the custom runners are located under:: - - .gitlab-ci.d/custom-runners.yml - -Custom runners entail custom machines. To see a list of the machines -currently deployed in the QEMU GitLab CI and their maintainers, please -refer to the QEMU `wiki `__. - -Machine Setup Howto -------------------- - -For all Linux based systems, the setup can be mostly automated by the -execution of two Ansible playbooks. Create an ``inventory`` file -under ``scripts/ci/setup``, such as this:: - - fully.qualified.domain - other.machine.hostname - -You may need to set some variables in the inventory file itself. One -very common need is to tell Ansible to use a Python 3 interpreter on -those hosts. This would look like:: - - fully.qualified.domain ansible_python_interpreter=/usr/bin/python3 - other.machine.hostname ansible_python_interpreter=/usr/bin/python3 - -Build environment -~~~~~~~~~~~~~~~~~ - -The ``scripts/ci/setup/build-environment.yml`` Ansible playbook will -set up machines with the environment needed to perform builds and run -QEMU tests. This playbook consists on the installation of various -required packages (and a general package update while at it). It -currently covers a number of different Linux distributions, but it can -be expanded to cover other systems. - -The minimum required version of Ansible successfully tested in this -playbook is 2.8.0 (a version check is embedded within the playbook -itself). To run the playbook, execute:: - - cd scripts/ci/setup - ansible-playbook -i inventory build-environment.yml - -Please note that most of the tasks in the playbook require superuser -privileges, such as those from the ``root`` account or those obtained -by ``sudo``. If necessary, please refer to ``ansible-playbook`` -options such as ``--become``, ``--become-method``, ``--become-user`` -and ``--ask-become-pass``. - -gitlab-runner setup and registration -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -The gitlab-runner agent needs to be installed on each machine that -will run jobs. The association between a machine and a GitLab project -happens with a registration token. To find the registration token for -your repository/project, navigate on GitLab's web UI to: - - * Settings (the gears-like icon at the bottom of the left hand side - vertical toolbar), then - * CI/CD, then - * Runners, and click on the "Expand" button, then - * Under "Set up a specific Runner manually", look for the value under - "And this registration token:" - -Copy the ``scripts/ci/setup/vars.yml.template`` file to -``scripts/ci/setup/vars.yml``. Then, set the -``gitlab_runner_registration_token`` variable to the value obtained -earlier. - -To run the playbook, execute:: - - cd scripts/ci/setup - ansible-playbook -i inventory gitlab-runner.yml - -Following the registration, it's necessary to configure the runner tags, -and optionally other configurations on the GitLab UI. Navigate to: - - * Settings (the gears like icon), then - * CI/CD, then - * Runners, and click on the "Expand" button, then - * "Runners activated for this project", then - * Click on the "Edit" icon (next to the "Lock" Icon) - -Tags are very important as they are used to route specific jobs to -specific types of runners, so it's a good idea to double check that -the automatically created tags are consistent with the OS and -architecture. For instance, an Ubuntu 20.04 aarch64 system should -have tags set as:: - - ubuntu_20.04,aarch64 - -Because the job definition at ``.gitlab-ci.d/custom-runners.yml`` -would contain:: - - ubuntu-20.04-aarch64-all: - tags: - - ubuntu_20.04 - - aarch64 - -It's also recommended to: - - * increase the "Maximum job timeout" to something like ``2h`` - * give it a better Description diff --git a/docs/devel/ci-runners.rst.inc b/docs/devel/ci-runners.rst.inc new file mode 100644 index 0000000..7817001 --- /dev/null +++ b/docs/devel/ci-runners.rst.inc @@ -0,0 +1,117 @@ +Jobs on Custom Runners +====================== + +Besides the jobs run under the various CI systems listed before, there +are a number additional jobs that will run before an actual merge. +These use the same GitLab CI's service/framework already used for all +other GitLab based CI jobs, but rely on additional systems, not the +ones provided by GitLab as "shared runners". + +The architecture of GitLab's CI service allows different machines to +be set up with GitLab's "agent", called gitlab-runner, which will take +care of running jobs created by events such as a push to a branch. +Here, the combination of a machine, properly configured with GitLab's +gitlab-runner, is called a "custom runner". + +The GitLab CI jobs definition for the custom runners are located under:: + + .gitlab-ci.d/custom-runners.yml + +Custom runners entail custom machines. To see a list of the machines +currently deployed in the QEMU GitLab CI and their maintainers, please +refer to the QEMU `wiki `__. + +Machine Setup Howto +------------------- + +For all Linux based systems, the setup can be mostly automated by the +execution of two Ansible playbooks. Create an ``inventory`` file +under ``scripts/ci/setup``, such as this:: + + fully.qualified.domain + other.machine.hostname + +You may need to set some variables in the inventory file itself. One +very common need is to tell Ansible to use a Python 3 interpreter on +those hosts. This would look like:: + + fully.qualified.domain ansible_python_interpreter=/usr/bin/python3 + other.machine.hostname ansible_python_interpreter=/usr/bin/python3 + +Build environment +~~~~~~~~~~~~~~~~~ + +The ``scripts/ci/setup/build-environment.yml`` Ansible playbook will +set up machines with the environment needed to perform builds and run +QEMU tests. This playbook consists on the installation of various +required packages (and a general package update while at it). It +currently covers a number of different Linux distributions, but it can +be expanded to cover other systems. + +The minimum required version of Ansible successfully tested in this +playbook is 2.8.0 (a version check is embedded within the playbook +itself). To run the playbook, execute:: + + cd scripts/ci/setup + ansible-playbook -i inventory build-environment.yml + +Please note that most of the tasks in the playbook require superuser +privileges, such as those from the ``root`` account or those obtained +by ``sudo``. If necessary, please refer to ``ansible-playbook`` +options such as ``--become``, ``--become-method``, ``--become-user`` +and ``--ask-become-pass``. + +gitlab-runner setup and registration +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The gitlab-runner agent needs to be installed on each machine that +will run jobs. The association between a machine and a GitLab project +happens with a registration token. To find the registration token for +your repository/project, navigate on GitLab's web UI to: + + * Settings (the gears-like icon at the bottom of the left hand side + vertical toolbar), then + * CI/CD, then + * Runners, and click on the "Expand" button, then + * Under "Set up a specific Runner manually", look for the value under + "And this registration token:" + +Copy the ``scripts/ci/setup/vars.yml.template`` file to +``scripts/ci/setup/vars.yml``. Then, set the +``gitlab_runner_registration_token`` variable to the value obtained +earlier. + +To run the playbook, execute:: + + cd scripts/ci/setup + ansible-playbook -i inventory gitlab-runner.yml + +Following the registration, it's necessary to configure the runner tags, +and optionally other configurations on the GitLab UI. Navigate to: + + * Settings (the gears like icon), then + * CI/CD, then + * Runners, and click on the "Expand" button, then + * "Runners activated for this project", then + * Click on the "Edit" icon (next to the "Lock" Icon) + +Tags are very important as they are used to route specific jobs to +specific types of runners, so it's a good idea to double check that +the automatically created tags are consistent with the OS and +architecture. For instance, an Ubuntu 20.04 aarch64 system should +have tags set as:: + + ubuntu_20.04,aarch64 + +Because the job definition at ``.gitlab-ci.d/custom-runners.yml`` +would contain:: + + ubuntu-20.04-aarch64-all: + tags: + - ubuntu_20.04 + - aarch64 + +It's also recommended to: + + * increase the "Maximum job timeout" to something like ``2h`` + * give it a better Description diff --git a/docs/devel/ci.rst b/docs/devel/ci.rst index 8d95247..d106610 100644 --- a/docs/devel/ci.rst +++ b/docs/devel/ci.rst @@ -8,6 +8,6 @@ found at:: https://wiki.qemu.org/Testing/CI -.. include:: ci-definitions.rst -.. include:: ci-jobs.rst -.. include:: ci-runners.rst +.. include:: ci-definitions.rst.inc +.. include:: ci-jobs.rst.inc +.. include:: ci-runners.rst.inc -- cgit v1.1 From f9df7aac758fed5cc2fb9210ad0edba79434aeed Mon Sep 17 00:00:00 2001 From: Paolo Bonzini Date: Tue, 7 Sep 2021 16:42:18 +0200 Subject: docs: move notes inside the body of the document Make all documents start with a heading. Signed-off-by: Paolo Bonzini --- docs/devel/multi-process.rst | 20 +++++++++++--------- 1 file changed, 11 insertions(+), 9 deletions(-) (limited to 'docs') diff --git a/docs/devel/multi-process.rst b/docs/devel/multi-process.rst index 6969932..e5758a7 100644 --- a/docs/devel/multi-process.rst +++ b/docs/devel/multi-process.rst @@ -1,15 +1,17 @@ -This is the design document for multi-process QEMU. It does not -necessarily reflect the status of the current implementation, which -may lack features or be considerably different from what is described -in this document. This document is still useful as a description of -the goals and general direction of this feature. - -Please refer to the following wiki for latest details: -https://wiki.qemu.org/Features/MultiProcessQEMU - Multi-process QEMU =================== +.. note:: + + This is the design document for multi-process QEMU. It does not + necessarily reflect the status of the current implementation, which + may lack features or be considerably different from what is described + in this document. This document is still useful as a description of + the goals and general direction of this feature. + + Please refer to the following wiki for latest details: + https://wiki.qemu.org/Features/MultiProcessQEMU + QEMU is often used as the hypervisor for virtual machines running in the Oracle cloud. Since one of the advantages of cloud computing is the ability to run many VMs from different tenants in the same cloud -- cgit v1.1 From 8b8939e44fc8315885598a9e1ee8deeea3b68a96 Mon Sep 17 00:00:00 2001 From: Paolo Bonzini Date: Tue, 7 Sep 2021 16:25:50 +0200 Subject: docs: put "make" information together in build-system.rst Signed-off-by: Paolo Bonzini --- docs/devel/build-system.rst | 21 ++++++++++----------- 1 file changed, 10 insertions(+), 11 deletions(-) (limited to 'docs') diff --git a/docs/devel/build-system.rst b/docs/devel/build-system.rst index 3baec15..0f636d6 100644 --- a/docs/devel/build-system.rst +++ b/docs/devel/build-system.rst @@ -380,6 +380,16 @@ phony target, while benchmarks are run with ``make bench``. Meson test suites such as ``unit`` can be ran with ``make check-unit`` too. It is also possible to run tests defined in meson.build with ``meson test``. +Useful make targets +------------------- + +``help`` + Print a help message for the most common build targets. + +``print-VAR`` + Print the value of the variable VAR. Useful for debugging the build + system. + Important files for the build system ==================================== @@ -473,14 +483,3 @@ Built by Makefile: meson.build. The rules are produced from Meson's JSON description of tests (obtained with "meson introspect --tests") through the script scripts/mtest2make.py. - - -Useful make targets -------------------- - -``help`` - Print a help message for the most common build targets. - -``print-VAR`` - Print the value of the variable VAR. Useful for debugging the build - system. -- cgit v1.1 From 768f14f94ec50fe57a3964fff75d8b3456b588b5 Mon Sep 17 00:00:00 2001 From: Paolo Bonzini Date: Tue, 7 Sep 2021 16:53:01 +0200 Subject: docs: reorganize qgraph.rst Clean up the heading levels to use === --- ~~~, and move the command line building near to the other execution steps. Signed-off-by: Paolo Bonzini --- docs/devel/qgraph.rst | 132 +++++++++++++++++++++++++------------------------- 1 file changed, 67 insertions(+), 65 deletions(-) (limited to 'docs') diff --git a/docs/devel/qgraph.rst b/docs/devel/qgraph.rst index c2882c3..db44d71 100644 --- a/docs/devel/qgraph.rst +++ b/docs/devel/qgraph.rst @@ -1,8 +1,7 @@ .. _qgraph: -======================================== Qtest Driver Framework -======================================== +====================== In order to test a specific driver, plain libqos tests need to take care of booting QEMU with the right machine and devices. @@ -31,13 +30,15 @@ so the sdhci-test should only care of linking its qgraph node with that interface. In this way, if the command line of a sdhci driver is changed, only the respective qgraph driver node has to be adjusted. +QGraph concepts +--------------- + The graph is composed by nodes that represent machines, drivers, tests and edges that define the relationships between them (``CONSUMES``, ``PRODUCES``, and ``CONTAINS``). - Nodes -^^^^^^ +~~~~~ A node can be of four types: @@ -64,7 +65,7 @@ Notes for the nodes: drivers name, otherwise they won't be discovered Edges -^^^^^^ +~~~~~ An edge relation between two nodes (drivers or machines) ``X`` and ``Y`` can be: @@ -73,7 +74,7 @@ An edge relation between two nodes (drivers or machines) ``X`` and ``Y`` can be: - ``X CONTAINS Y``: ``Y`` is part of ``X`` component Execution steps -^^^^^^^^^^^^^^^ +~~~~~~~~~~~~~~~ The basic framework steps are the following: @@ -92,8 +93,64 @@ The basic framework steps are the following: Depending on the QEMU binary used, only some drivers/machines will be available and only test that are reached by them will be executed. +Command line +~~~~~~~~~~~~ + +Command line is built by using node names and optional arguments +passed by the user when building the edges. + +There are three types of command line arguments: + +- ``in node`` : created from the node name. For example, machines will + have ``-M `` to its command line, while devices + ``-device ``. It is automatically done by the framework. +- ``after node`` : added as additional argument to the node name. + This argument is added optionally when creating edges, + by setting the parameter ``after_cmd_line`` and + ``extra_edge_opts`` in ``QOSGraphEdgeOptions``. + The framework automatically adds + a comma before ``extra_edge_opts``, + because it is going to add attributes + after the destination node pointed by + the edge containing these options, and automatically + adds a space before ``after_cmd_line``, because it + adds an additional device, not an attribute. +- ``before node`` : added as additional argument to the node name. + This argument is added optionally when creating edges, + by setting the parameter ``before_cmd_line`` in + ``QOSGraphEdgeOptions``. This attribute + is going to add attributes before the destination node + pointed by the edge containing these options. It is + helpful to commands that are not node-representable, + such as ``-fdsev`` or ``-netdev``. + +While adding command line in edges is always used, not all nodes names are +used in every path walk: this is because the contained or produced ones +are already added by QEMU, so only nodes that "consumes" will be used to +build the command line. Also, nodes that will have ``{ "abstract" : true }`` +as QMP attribute will loose their command line, since they are not proper +devices to be added in QEMU. + +Example:: + + QOSGraphEdgeOptions opts = { + .before_cmd_line = "-drive id=drv0,if=none,file=null-co://," + "file.read-zeroes=on,format=raw", + .after_cmd_line = "-device scsi-hd,bus=vs0.0,drive=drv0", + + opts.extra_device_opts = "id=vs0"; + }; + + qos_node_create_driver("virtio-scsi-device", + virtio_scsi_device_create); + qos_node_consumes("virtio-scsi-device", "virtio-bus", &opts); + +Will produce the following command line: +``-drive id=drv0,if=none,file=null-co://, -device virtio-scsi-device,id=vs0 -device scsi-hd,bus=vs0.0,drive=drv0`` + Troubleshooting unavailable tests -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + If there is no path from an available machine to a test then that test will be unavailable and won't execute. This can happen if a test or driver did not set up its qgraph node correctly. It can also happen if the necessary machine type @@ -151,7 +208,7 @@ Typically this is because the QEMU binary lacks support for the necessary machine type or device. Creating a new driver and its interface -""""""""""""""""""""""""""""""""""""""""" +--------------------------------------- Here we continue the ``sdhci`` use case, with the following scenario: @@ -489,7 +546,7 @@ or inverting the consumes edge in consumed_by:: arm/raspi2b --contains--> generic-sdhci Adding a new test -""""""""""""""""" +----------------- Given the above setup, adding a new test is very simple. ``sdhci-test``, taken from ``tests/qtest/sdhci-test.c``:: @@ -565,62 +622,7 @@ and for the binary ``QTEST_QEMU_BINARY=./qemu-system-arm``: Additional examples are also in ``test-qgraph.c`` -Command line: -"""""""""""""" - -Command line is built by using node names and optional arguments -passed by the user when building the edges. - -There are three types of command line arguments: - -- ``in node`` : created from the node name. For example, machines will - have ``-M `` to its command line, while devices - ``-device ``. It is automatically done by the framework. -- ``after node`` : added as additional argument to the node name. - This argument is added optionally when creating edges, - by setting the parameter ``after_cmd_line`` and - ``extra_edge_opts`` in ``QOSGraphEdgeOptions``. - The framework automatically adds - a comma before ``extra_edge_opts``, - because it is going to add attributes - after the destination node pointed by - the edge containing these options, and automatically - adds a space before ``after_cmd_line``, because it - adds an additional device, not an attribute. -- ``before node`` : added as additional argument to the node name. - This argument is added optionally when creating edges, - by setting the parameter ``before_cmd_line`` in - ``QOSGraphEdgeOptions``. This attribute - is going to add attributes before the destination node - pointed by the edge containing these options. It is - helpful to commands that are not node-representable, - such as ``-fdsev`` or ``-netdev``. - -While adding command line in edges is always used, not all nodes names are -used in every path walk: this is because the contained or produced ones -are already added by QEMU, so only nodes that "consumes" will be used to -build the command line. Also, nodes that will have ``{ "abstract" : true }`` -as QMP attribute will loose their command line, since they are not proper -devices to be added in QEMU. - -Example:: - - QOSGraphEdgeOptions opts = { - .before_cmd_line = "-drive id=drv0,if=none,file=null-co://," - "file.read-zeroes=on,format=raw", - .after_cmd_line = "-device scsi-hd,bus=vs0.0,drive=drv0", - - opts.extra_device_opts = "id=vs0"; - }; - - qos_node_create_driver("virtio-scsi-device", - virtio_scsi_device_create); - qos_node_consumes("virtio-scsi-device", "virtio-bus", &opts); - -Will produce the following command line: -``-drive id=drv0,if=none,file=null-co://, -device virtio-scsi-device,id=vs0 -device scsi-hd,bus=vs0.0,drive=drv0`` - Qgraph API reference -^^^^^^^^^^^^^^^^^^^^ +-------------------- .. kernel-doc:: tests/qtest/libqos/qgraph.h -- cgit v1.1 From e9adb4ace229dfa742176e9ddb629dbb6a6081bc Mon Sep 17 00:00:00 2001 From: Paolo Bonzini Date: Tue, 7 Sep 2021 17:06:07 +0200 Subject: docs: reorganize tcg-plugins.rst Clean up the heading levels to use === --- ~~~, and create a new "writing plugins" section. Signed-off-by: Paolo Bonzini --- docs/devel/tcg-plugins.rst | 93 +++++++++++++++++++++++----------------------- 1 file changed, 46 insertions(+), 47 deletions(-) (limited to 'docs') diff --git a/docs/devel/tcg-plugins.rst b/docs/devel/tcg-plugins.rst index dac5101..842ae01 100644 --- a/docs/devel/tcg-plugins.rst +++ b/docs/devel/tcg-plugins.rst @@ -3,7 +3,6 @@ Copyright (c) 2019, Linaro Limited Written by Emilio Cota and Alex Bennée -================ QEMU TCG Plugins ================ @@ -16,8 +15,30 @@ only monitor it passively. However they can do this down to an individual instruction granularity including potentially subscribing to all load and store operations. -API Stability -============= +Usage +----- + +Any QEMU binary with TCG support has plugins enabled by default. +Earlier releases needed to be explicitly enabled with:: + + configure --enable-plugins + +Once built a program can be run with multiple plugins loaded each with +their own arguments:: + + $QEMU $OTHER_QEMU_ARGS \ + -plugin tests/plugin/libhowvec.so,inline=on,count=hint \ + -plugin tests/plugin/libhotblocks.so + +Arguments are plugin specific and can be used to modify their +behaviour. In this case the howvec plugin is being asked to use inline +ops to count and break down the hint instructions by type. + +Writing plugins +--------------- + +API versioning +~~~~~~~~~~~~~~ This is a new feature for QEMU and it does allow people to develop out-of-tree plugins that can be dynamically linked into a running QEMU @@ -25,9 +46,6 @@ process. However the project reserves the right to change or break the API should it need to do so. The best way to avoid this is to submit your plugin upstream so they can be updated if/when the API changes. -API versioning --------------- - All plugins need to declare a symbol which exports the plugin API version they were built against. This can be done simply by:: @@ -43,18 +61,8 @@ current API versions supported by QEMU. The API version will be incremented if new APIs are added. The minimum API version will be incremented if existing APIs are changed or removed. -Exposure of QEMU internals --------------------------- - -The plugin architecture actively avoids leaking implementation details -about how QEMU's translation works to the plugins. While there are -conceptions such as translation time and translation blocks the -details are opaque to plugins. The plugin is able to query select -details of instructions and system configuration only through the -exported *qemu_plugin* functions. - -Query Handle Lifetime ---------------------- +Lifetime of the query handle +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Each callback provides an opaque anonymous information handle which can usually be further queried to find out information about a @@ -63,32 +71,8 @@ valid during the lifetime of the callback so it is important that any information that is needed is extracted during the callback and saved by the plugin. -API -=== - -.. kernel-doc:: include/qemu/qemu-plugin.h - -Usage -===== - -Any QEMU binary with TCG support has plugins enabled by default. -Earlier releases needed to be explicitly enabled with:: - - configure --enable-plugins - -Once built a program can be run with multiple plugins loaded each with -their own arguments:: - - $QEMU $OTHER_QEMU_ARGS \ - -plugin tests/plugin/libhowvec.so,inline=on,count=hint \ - -plugin tests/plugin/libhotblocks.so - -Arguments are plugin specific and can be used to modify their -behaviour. In this case the howvec plugin is being asked to use inline -ops to count and break down the hint instructions by type. - -Plugin Life cycle -================= +Plugin life cycle +~~~~~~~~~~~~~~~~~ First the plugin is loaded and the public qemu_plugin_install function is called. The plugin will then register callbacks for various plugin @@ -111,11 +95,26 @@ callback which can then ensure atomicity itself. Finally when QEMU exits all the registered *atexit* callbacks are invoked. +Exposure of QEMU internals +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The plugin architecture actively avoids leaking implementation details +about how QEMU's translation works to the plugins. While there are +conceptions such as translation time and translation blocks the +details are opaque to plugins. The plugin is able to query select +details of instructions and system configuration only through the +exported *qemu_plugin* functions. + +API +~~~ + +.. kernel-doc:: include/qemu/qemu-plugin.h + Internals -========= +--------- Locking -------- +~~~~~~~ We have to ensure we cannot deadlock, particularly under MTTCG. For this we acquire a lock when called from plugin code. We also keep the @@ -142,7 +141,7 @@ requested. The plugin isn't completely uninstalled until the safe work has executed while all vCPUs are quiescent. Example Plugins -=============== +--------------- There are a number of plugins included with QEMU and you are encouraged to contribute your own plugins plugins upstream. There is a -- cgit v1.1 From 9fce3601761779f14d8d1ea32e2b6abf2f704edb Mon Sep 17 00:00:00 2001 From: Paolo Bonzini Date: Wed, 22 Sep 2021 17:06:57 +0200 Subject: docs: move gcov section at the end of testing.rst gcov testing applies to all tests, not just make check. Move it out of the make check section. Signed-off-by: Paolo Bonzini --- docs/devel/testing.rst | 38 +++++++++++++++++++------------------- 1 file changed, 19 insertions(+), 19 deletions(-) (limited to 'docs') diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst index 64c9744..a80df07 100644 --- a/docs/devel/testing.rst +++ b/docs/devel/testing.rst @@ -114,25 +114,6 @@ check-block are in the "auto" group). See the "QEMU iotests" section below for more information. -GCC gcov support ----------------- - -``gcov`` is a GCC tool to analyze the testing coverage by -instrumenting the tested code. To use it, configure QEMU with -``--enable-gcov`` option and build. Then run ``make check`` as usual. - -If you want to gather coverage information on a single test the ``make -clean-gcda`` target can be used to delete any existing coverage -information before running a single test. - -You can generate a HTML coverage report by executing ``make -coverage-html`` which will create -``meson-logs/coveragereport/index.html``. - -Further analysis can be conducted by running the ``gcov`` command -directly on the various .gcda output files. Please read the ``gcov`` -documentation for more information. - QEMU iotests ============ @@ -1302,3 +1283,22 @@ exercise as many corner cases as possible. It is a useful test suite to run to exercise QEMU's linux-user code:: https://linux-test-project.github.io/ + +GCC gcov support +================ + +``gcov`` is a GCC tool to analyze the testing coverage by +instrumenting the tested code. To use it, configure QEMU with +``--enable-gcov`` option and build. Then run the tests as usual. + +If you want to gather coverage information on a single test the ``make +clean-gcda`` target can be used to delete any existing coverage +information before running a single test. + +You can generate a HTML coverage report by executing ``make +coverage-html`` which will create +``meson-logs/coveragereport/index.html``. + +Further analysis can be conducted by running the ``gcov`` command +directly on the various .gcda output files. Please read the ``gcov`` +documentation for more information. -- cgit v1.1 From 16e79e1b01a698908e14eda3078d4a8e7b1b9c2b Mon Sep 17 00:00:00 2001 From: Paolo Bonzini Date: Tue, 7 Sep 2021 17:21:58 +0200 Subject: docs: reorganize testing.rst Clean up the heading levels to use === --- ~~~ ^^^ '''. Reorganize the outline for the Avocado part, and always include headings for the class names. Signed-off-by: Paolo Bonzini --- docs/devel/testing.rst | 146 +++++++++++++++++++++++++------------------------ 1 file changed, 76 insertions(+), 70 deletions(-) (limited to 'docs') diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst index a80df07..7500f07 100644 --- a/docs/devel/testing.rst +++ b/docs/devel/testing.rst @@ -1,11 +1,10 @@ -=============== Testing in QEMU =============== This document describes the testing infrastructure in QEMU. Testing with "make check" -========================= +------------------------- The "make check" testing family includes most of the C based tests in QEMU. For a quick help, run ``make check-help`` from the source tree. @@ -24,7 +23,7 @@ expect the executables to exist and will fail with obscure messages if they cannot find them. Unit tests ----------- +~~~~~~~~~~ Unit tests, which can be invoked with ``make check-unit``, are simple C tests that typically link to individual QEMU object files and exercise them by @@ -67,7 +66,7 @@ and copy the actual command line which executes the unit test, then run it from the command line. QTest ------ +~~~~~ QTest is a device emulation testing framework. It can be very useful to test device models; it could also control certain aspects of QEMU (such as virtual @@ -81,7 +80,7 @@ QTest cases can be executed with make check-qtest QAPI schema tests ------------------ +~~~~~~~~~~~~~~~~~ The QAPI schema tests validate the QAPI parser used by QMP, by feeding predefined input to the parser and comparing the result with the reference @@ -108,14 +107,14 @@ parser (either fixing a bug or extending/modifying the syntax). To do this: ``qapi-schema += foo.json`` check-block ------------ +~~~~~~~~~~~ ``make check-block`` runs a subset of the block layer iotests (the tests that are in the "auto" group). See the "QEMU iotests" section below for more information. QEMU iotests -============ +------------ QEMU iotests, under the directory ``tests/qemu-iotests``, is the testing framework widely used to test block layer related features. It is higher level @@ -152,7 +151,7 @@ More options are supported by the ``./check`` script, run ``./check -h`` for help. Writing a new test case ------------------------ +~~~~~~~~~~~~~~~~~~~~~~~ Consider writing a tests case when you are making any changes to the block layer. An iotest case is usually the choice for that. There are already many @@ -206,7 +205,8 @@ test failure. If using such devices are explicitly desired, consider adding ``locking=off`` option to disable image locking. Debugging a test case ------------------------ +~~~~~~~~~~~~~~~~~~~~~ + The following options to the ``check`` script can be useful when debugging a failing test: @@ -235,7 +235,7 @@ a failing test: ``$TEST_DIR/qemu-machine-``. Test case groups ----------------- +~~~~~~~~~~~~~~~~ "Tests may belong to one or more test groups, which are defined in the form of a comment in the test source file. By convention, test groups are listed @@ -285,10 +285,10 @@ Note that the following group names have a special meaning: .. _container-ref: Container based tests -===================== +--------------------- Introduction ------------- +~~~~~~~~~~~~ The container testing framework in QEMU utilizes public images to build and test QEMU in predefined and widely accessible Linux @@ -303,7 +303,7 @@ The container images are also used to augment the generation of tests for testing TCG. See :ref:`checktcg-ref` for more details. Docker Prerequisites --------------------- +~~~~~~~~~~~~~~~~~~~~ Install "docker" with the system package manager and start the Docker service on your development machine, then make sure you have the privilege to run @@ -334,7 +334,7 @@ exploit the whole host with Docker bind mounting or other privileged operations. So only do it on development machines. Podman Prerequisites --------------------- +~~~~~~~~~~~~~~~~~~~~ Install "podman" with the system package manager. @@ -346,7 +346,7 @@ Install "podman" with the system package manager. The last command should print an empty table, to verify the system is ready. Quickstart ----------- +~~~~~~~~~~ From source tree, type ``make docker-help`` to see the help. Testing can be started without configuring or building QEMU (``configure`` and @@ -362,7 +362,7 @@ is downloaded and initialized automatically), in which the ``test-build`` job is executed. Registry --------- +~~~~~~~~ The QEMU project has a container registry hosted by GitLab at ``registry.gitlab.com/qemu-project/qemu`` which will automatically be @@ -376,7 +376,7 @@ locally by using the ``NOCACHE`` build option: make docker-image-debian10 NOCACHE=1 Images ------- +~~~~~~ Along with many other images, the ``centos8`` image is defined in a Dockerfile in ``tests/docker/dockerfiles/``, called ``centos8.docker``. ``make docker-help`` @@ -391,7 +391,7 @@ mainly used to do necessary host side setup. One such setup is ``binfmt_misc``, for example, to make qemu-user powered cross build containers work. Tests ------ +~~~~~ Different tests are added to cover various configurations to build and test QEMU. Docker tests are the executables under ``tests/docker`` named @@ -402,7 +402,7 @@ source and build it. The full list of tests is printed in the ``make docker-help`` help. Debugging a Docker test failure -------------------------------- +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ When CI tasks, maintainers or yourself report a Docker test failure, follow the below steps to debug it: @@ -419,7 +419,7 @@ below steps to debug it: the prompt for debug. Options -------- +~~~~~~~ Various options can be used to affect how Docker tests are done. The full list is in the ``make docker`` help text. The frequently used ones are: @@ -433,7 +433,7 @@ list is in the ``make docker`` help text. The frequently used ones are: failure" section. Thread Sanitizer -================ +---------------- Thread Sanitizer (TSan) is a tool which can detect data races. QEMU supports building and testing with this tool. @@ -443,7 +443,7 @@ For more information on TSan: https://github.com/google/sanitizers/wiki/ThreadSanitizerCppManual Thread Sanitizer in Docker ---------------------------- +~~~~~~~~~~~~~~~~~~~~~~~~~~ TSan is currently supported in the ubuntu2004 docker. The test-tsan test will build using TSan and then run make check. @@ -458,7 +458,7 @@ We recommend using DEBUG=1 to allow launching the test from inside the docker, and to allow review of the warnings generated by TSan. Building and Testing with TSan ------------------------------- +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ It is possible to build and test with TSan, with a few additional steps. These steps are normally done automatically in the docker. @@ -497,7 +497,7 @@ This allows for running the test and then checking the warnings afterwards. If you want TSan to stop and exit with error on warnings, use exitcode=66. TSan Suppressions ------------------ +~~~~~~~~~~~~~~~~~ Keep in mind that for any data race warning, although there might be a data race detected by TSan, there might be no actual bug here. TSan provides several different mechanisms for suppressing warnings. In general it is recommended @@ -523,7 +523,7 @@ More information on the file format can be found here under "Blacklist Format": https://github.com/google/sanitizers/wiki/ThreadSanitizerFlags TSan Annotations ----------------- +~~~~~~~~~~~~~~~~ include/qemu/tsan.h defines annotations. See this file for more descriptions of the annotations themselves. Annotations can be used to suppress TSan warnings or give TSan more information so that it can detect proper @@ -540,14 +540,14 @@ The full set of annotations can be found here: https://github.com/llvm/llvm-project/blob/master/compiler-rt/lib/tsan/rtl/tsan_interface_ann.cpp VM testing -========== +---------- This test suite contains scripts that bootstrap various guest images that have necessary packages to build QEMU. The basic usage is documented in ``Makefile`` help which is displayed with ``make vm-help``. Quickstart ----------- +~~~~~~~~~~ Run ``make vm-help`` to list available make targets. Invoke a specific make command to run build test in an image. For example, ``make vm-build-freebsd`` @@ -562,7 +562,7 @@ concerned about attackers taking control of the guest and potentially exploiting a QEMU security bug to compromise the host. QEMU binaries -------------- +~~~~~~~~~~~~~ By default, qemu-system-x86_64 is searched in $PATH to run the guest. If there isn't one, or if it is older than 2.10, the test won't work. In this case, @@ -571,20 +571,20 @@ provide the QEMU binary in env var: ``QEMU=/path/to/qemu-2.10+``. Likewise the path to qemu-img can be set in QEMU_IMG environment variable. Make jobs ---------- +~~~~~~~~~ The ``-j$X`` option in the make command line is not propagated into the VM, specify ``J=$X`` to control the make jobs in the guest. Debugging ---------- +~~~~~~~~~ Add ``DEBUG=1`` and/or ``V=1`` to the make command to allow interactive debugging and verbose output. If this is not enough, see the next section. ``V=1`` will be propagated down into the make jobs in the guest. Manual invocation ------------------ +~~~~~~~~~~~~~~~~~ Each guest script is an executable script with the same command line options. For example to work with the netbsd guest, use ``$QEMU_SRC/tests/vm/netbsd``: @@ -608,7 +608,7 @@ For example to work with the netbsd guest, use ``$QEMU_SRC/tests/vm/netbsd``: $ ./netbsd --interactive --image /var/tmp/netbsd.img sh Adding new guests ------------------ +~~~~~~~~~~~~~~~~~ Please look at existing guest scripts for how to add new guests. @@ -641,7 +641,7 @@ the script's ``main()``. recommended. Image fuzzer testing -==================== +-------------------- An image fuzzer was added to exercise format drivers. Currently only qcow2 is supported. To start the fuzzer, run @@ -654,7 +654,7 @@ Alternatively, some command different from "qemu-img info" can be tested, by changing the ``-c`` option. Acceptance tests using the Avocado Framework -============================================ +-------------------------------------------- The ``tests/acceptance`` directory hosts functional tests, also known as acceptance level tests. They're usually higher level tests, and @@ -693,7 +693,7 @@ Tests based on ``avocado_qemu.Test`` can easily: - http://avocado-framework.readthedocs.io/en/latest/api/utils/avocado.utils.html Running tests -------------- +~~~~~~~~~~~~~ You can run the acceptance tests simply by executing: @@ -791,7 +791,7 @@ of Avocado or ``make check-acceptance``, and can also be queried using: tests/venv/bin/avocado list tests/acceptance Manual Installation -------------------- +~~~~~~~~~~~~~~~~~~~ To manually install Avocado and its dependencies, run: @@ -804,7 +804,7 @@ Alternatively, follow the instructions on this link: https://avocado-framework.readthedocs.io/en/latest/guides/user/chapters/installing.html Overview --------- +~~~~~~~~ The ``tests/acceptance/avocado_qemu`` directory provides the ``avocado_qemu`` Python module, containing the ``avocado_qemu.Test`` @@ -840,7 +840,7 @@ in the current directory, tagged as "quick", run: avocado run -t quick . The ``avocado_qemu.Test`` base test class ------------------------------------------ +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ The ``avocado_qemu.Test`` class has a number of characteristics that are worth being mentioned right away. @@ -890,7 +890,7 @@ At test "tear down", ``avocado_qemu.Test`` handles all the QEMUMachines shutdown. The ``avocado_qemu.LinuxTest`` base test class -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ The ``avocado_qemu.LinuxTest`` is further specialization of the ``avocado_qemu.Test`` class, so it contains all the characteristics of @@ -933,7 +933,7 @@ execution of a QEMU binary, giving its users: a more succinct and intuitive way QEMU binary selection -~~~~~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^^^^^^^^ The QEMU binary used for the ``self.vm`` QEMUMachine instance will primarily depend on the value of the ``qemu_bin`` parameter. If it's @@ -954,20 +954,23 @@ The resulting ``qemu_bin`` value will be preserved in the ``avocado_qemu.Test`` as an attribute with the same name. Attribute reference -------------------- +~~~~~~~~~~~~~~~~~~~ + +Test +^^^^ Besides the attributes and methods that are part of the base ``avocado.Test`` class, the following attributes are available on any ``avocado_qemu.Test`` instance. vm -~~ +'' A QEMUMachine instance, initially configured according to the given ``qemu_bin`` parameter. arch -~~~~ +'''' The architecture can be used on different levels of the stack, e.g. by the framework or by the test itself. At the framework level, it will @@ -984,7 +987,7 @@ name. If one is not given explicitly, it will either be set to ``:avocado: tags=arch:VALUE`` tag, it will be set to ``VALUE``. cpu -~~~ +''' The cpu model that will be set to all QEMUMachine instances created by the test. @@ -995,7 +998,7 @@ name. If one is not given explicitly, it will either be set to ``:avocado: tags=cpu:VALUE`` tag, it will be set to ``VALUE``. machine -~~~~~~~ +''''''' The machine type that will be set to all QEMUMachine instances created by the test. @@ -1006,20 +1009,20 @@ name. If one is not given explicitly, it will either be set to ``:avocado: tags=machine:VALUE`` tag, it will be set to ``VALUE``. qemu_bin -~~~~~~~~ +'''''''' The preserved value of the ``qemu_bin`` parameter or the result of the dynamic probe for a QEMU binary in the current working directory or source tree. LinuxTest -~~~~~~~~~ +^^^^^^^^^ Besides the attributes present on the ``avocado_qemu.Test`` base class, the ``avocado_qemu.LinuxTest`` adds the following attributes: distro -...... +'''''' The name of the Linux distribution used as the guest image for the test. The name should match the **Provider** column on the list @@ -1028,7 +1031,7 @@ of images supported by the avocado.utils.vmimage library: https://avocado-framework.readthedocs.io/en/latest/guides/writer/libs/vmimage.html#supported-images distro_version -.............. +'''''''''''''' The version of the Linux distribution as the guest image for the test. The name should match the **Version** column on the list @@ -1037,7 +1040,7 @@ of images supported by the avocado.utils.vmimage library: https://avocado-framework.readthedocs.io/en/latest/guides/writer/libs/vmimage.html#supported-images distro_checksum -............... +''''''''''''''' The sha256 hash of the guest image file used for the test. @@ -1046,7 +1049,7 @@ same name), no validation on the integrity of the image will be performed. Parameter reference -------------------- +~~~~~~~~~~~~~~~~~~~ To understand how Avocado parameters are accessed by tests, and how they can be passed to tests, please refer to:: @@ -1060,8 +1063,11 @@ like the following: PARAMS (key=qemu_bin, path=*, default=./qemu-system-x86_64) => './qemu-system-x86_64 +Test +^^^^ + arch -~~~~ +'''' The architecture that will influence the selection of a QEMU binary (when one is not explicitly given). @@ -1074,31 +1080,30 @@ This parameter has a direct relation with the ``arch`` attribute. If not given, it will default to None. cpu -~~~ +''' The cpu model that will be set to all QEMUMachine instances created by the test. machine -~~~~~~~ +''''''' The machine type that will be set to all QEMUMachine instances created by the test. - qemu_bin -~~~~~~~~ +'''''''' The exact QEMU binary to be used on QEMUMachine. LinuxTest -~~~~~~~~~ +^^^^^^^^^ Besides the parameters present on the ``avocado_qemu.Test`` base class, the ``avocado_qemu.LinuxTest`` adds the following parameters: distro -...... +'''''' The name of the Linux distribution used as the guest image for the test. The name should match the **Provider** column on the list @@ -1107,7 +1112,7 @@ of images supported by the avocado.utils.vmimage library: https://avocado-framework.readthedocs.io/en/latest/guides/writer/libs/vmimage.html#supported-images distro_version -.............. +'''''''''''''' The version of the Linux distribution as the guest image for the test. The name should match the **Version** column on the list @@ -1116,7 +1121,7 @@ of images supported by the avocado.utils.vmimage library: https://avocado-framework.readthedocs.io/en/latest/guides/writer/libs/vmimage.html#supported-images distro_checksum -............... +''''''''''''''' The sha256 hash of the guest image file used for the test. @@ -1124,7 +1129,8 @@ If this value is not set in the code or by this parameter no validation on the integrity of the image will be performed. Skipping tests --------------- +~~~~~~~~~~~~~~ + The Avocado framework provides Python decorators which allow for easily skip tests running under certain conditions. For example, on the lack of a binary on the test system or when the running environment is a CI system. For further @@ -1139,7 +1145,7 @@ environment variables became a kind of standard way to enable/disable tests. Here is a list of the most used variables: AVOCADO_ALLOW_LARGE_STORAGE -~~~~~~~~~~~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^^^^^^^^^^^^^^ Tests which are going to fetch or produce assets considered *large* are not going to run unless that ``AVOCADO_ALLOW_LARGE_STORAGE=1`` is exported on the environment. @@ -1148,7 +1154,7 @@ The definition of *large* is a bit arbitrary here, but it usually means an asset which occupies at least 1GB of size on disk when uncompressed. AVOCADO_ALLOW_UNTRUSTED_CODE -~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ There are tests which will boot a kernel image or firmware that can be considered not safe to run on the developer's workstation, thus they are skipped by default. The definition of *not safe* is also arbitrary but @@ -1159,7 +1165,7 @@ You should export ``AVOCADO_ALLOW_UNTRUSTED_CODE=1`` on the environment in order to allow tests which make use of those kind of assets. AVOCADO_TIMEOUT_EXPECTED -~~~~~~~~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^^^^^^^^^^^ The Avocado framework has a timeout mechanism which interrupts tests to avoid the test suite of getting stuck. The timeout value can be set via test parameter or property defined in the test class, for further details:: @@ -1173,7 +1179,7 @@ compiled with debug flags. Therefore, the ``AVOCADO_TIMEOUT_EXPECTED`` variable has been used to determine whether those tests should run or not. GITLAB_CI -~~~~~~~~~ +^^^^^^^^^ A number of tests are flagged to not run on the GitLab CI. Usually because they proved to the flaky or there are constraints on the CI environment which would make them fail. If you encounter a similar situation then use that @@ -1186,7 +1192,7 @@ variable as shown on the code snippet below to skip the test: do_something() Uninstalling Avocado --------------------- +~~~~~~~~~~~~~~~~~~~~ If you've followed the manual installation instructions above, you can easily uninstall Avocado. Start by listing the packages you have @@ -1204,7 +1210,7 @@ Avocado is installed will be cleaned up as part of ``make check-clean``. .. _checktcg-ref: Testing with "make check-tcg" -============================= +----------------------------- The check-tcg tests are intended for simple smoke tests of both linux-user and softmmu TCG functionality. However to build test @@ -1237,7 +1243,7 @@ itself. See :ref:`container-ref` for more details. Running subset of tests ------------------------ +~~~~~~~~~~~~~~~~~~~~~~~ You can build the tests for one architecture:: @@ -1251,7 +1257,7 @@ Adding ``V=1`` to the invocation will show the details of how to invoke QEMU for the test which is useful for debugging tests. TCG test dependencies ---------------------- +~~~~~~~~~~~~~~~~~~~~~ The TCG tests are deliberately very light on dependencies and are either totally bare with minimal gcc lib support (for softmmu tests) @@ -1285,7 +1291,7 @@ to run to exercise QEMU's linux-user code:: https://linux-test-project.github.io/ GCC gcov support -================ +---------------- ``gcov`` is a GCC tool to analyze the testing coverage by instrumenting the tested code. To use it, configure QEMU with -- cgit v1.1