diff options
Diffstat (limited to 'docs')
| -rw-r--r-- | docs/about/build-platforms.rst | 10 | ||||
| -rw-r--r-- | docs/devel/build-system.rst | 11 | ||||
| -rw-r--r-- | docs/devel/codebase.rst | 5 | ||||
| -rw-r--r-- | docs/devel/memory.rst | 2 | ||||
| -rw-r--r-- | docs/devel/multi-thread-tcg.rst | 6 | ||||
| -rw-r--r-- | docs/devel/qom.rst | 8 | ||||
| -rw-r--r-- | docs/devel/reset.rst | 2 | ||||
| -rw-r--r-- | docs/devel/tcg-ops.rst | 228 | ||||
| -rw-r--r-- | docs/devel/testing/avocado.rst | 581 | ||||
| -rw-r--r-- | docs/devel/testing/ci-definitions.rst.inc | 121 | ||||
| -rw-r--r-- | docs/devel/testing/ci-jobs.rst.inc | 19 | ||||
| -rw-r--r-- | docs/devel/testing/ci.rst | 28 | ||||
| -rw-r--r-- | docs/devel/testing/functional.rst | 3 | ||||
| -rw-r--r-- | docs/devel/testing/index.rst | 1 | ||||
| -rw-r--r-- | docs/devel/testing/main.rst | 80 | ||||
| -rw-r--r-- | docs/devel/virtio-backends.rst | 2 | ||||
| -rw-r--r-- | docs/igd-assign.txt | 265 |
17 files changed, 445 insertions, 927 deletions
diff --git a/docs/about/build-platforms.rst b/docs/about/build-platforms.rst index 1552b1a..5252155 100644 --- a/docs/about/build-platforms.rst +++ b/docs/about/build-platforms.rst @@ -123,11 +123,11 @@ Rust build dependencies to build QEMU. Optional build dependencies - Build components whose absence does not affect the ability to build - QEMU may not be available in distros, or may be too old for QEMU's - requirements. Many of these, such as the Avocado testing framework - or various linters, are written in Python and therefore can also - be installed using ``pip``. Cross compilers are another example + Build components whose absence does not affect the ability to build QEMU + may not be available in distros, or may be too old for our requirements. + Many of these, such as additional modules for the functional testing + framework or various linters, are written in Python and therefore can + also be installed using ``pip``. Cross compilers are another example of optional build-time dependency; in this case it is possible to download them from repositories such as EPEL, to use container-based cross compilation using ``docker`` or ``podman``, or to use pre-built diff --git a/docs/devel/build-system.rst b/docs/devel/build-system.rst index a759982..258cfad 100644 --- a/docs/devel/build-system.rst +++ b/docs/devel/build-system.rst @@ -134,7 +134,7 @@ in how the build process runs Python code. At this stage, ``configure`` also queries the chosen Python interpreter about QEMU's build dependencies. Note that the build process does *not* -look for ``meson``, ``sphinx-build`` or ``avocado`` binaries in the PATH; +look for ``meson`` or ``sphinx-build`` binaries in the PATH; likewise, there are no options such as ``--meson`` or ``--sphinx-build``. This avoids a potential mismatch, where Meson and Sphinx binaries on the PATH might operate in a different Python environment than the one chosen @@ -151,7 +151,7 @@ virtual environment with ``pip``, either from wheels in ``python/wheels`` or by downloading the package with PyPI. Downloading can be disabled with ``--disable-download``; and anyway, it only happens when a ``configure`` option (currently, only ``--enable-docs``) is explicitly enabled but -the dependencies are not present\ [#pip]_. +the dependencies are not present. .. [#distlib] The scripts are created based on the package's metadata, specifically the ``console_script`` entry points. This is the @@ -164,10 +164,6 @@ the dependencies are not present\ [#pip]_. because the Python Packaging Authority provides a package ``distlib.scripts`` to perform this task. -.. [#pip] ``pip`` might also be used when running ``make check-avocado`` - if downloading is enabled, to ensure that Avocado is - available. - The required versions of the packages are stored in a configuration file ``pythondeps.toml``. The format is custom to QEMU, but it is documented at the top of the file itself and it should be easy to understand. The @@ -497,8 +493,7 @@ number of dynamically created files listed later. ``pyvenv/bin``, and calling ``pip`` to install dependencies. ``tests/Makefile.include`` - Rules for external test harnesses. These include the TCG tests - and the Avocado-based integration tests. + Rules for external test harnesses like the TCG tests. ``tests/docker/Makefile.include`` Rules for Docker tests. Like ``tests/Makefile.include``, this file is diff --git a/docs/devel/codebase.rst b/docs/devel/codebase.rst index ef98578..40273e7 100644 --- a/docs/devel/codebase.rst +++ b/docs/devel/codebase.rst @@ -175,11 +175,6 @@ yet, so sometimes the source code is all you have. * `tests <https://gitlab.com/qemu-project/qemu/-/tree/master/tests>`_: QEMU `test <testing>` suite - - `avocado <https://gitlab.com/qemu-project/qemu/-/tree/master/tests/avocado>`_: - Functional tests booting full VM using `Avocado framework <checkavocado-ref>`. - Those tests will be transformed and moved into - `tests/functional <https://gitlab.com/qemu-project/qemu/-/tree/master/tests/functional>`_ - in the future. - `data <https://gitlab.com/qemu-project/qemu/-/tree/master/tests/data>`_: Data for various tests. - `decode <https://gitlab.com/qemu-project/qemu/-/tree/master/tests/decode>`_: diff --git a/docs/devel/memory.rst b/docs/devel/memory.rst index 69c5e3f..57fb2ae 100644 --- a/docs/devel/memory.rst +++ b/docs/devel/memory.rst @@ -369,4 +369,4 @@ callbacks are called: API Reference ------------- -.. kernel-doc:: include/exec/memory.h +.. kernel-doc:: include/system/memory.h diff --git a/docs/devel/multi-thread-tcg.rst b/docs/devel/multi-thread-tcg.rst index b0f4739..da9a153 100644 --- a/docs/devel/multi-thread-tcg.rst +++ b/docs/devel/multi-thread-tcg.rst @@ -28,9 +28,9 @@ vCPU Scheduling We introduce a new running mode where each vCPU will run on its own user-space thread. This is enabled by default for all FE/BE combinations where the host memory model is able to accommodate the -guest (TCG_GUEST_DEFAULT_MO & ~TCG_TARGET_DEFAULT_MO is zero) and the -guest has had the required work done to support this safely -(TARGET_SUPPORTS_MTTCG). +guest (TCGCPUOps::guest_default_memory_order & ~TCG_TARGET_DEFAULT_MO is zero) +and the guest has had the required work done to support this safely +(TCGCPUOps::mttcg_supported). System emulation will fall back to the original round robin approach if: diff --git a/docs/devel/qom.rst b/docs/devel/qom.rst index 0889ca9..5870745 100644 --- a/docs/devel/qom.rst +++ b/docs/devel/qom.rst @@ -147,7 +147,7 @@ to introduce an overridden virtual function: #include "qdev.h" - void my_device_class_init(ObjectClass *klass, void *class_data) + void my_device_class_init(ObjectClass *klass, const void *class_data) { DeviceClass *dc = DEVICE_CLASS(klass); dc->reset = my_device_reset; @@ -249,7 +249,7 @@ class, which someone might choose to change at some point. // do something } - static void my_class_init(ObjectClass *oc, void *data) + static void my_class_init(ObjectClass *oc, const void *data) { MyClass *mc = MY_CLASS(oc); @@ -279,7 +279,7 @@ class, which someone might choose to change at some point. // do something else here } - static void derived_class_init(ObjectClass *oc, void *data) + static void derived_class_init(ObjectClass *oc, const void *data) { MyClass *mc = MY_CLASS(oc); DerivedClass *dc = DERIVED_CLASS(oc); @@ -363,7 +363,7 @@ This is equivalent to the following: :caption: Expansion from defining a simple type static void my_device_finalize(Object *obj); - static void my_device_class_init(ObjectClass *oc, void *data); + static void my_device_class_init(ObjectClass *oc, const void *data); static void my_device_init(Object *obj); static const TypeInfo my_device_info = { diff --git a/docs/devel/reset.rst b/docs/devel/reset.rst index 0b8b2fa..c02fe0a 100644 --- a/docs/devel/reset.rst +++ b/docs/devel/reset.rst @@ -216,7 +216,7 @@ in reset. ResettablePhases parent_phases; } MyDevClass; - static void mydev_class_init(ObjectClass *class, void *data) + static void mydev_class_init(ObjectClass *class, const void *data) { MyDevClass *myclass = MYDEV_CLASS(class); ResettableClass *rc = RESETTABLE_CLASS(class); diff --git a/docs/devel/tcg-ops.rst b/docs/devel/tcg-ops.rst index 688984f..f26b837 100644 --- a/docs/devel/tcg-ops.rst +++ b/docs/devel/tcg-ops.rst @@ -239,7 +239,7 @@ Jumps/Labels - | Jump to label. - * - brcond_i32/i64 *t0*, *t1*, *cond*, *label* + * - brcond *t0*, *t1*, *cond*, *label* - | Conditional jump if *t0* *cond* *t1* is true. *cond* can be: | @@ -261,98 +261,117 @@ Arithmetic .. list-table:: - * - add_i32/i64 *t0*, *t1*, *t2* + * - add *t0*, *t1*, *t2* - | *t0* = *t1* + *t2* - * - sub_i32/i64 *t0*, *t1*, *t2* + * - sub *t0*, *t1*, *t2* - | *t0* = *t1* - *t2* - * - neg_i32/i64 *t0*, *t1* + * - neg *t0*, *t1* - | *t0* = -*t1* (two's complement) - * - mul_i32/i64 *t0*, *t1*, *t2* + * - mul *t0*, *t1*, *t2* - | *t0* = *t1* * *t2* - * - div_i32/i64 *t0*, *t1*, *t2* + * - divs *t0*, *t1*, *t2* - | *t0* = *t1* / *t2* (signed) | Undefined behavior if division by zero or overflow. - * - divu_i32/i64 *t0*, *t1*, *t2* + * - divu *t0*, *t1*, *t2* - | *t0* = *t1* / *t2* (unsigned) | Undefined behavior if division by zero. - * - rem_i32/i64 *t0*, *t1*, *t2* + * - rems *t0*, *t1*, *t2* - | *t0* = *t1* % *t2* (signed) | Undefined behavior if division by zero or overflow. - * - remu_i32/i64 *t0*, *t1*, *t2* + * - remu *t0*, *t1*, *t2* - | *t0* = *t1* % *t2* (unsigned) | Undefined behavior if division by zero. + * - divs2 *q*, *r*, *nl*, *nh*, *d* + + - | *q* = *nh:nl* / *d* (signed) + | *r* = *nh:nl* % *d* + | Undefined behaviour if division by zero, or the double-word + numerator divided by the single-word divisor does not fit + within the single-word quotient. The code generator will + pass *nh* as a simple sign-extension of *nl*, so the only + overflow should be *INT_MIN* / -1. + + * - divu2 *q*, *r*, *nl*, *nh*, *d* + + - | *q* = *nh:nl* / *d* (unsigned) + | *r* = *nh:nl* % *d* + | Undefined behaviour if division by zero, or the double-word + numerator divided by the single-word divisor does not fit + within the single-word quotient. The code generator will + pass 0 to *nh* to make a simple zero-extension of *nl*, + so overflow should never occur. Logical ------- .. list-table:: - * - and_i32/i64 *t0*, *t1*, *t2* + * - and *t0*, *t1*, *t2* - | *t0* = *t1* & *t2* - * - or_i32/i64 *t0*, *t1*, *t2* + * - or *t0*, *t1*, *t2* - | *t0* = *t1* | *t2* - * - xor_i32/i64 *t0*, *t1*, *t2* + * - xor *t0*, *t1*, *t2* - | *t0* = *t1* ^ *t2* - * - not_i32/i64 *t0*, *t1* + * - not *t0*, *t1* - | *t0* = ~\ *t1* - * - andc_i32/i64 *t0*, *t1*, *t2* + * - andc *t0*, *t1*, *t2* - | *t0* = *t1* & ~\ *t2* - * - eqv_i32/i64 *t0*, *t1*, *t2* + * - eqv *t0*, *t1*, *t2* - | *t0* = ~(*t1* ^ *t2*), or equivalently, *t0* = *t1* ^ ~\ *t2* - * - nand_i32/i64 *t0*, *t1*, *t2* + * - nand *t0*, *t1*, *t2* - | *t0* = ~(*t1* & *t2*) - * - nor_i32/i64 *t0*, *t1*, *t2* + * - nor *t0*, *t1*, *t2* - | *t0* = ~(*t1* | *t2*) - * - orc_i32/i64 *t0*, *t1*, *t2* + * - orc *t0*, *t1*, *t2* - | *t0* = *t1* | ~\ *t2* - * - clz_i32/i64 *t0*, *t1*, *t2* + * - clz *t0*, *t1*, *t2* - | *t0* = *t1* ? clz(*t1*) : *t2* - * - ctz_i32/i64 *t0*, *t1*, *t2* + * - ctz *t0*, *t1*, *t2* - | *t0* = *t1* ? ctz(*t1*) : *t2* - * - ctpop_i32/i64 *t0*, *t1* + * - ctpop *t0*, *t1* - | *t0* = number of bits set in *t1* | - | With *ctpop* short for "count population", matching - | the function name used in ``include/qemu/host-utils.h``. + | The name *ctpop* is short for "count population", and matches + the function name used in ``include/qemu/host-utils.h``. Shifts/Rotates @@ -360,30 +379,30 @@ Shifts/Rotates .. list-table:: - * - shl_i32/i64 *t0*, *t1*, *t2* + * - shl *t0*, *t1*, *t2* - | *t0* = *t1* << *t2* - | Unspecified behavior if *t2* < 0 or *t2* >= 32 (resp 64) + | Unspecified behavior for negative or out-of-range shifts. - * - shr_i32/i64 *t0*, *t1*, *t2* + * - shr *t0*, *t1*, *t2* - | *t0* = *t1* >> *t2* (unsigned) - | Unspecified behavior if *t2* < 0 or *t2* >= 32 (resp 64) + | Unspecified behavior for negative or out-of-range shifts. - * - sar_i32/i64 *t0*, *t1*, *t2* + * - sar *t0*, *t1*, *t2* - | *t0* = *t1* >> *t2* (signed) - | Unspecified behavior if *t2* < 0 or *t2* >= 32 (resp 64) + | Unspecified behavior for negative or out-of-range shifts. - * - rotl_i32/i64 *t0*, *t1*, *t2* + * - rotl *t0*, *t1*, *t2* - | Rotation of *t2* bits to the left - | Unspecified behavior if *t2* < 0 or *t2* >= 32 (resp 64) + | Unspecified behavior for negative or out-of-range shifts. - * - rotr_i32/i64 *t0*, *t1*, *t2* + * - rotr *t0*, *t1*, *t2* - | Rotation of *t2* bits to the right. - | Unspecified behavior if *t2* < 0 or *t2* >= 32 (resp 64) + | Unspecified behavior for negative or out-of-range shifts. Misc @@ -391,26 +410,12 @@ Misc .. list-table:: - * - mov_i32/i64 *t0*, *t1* + * - mov *t0*, *t1* - | *t0* = *t1* - | Move *t1* to *t0* (both operands must have the same type). - - * - ext8s_i32/i64 *t0*, *t1* - - ext8u_i32/i64 *t0*, *t1* - - ext16s_i32/i64 *t0*, *t1* - - ext16u_i32/i64 *t0*, *t1* + | Move *t1* to *t0*. - ext32s_i64 *t0*, *t1* - - ext32u_i64 *t0*, *t1* - - - | 8, 16 or 32 bit sign/zero extension (both operands must have the same type) - - * - bswap16_i32/i64 *t0*, *t1*, *flags* + * - bswap16 *t0*, *t1*, *flags* - | 16 bit byte swap on the low bits of a 32/64 bit input. | @@ -420,24 +425,24 @@ Misc | | If neither ``TCG_BSWAP_OZ`` nor ``TCG_BSWAP_OS`` are set, then the bits of *t0* above bit 15 may contain any value. - * - bswap32_i64 *t0*, *t1*, *flags* - - - | 32 bit byte swap on a 64-bit value. The flags are the same as for bswap16, - except they apply from bit 31 instead of bit 15. + * - bswap32 *t0*, *t1*, *flags* - * - bswap32_i32 *t0*, *t1*, *flags* + - | 32 bit byte swap. The flags are the same as for bswap16, except + they apply from bit 31 instead of bit 15. On TCG_TYPE_I32, the + flags should be zero. - bswap64_i64 *t0*, *t1*, *flags* + * - bswap64 *t0*, *t1*, *flags* - - | 32/64 bit byte swap. The flags are ignored, but still present - for consistency with the other bswap opcodes. + - | 64 bit byte swap. The flags are ignored, but still present + for consistency with the other bswap opcodes. For future + compatibility, the flags should be zero. * - discard_i32/i64 *t0* - | Indicate that the value of *t0* won't be used later. It is useful to force dead code elimination. - * - deposit_i32/i64 *dest*, *t1*, *t2*, *pos*, *len* + * - deposit *dest*, *t1*, *t2*, *pos*, *len* - | Deposit *t2* as a bitfield into *t1*, placing the result in *dest*. | @@ -446,14 +451,16 @@ Misc | *len* - the length of the bitfield | *pos* - the position of the first bit, counting from the LSB | - | For example, "deposit_i32 dest, t1, t2, 8, 4" indicates a 4-bit field + | For example, "deposit dest, t1, t2, 8, 4" indicates a 4-bit field at bit 8. This operation would be equivalent to | | *dest* = (*t1* & ~0x0f00) | ((*t2* << 8) & 0x0f00) + | + | on TCG_TYPE_I32. - * - extract_i32/i64 *dest*, *t1*, *pos*, *len* + * - extract *dest*, *t1*, *pos*, *len* - sextract_i32/i64 *dest*, *t1*, *pos*, *len* + sextract *dest*, *t1*, *pos*, *len* - | Extract a bitfield from *t1*, placing the result in *dest*. | @@ -462,16 +469,16 @@ Misc to the left with zeros; for sextract_*, the result will be extended to the left with copies of the bitfield sign bit at *pos* + *len* - 1. | - | For example, "sextract_i32 dest, t1, 8, 4" indicates a 4-bit field + | For example, "sextract dest, t1, 8, 4" indicates a 4-bit field at bit 8. This operation would be equivalent to | | *dest* = (*t1* << 20) >> 28 | - | (using an arithmetic right shift). + | (using an arithmetic right shift) on TCG_TYPE_I32. - * - extract2_i32/i64 *dest*, *t1*, *t2*, *pos* + * - extract2 *dest*, *t1*, *t2*, *pos* - - | For N = {32,64}, extract an N-bit quantity from the concatenation + - | For TCG_TYPE_I{N}, extract an N-bit quantity from the concatenation of *t2*:*t1*, beginning at *pos*. The tcg_gen_extract2_{i32,i64} expander accepts 0 <= *pos* <= N as inputs. The backend code generator will not see either 0 or N as inputs for these opcodes. @@ -494,19 +501,19 @@ Conditional moves .. list-table:: - * - setcond_i32/i64 *dest*, *t1*, *t2*, *cond* + * - setcond *dest*, *t1*, *t2*, *cond* - | *dest* = (*t1* *cond* *t2*) | | Set *dest* to 1 if (*t1* *cond* *t2*) is true, otherwise set to 0. - * - negsetcond_i32/i64 *dest*, *t1*, *t2*, *cond* + * - negsetcond *dest*, *t1*, *t2*, *cond* - | *dest* = -(*t1* *cond* *t2*) | | Set *dest* to -1 if (*t1* *cond* *t2*) is true, otherwise set to 0. - * - movcond_i32/i64 *dest*, *c1*, *c2*, *v1*, *v2*, *cond* + * - movcond *dest*, *c1*, *c2*, *v1*, *v2*, *cond* - | *dest* = (*c1* *cond* *c2* ? *v1* : *v2*) | @@ -586,26 +593,79 @@ Multiword arithmetic support .. list-table:: - * - add2_i32/i64 *t0_low*, *t0_high*, *t1_low*, *t1_high*, *t2_low*, *t2_high* + * - addco *t0*, *t1*, *t2* + + - | Compute *t0* = *t1* + *t2* and in addition output to the + carry bit provided by the host architecture. + + * - addci *t0, *t1*, *t2* - sub2_i32/i64 *t0_low*, *t0_high*, *t1_low*, *t1_high*, *t2_low*, *t2_high* + - | Compute *t0* = *t1* + *t2* + *C*, where *C* is the + input carry bit provided by the host architecture. + The output carry bit need not be computed. - - | Similar to add/sub, except that the double-word inputs *t1* and *t2* are - formed from two single-word arguments, and the double-word output *t0* - is returned in two single-word outputs. + * - addcio *t0, *t1*, *t2* - * - mulu2_i32/i64 *t0_low*, *t0_high*, *t1*, *t2* + - | Compute *t0* = *t1* + *t2* + *C*, where *C* is the + input carry bit provided by the host architecture, + and also compute the output carry bit. + + * - addc1o *t0, *t1*, *t2* + + - | Compute *t0* = *t1* + *t2* + 1, and in addition output to the + carry bit provided by the host architecture. This is akin to + *addcio* with a fixed carry-in value of 1. + | This is intended to be used by the optimization pass, + intermediate to complete folding of the addition chain. + In some cases complete folding is not possible and this + opcode will remain until output. If this happens, the + code generator will use ``tcg_out_set_carry`` and then + the output routine for *addcio*. + + * - subbo *t0*, *t1*, *t2* + + - | Compute *t0* = *t1* - *t2* and in addition output to the + borrow bit provided by the host architecture. + | Depending on the host architecture, the carry bit may or may not be + identical to the borrow bit. Thus the addc\* and subb\* + opcodes must not be mixed. + + * - subbi *t0, *t1*, *t2* + + - | Compute *t0* = *t1* - *t2* - *B*, where *B* is the + input borrow bit provided by the host architecture. + The output borrow bit need not be computed. + + * - subbio *t0, *t1*, *t2* + + - | Compute *t0* = *t1* - *t2* - *B*, where *B* is the + input borrow bit provided by the host architecture, + and also compute the output borrow bit. + + * - subb1o *t0, *t1*, *t2* + + - | Compute *t0* = *t1* - *t2* - 1, and in addition output to the + borrow bit provided by the host architecture. This is akin to + *subbio* with a fixed borrow-in value of 1. + | This is intended to be used by the optimization pass, + intermediate to complete folding of the subtraction chain. + In some cases complete folding is not possible and this + opcode will remain until output. If this happens, the + code generator will use ``tcg_out_set_borrow`` and then + the output routine for *subbio*. + + * - mulu2 *t0_low*, *t0_high*, *t1*, *t2* - | Similar to mul, except two unsigned inputs *t1* and *t2* yielding the full double-word product *t0*. The latter is returned in two single-word outputs. - * - muls2_i32/i64 *t0_low*, *t0_high*, *t1*, *t2* + * - muls2 *t0_low*, *t0_high*, *t1*, *t2* - | Similar to mulu2, except the two inputs *t1* and *t2* are signed. - * - mulsh_i32/i64 *t0*, *t1*, *t2* + * - mulsh *t0*, *t1*, *t2* - muluh_i32/i64 *t0*, *t1*, *t2* + muluh *t0*, *t1*, *t2* - | Provide the high part of a signed or unsigned multiply, respectively. | @@ -684,8 +744,6 @@ QEMU specific operations qemu_st_i32/i64/i128 *t0*, *t1*, *flags*, *memidx* - qemu_st8_i32 *t0*, *t1*, *flags*, *memidx* - - | Load data at the guest address *t1* into *t0*, or store data in *t0* at guest address *t1*. The _i32/_i64/_i128 size applies to the size of the input/output register *t0* only. The address *t1* is always sized according to the guest, @@ -703,10 +761,6 @@ QEMU specific operations 64-bit memory access specified in *flags*. | | For qemu_ld/st_i128, these are only supported for a 64-bit host. - | - | For i386, qemu_st8_i32 is exactly like qemu_st_i32, except the size of - the memory operation is known to be 8-bit. This allows the backend to - provide a different set of register constraints. Host vector operations @@ -884,9 +938,9 @@ Assumptions The target word size (``TCG_TARGET_REG_BITS``) is expected to be 32 bit or 64 bit. It is expected that the pointer has the same size as the word. -On a 32 bit target, all 64 bit operations are converted to 32 bits. A -few specific operations must be implemented to allow it (see add2_i32, -sub2_i32, brcond2_i32). +On a 32 bit target, all 64 bit operations are converted to 32 bits. +A few specific operations must be implemented to allow it +(see brcond2_i32, setcond2_i32). On a 64 bit target, the values are transferred between 32 and 64-bit registers using the following ops: diff --git a/docs/devel/testing/avocado.rst b/docs/devel/testing/avocado.rst deleted file mode 100644 index eda76fe..0000000 --- a/docs/devel/testing/avocado.rst +++ /dev/null @@ -1,581 +0,0 @@ -.. _checkavocado-ref: - - -Integration testing with Avocado -================================ - -The ``tests/avocado`` directory hosts integration tests. They're usually -higher level tests, and may interact with external resources and with -various guest operating systems. - -These tests are written using the Avocado Testing Framework (which must be -installed separately) in conjunction with a the ``avocado_qemu.QemuSystemTest`` -class, implemented at ``tests/avocado/avocado_qemu``. - -Tests based on ``avocado_qemu.QemuSystemTest`` can easily: - - * Customize the command line arguments given to the convenience - ``self.vm`` attribute (a QEMUMachine instance) - - * Interact with the QEMU monitor, send QMP commands and check - their results - - * Interact with the guest OS, using the convenience console device - (which may be useful to assert the effectiveness and correctness of - command line arguments or QMP commands) - - * Interact with external data files that accompany the test itself - (see ``self.get_data()``) - - * Download (and cache) remote data files, such as firmware and kernel - images - - * Have access to a library of guest OS images (by means of the - ``avocado.utils.vmimage`` library) - - * Make use of various other test related utilities available at the - test class itself and at the utility library: - - - http://avocado-framework.readthedocs.io/en/latest/api/test/avocado.html#avocado.Test - - http://avocado-framework.readthedocs.io/en/latest/api/utils/avocado.utils.html - -Running tests -------------- - -You can run the avocado tests simply by executing: - -.. code:: - - make check-avocado - -This involves the automatic installation, from PyPI, of all the -necessary avocado-framework dependencies into the QEMU venv within the -build tree (at ``./pyvenv``). Test results are also saved within the -build tree (at ``tests/results``). - -Note: the build environment must be using a Python 3 stack, and have -the ``venv`` and ``pip`` packages installed. If necessary, make sure -``configure`` is called with ``--python=`` and that those modules are -available. On Debian and Ubuntu based systems, depending on the -specific version, they may be on packages named ``python3-venv`` and -``python3-pip``. - -It is also possible to run tests based on tags using the -``make check-avocado`` command and the ``AVOCADO_TAGS`` environment -variable: - -.. code:: - - make check-avocado AVOCADO_TAGS=quick - -Note that tags separated with commas have an AND behavior, while tags -separated by spaces have an OR behavior. For more information on Avocado -tags, see: - - https://avocado-framework.readthedocs.io/en/latest/guides/user/chapters/tags.html - -To run a single test file, a couple of them, or a test within a file -using the ``make check-avocado`` command, set the ``AVOCADO_TESTS`` -environment variable with the test files or test names. To run all -tests from a single file, use: - - .. code:: - - make check-avocado AVOCADO_TESTS=$FILEPATH - -The same is valid to run tests from multiple test files: - - .. code:: - - make check-avocado AVOCADO_TESTS='$FILEPATH1 $FILEPATH2' - -To run a single test within a file, use: - - .. code:: - - make check-avocado AVOCADO_TESTS=$FILEPATH:$TESTCLASS.$TESTNAME - -The same is valid to run single tests from multiple test files: - - .. code:: - - make check-avocado AVOCADO_TESTS='$FILEPATH1:$TESTCLASS1.$TESTNAME1 $FILEPATH2:$TESTCLASS2.$TESTNAME2' - -The scripts installed inside the virtual environment may be used -without an "activation". For instance, the Avocado test runner -may be invoked by running: - - .. code:: - - pyvenv/bin/avocado run $OPTION1 $OPTION2 tests/avocado/ - -Note that if ``make check-avocado`` was not executed before, it is -possible to create the Python virtual environment with the dependencies -needed running: - - .. code:: - - make check-venv - -It is also possible to run tests from a single file or a single test within -a test file. To run tests from a single file within the build tree, use: - - .. code:: - - pyvenv/bin/avocado run tests/avocado/$TESTFILE - -To run a single test within a test file, use: - - .. code:: - - pyvenv/bin/avocado run tests/avocado/$TESTFILE:$TESTCLASS.$TESTNAME - -Valid test names are visible in the output from any previous execution -of Avocado or ``make check-avocado``, and can also be queried using: - - .. code:: - - pyvenv/bin/avocado list tests/avocado - -Manual Installation -------------------- - -To manually install Avocado and its dependencies, run: - -.. code:: - - pip install --user avocado-framework - -Alternatively, follow the instructions on this link: - - https://avocado-framework.readthedocs.io/en/latest/guides/user/chapters/installing.html - -Overview --------- - -The ``tests/avocado/avocado_qemu`` directory provides the -``avocado_qemu`` Python module, containing the ``avocado_qemu.QemuSystemTest`` -class. Here's a simple usage example: - -.. code:: - - from avocado_qemu import QemuSystemTest - - - class Version(QemuSystemTest): - """ - :avocado: tags=quick - """ - def test_qmp_human_info_version(self): - self.vm.launch() - res = self.vm.cmd('human-monitor-command', - command_line='info version') - self.assertRegex(res, r'^(\d+\.\d+\.\d)') - -To execute your test, run: - -.. code:: - - avocado run version.py - -Tests may be classified according to a convention by using docstring -directives such as ``:avocado: tags=TAG1,TAG2``. To run all tests -in the current directory, tagged as "quick", run: - -.. code:: - - avocado run -t quick . - -The ``avocado_qemu.QemuSystemTest`` base test class -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -The ``avocado_qemu.QemuSystemTest`` class has a number of characteristics -that are worth being mentioned right away. - -First of all, it attempts to give each test a ready to use QEMUMachine -instance, available at ``self.vm``. Because many tests will tweak the -QEMU command line, launching the QEMUMachine (by using ``self.vm.launch()``) -is left to the test writer. - -The base test class has also support for tests with more than one -QEMUMachine. The way to get machines is through the ``self.get_vm()`` -method which will return a QEMUMachine instance. The ``self.get_vm()`` -method accepts arguments that will be passed to the QEMUMachine creation -and also an optional ``name`` attribute so you can identify a specific -machine and get it more than once through the tests methods. A simple -and hypothetical example follows: - -.. code:: - - from avocado_qemu import QemuSystemTest - - - class MultipleMachines(QemuSystemTest): - def test_multiple_machines(self): - first_machine = self.get_vm() - second_machine = self.get_vm() - self.get_vm(name='third_machine').launch() - - first_machine.launch() - second_machine.launch() - - first_res = first_machine.cmd( - 'human-monitor-command', - command_line='info version') - - second_res = second_machine.cmd( - 'human-monitor-command', - command_line='info version') - - third_res = self.get_vm(name='third_machine').cmd( - 'human-monitor-command', - command_line='info version') - - self.assertEqual(first_res, second_res, third_res) - -At test "tear down", ``avocado_qemu.QemuSystemTest`` handles all the -QEMUMachines shutdown. - -The ``avocado_qemu.LinuxTest`` base test class -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -The ``avocado_qemu.LinuxTest`` is further specialization of the -``avocado_qemu.QemuSystemTest`` class, so it contains all the characteristics -of the later plus some extra features. - -First of all, this base class is intended for tests that need to -interact with a fully booted and operational Linux guest. At this -time, it uses a Fedora 31 guest image. The most basic example looks -like this: - -.. code:: - - from avocado_qemu import LinuxTest - - - class SomeTest(LinuxTest): - - def test(self): - self.launch_and_wait() - self.ssh_command('some_command_to_be_run_in_the_guest') - -Please refer to tests that use ``avocado_qemu.LinuxTest`` under -``tests/avocado`` for more examples. - -QEMUMachine ------------ - -The QEMUMachine API is already widely used in the Python iotests, -device-crash-test and other Python scripts. It's a wrapper around the -execution of a QEMU binary, giving its users: - - * the ability to set command line arguments to be given to the QEMU - binary - - * a ready to use QMP connection and interface, which can be used to - send commands and inspect its results, as well as asynchronous - events - - * convenience methods to set commonly used command line arguments in - a more succinct and intuitive way - -QEMU binary selection -^^^^^^^^^^^^^^^^^^^^^ - -The QEMU binary used for the ``self.vm`` QEMUMachine instance will -primarily depend on the value of the ``qemu_bin`` parameter. If it's -not explicitly set, its default value will be the result of a dynamic -probe in the same source tree. A suitable binary will be one that -targets the architecture matching host machine. - -Based on this description, test writers will usually rely on one of -the following approaches: - -1) Set ``qemu_bin``, and use the given binary - -2) Do not set ``qemu_bin``, and use a QEMU binary named like - "qemu-system-${arch}", either in the current - working directory, or in the current source tree. - -The resulting ``qemu_bin`` value will be preserved in the -``avocado_qemu.QemuSystemTest`` as an attribute with the same name. - -Attribute reference -------------------- - -Test -^^^^ - -Besides the attributes and methods that are part of the base -``avocado.Test`` class, the following attributes are available on any -``avocado_qemu.QemuSystemTest`` instance. - -vm -"" - -A QEMUMachine instance, initially configured according to the given -``qemu_bin`` parameter. - -arch -"""" - -The architecture can be used on different levels of the stack, e.g. by -the framework or by the test itself. At the framework level, it will -currently influence the selection of a QEMU binary (when one is not -explicitly given). - -Tests are also free to use this attribute value, for their own needs. -A test may, for instance, use the same value when selecting the -architecture of a kernel or disk image to boot a VM with. - -The ``arch`` attribute will be set to the test parameter of the same -name. If one is not given explicitly, it will either be set to -``None``, or, if the test is tagged with one (and only one) -``:avocado: tags=arch:VALUE`` tag, it will be set to ``VALUE``. - -cpu -""" - -The cpu model that will be set to all QEMUMachine instances created -by the test. - -The ``cpu`` attribute will be set to the test parameter of the same -name. If one is not given explicitly, it will either be set to -``None ``, or, if the test is tagged with one (and only one) -``:avocado: tags=cpu:VALUE`` tag, it will be set to ``VALUE``. - -machine -""""""" - -The machine type that will be set to all QEMUMachine instances created -by the test. - -The ``machine`` attribute will be set to the test parameter of the same -name. If one is not given explicitly, it will either be set to -``None``, or, if the test is tagged with one (and only one) -``:avocado: tags=machine:VALUE`` tag, it will be set to ``VALUE``. - -qemu_bin -"""""""" - -The preserved value of the ``qemu_bin`` parameter or the result of the -dynamic probe for a QEMU binary in the current working directory or -source tree. - -LinuxTest -^^^^^^^^^ - -Besides the attributes present on the ``avocado_qemu.QemuSystemTest`` base -class, the ``avocado_qemu.LinuxTest`` adds the following attributes: - -distro -"""""" - -The name of the Linux distribution used as the guest image for the -test. The name should match the **Provider** column on the list -of images supported by the avocado.utils.vmimage library: - -https://avocado-framework.readthedocs.io/en/latest/guides/writer/libs/vmimage.html#supported-images - -distro_version -"""""""""""""" - -The version of the Linux distribution as the guest image for the -test. The name should match the **Version** column on the list -of images supported by the avocado.utils.vmimage library: - -https://avocado-framework.readthedocs.io/en/latest/guides/writer/libs/vmimage.html#supported-images - -distro_checksum -""""""""""""""" - -The sha256 hash of the guest image file used for the test. - -If this value is not set in the code or by a test parameter (with the -same name), no validation on the integrity of the image will be -performed. - -Parameter reference -------------------- - -To understand how Avocado parameters are accessed by tests, and how -they can be passed to tests, please refer to:: - - https://avocado-framework.readthedocs.io/en/latest/guides/writer/chapters/writing.html#accessing-test-parameters - -Parameter values can be easily seen in the log files, and will look -like the following: - -.. code:: - - PARAMS (key=qemu_bin, path=*, default=./qemu-system-x86_64) => './qemu-system-x86_64 - -Test -^^^^ - -arch -"""" - -The architecture that will influence the selection of a QEMU binary -(when one is not explicitly given). - -Tests are also free to use this parameter value, for their own needs. -A test may, for instance, use the same value when selecting the -architecture of a kernel or disk image to boot a VM with. - -This parameter has a direct relation with the ``arch`` attribute. If -not given, it will default to None. - -cpu -""" - -The cpu model that will be set to all QEMUMachine instances created -by the test. - -machine -""""""" - -The machine type that will be set to all QEMUMachine instances created -by the test. - -qemu_bin -"""""""" - -The exact QEMU binary to be used on QEMUMachine. - -LinuxTest -^^^^^^^^^ - -Besides the parameters present on the ``avocado_qemu.QemuSystemTest`` base -class, the ``avocado_qemu.LinuxTest`` adds the following parameters: - -distro -"""""" - -The name of the Linux distribution used as the guest image for the -test. The name should match the **Provider** column on the list -of images supported by the avocado.utils.vmimage library: - -https://avocado-framework.readthedocs.io/en/latest/guides/writer/libs/vmimage.html#supported-images - -distro_version -"""""""""""""" - -The version of the Linux distribution as the guest image for the -test. The name should match the **Version** column on the list -of images supported by the avocado.utils.vmimage library: - -https://avocado-framework.readthedocs.io/en/latest/guides/writer/libs/vmimage.html#supported-images - -distro_checksum -""""""""""""""" - -The sha256 hash of the guest image file used for the test. - -If this value is not set in the code or by this parameter no -validation on the integrity of the image will be performed. - -Skipping tests --------------- - -The Avocado framework provides Python decorators which allow for easily skip -tests running under certain conditions. For example, on the lack of a binary -on the test system or when the running environment is a CI system. For further -information about those decorators, please refer to:: - - https://avocado-framework.readthedocs.io/en/latest/guides/writer/chapters/writing.html#skipping-tests - -While the conditions for skipping tests are often specifics of each one, there -are recurring scenarios identified by the QEMU developers and the use of -environment variables became a kind of standard way to enable/disable tests. - -Here is a list of the most used variables: - -AVOCADO_ALLOW_LARGE_STORAGE -^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Tests which are going to fetch or produce assets considered *large* are not -going to run unless that ``AVOCADO_ALLOW_LARGE_STORAGE=1`` is exported on -the environment. - -The definition of *large* is a bit arbitrary here, but it usually means an -asset which occupies at least 1GB of size on disk when uncompressed. - -SPEED -^^^^^ -Tests which have a long runtime will not be run unless ``SPEED=slow`` is -exported on the environment. - -The definition of *long* is a bit arbitrary here, and it depends on the -usefulness of the test too. A unique test is worth spending more time on, -small variations on existing tests perhaps less so. As a rough guide, -a test or set of similar tests which take more than 100 seconds to -complete. - -AVOCADO_ALLOW_UNTRUSTED_CODE -^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -There are tests which will boot a kernel image or firmware that can be -considered not safe to run on the developer's workstation, thus they are -skipped by default. The definition of *not safe* is also arbitrary but -usually it means a blob which either its source or build process aren't -public available. - -You should export ``AVOCADO_ALLOW_UNTRUSTED_CODE=1`` on the environment in -order to allow tests which make use of those kind of assets. - -AVOCADO_TIMEOUT_EXPECTED -^^^^^^^^^^^^^^^^^^^^^^^^ -The Avocado framework has a timeout mechanism which interrupts tests to avoid the -test suite of getting stuck. The timeout value can be set via test parameter or -property defined in the test class, for further details:: - - https://avocado-framework.readthedocs.io/en/latest/guides/writer/chapters/writing.html#setting-a-test-timeout - -Even though the timeout can be set by the test developer, there are some tests -that may not have a well-defined limit of time to finish under certain -conditions. For example, tests that take longer to execute when QEMU is -compiled with debug flags. Therefore, the ``AVOCADO_TIMEOUT_EXPECTED`` variable -has been used to determine whether those tests should run or not. - -QEMU_TEST_FLAKY_TESTS -^^^^^^^^^^^^^^^^^^^^^ -Some tests are not working reliably and thus are disabled by default. -This includes tests that don't run reliably on GitLab's CI which -usually expose real issues that are rarely seen on developer machines -due to the constraints of the CI environment. If you encounter a -similar situation then raise a bug and then mark the test as shown on -the code snippet below: - -.. code:: - - # See https://gitlab.com/qemu-project/qemu/-/issues/nnnn - @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab') - def test(self): - do_something() - -You can also add ``:avocado: tags=flaky`` to the test meta-data so -only the flaky tests can be run as a group: - -.. code:: - - env QEMU_TEST_FLAKY_TESTS=1 ./pyvenv/bin/avocado \ - run tests/avocado -filter-by-tags=flaky - -Tests should not live in this state forever and should either be fixed -or eventually removed. - - -Uninstalling Avocado --------------------- - -If you've followed the manual installation instructions above, you can -easily uninstall Avocado. Start by listing the packages you have -installed:: - - pip list --user - -And remove any package you want with:: - - pip uninstall <package_name> - -If you've used ``make check-avocado``, the Python virtual environment where -Avocado is installed will be cleaned up as part of ``make check-clean``. diff --git a/docs/devel/testing/ci-definitions.rst.inc b/docs/devel/testing/ci-definitions.rst.inc deleted file mode 100644 index 6d5c6fd..0000000 --- a/docs/devel/testing/ci-definitions.rst.inc +++ /dev/null @@ -1,121 +0,0 @@ -Definition of terms -=================== - -This section defines the terms used in this document and correlates them with -what is currently used on QEMU. - -Automated tests ---------------- - -An automated test is written on a test framework using its generic test -functions/classes. The test framework can run the tests and report their -success or failure [1]_. - -An automated test has essentially three parts: - -1. The test initialization of the parameters, where the expected parameters, - like inputs and expected results, are set up; -2. The call to the code that should be tested; -3. An assertion, comparing the result from the previous call with the expected - result set during the initialization of the parameters. If the result - matches the expected result, the test has been successful; otherwise, it has - failed. - -Unit testing ------------- - -A unit test is responsible for exercising individual software components as a -unit, like interfaces, data structures, and functionality, uncovering errors -within the boundaries of a component. The verification effort is in the -smallest software unit and focuses on the internal processing logic and data -structures. A test case of unit tests should be designed to uncover errors due -to erroneous computations, incorrect comparisons, or improper control flow [2]_. - -On QEMU, unit testing is represented by the 'check-unit' target from 'make'. - -Functional testing ------------------- - -A functional test focuses on the functional requirement of the software. -Deriving sets of input conditions, the functional tests should fully exercise -all the functional requirements for a program. Functional testing is -complementary to other testing techniques, attempting to find errors like -incorrect or missing functions, interface errors, behavior errors, and -initialization and termination errors [3]_. - -On QEMU, functional testing is represented by the 'check-qtest' target from -'make'. - -System testing --------------- - -System tests ensure all application elements mesh properly while the overall -functionality and performance are achieved [4]_. Some or all system components -are integrated to create a complete system to be tested as a whole. System -testing ensures that components are compatible, interact correctly, and -transfer the right data at the right time across their interfaces. As system -testing focuses on interactions, use case-based testing is a practical approach -to system testing [5]_. Note that, in some cases, system testing may require -interaction with third-party software, like operating system images, databases, -networks, and so on. - -On QEMU, system testing is represented by the 'check-avocado' target from -'make'. - -Flaky tests ------------ - -A flaky test is defined as a test that exhibits both a passing and a failing -result with the same code on different runs. Some usual reasons for an -intermittent/flaky test are async wait, concurrency, and test order dependency -[6]_. - -Gating ------- - -A gate restricts the move of code from one stage to another on a -test/deployment pipeline. The step move is granted with approval. The approval -can be a manual intervention or a set of tests succeeding [7]_. - -On QEMU, the gating process happens during the pull request. The approval is -done by the project leader running its own set of tests. The pull request gets -merged when the tests succeed. - -Continuous Integration (CI) ---------------------------- - -Continuous integration (CI) requires the builds of the entire application and -the execution of a comprehensive set of automated tests every time there is a -need to commit any set of changes [8]_. The automated tests can be composed of -the unit, functional, system, and other tests. - -Keynotes about continuous integration (CI) [9]_: - -1. System tests may depend on external software (operating system images, - firmware, database, network). -2. It may take a long time to build and test. It may be impractical to build - the system being developed several times per day. -3. If the development platform is different from the target platform, it may - not be possible to run system tests in the developer’s private workspace. - There may be differences in hardware, operating system, or installed - software. Therefore, more time is required for testing the system. - -References ----------- - -.. [1] Sommerville, Ian (2016). Software Engineering. p. 233. -.. [2] Pressman, Roger S. & Maxim, Bruce R. (2020). Software Engineering, - A Practitioner’s Approach. p. 48, 376, 378, 381. -.. [3] Pressman, Roger S. & Maxim, Bruce R. (2020). Software Engineering, - A Practitioner’s Approach. p. 388. -.. [4] Pressman, Roger S. & Maxim, Bruce R. (2020). Software Engineering, - A Practitioner’s Approach. Software Engineering, p. 377. -.. [5] Sommerville, Ian (2016). Software Engineering. p. 59, 232, 240. -.. [6] Luo, Qingzhou, et al. An empirical analysis of flaky tests. - Proceedings of the 22nd ACM SIGSOFT International Symposium on - Foundations of Software Engineering. 2014. -.. [7] Humble, Jez & Farley, David (2010). Continuous Delivery: - Reliable Software Releases Through Build, Test, and Deployment, p. 122. -.. [8] Humble, Jez & Farley, David (2010). Continuous Delivery: - Reliable Software Releases Through Build, Test, and Deployment, p. 55. -.. [9] Sommerville, Ian (2016). Software Engineering. p. 743. diff --git a/docs/devel/testing/ci-jobs.rst.inc b/docs/devel/testing/ci-jobs.rst.inc index 3756bbe..f1c541c 100644 --- a/docs/devel/testing/ci-jobs.rst.inc +++ b/docs/devel/testing/ci-jobs.rst.inc @@ -126,10 +126,10 @@ QEMU_JOB_PUBLISH The job is for publishing content after a branch has been merged into the upstream default branch. -QEMU_JOB_AVOCADO -~~~~~~~~~~~~~~~~ +QEMU_JOB_FUNCTIONAL +~~~~~~~~~~~~~~~~~~~ -The job runs the Avocado integration test suite +The job runs the functional test suite Contributor controlled runtime variables ---------------------------------------- @@ -149,13 +149,12 @@ the jobs to be manually started from the UI Set this variable to 2 to create the pipelines and run all the jobs immediately, as was the historical behaviour -QEMU_CI_AVOCADO_TESTING -~~~~~~~~~~~~~~~~~~~~~~~ -By default, tests using the Avocado framework are not run automatically in -the pipelines (because multiple artifacts have to be downloaded, and if -these artifacts are not already cached, downloading them make the jobs -reach the timeout limit). Set this variable to have the tests using the -Avocado framework run automatically. +QEMU_CI_FUNCTIONAL +~~~~~~~~~~~~~~~~~~ +By default, tests using the functional framework are not run automatically +in the pipelines (because multiple artifacts have to be downloaded, which +might cause a lot of network traffic). Set this variable to have the tests +using the functional framework run automatically. Other misc variables -------------------- diff --git a/docs/devel/testing/ci.rst b/docs/devel/testing/ci.rst index ed88a20..e21d39d 100644 --- a/docs/devel/testing/ci.rst +++ b/docs/devel/testing/ci.rst @@ -1,14 +1,34 @@ .. _ci: -== -CI -== +Continuous Integration (CI) +=========================== + +Continuous integration (CI) requires the builds of the entire application and +the execution of a comprehensive set of automated tests every time there is a +need to commit any set of changes [1]_. The automated tests are composed +of unit, functional and other tests. Most of QEMU's CI is run on GitLab's infrastructure although a number of other CI services are used for specialised purposes. The most up to date information about them and their status can be found on the `project wiki testing page <https://wiki.qemu.org/Testing/CI>`_. -.. include:: ci-definitions.rst.inc +These tests are also used as gating tests before merging pull requests. +A gating test restricts the move of code from one stage to another on a +test/deployment pipeline. The step move is granted with approval. The approval +can be a manual intervention or a set of tests succeeding [2]_. + +On QEMU, the gating process happens during the pull request. The approval is +done by the project leader running its own set of tests. The pull request gets +merged when the tests succeed. + .. include:: ci-jobs.rst.inc .. include:: ci-runners.rst.inc + +References +---------- + +.. [1] Humble, Jez & Farley, David (2010). Continuous Delivery: + Reliable Software Releases Through Build, Test, and Deployment, p. 55. +.. [2] Humble, Jez & Farley, David (2010). Continuous Delivery: + Reliable Software Releases Through Build, Test, and Deployment, p. 122. diff --git a/docs/devel/testing/functional.rst b/docs/devel/testing/functional.rst index 9bc9733..8030cb4 100644 --- a/docs/devel/testing/functional.rst +++ b/docs/devel/testing/functional.rst @@ -6,9 +6,6 @@ Functional testing with Python The ``tests/functional`` directory hosts functional tests written in Python. They are usually higher level tests, and may interact with external resources and with various guest operating systems. -The functional tests have initially evolved from the Avocado tests, so there -is a lot of similarity to those tests here (see :ref:`checkavocado-ref` for -details about the Avocado tests). The tests should be written in the style of the Python `unittest`_ framework, using stdio for the TAP protocol. The folder ``tests/functional/qemu_test`` diff --git a/docs/devel/testing/index.rst b/docs/devel/testing/index.rst index 1171f7d..ccc2fc6 100644 --- a/docs/devel/testing/index.rst +++ b/docs/devel/testing/index.rst @@ -10,7 +10,6 @@ testing infrastructure. main qtest functional - avocado acpi-bits ci fuzzing diff --git a/docs/devel/testing/main.rst b/docs/devel/testing/main.rst index 9869bcf..6b18ed8 100644 --- a/docs/devel/testing/main.rst +++ b/docs/devel/testing/main.rst @@ -5,19 +5,32 @@ Testing in QEMU QEMU's testing infrastructure is fairly complex as it covers everything from unit testing and exercising specific sub-systems all -the way to full blown acceptance tests. To get an overview of the +the way to full blown functional tests. To get an overview of the tests you can run ``make check-help`` from either the source or build tree. -Most (but not all) tests are also integrated into the meson build -system so can be run directly from the build tree, for example: - -.. code:: +Most (but not all) tests are also integrated as an automated test into +the meson build system so can be run directly from the build tree, +for example:: [./pyvenv/bin/]meson test --suite qemu:softfloat will run just the softfloat tests. +An automated test is written with one of the test frameworks using its +generic test functions/classes. The test framework can run the tests and +report their success or failure [1]_. + +An automated test has essentially three parts: + +1. The test initialization of the parameters, where the expected parameters, + like inputs and expected results, are set up; +2. The call to the code that should be tested; +3. An assertion, comparing the result from the previous call with the expected + result set during the initialization of the parameters. If the result + matches the expected result, the test has been successful; otherwise, it has + failed. + The rest of this document will cover the details for specific test groups. @@ -44,9 +57,17 @@ cannot find them. Unit tests ~~~~~~~~~~ -Unit tests, which can be invoked with ``make check-unit``, are simple C tests -that typically link to individual QEMU object files and exercise them by -calling exported functions. +A unit test is responsible for exercising individual software components as a +unit, like interfaces, data structures, and functionality, uncovering errors +within the boundaries of a component. The verification effort is in the +smallest software unit and focuses on the internal processing logic and data +structures. A test case of unit tests should be designed to uncover errors +due to erroneous computations, incorrect comparisons, or improper control +flow [2]_. + +In QEMU, unit tests can be invoked with ``make check-unit``. They are +simple C tests that typically link to individual QEMU object files and +exercise them by calling exported functions. If you are writing new code in QEMU, consider adding a unit test, especially for utility modules that are relatively stateless or have few dependencies. To @@ -885,6 +906,10 @@ changing the ``-c`` option. Functional tests using Python ----------------------------- +A functional test focuses on the functional requirement of the software, +attempting to find errors like incorrect functions, interface errors, +behavior errors, and initialization and termination errors [3]_. + The ``tests/functional`` directory hosts functional tests written in Python. You can run the functional tests simply by executing: @@ -894,21 +919,6 @@ Python. You can run the functional tests simply by executing: See :ref:`checkfunctional-ref` for more details. -Integration tests using the Avocado Framework ---------------------------------------------- - -The ``tests/avocado`` directory hosts integration tests. They're usually -higher level tests, and may interact with external resources and with -various guest operating systems. - -You can run the avocado tests simply by executing: - -.. code:: - - make check-avocado - -See :ref:`checkavocado-ref` for more details. - .. _checktcg-ref: Testing with "make check-tcg" @@ -1023,3 +1033,27 @@ coverage-html`` which will create Further analysis can be conducted by running the ``gcov`` command directly on the various .gcda output files. Please read the ``gcov`` documentation for more information. + +Flaky tests +----------- + +A flaky test is defined as a test that exhibits both a passing and a failing +result with the same code on different runs. Some usual reasons for an +intermittent/flaky test are async wait, concurrency, and test order dependency +[4]_. + +In QEMU, tests that are identified to be flaky are normally disabled by +default. Set the QEMU_TEST_FLAKY_TESTS environment variable before running +the tests to enable them. + +References +---------- + +.. [1] Sommerville, Ian (2016). Software Engineering. p. 233. +.. [2] Pressman, Roger S. & Maxim, Bruce R. (2020). Software Engineering, + A Practitioner’s Approach. p. 48, 376, 378, 381. +.. [3] Pressman, Roger S. & Maxim, Bruce R. (2020). Software Engineering, + A Practitioner’s Approach. p. 388. +.. [4] Luo, Qingzhou, et al. An empirical analysis of flaky tests. + Proceedings of the 22nd ACM SIGSOFT International Symposium on + Foundations of Software Engineering. 2014. diff --git a/docs/devel/virtio-backends.rst b/docs/devel/virtio-backends.rst index 679d754..ebddc3b 100644 --- a/docs/devel/virtio-backends.rst +++ b/docs/devel/virtio-backends.rst @@ -119,7 +119,7 @@ manually instantiated: qdev_realize(vdev, BUS(&vpci_dev->bus), errp); } - static void virtio_blk_pci_class_init(ObjectClass *klass, void *data) + static void virtio_blk_pci_class_init(ObjectClass *klass, const void *data) { DeviceClass *dc = DEVICE_CLASS(klass); VirtioPCIClass *k = VIRTIO_PCI_CLASS(klass); diff --git a/docs/igd-assign.txt b/docs/igd-assign.txt index e17bb50..3aed795 100644 --- a/docs/igd-assign.txt +++ b/docs/igd-assign.txt @@ -1,44 +1,69 @@ Intel Graphics Device (IGD) assignment with vfio-pci ==================================================== -IGD has two different modes for assignment using vfio-pci: - -1) Universal Pass-Through (UPT) mode: - - In this mode the IGD device is added as a *secondary* (ie. non-primary) - graphics device in combination with an emulated primary graphics device. - This mode *requires* guest driver support to remove the external - dependencies generally associated with IGD (see below). Those guest - drivers only support this mode for Broadwell and newer IGD, according to - Intel. Additionally, this mode by default, and as officially supported - by Intel, does not support direct video output. The intention is to use - this mode either to provide hardware acceleration to the emulated graphics - or to use this mode in combination with guest-based remote access software, - for example VNC (see below for optional output support). This mode - theoretically has no device specific handling dependencies on vfio-pci or - the VM firmware. - -2) "Legacy" mode: - - In this mode the IGD device is intended to be the primary and exclusive - graphics device in the VM[1], as such QEMU does not facilitate any sort - of remote graphics to the VM in this mode. A connected physical monitor - is the intended output device for IGD. This mode includes several - requirements and restrictions: - - * IGD must be given address 02.0 on the PCI root bus in the VM - * The host kernel must support vfio extensions for IGD (v4.6) - * vfio VGA support very likely needs to be enabled in the host kernel - * The VM firmware must support specific fw_cfg enablers for IGD - * The VM machine type must support a PCI host bridge at 00.0 (standard) - * The VM machine type must provide or allow to be created a special - ISA/LPC bridge device (vfio-pci-igd-lpc-bridge) on the root bus at - PCI address 1f.0. - * The IGD device must have a VGA ROM, either provided via the romfile - option or loaded automatically through vfio (standard). rombar=0 - will disable legacy mode support. - * Hotplug of the IGD device is not supported. - * The IGD device must be a SandyBridge or newer model device. +Using vfio-pci, we can passthrough Intel Graphics Device (IGD) to guest, either +serve as primary and exclusive graphics adapter, or used in combination with an +emulated primary graphics device, depending on the config and guest driver +support. However, IGD devices are not "clean" PCI devices, they use extra +memory regions other than BARs. Special handling is required to make them work +properly, including: + +* OpRegion for accessing Virtual BIOS Table (VBT) that contains display output + information. +* Data Stolen Memory (DSM) region used as VRAM at early stage (BIOS/UEFI) + +Certain guest software also depends on following conditions to work: +(*-Required by) + +| Condition | Linux | Windows | VBIOS | EFI GOP | +|---------------------------------------------|-------|---------|-------|---------| +| #1 IGD has a valid OpRegion containing VBT | * ^1 | * | * | * | +| #2 VID/DID of LPC bridge at 00:1f.0 matches | | | * | * | +| #3 IGD is assigned to BDF 00:02.0 | | | * | * | +| #4 IGD has VGA controller device class | | | * | * | +| #5 Host's VGA ranges are mapped to IGD | | | * | | +| #6 Guest has valid VBIOS or UEFI Option ROM | | | * | * | + +^1 Though i915 driver is able to mock a OpRegion, it is still recommended to + use the VBT copied from host OpRegion to prevent incorrect configuration. + +For #1, the "x-igd-opregion=on" option exposes a copy of host IGD OpRegion to +guest via fw_cfg, where guest firmware can set up guest OpRegion with it. + +For #2, "x-igd-lpc=on" option copies the IDs of host LPC bridge and host bridge +to guest. Currently this is only supported on i440fx machines as there is +already an ICH9 LPC bridge present on q35 machines, overwriting its IDs may +lead to unexpected behavior. + +For #3, "addr=2.0" assigns IGD to 00:02.0. + +For #4, the primary display must be set to IGD in host BIOS. + +For #5, "x-vga=on" enables guest access to standard VGA IO/MMIO ranges. + +For #6, ROM either provided via the ROM BAR or romfile= option is needed, this +Intel document [1] shows how to dump VBIOS to file. For UEFI Option ROM, see +"Guest firmware" section. + +QEMU also provides a "Legacy" mode that implicitly enables full functionality +on IGD, it is automatically enabled when +* Machine type is i440fx +* IGD is assigned to guest BDF 00:02.0 +* ROM BAR or romfile is present + +In "Legacy" mode, QEMU will automatically setup OpRegion, LPC bridge IDs and +VGA range access, which is equivalent to: + x-igd-opregion=on,x-igd-lpc=on,x-vga=on + +By default, "Legacy" mode won't fail, it continues on error. User can set +"x-igd-legacy-mode=on" to force enabling legacy mode, this also checks if the +conditions above for legacy mode is met, and if any error occurs, QEMU will +fail immediately. Users can also set "x-igd-legacy-mode=off" to disable legacy +mode. + +In legacy mode, as the guest VGA ranges are assigned to IGD device, all other +graphics devices should be removed, this can be done using "-nographic" or +"-vga none" or "-nodefaults", along with adding the device using vfio-pci. For either mode, depending on the host kernel, the i915 driver in the host may generate faults and errors upon re-binding to an IGD device after it @@ -73,31 +98,39 @@ DVI, or DisplayPort) may be unsupported in some use cases. In the author's experience, even DP to VGA adapters can be troublesome while adapters between digital formats work well. -Usage -===== -The intention is for IGD assignment to be transparent for users and thus for -management tools like libvirt. To make use of legacy mode, simply remove all -other graphics options and use "-nographic" and either "-vga none" or -"-nodefaults", along with adding the device using vfio-pci: - -device vfio-pci,host=00:02.0,id=hostdev0,bus=pci.0,addr=0x2 +Options +======= +* x-igd-opregion=[on|*off*] + Copy host IGD OpRegion and expose it to guest with fw_cfg + +* x-igd-lpc=[on|*off*] + Creates a dummy LPC bridge at 00:1f:0 with host VID/DID (i440fx only) + +* x-igd-legacy-mode=[on|off|*auto*] + Enable/Disable legacy mode + +* x-igd-gms=[hex, default 0] + Overriding DSM region size in GGC register, 0 means uses host value. + Use this only when the DSM size cannot be changed through the + 'DVMT Pre-Allocated' option in host BIOS. -For UPT mode, retain the default emulated graphics and simply add the vfio-pci -device making use of any other bus address other than 02.0. libvirt will -default to assigning the device a UPT compatible address while legacy mode -users will need to manually edit the XML if using a tool like virt-manager -where the VM device address is not expressly specified. -An experimental vfio-pci option also exists to enable OpRegion, and thus -external monitor support, for UPT mode. This can be enabled by adding -"x-igd-opregion=on" to the vfio-pci device options for the IGD device. As -with legacy mode, this requires the host to support features introduced in -the v4.6 kernel. If Intel chooses to embrace this support, the option may -be made non-experimental in the future, opening it to libvirt support. +Examples +======== +* Adding IGD with automatically legacy mode support + -device vfio-pci,host=00:02.0,id=hostdev0,addr=2.0 -Developer ABI -============= -Legacy mode IGD support imposes two fw_cfg requirements on the VM firmware: +* Adding IGD with OpRegion and LPC ID hack, but without VGA ranges + (For UEFI guests) + -device vfio-pci,host=00:02.0,id=hostdev0,addr=2.0,x-igd-legacy-mode=off,x-igd-opregion=on,x-igd-lpc=on,romfile=efi_oprom.rom + + +Guest firmware +============== +Guest firmware is responsible for setting up OpRegion and Base of Data Stolen +Memory (BDSM) in guest address space. IGD passthrough support imposes two +fw_cfg requirements on the VM firmware: 1) "etc/igd-opregion" @@ -117,17 +150,111 @@ Legacy mode IGD support imposes two fw_cfg requirements on the VM firmware: Firmware must allocate a reserved memory below 4GB with required 1MB alignment equal to this size. Additionally the base address of this reserved region must be written to the dword BDSM register in PCI config - space of the IGD device at offset 0x5C. As this support is related to - running the IGD ROM, which has other dependencies on the device appearing - at guest address 00:02.0, it's expected that this fw_cfg file is only - relevant to a single PCI class VGA device with Intel vendor ID, appearing - at PCI bus address 00:02.0. + space of the IGD device at offset 0x5C (or 0xC0 for Gen 11+ devices using + 64-bit BDSM). As this support is related to running the IGD ROM, which + has other dependencies on the device appearing at guest address 00:02.0, + it's expected that this fw_cfg file is only relevant to a single PCI + class VGA device with Intel vendor ID, appearing at PCI bus address 00:02.0. + +Upstream Seabios has OpRegion and BDSM (pre-Gen11 device only) support. +However, the support is not accepted by upstream EDK2/OVMF. A recommended +solution is to create a virtual OpRom with following DXE drivers: + +* IgdAssignmentDxe: Set up OpRegion and BDSM according to fw_cfg (must) +* IntelGopDriver: Closed-source Intel GOP driver +* PlatformGopPolicy: Protocol required by IntelGopDriver + +IntelGopDriver and PlatformGopPolicy is only required when enabling GOP on IGD. + +The original IgdAssignmentDxe can be found at [3]. A Intel maintained version +with PlatformGopPolicy for industrial computing is at [4]. There is also an +unofficially maintained version with newer Gen11+ device support at [5]. +You need to build them with EDK2. + +For the IntelGopDriver, Intel never released it to public. You may contact +Intel support to get one as [4] said, if you are an Intel Premier Support +customer, or you can try extracting it from your host firmware using +"UEFI BIOS Updater"[6]. + +Once you got all the required DXE drivers, a Option ROM can be generated with +EfiRom utility in EDK2, using + EfiRom -f 0x8086 -i <Device ID of your IGD> -o output.rom \ + -e IgdAssignmentDxe.efi PlatformGOPPolicy.efi IntelGopDriver.efi + + +Known issues +============ +When using OVMF as guest firmware, you may encounter the following warning: +warning: vfio_container_dma_map(0x55fab36ce610, 0x380010000000, 0x108000, 0x7fd336000000) = -22 (Invalid argument) + +Solution: +Set the host physical address bits to IOMMU address width using + -cpu host,host-phys-bits-limit=<IOMMU address width> +Or in libvirt XML with + <cpu> + <maxphysaddr mode='passthrough' limit='<IOMMU address width>'/> + </cpu> +The IOMMU address width can be determined with + echo $(( ((0x$(cat /sys/devices/virtual/iommu/dmar0/intel-iommu/cap) & 0x3F0000) >> 16) + 1 )) +Refer https://edk2.groups.io/g/devel/topic/patch_v1/102359124 for more details + + +Memory View +=========== +IGD has it own address space. To use system RAM as VRAM, a single-level page +table named Global Graphics Translation Table (GTT) is used for the address +translation. Each page table entry points a 4KB page. Illustration below shows +the translation flow on IGD with 64-bit GTT PTEs. + +(PTE_SIZE == 8) +-------------+---+ + | Address | V | V: Valid Bit + +-------------+---+ + | ... | | +IGD:0x01ae9010 0xd740| 0x70ffc000 | 1 | Mem:0x42ba3e010^ +-----------------------> 0xd748| 0x42ba3e000 | 1 +------------------> +(addr >> 12) * PTE_SIZE 0xd750| 0x42ba3f000 | 1 | + | ... | | + +-------------+---+ +^ The address may be remapped by IOMMU + +The memory region store GTT is called GTT Stolen Memory (GSM) it is located +right below the Data Stolen Memory (DSM). Accessing this region directly is +not allowed, any access will immediately freeze the whole system. The only way +to access it is through the second half of MMIO BAR0. + +The Data Stolen Memory is reserved by firmware, and acts as the VRAM in pre-OS +environments. In QEMU, guest firmware (Seabios/OVMF) is responsible for +reserving a continuous region and program its base address to BDSM register, +then let VBIOS/GOP driver initializing this region. Illustration below shows +how DSM is mapped. + + IGD Addr Space Host Addr Space Guest Addr Space + +-------------+ +-------------+ +-------------+ + | | | | | | + | | | | | | + | | +-------------+ +-------------+ + | | | Data Stolen | | Data Stolen | + | | | (Guest) | | (Guest) | + | | +------------>+-------------+<------->+-------------+<--Guest BDSM + | | | Passthrough | | EPT | | Emulated by QEMU +DSMSIZE+-------------+ | with IOMMU | | Mapping | | Programmed by guest FW + | | | | | | | + | | | | | | | + 0+-------------+--+ | | | | + | +-------------+ | | + | | Data Stolen | +-------------+ + | | (Host) | + +------------>+-------------+<--Host BDSM + Non- | | "real" one in HW + Passthrough | | Programmed by host FW + +-------------+ Footnotes ========= -[1] Nothing precludes adding additional emulated or assigned graphics devices - as non-primary, other than the combination typically not working. I only - intend to set user expectations, others are welcome to find working - combinations or fix whatever issues prevent this from working in the common - case. +[1] https://www.intel.com/content/www/us/en/docs/graphics-for-linux/developer-reference/1-0/dump-video-bios.html [2] # echo "vfio-pci" > /sys/bus/pci/devices/0000:00:02.0/driver_override +[3] https://web.archive.org/web/20240827012422/https://bugzilla.tianocore.org/show_bug.cgi?id=935 + Tianocore bugzilla was down since Jan 2025 :( +[4] https://eci.intel.com/docs/3.3/components/kvm-hypervisor.html, Patch 0001-0004 +[5] https://github.com/tomitamoeko/VfioIgdPkg +[6] https://winraid.level1techs.com/t/tool-guide-news-uefi-bios-updater-ubu/30357 |