diff options
Diffstat (limited to 'docs')
-rw-r--r-- | docs/about/emulation.rst | 199 | ||||
-rw-r--r-- | docs/devel/build-system.rst | 2 | ||||
-rw-r--r-- | docs/devel/code-provenance.rst | 33 | ||||
-rw-r--r-- | docs/devel/memory.rst | 17 | ||||
-rw-r--r-- | docs/devel/rust.rst | 2 | ||||
-rw-r--r-- | docs/system/devices/igb.rst | 2 |
6 files changed, 230 insertions, 25 deletions
diff --git a/docs/about/emulation.rst b/docs/about/emulation.rst index 456d01d..8a5e128 100644 --- a/docs/about/emulation.rst +++ b/docs/about/emulation.rst @@ -816,6 +816,205 @@ This plugin can limit the number of Instructions Per Second that are executed:: The lower the number the more accurate time will be, but the less efficient the plugin. Defaults to ips/10 +Uftrace +....... + +``contrib/plugins/uftrace.c`` + +This plugin generates a binary trace compatible with +`uftrace <https://github.com/namhyung/uftrace>`_. + +Plugin supports aarch64 and x64, and works in user and system mode, allowing to +trace a system boot, which is not something possible usually. + +In user mode, the memory mapping is directly copied from ``/proc/self/maps`` at +the end of execution. Uftrace should be able to retrieve symbols by itself, +without any additional step. +In system mode, the default memory mapping is empty, and you can generate +one (and associated symbols) using ``contrib/plugins/uftrace_symbols.py``. +Symbols must be present in ELF binaries. + +It tracks the call stack (based on frame pointer analysis). Thus, your program +and its dependencies must be compiled using ``-fno-omit-frame-pointer +-mno-omit-leaf-frame-pointer``. In 2024, `Ubuntu and Fedora enabled it by +default again on x64 +<https://www.brendangregg.com/blog/2024-03-17/the-return-of-the-frame-pointers.html>`_. +On aarch64, this is less of a problem, as they are usually part of the ABI, +except for leaf functions. That's true for user space applications, but not +necessarily for bare metal code. You can read this `section +<uftrace_build_system_example>` to easily build a system with frame pointers. + +When tracing long scenarios (> 1 min), the generated trace can become very long, +making it hard to extract data from it. In this case, a simple solution is to +trace execution while generating a timestamped output log using +``qemu-system-aarch64 ... | ts "%s"``. Then, ``uftrace --time-range=start~end`` +can be used to reduce trace for only this part of execution. + +Performance wise, overhead compared to normal tcg execution is around x5-x15. + +.. list-table:: Uftrace plugin arguments + :widths: 20 80 + :header-rows: 1 + + * - Option + - Description + * - trace-privilege-level=[on|off] + - Generate separate traces for each privilege level (Exception Level + + Security State on aarch64, Rings on x64). + +.. list-table:: uftrace_symbols.py arguments + :widths: 20 80 + :header-rows: 1 + + * - Option + - Description + * - elf_file [elf_file ...] + - path to an ELF file. Use /path/to/file:0xdeadbeef to add a mapping offset. + * - --prefix-symbols + - prepend binary name to symbols + +Example user trace +++++++++++++++++++ + +As an example, we can trace qemu itself running git:: + + $ ./build/qemu-aarch64 -plugin \ + build/contrib/plugins/libuftrace.so \ + ./build/qemu-aarch64 /usr/bin/git --help + + # and generate a chrome trace directly + $ uftrace dump --chrome | gzip > ~/qemu_aarch64_git_help.json.gz + +For convenience, you can download this trace `qemu_aarch64_git_help.json.gz +<https://fileserver.linaro.org/s/N8X8fnZ5yGRZLsT/download/qemu_aarch64_git_help.json.gz>`_. +Download it and open this trace on https://ui.perfetto.dev/. You can zoom in/out +using :kbd:`W`, :kbd:`A`, :kbd:`S`, :kbd:`D` keys. +Some sequences taken from this trace: + +- Loading program and its interpreter + +.. image:: https://fileserver.linaro.org/s/fie8JgX76yyL5cq/preview + :height: 200px + +- open syscall + +.. image:: https://fileserver.linaro.org/s/rsXPTeZZPza4PcE/preview + :height: 200px + +- TB creation + +.. image:: https://fileserver.linaro.org/s/GXY6NKMw5EeRCew/preview + :height: 200px + +It's usually better to use ``uftrace record`` directly. However, tracing +binaries through qemu-user can be convenient when you don't want to recompile +them (``uftrace record`` requires instrumentation), as long as symbols are +present. + +Example system trace +++++++++++++++++++++ + +A full trace example (chrome trace, from instructions below) generated from a +system boot can be found `here +<https://fileserver.linaro.org/s/WsemLboPEzo24nw/download/aarch64_boot.json.gz>`_. +Download it and open this trace on https://ui.perfetto.dev/. You can see code +executed for all privilege levels, and zoom in/out using +:kbd:`W`, :kbd:`A`, :kbd:`S`, :kbd:`D` keys. You can find below some sequences +taken from this trace: + +- Two first stages of boot sequence in Arm Trusted Firmware (EL3 and S-EL1) + +.. image:: https://fileserver.linaro.org/s/kkxBS552W7nYESX/preview + :height: 200px + +- U-boot initialization (until code relocation, after which we can't track it) + +.. image:: https://fileserver.linaro.org/s/LKTgsXNZFi5GFNC/preview + :height: 200px + +- Stat and open syscalls in kernel + +.. image:: https://fileserver.linaro.org/s/dXe4MfraKg2F476/preview + :height: 200px + +- Timer interrupt + +.. image:: https://fileserver.linaro.org/s/TM5yobYzJtP7P3C/preview + :height: 200px + +- Poweroff sequence (from kernel back to firmware, NS-EL2 to EL3) + +.. image:: https://fileserver.linaro.org/s/oR2PtyGKJrqnfRf/preview + :height: 200px + +Build and run system example +++++++++++++++++++++++++++++ + +.. _uftrace_build_system_example: + +Building a full system image with frame pointers is not trivial. + +We provide a `simple way <https://github.com/pbo-linaro/qemu-linux-stack>`_ to +build an aarch64 system, combining Arm Trusted firmware, U-boot, Linux kernel +and debian userland. It's based on containers (``podman`` only) and +``qemu-user-static (binfmt)`` to make sure it's easily reproducible and does not depend +on machine where you build it. + +You can follow the exact same instructions for a x64 system, combining edk2, +Linux, and Ubuntu, simply by switching to +`x86_64 <https://github.com/pbo-linaro/qemu-linux-stack/tree/x86_64>`_ branch. + +To build the system:: + + # Install dependencies + $ sudo apt install -y podman qemu-user-static + + $ git clone https://github.com/pbo-linaro/qemu-linux-stack + $ cd qemu-linux-stack + $ ./build.sh + + # system can be started using: + $ ./run.sh /path/to/qemu-system-aarch64 + +To generate a uftrace for a system boot from that:: + + # run true and poweroff the system + $ env INIT=true ./run.sh path/to/qemu-system-aarch64 \ + -plugin path/to/contrib/plugins/libuftrace.so,trace-privilege-level=on + + # generate symbols and memory mapping + $ path/to/contrib/plugins/uftrace_symbols.py \ + --prefix-symbols \ + arm-trusted-firmware/build/qemu/debug/bl1/bl1.elf \ + arm-trusted-firmware/build/qemu/debug/bl2/bl2.elf \ + arm-trusted-firmware/build/qemu/debug/bl31/bl31.elf \ + u-boot/u-boot:0x60000000 \ + linux/vmlinux + + # inspect trace with + $ uftrace replay + +Uftrace allows to filter the trace, and dump flamegraphs, or a chrome trace. +This last one is very interesting to see visually the boot process:: + + $ uftrace dump --chrome > boot.json + # Open your browser, and load boot.json on https://ui.perfetto.dev/. + +Long visual chrome traces can't be easily opened, thus, it might be +interesting to generate them around a particular point of execution:: + + # execute qemu and timestamp output log + $ env INIT=true ./run.sh path/to/qemu-system-aarch64 \ + -plugin path/to/contrib/plugins/libuftrace.so,trace-privilege-level=on |& + ts "%s" | tee exec.log + + $ cat exec.log | grep 'Run /init' + 1753122320 [ 11.834391] Run /init as init process + # init was launched at 1753122320 + + # generate trace around init execution (2 seconds): + $ uftrace dump --chrome --time-range=1753122320~1753122322 > init.json + Other emulation features ------------------------ diff --git a/docs/devel/build-system.rst b/docs/devel/build-system.rst index 2c88419..6204aa6 100644 --- a/docs/devel/build-system.rst +++ b/docs/devel/build-system.rst @@ -450,7 +450,7 @@ are run with ``make bench``. Meson test suites such as ``unit`` can be ran with ``make check-unit``, and ``make check-tcg`` builds and runs "non-Meson" tests for all targets. -If desired, it is also possible to use ``ninja`` and ``meson test``, +If desired, it is also possible to use ``ninja`` and ``pyvenv/bin/meson test``, respectively to build emulators and run tests defined in meson.build. The main difference is that ``make`` needs the ``-jN`` flag in order to enable parallel builds or tests. diff --git a/docs/devel/code-provenance.rst b/docs/devel/code-provenance.rst index b5aae2e..8cdc56f 100644 --- a/docs/devel/code-provenance.rst +++ b/docs/devel/code-provenance.rst @@ -285,8 +285,8 @@ Such tools are acceptable to use, provided there is clearly defined copyright and licensing for their output. Note in particular the caveats applying to AI content generators below. -Use of AI content generators -~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Use of AI-generated content +~~~~~~~~~~~~~~~~~~~~~~~~~~~ TL;DR: @@ -294,6 +294,10 @@ TL;DR: believed to include or derive from AI generated content. This includes ChatGPT, Claude, Copilot, Llama and similar tools.** + **This policy does not apply to other uses of AI, such as researching APIs + or algorithms, static analysis, or debugging, provided their output is not + included in contributions.** + The increasing prevalence of AI-assisted software development results in a number of difficult legal questions and risks for software projects, including QEMU. Of particular concern is content generated by `Large Language Models @@ -322,17 +326,24 @@ The QEMU project thus requires that contributors refrain from using AI content generators on patches intended to be submitted to the project, and will decline any contribution if use of AI is either known or suspected. -This policy does not apply to other uses of AI, such as researching APIs or -algorithms, static analysis, or debugging, provided their output is not to be -included in contributions. - Examples of tools impacted by this policy includes GitHub's CoPilot, OpenAI's ChatGPT, Anthropic's Claude, and Meta's Code Llama, and code/content generation agents which are built on top of such tools. This policy may evolve as AI tools mature and the legal situation is -clarifed. In the meanwhile, requests for exceptions to this policy will be -evaluated by the QEMU project on a case by case basis. To be granted an -exception, a contributor will need to demonstrate clarity of the license and -copyright status for the tool's output in relation to its training model and -code, to the satisfaction of the project maintainers. +clarified. + +Exceptions +^^^^^^^^^^ + +The QEMU project welcomes discussion on any exceptions to this policy, +or more general revisions. This can be done by contacting the qemu-devel +mailing list with details of a proposed tool, model, usage scenario, etc. +that is beneficial to QEMU, while still mitigating issues around compliance +with the DCO. After discussion, any exception will be listed below. + +Exceptions do not remove the need for authors to comply with all other +requirements for contribution. In particular, the "Signed-off-by" +label in a patch submission is a statement that the author takes +responsibility for the entire contents of the patch, including any parts +that were generated or assisted by AI tools or other tools. diff --git a/docs/devel/memory.rst b/docs/devel/memory.rst index 42d3ca2..f22146e 100644 --- a/docs/devel/memory.rst +++ b/docs/devel/memory.rst @@ -165,17 +165,14 @@ and finalized one by one. The order in which memory regions will be finalized is not guaranteed. If however the memory region is part of a dynamically allocated data -structure, you should call object_unparent() to destroy the memory region -before the data structure is freed. For an example see VFIOMSIXInfo -and VFIOQuirk in hw/vfio/pci.c. +structure, you should free the memory region in the instance_finalize +callback. For an example see VFIOMSIXInfo and VFIOQuirk in +hw/vfio/pci.c. You must not destroy a memory region as long as it may be in use by a device or CPU. In order to do this, as a general rule do not create or -destroy memory regions dynamically during a device's lifetime, and only -call object_unparent() in the memory region owner's instance_finalize -callback. The dynamically allocated data structure that contains the -memory region then should obviously be freed in the instance_finalize -callback as well. +destroy memory regions dynamically during a device's lifetime, and never +call object_unparent(). If you break this rule, the following situation can happen: @@ -201,9 +198,7 @@ this exception is rarely necessary, and therefore it is discouraged, but nevertheless it is used in a few places. For regions that "have no owner" (NULL is passed at creation time), the -machine object is actually used as the owner. Since instance_finalize is -never called for the machine object, you must never call object_unparent -on regions that have no owner, unless they are aliases or containers. +machine object is actually used as the owner. Overlapping regions and priority diff --git a/docs/devel/rust.rst b/docs/devel/rust.rst index 13a20e8..2f0ab2e 100644 --- a/docs/devel/rust.rst +++ b/docs/devel/rust.rst @@ -66,7 +66,7 @@ __ https://mesonbuild.com/Commands.html#devenv As shown above, you can use the ``--tests`` option as usual to operate on test code. Note however that you cannot *build* or run tests via ``cargo``, because they need support C code from QEMU that Cargo does not know about. Tests can -be run via ``meson test`` or ``make``:: +be run via Meson (``pyvenv/bin/meson test``) or ``make``:: make check-rust diff --git a/docs/system/devices/igb.rst b/docs/system/devices/igb.rst index 71f31cb..50f625f 100644 --- a/docs/system/devices/igb.rst +++ b/docs/system/devices/igb.rst @@ -54,7 +54,7 @@ directory: .. code-block:: shell - meson test qtest-x86_64/qos-test + pyvenv/bin/meson test qtest-x86_64/qos-test ethtool can test register accesses, interrupts, etc. It is automated as an functional test and can be run from the build directory with the following |