aboutsummaryrefslogtreecommitdiff
path: root/docs/testing.md
blob: adf2a5557f87d5d6f2d61e8b3a9844d4e3b42eda (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
Testing
=======

The tests in libvfio-user are organized into a number of suites

* unit - old unit tests written in C
* functional - old functional tests using binaries written in C
* pyunit - modern tests written in python
* style - code style checking (eg flake8, rstlint, etc)

Running `meson test -C build` runs all the suites:

```
meson build
meson test -C build
```

It is possible to be selective about which tests are run
by including or excluding suites:

```
meson test -C build --suite=pyunit
meson test -C build --no-suite=pyunit
```

The `unit` and `pyunit` suites support optional execution
under valgrind

```
meson test -C build --suite=unit --setup=valgrind
meson test -C build --suite=pyunit --setup=pyvalgrind
```

To run with ASAN enabled, pass the `-Db_sanitize=address`
option to meson. Note, this is incompatible with enabling
valgrind

```
meson build -Db_sanitize=address
meson test -C build
```

The `.github/workflows/pull_request.sh` script runs a
sequence of builds in various configurations. This is
invoked for all pull requests, but can be launched
manually by contributors ahead of opening a pull
request.

The master branch is run through [Coverity](scan.coverity.com) when a new PR
lands.

Coverage reports can be enabled via meson, provided `gcovr` is installed:

```
meson build -Db_coverage=true
meson test -C build
ninja -C build coverage
```

The resulting coverage report can be viewed with

```
firefox file:`pwd`/build/meson-logs/coveragereport/index.html
```

Debugging Test Errors
---------------------

Sometimes debugging Valgrind errors on Python unit tests can be tricky. To
run specific tests, pass their name on the command line to `meson`:

```
meson build
meson test -C build test_quiesce.py
```

For even better control , use something like the following:
```
LIBVFIO_SO_DIR='/root/src/libvfio-user/build/lib' pytest-3 test/py/test_quiesce.py
```

To print libvfio-user's log messages, append the `--capture=tee-sys` option.

AFL++
-----

You can run [American Fuzzy Lop](https://github.com/AFLplusplus/AFLplusplus)
against `libvfio-user`. It's easiest to use the Docker container:

```
cd /path/to/libvfio-user/src
docker pull aflplusplus/aflplusplus
docker run -ti -v $(pwd):/src aflplusplus/aflplusplus
```

Set up and build:

```
apt update
apt-get -y install libjson-c-dev libcmocka-dev clang valgrind \
                   python3-pytest debianutils flake8 meson

cd /src
export AFL_LLVM_LAF_ALL=1
CC=afl-clang-fast meson build -Dtran-pipe=true
ninja -C build

mkdir inputs
# don't yet have a better starting point
echo "1" >inputs/start
mkdir outputs
```

The `VFU_TRAN_PIPE` is a special `libvfio-user` transport that reads from
`stdin` instead of a socket, we'll use this with the sample server to do our
fuzzing:

```
afl-fuzz -i inputs/ -o outputs/ -- ./build/dbg/samples/server pipe
```