1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
|
Mediated User space device
==========================
Overview
--------
muser is a framework that allows mediated device drivers to be implemented in
user space. The device driver can by a completely virtual one without driving
an actual device of that type. This can greatly simplify the initial
development and prototyping of kernel drivers as no kernel code needs to be
written, and failures result in the user space process crashing in the worst
case. The mediated device can be passed to a virtual machine for proper
testing. Device drivers are typically implemented entirely in kernel space for
various reasons, however in early development stages it's acceptable to do it
in user space.
muser is implemented by a small kernel module, muser.ko, that registers itself
with mdev. Every request is forwarded to a user space application via a small,
custom ioctl interface on a control device. The application must be externally
provided and needs to contain the actual device implementation by using the API
of libmuser. See src/samples on how to build such an application. Currently
there is a one, single-threaded application instance per device, however the
application can employ any form of concurrency needed. In the future we plan to
make libmuser multi-threaded. The application can be implemented in whatever
way is convenient, e.g. as a Python script using bindings, on the cloud, etc.
Memory Mapping the Device
-------------------------
The device driver can allow parts of the virtual device to be memory mapped by
the virtual machine (e.g. the PCI BARs). The business logic needs to implement
the mmap callback and reply to the request passing the memory address whose
backing pages are then used to satisfy the original mmap call. Currently
reading and writing of the memory mapped memory by the client goes undetected
by libmuser, the business logic needs to poll. In the future we plan to
implement a mechanism in order to provide notifications to libmuser whenever a
page is written to.
Interrupts
----------
Interrupts are implemented by installing the event file descriptor in libmuser
and then notifying it about it. libmuser can then trigger interrupts simply by
writing to it. This can be much more expensive compared to triggering interrupts
from the kernel, however this performance penalty is perfectly acceptable when
prototyping the functional aspect of a device driver.
System Architecture
-------------------
muser.ko and libmuser communicate via ioctl on a control device. This control
device is create when the mediated device is created and appears as
/dev/muser/<UUID>. libmuser opens this device and then executes a "wait
command" ioctl. Whenever a callback of muser.ko is executed, it fills a struct
with the command details and then completes the ioctl, unblocking libmuser. It
then waits to receive another ioctl from libmuser with the result. Currently
there can be only one command pending, we plan to allow multiple commands to be
executed in parallel.
Building muser
==============
vfio/mdev needs to be patched. To generate the patch run:
git diff 869e3305f23dfeacdaa234717c92ccb237815d90 --diff-filter=M > vfio.patch
Apply the patch and rebuild the vfio/mdev modules:
make SUBDIRS=drivers/vfio/ modules
Reload the relevant kernel modules:
drivers/vfio/vfio_iommu_type1.ko
drivers/vfio/vfio.ko
drivers/vfio/mdev/mdev.ko
drivers/vfio/mdev/vfio_mdev.ko
Build the kernel module:
cd src/kmod
make
Build the library:
mkdir build
cd build
cmake ..
make
make install
Finally build your program and link it to libmuser.so.
Running QEMU
============
To pass the device to QEMU add the following options:
-device vfio-pci,sysfsdev=/sys/bus/mdev/devices/<UUID>
-object memory-backend-file,id=ram-node0,prealloc=yes,mem-path=mem,share=yes,size=1073741824 -numa node,nodeid=0,cpus=0,memdev=ram-node0
Guest RAM must be shared (share=yes) otherwise libmuser won't be able to do DMA
transfers from/to it. If you're not using QEMU then any memory that must be
accessed by libmuser must be allocate MAP_SHARED. Registering memory for DMA
that has not been allocated with MAP_SHARED is ignored and any attempts to
access that memory will result in an error.
Example
=======
samples/gpio-pci-idio-16.c implements a tiny part of the PCI-IDIO-16 GPIO
(https://www.accesio.com/?p=/pci/pci_idio_16.html). In this sample it's a simple
device that toggles the input every 3 times it's read.
Running gpio-pci-idio-16
------------------------
First, follow the instructions to build and load muser.
Then, start the gpio-pci-idio-16 device emulation:
# echo 00000000-0000-0000-0000-000000000000 > /sys/class/muser/muser/mdev_supported_types/muser-1/create
# build/dbg/samples/gpio-pci-idio-16 00000000-0000-0000-0000-000000000000
Finally, start the VM adding the command line explained earlier and then
execute:
# insmod gpio-pci-idio-16.ko
# cat /sys/class/gpio/gpiochip480/base > /sys/class/gpio/export
# for ((i=0;i<12;i++)); do cat /sys/class/gpio/OUT0/value; done
0
0
0
1
1
1
0
0
0
1
1
1
Future Work
===========
Making libmuser Restartable
----------------------------
muser can be made restartable so that (a) it can recover from failures, and
(b) upgrades are less disrupting. This is something we plan to implement in the
future. To make it restarable muser needs to reconfigure eventfds and DMA
region mmaps first thing when the device is re-opened by libmuser. After muser
has finished reconfiguring it will send a "ready" command, after which normal
operation will be resumed. This "ready" command will always be sent when the
device is opened, even if this is the first time, as this way we don't need to
differentiate between normal operation and restarted operation. libmuser will
store the PCI BAR on /dev/shm (named after e.g. the device UUID) so that it can
easily find them on restart.
Making libmuser Multi-threaded
-------------------------------
libmuser can be made multi-threaded in order to improve performance. To
implement this we'll have to maintain a private context in struct file.
|
