Initial commit

1 files changed, 134 insertions, 0 deletions

diff --git a/README.md b/README.md
new file mode 100644
index 0000000..b744c26
--- /dev/null
+++ b/README.md
@@ -0,0 +1,134 @@
+Mediated User space device
+==========================
+
+Overview
+--------
+
+muser is a framework that allows mediated device drivers to be implemented in
+user space. The device driver can by a completely virtual one without driving
+an actual device of that type. This can greatly simplify the initial
+development and prototyping of kernel drivers as no kernel code needs to be
+written, and failures result in the user space process crashing in the worst
+case. The mediated device can be passed to a virtual machine for proper
+testing. Device drivers are typically implemented entirely in kernel space for
+various reasons, however in early development stages it's acceptable to do it
+in user space.
+
+muser is implemented by a small kernel module, muser.ko, that registers itself
+with mdev. Every request is forwarded to a user space application via a small,
+custom ioctl interface on a control device. The application must be externally
+provided and needs to contain the actual device implementation by using the API
+of libmuser. See src/samples on how to build such an application.  Currently
+there is a one, single-threaded application instance per device, however the
+application can employ any form of concurrency needed. In the future we plan to
+make libmuser multi-threaded.  The application can be implemented in whatever
+way is convenient, e.g. as a Python script using bindings, on the cloud, etc.
+
+
+Memory Mapping the Device
+-------------------------
+
+The device driver can allow parts of the virtual device to be memory mapped by
+the virtual machine (e.g. the PCI BARs). The business logic needs to implement
+the mmap callback and reply to the request passing the memory address whose
+backing pages are then used to satisfy the original mmap call. Currently
+reading and writing of the memory mapped memory by the client goes undetected
+by libmuser, the business logic needs to poll. In the future we plan to
+implement a mechanism in order to provide notifications to libmuser whenever a
+page is written to.
+
+
+Interrupts
+----------
+
+Interrupts are implemented by installing the event file descriptor in libmuser
+and then notifying it about it. libmuser can then trigger interrupts simply by
+writing to it. This can be much more expensive compared to triggering interrupts
+from the kernel, however this performance penalty is perfectly acceptable when
+prototyping the functional aspect of a device driver.
+
+
+System Architecture
+-------------------
+
+muser.ko and libmuser communicate via ioctl on a control device. This control
+device is create when the mediated device is created and appears as
+/dev/muser/<UUID>. libmuser opens this device and then executes a "wait
+command" ioctl. Whenever a callback of muser.ko is executed, it fills a struct
+with the command details and then completes the ioctl, unblocking libmuser. It
+then waits to receive another ioctl from libmuser with the result. Currently
+there can be only one command pending, we plan to allow multiple commands to be
+executed in parallel.
+
+
+Building muser
+==============
+
+vfio/mdev needs to be patched. To generate the patch run:
+
+	git diff 869e3305f23dfeacdaa234717c92ccb237815d90 --diff-filter=M > vfio.patch
+
+Apply the patch and rebuild the vfio/mdev modules:
+
+	make SUBDIRS=drivers/vfio/ modules
+
+Reload the relevant kernel modules:
+
+	drivers/vfio/vfio_iommu_type1.ko
+	drivers/vfio/vfio.ko
+	drivers/vfio/mdev/mdev.ko
+	drivers/vfio/mdev/vfio_mdev.ko
+
+Build the kernel module:
+
+	cd src/kmod
+	make
+
+Build the library:
+
+	mkdir build
+	cd build
+	cmake ..
+	make
+	make install
+
+Finally build your program and link it to libmuser.so.
+
+Running QEMU
+============
+
+To pass the device to QEMU add the following options:
+
+		-device vfio-pci,sysfsdev=/sys/bus/mdev/devices/00000000-0000-0000-0000-000000000000
+		-object memory-backend-file,id=ram-node0,prealloc=yes,mem-path=mem,share=yes,size=1073741824 -numa node,nodeid=0,cpus=0,memdev=ram-node0
+
+Guest RAM must be shared (share=yes) otherwise libmuser won't be able to do DMA
+transfers from/to it. If you're not using QEMU then any memory that must be
+accessed by libmuser must be allocate MAP_SHARED. Registering memory for DMA
+that has not been allocated with MAP_SHARED is ignored and any attempts to
+access that memory will result in an error.
+
+
+Future Work
+===========
+
+Making libmuser Restartable
+----------------------------
+
+muser can be made restartable so that (a) it can recover from failures, and
+(b) upgrades are less disrupting. This is something we plan to implement in the
+future. To make it restarable muser needs to reconfigure eventfds and DMA
+region mmaps first thing when the device is re-opened by libmuser. After muser
+has finished reconfiguring it will send a "ready" command, after which normal
+operation will be resumed. This "ready" command will always be sent when the
+device is opened, even if this is the first time, as this way we don't need to
+differentiate between normal operation and restarted operation. libmuser will
+store the PCI BAR on /dev/shm (named after e.g. the device UUID) so that it can
+easily find them on restart.
+
+
+Making libmuser Multi-threaded
+-------------------------------
+
+libmuser can be made multi-threaded in order to improve performance. To
+implement this we'll have to maintain a private context in struct file.