aboutsummaryrefslogtreecommitdiff
path: root/docs
diff options
context:
space:
mode:
authorPeter Maydell <peter.maydell@linaro.org>2018-03-20 15:48:34 +0000
committerPeter Maydell <peter.maydell@linaro.org>2018-03-20 15:48:34 +0000
commited627b2ad37469eeba9e9ed5fecfe315df9ecc60 (patch)
treeaca1c6bddbaa61ffe2d029b123539fe20e6ecddc /docs
parent4aafb1b192e5d3685e94cefdce63343a86d64647 (diff)
parent1dc61e7b37d339c42ec9bd7a7eec1ef2c22f351c (diff)
downloadqemu-ed627b2ad37469eeba9e9ed5fecfe315df9ecc60.zip
qemu-ed627b2ad37469eeba9e9ed5fecfe315df9ecc60.tar.gz
qemu-ed627b2ad37469eeba9e9ed5fecfe315df9ecc60.tar.bz2
Merge remote-tracking branch 'remotes/mst/tags/for_upstream' into staging
virtio,vhost,pci,pc: features, cleanups SRAT tables for DIMM devices new virtio net flags for speed/duplex post-copy migration support in vhost cleanups in pci Signed-off-by: Michael S. Tsirkin <mst@redhat.com> # gpg: Signature made Tue 20 Mar 2018 14:40:43 GMT # gpg: using RSA key 281F0DB8D28D5469 # gpg: Good signature from "Michael S. Tsirkin <mst@kernel.org>" # gpg: aka "Michael S. Tsirkin <mst@redhat.com>" # Primary key fingerprint: 0270 606B 6F3C DF3D 0B17 0970 C350 3912 AFBE 8E67 # Subkey fingerprint: 5D09 FD08 71C8 F85B 94CA 8A0D 281F 0DB8 D28D 5469 * remotes/mst/tags/for_upstream: (51 commits) postcopy shared docs libvhost-user: Claim support for postcopy postcopy: Allow shared memory vhost: Huge page align and merge vhost+postcopy: Wire up POSTCOPY_END notify vhost-user: Add VHOST_USER_POSTCOPY_END message libvhost-user: mprotect & madvises for postcopy vhost+postcopy: Call wakeups vhost+postcopy: Add vhost waker postcopy: postcopy_notify_shared_wake postcopy: helper for waking shared vhost+postcopy: Resolve client address postcopy-ram: add a stub for postcopy_request_shared_page vhost+postcopy: Helper to send requests to source for shared pages vhost+postcopy: Stash RAMBlock and offset vhost+postcopy: Send address back to qemu libvhost-user+postcopy: Register new regions with the ufd migration/ram: ramblock_recv_bitmap_test_byte_offset postcopy+vhost-user: Split set_mem_table for postcopy vhost+postcopy: Transmit 'listen' to slave ... Signed-off-by: Peter Maydell <peter.maydell@linaro.org> # Conflicts: # scripts/update-linux-headers.sh
Diffstat (limited to 'docs')
-rw-r--r--docs/devel/migration.rst41
-rw-r--r--docs/interop/vhost-user.txt52
2 files changed, 93 insertions, 0 deletions
diff --git a/docs/devel/migration.rst b/docs/devel/migration.rst
index 9d1b765..e32b087 100644
--- a/docs/devel/migration.rst
+++ b/docs/devel/migration.rst
@@ -577,3 +577,44 @@ Postcopy now works with hugetlbfs backed memory:
hugepages works well, however 1GB hugepages are likely to be problematic
since it takes ~1 second to transfer a 1GB hugepage across a 10Gbps link,
and until the full page is transferred the destination thread is blocked.
+
+Postcopy with shared memory
+---------------------------
+
+Postcopy migration with shared memory needs explicit support from the other
+processes that share memory and from QEMU. There are restrictions on the type of
+memory that userfault can support shared.
+
+The Linux kernel userfault support works on `/dev/shm` memory and on `hugetlbfs`
+(although the kernel doesn't provide an equivalent to `madvise(MADV_DONTNEED)`
+for hugetlbfs which may be a problem in some configurations).
+
+The vhost-user code in QEMU supports clients that have Postcopy support,
+and the `vhost-user-bridge` (in `tests/`) and the DPDK package have changes
+to support postcopy.
+
+The client needs to open a userfaultfd and register the areas
+of memory that it maps with userfault. The client must then pass the
+userfaultfd back to QEMU together with a mapping table that allows
+fault addresses in the clients address space to be converted back to
+RAMBlock/offsets. The client's userfaultfd is added to the postcopy
+fault-thread and page requests are made on behalf of the client by QEMU.
+QEMU performs 'wake' operations on the client's userfaultfd to allow it
+to continue after a page has arrived.
+
+.. note::
+ There are two future improvements that would be nice:
+ a) Some way to make QEMU ignorant of the addresses in the clients
+ address space
+ b) Avoiding the need for QEMU to perform ufd-wake calls after the
+ pages have arrived
+
+Retro-fitting postcopy to existing clients is possible:
+ a) A mechanism is needed for the registration with userfault as above,
+ and the registration needs to be coordinated with the phases of
+ postcopy. In vhost-user extra messages are added to the existing
+ control channel.
+ b) Any thread that can block due to guest memory accesses must be
+ identified and the implication understood; for example if the
+ guest memory access is made while holding a lock then all other
+ threads waiting for that lock will also be blocked.
diff --git a/docs/interop/vhost-user.txt b/docs/interop/vhost-user.txt
index cb3a759..c058c40 100644
--- a/docs/interop/vhost-user.txt
+++ b/docs/interop/vhost-user.txt
@@ -290,6 +290,15 @@ Once the source has finished migration, rings will be stopped by
the source. No further update must be done before rings are
restarted.
+In postcopy migration the slave is started before all the memory has been
+received from the source host, and care must be taken to avoid accessing pages
+that have yet to be received. The slave opens a 'userfault'-fd and registers
+the memory with it; this fd is then passed back over to the master.
+The master services requests on the userfaultfd for pages that are accessed
+and when the page is available it performs WAKE ioctl's on the userfaultfd
+to wake the stalled slave. The client indicates support for this via the
+VHOST_USER_PROTOCOL_F_PAGEFAULT feature.
+
Memory access
-------------
@@ -369,6 +378,7 @@ Protocol features
#define VHOST_USER_PROTOCOL_F_SLAVE_REQ 5
#define VHOST_USER_PROTOCOL_F_CROSS_ENDIAN 6
#define VHOST_USER_PROTOCOL_F_CRYPTO_SESSION 7
+#define VHOST_USER_PROTOCOL_F_PAGEFAULT 8
Master message types
--------------------
@@ -445,12 +455,21 @@ Master message types
Id: 5
Equivalent ioctl: VHOST_SET_MEM_TABLE
Master payload: memory regions description
+ Slave payload: (postcopy only) memory regions description
Sets the memory map regions on the slave so it can translate the vring
addresses. In the ancillary data there is an array of file descriptors
for each memory mapped region. The size and ordering of the fds matches
the number and ordering of memory regions.
+ When VHOST_USER_POSTCOPY_LISTEN has been received, SET_MEM_TABLE replies with
+ the bases of the memory mapped regions to the master. The slave must
+ have mmap'd the regions but not yet accessed them and should not yet generate
+ a userfault event. Note NEED_REPLY_MASK is not set in this case.
+ QEMU will then reply back to the list of mappings with an empty
+ VHOST_USER_SET_MEM_TABLE as an acknowledgment; only upon reception of this
+ message may the guest start accessing the memory and generating faults.
+
* VHOST_USER_SET_LOG_BASE
Id: 6
@@ -689,6 +708,39 @@ Master message types
feature has been successfully negotiated.
It's a required feature for crypto devices.
+ * VHOST_USER_POSTCOPY_ADVISE
+ Id: 28
+ Master payload: N/A
+ Slave payload: userfault fd
+
+ When VHOST_USER_PROTOCOL_F_PAGEFAULT is supported, the
+ master advises slave that a migration with postcopy enabled is underway,
+ the slave must open a userfaultfd for later use.
+ Note that at this stage the migration is still in precopy mode.
+
+ * VHOST_USER_POSTCOPY_LISTEN
+ Id: 29
+ Master payload: N/A
+
+ Master advises slave that a transition to postcopy mode has happened.
+ The slave must ensure that shared memory is registered with userfaultfd
+ to cause faulting of non-present pages.
+
+ This is always sent sometime after a VHOST_USER_POSTCOPY_ADVISE, and
+ thus only when VHOST_USER_PROTOCOL_F_PAGEFAULT is supported.
+
+ * VHOST_USER_POSTCOPY_END
+ Id: 30
+ Slave payload: u64
+
+ Master advises that postcopy migration has now completed. The
+ slave must disable the userfaultfd. The response is an acknowledgement
+ only.
+ When VHOST_USER_PROTOCOL_F_PAGEFAULT is supported, this message
+ is sent at the end of the migration, after VHOST_USER_POSTCOPY_LISTEN
+ was previously sent.
+ The value returned is an error indication; 0 is success.
+
Slave message types
-------------------