diff options
Diffstat (limited to 'docs/devel/migration')
| -rw-r--r-- | docs/devel/migration/CPR.rst | 117 | ||||
| -rw-r--r-- | docs/devel/migration/main.rst | 23 | ||||
| -rw-r--r-- | docs/devel/migration/postcopy.rst | 36 | ||||
| -rw-r--r-- | docs/devel/migration/vfio.rst | 35 |
4 files changed, 178 insertions, 33 deletions
diff --git a/docs/devel/migration/CPR.rst b/docs/devel/migration/CPR.rst index 7897873..b617856 100644 --- a/docs/devel/migration/CPR.rst +++ b/docs/devel/migration/CPR.rst @@ -5,7 +5,7 @@ CPR is the umbrella name for a set of migration modes in which the VM is migrated to a new QEMU instance on the same host. It is intended for use when the goal is to update host software components that run the VM, such as QEMU or even the host kernel. At this time, -the cpr-reboot and cpr-transfer modes are available. +the cpr-reboot, cpr-transfer, and cpr-exec modes are available. Because QEMU is restarted on the same host, with access to the same local devices, CPR is allowed in certain cases where normal migration @@ -152,8 +152,7 @@ cpr-transfer mode This mode allows the user to transfer a guest to a new QEMU instance on the same host with minimal guest pause time, by preserving guest RAM in place, albeit with new virtual addresses in new QEMU. Devices -and their pinned memory pages will also be preserved in a future QEMU -release. +and their pinned memory pages are also preserved for VFIO and IOMMUFD. The user starts new QEMU on the same host as old QEMU, with command- line arguments to create the same machine, plus the ``-incoming`` @@ -322,6 +321,116 @@ Futures cpr-transfer mode is based on a capability to transfer open file descriptors from old to new QEMU. In the future, descriptors for -vfio, iommufd, vhost, and char devices could be transferred, +vhost, and char devices could be transferred, preserving those devices and their kernel state without interruption, even if they do not explicitly support live migration. + +cpr-exec mode +------------- + +In this mode, QEMU stops the VM, writes VM state to the migration +URI, and directly exec's a new version of QEMU on the same host, +replacing the original process while retaining its PID. Guest RAM is +preserved in place, albeit with new virtual addresses. The user +completes the migration by specifying the ``-incoming`` option, and +by issuing the ``migrate-incoming`` command if necessary; see details +below. + +This mode supports VFIO/IOMMUFD devices by preserving device +descriptors and hence kernel state across the exec, even for devices +that do not support live migration. + +Because the old and new QEMU instances are not active concurrently, +the URI cannot be a type that streams data from one instance to the +other. + +This mode does not require a channel of type ``cpr``. The information +that is passed over that channel for cpr-transfer mode is instead +serialized to a memfd, the number of the fd is saved in the +QEMU_CPR_EXEC_STATE environment variable during the exec of new QEMU. +and new QEMU mmaps the memfd. + +Usage +^^^^^ + +Arguments for the new QEMU process are taken from the +@cpr-exec-command parameter. The first argument should be the +path of a new QEMU binary, or a prefix command that exec's the +new QEMU binary, and the arguments should include the ''-incoming'' +option. + +Memory backend objects must have the ``share=on`` attribute. +The VM must be started with the ``-machine aux-ram-share=on`` option. + +Outgoing: + * Set the migration mode parameter to ``cpr-exec``. + * Set the ``cpr-exec-command`` parameter. + * Issue the ``migrate`` command. It is recommended that the URI be + a ``file`` type, but one can use other types such as ``exec``, + provided the command captures all the data from the outgoing side, + and provides all the data to the incoming side. + +Incoming: + * You do not need to explicitly start new QEMU. It is started as + a side effect of the migrate command above. + * If the VM was running when the outgoing ``migrate`` command was + issued, then QEMU automatically resumes VM execution. + +Example 1: incoming URI +^^^^^^^^^^^^^^^^^^^^^^^ + +In these examples, we simply restart the same version of QEMU, but in +a real scenario one would set a new QEMU binary path in +cpr-exec-command. + +:: + + # qemu-kvm -monitor stdio + -object memory-backend-memfd,id=ram0,size=4G + -machine memory-backend=ram0 + -machine aux-ram-share=on + ... + + QEMU 10.2.50 monitor - type 'help' for more information + (qemu) info status + VM status: running + (qemu) migrate_set_parameter mode cpr-exec + (qemu) migrate_set_parameter cpr-exec-command qemu-kvm ... -incoming file:vm.state + (qemu) migrate -d file:vm.state + (qemu) QEMU 10.2.50 monitor - type 'help' for more information + (qemu) info status + VM status: running + +Example 2: incoming defer +^^^^^^^^^^^^^^^^^^^^^^^^^ +:: + + # qemu-kvm -monitor stdio + -object memory-backend-memfd,id=ram0,size=4G + -machine memory-backend=ram0 + -machine aux-ram-share=on + ... + + QEMU 10.2.50 monitor - type 'help' for more information + (qemu) info status + VM status: running + (qemu) migrate_set_parameter mode cpr-exec + (qemu) migrate_set_parameter cpr-exec-command qemu-kvm ... -incoming defer + (qemu) migrate -d file:vm.state + (qemu) QEMU 10.2.50 monitor - type 'help' for more information + (qemu) info status + status: paused (inmigrate) + (qemu) migrate_incoming file:vm.state + (qemu) info status + VM status: running + +Caveats +^^^^^^^ + +cpr-exec mode may not be used with postcopy, background-snapshot, +or COLO. + +cpr-exec mode requires permission to use the exec system call, which +is denied by certain sandbox options, such as spawn. + +The guest pause time increases for large guest RAM backed by small pages. diff --git a/docs/devel/migration/main.rst b/docs/devel/migration/main.rst index cdd4f4a..1afe7b9 100644 --- a/docs/devel/migration/main.rst +++ b/docs/devel/migration/main.rst @@ -444,6 +444,25 @@ The functions to do that are inside a vmstate definition, and are called: This function is called after we save the state of one device (even upon failure, unless the call to pre_save returned an error). +Following are the errp variants of these functions. + +- ``int (*pre_load_errp)(void *opaque, Error **errp);`` + + This function is called before we load the state of one device. + +- ``int (*post_load_errp)(void *opaque, int version_id, Error **errp);`` + + This function is called after we load the state of one device. + +- ``int (*pre_save_errp)(void *opaque, Error **errp);`` + + This function is called before we save the state of one device. + +New impls should preferentally use 'errp' variants of these +methods and existing impls incrementally converted. +The variants without 'errp' are intended to be removed +once all usage is converted. + Example: You can look at hpet.c, that uses the first three functions to massage the state that is transferred. @@ -508,8 +527,8 @@ An iterative device must provide: the point that stream bandwidth limits tell it to stop. Each call generates one section. - - A ``save_live_complete_precopy`` function that must transmit the - last section for the device containing any remaining data. + - A ``save_complete`` function that must transmit the last section for + the device containing any remaining data. - A ``load_state`` function used to load sections generated by any of the save functions that generate sections. diff --git a/docs/devel/migration/postcopy.rst b/docs/devel/migration/postcopy.rst index 82e7a84..e319388 100644 --- a/docs/devel/migration/postcopy.rst +++ b/docs/devel/migration/postcopy.rst @@ -33,25 +33,6 @@ will now cause the transition from precopy to postcopy. It can be issued immediately after migration is started or any time later on. Issuing it after the end of a migration is harmless. -Blocktime is a postcopy live migration metric, intended to show how -long the vCPU was in state of interruptible sleep due to pagefault. -That metric is calculated both for all vCPUs as overlapped value, and -separately for each vCPU. These values are calculated on destination -side. To enable postcopy blocktime calculation, enter following -command on destination monitor: - -``migrate_set_capability postcopy-blocktime on`` - -Postcopy blocktime can be retrieved by query-migrate qmp command. -postcopy-blocktime value of qmp command will show overlapped blocking -time for all vCPU, postcopy-vcpu-blocktime will show list of blocking -time per vCPU. - -.. note:: - During the postcopy phase, the bandwidth limits set using - ``migrate_set_parameter`` is ignored (to avoid delaying requested pages that - the destination is waiting for). - Postcopy internals ================== @@ -312,3 +293,20 @@ explicitly) to be sent in a separate preempt channel, rather than queued in the background migration channel. Anyone who cares about latencies of page faults during a postcopy migration should enable this feature. By default, it's not enabled. + +Postcopy blocktime statistics +----------------------------- + +Blocktime is a postcopy live migration metric, intended to show how +long the vCPU was in state of interruptible sleep due to pagefault. +That metric is calculated both for all vCPUs as overlapped value, and +separately for each vCPU. These values are calculated on destination +side. To enable postcopy blocktime calculation, enter following +command on destination monitor: + +``migrate_set_capability postcopy-blocktime on`` + +Postcopy blocktime can be retrieved by query-migrate qmp command. +postcopy-blocktime value of qmp command will show overlapped blocking +time for all vCPU, postcopy-vcpu-blocktime will show list of blocking +time per vCPU. diff --git a/docs/devel/migration/vfio.rst b/docs/devel/migration/vfio.rst index 673e354..0790e50 100644 --- a/docs/devel/migration/vfio.rst +++ b/docs/devel/migration/vfio.rst @@ -75,12 +75,12 @@ VFIO implements the device hooks for the iterative approach as follows: in the non-multifd mode. In the multifd mode it just emits either a dummy EOS marker. -* A ``save_live_complete_precopy`` function that sets the VFIO device in - _STOP_COPY state and iteratively copies the data for the VFIO device until - the vendor driver indicates that no data remains. - In the multifd mode it just emits a dummy EOS marker. +* A ``save_complete`` function that sets the VFIO device in _STOP_COPY + state and iteratively copies the data for the VFIO device until the + vendor driver indicates that no data remains. In the multifd mode it + just emits a dummy EOS marker. -* A ``save_live_complete_precopy_thread`` function that in the multifd mode +* A ``save_complete_precopy_thread`` function that in the multifd mode provides thread handler performing multifd device state transfer. It sets the VFIO device to _STOP_COPY state, iteratively reads the data from the VFIO device and queues it for multifd transmission until the vendor @@ -195,12 +195,12 @@ Live migration save path | Then the VFIO device is put in _STOP_COPY state (FINISH_MIGRATE, _ACTIVE, _STOP_COPY) - .save_live_complete_precopy() is called for each active device + .save_complete() is called for each active device For the VFIO device: in the non-multifd mode iterate in - .save_live_complete_precopy() until + .save_complete() until pending data is 0 In the multifd mode this iteration is done in - .save_live_complete_precopy_thread() instead. + .save_complete_precopy_thread() instead. | (POSTMIGRATE, _COMPLETED, _STOP_COPY) Migraton thread schedules cleanup bottom half and exits @@ -247,3 +247,22 @@ The multifd VFIO device state transfer is controlled by "x-migration-multifd-transfer" VFIO device property. This property defaults to AUTO, which means that VFIO device state transfer via multifd channels is attempted in configurations that otherwise support it. + +Since the target QEMU needs to load device state buffers in-order it needs to +queue incoming buffers until they can be loaded into the device. +This means that a malicious QEMU source could theoretically cause the target +QEMU to allocate unlimited amounts of memory for such buffers-in-flight. + +The "x-migration-max-queued-buffers-size" property allows capping the total size +of these VFIO device state buffers queued at the destination. + +Because a malicious QEMU source causing OOM on the target is not expected to be +a realistic threat in most of VFIO live migration use cases and the right value +depends on the particular setup by default this queued buffers size limit is +disabled by setting it to UINT64_MAX. + +Some host platforms (like ARM64) require that VFIO device config is loaded only +after all iterables were loaded, during non-iterables loading phase. +Such interlocking is controlled by "x-migration-load-config-after-iter" VFIO +device property, which in its default setting (AUTO) does so only on platforms +that actually require it. |