Commit 41ac3c2a authored by Jason Gunthorpe's avatar Jason Gunthorpe
Browse files

Merge branch 'v6.6/vfio/cdev' of https://github.com/awilliam/linux-vfio into iommufd for-next 

Shared branch with VFIO containing the enablement for VFIO "cdev"
devices. This is required by following iommufd patches which add new
ioctls to the VFIO cdev.

=======
Existing VFIO provides group-centric user APIs for userspace. Userspace
opens the /dev/vfio/$group_id first before getting device fd and hence
getting access to device. This is not the desired model for iommufd. Per
the conclusion of community discussion[1], iommufd provides device-centric
kAPIs and requires its consumer (like VFIO) to be device-centric user
APIs. Such user APIs are used to associate device with iommufd and also
the I/O address spaces managed by the iommufd.

This series first introduces a per device file structure to be prepared
for further enhancement and refactors the kvm-vfio code to be prepared
for accepting device file from userspace. After this, adds a mechanism for
blocking device access before iommufd bind. Then refactors the vfio to be
able to handle cdev paths (e.g. iommufd binding, no-iommufd, [de]attach ioas).
This refactor includes making the device_open exclusive between the group
and the cdev path, only allow single device open in cdev path; vfio-iommufd
code is also refactored to support cdev. e.g. split the vfio_iommufd_bind()
into two steps. Eventually, adds the cdev support for vfio device and the
new ioctls, then makes group infrastructure optional as it is not needed
when vfio device cdev is compiled.

This series is based on some preparation works done to vfio emulated devices[2]
and vfio pci hot reset enhancements[3]. Per discussion[4], this series does not
support cdev for physical devices that do not have IOMMU. Such devices only
have group-centric user APIs.

This series is a prerequisite for iommu nesting for vfio device[5] [6].

[1] https://lore.kernel.org/kvm/BN9PR11MB5433B1E4AE5B0480369F97178C189@BN9PR11MB5433.namprd11.prod.outlook.com/
[2] https://lore.kernel.org/kvm/20230327093351.44505-1-yi.l.liu@intel.com/ - merged
[3] https://lore.kernel.org/kvm/20230718105542.4138-1-yi.l.liu@intel.com/
[4] https://lore.kernel.org/kvm/20230525095939.37ddb8ce.alex.williamson@redhat.com/
[5] https://lore.kernel.org/linux-iommu/20230511143844.22693-1-yi.l.liu@intel.com/
[6] https://lore.kernel.org/linux-iommu/20230511145110.27707-1-yi.l.liu@intel.com/#t
=======

* 'v6.6/vfio/cdev' of https://github.com/awilliam/linux-vfio: (36 commits)
  docs: vfio: Add vfio device cdev description
  vfio: Compile vfio_group infrastructure optionally
  vfio: Move the IOMMU_CAP_CACHE_COHERENCY check in __vfio_register_dev()
  vfio: Add VFIO_DEVICE_[AT|DE]TACH_IOMMUFD_PT
  vfio: Add VFIO_DEVICE_BIND_IOMMUFD
  vfio: Avoid repeated user pointer cast in vfio_device_fops_unl_ioctl()
  iommufd: Add iommufd_ctx_from_fd()
  vfio: Test kvm pointer in _vfio_device_get_kvm_safe()
  vfio: Add cdev for vfio_device
  vfio: Move device_del() before waiting for the last vfio_device registration refcount
  vfio: Move vfio_device_group_unregister() to be the first operation in unregister
  vfio-iommufd: Add detach_ioas support for emulated VFIO devices
  iommufd/device: Add iommufd_access_detach() API
  vfio-iommufd: Add detach_ioas support for physical VFIO devices
  vfio: Record devid in vfio_device_file
  vfio-iommufd: Split bind/attach into two steps
  vfio-iommufd: Move noiommu compat validation out of vfio_iommufd_bind()
  vfio: Make vfio_df_open() single open for device cdev path
  vfio: Add cdev_device_open_cnt to vfio_group
  vfio: Block device access via device fd until device is opened
  vfio: Pass struct vfio_device_file * to vfio_device_open/close()
  kvm/vfio: Accept vfio device file from userspace
  kvm/vfio: Prepare for accepting vfio device fd
  vfio: Accept vfio device file in the KVM facing kAPI
  vfio: Refine vfio file kAPIs for KVM
  vfio: Allocate per device file structure
  vfio/pci: Allow passing zero-length fd array in VFIO_DEVICE_PCI_HOT_RESET
  vfio/pci: Copy hot-reset device info to userspace in the devices loop
  vfio/pci: Extend VFIO_DEVICE_GET_PCI_HOT_RESET_INFO for vfio device cdev
  vfio: Add helper to search vfio_device in a dev_set
  ...
parents 6eaae198 09467130

Documentation/driver-api/vfio.rst

+144 −3
Original line number Diff line number Diff line
@@ -239,6 +239,137 @@ group and can access them as follows:: 
	/* Gratuitous device reset and go... */

	ioctl(device, VFIO_DEVICE_RESET);



IOMMUFD and vfio_iommu_type1

----------------------------



IOMMUFD is the new user API to manage I/O page tables from userspace.

It intends to be the portal of delivering advanced userspace DMA

features (nested translation [5]_, PASID [6]_, etc.) while also providing

a backwards compatibility interface for existing VFIO_TYPE1v2_IOMMU use

cases.  Eventually the vfio_iommu_type1 driver, as well as the legacy

vfio container and group model is intended to be deprecated.



The IOMMUFD backwards compatibility interface can be enabled two ways.

In the first method, the kernel can be configured with

CONFIG_IOMMUFD_VFIO_CONTAINER, in which case the IOMMUFD subsystem

transparently provides the entire infrastructure for the VFIO

container and IOMMU backend interfaces.  The compatibility mode can

also be accessed if the VFIO container interface, ie. /dev/vfio/vfio is

simply symlink'd to /dev/iommu.  Note that at the time of writing, the

compatibility mode is not entirely feature complete relative to

VFIO_TYPE1v2_IOMMU (ex. DMA mapping MMIO) and does not attempt to

provide compatibility to the VFIO_SPAPR_TCE_IOMMU interface.  Therefore

it is not generally advisable at this time to switch from native VFIO

implementations to the IOMMUFD compatibility interfaces.



Long term, VFIO users should migrate to device access through the cdev

interface described below, and native access through the IOMMUFD

provided interfaces.



VFIO Device cdev

----------------



Traditionally user acquires a device fd via VFIO_GROUP_GET_DEVICE_FD

in a VFIO group.



With CONFIG_VFIO_DEVICE_CDEV=y the user can now acquire a device fd

by directly opening a character device /dev/vfio/devices/vfioX where

"X" is the number allocated uniquely by VFIO for registered devices.

cdev interface does not support noiommu devices, so user should use

the legacy group interface if noiommu is wanted.



The cdev only works with IOMMUFD.  Both VFIO drivers and applications

must adapt to the new cdev security model which requires using

VFIO_DEVICE_BIND_IOMMUFD to claim DMA ownership before starting to

actually use the device.  Once BIND succeeds then a VFIO device can

be fully accessed by the user.



VFIO device cdev doesn't rely on VFIO group/container/iommu drivers.

Hence those modules can be fully compiled out in an environment

where no legacy VFIO application exists.



So far SPAPR does not support IOMMUFD yet.  So it cannot support device

cdev either.



vfio device cdev access is still bound by IOMMU group semantics, ie. there

can be only one DMA owner for the group.  Devices belonging to the same

group can not be bound to multiple iommufd_ctx or shared between native

kernel and vfio bus driver or other driver supporting the driver_managed_dma

flag.  A violation of this ownership requirement will fail at the

VFIO_DEVICE_BIND_IOMMUFD ioctl, which gates full device access.



Device cdev Example

-------------------



Assume user wants to access PCI device 0000:6a:01.0::



	$ ls /sys/bus/pci/devices/0000:6a:01.0/vfio-dev/

	vfio0



This device is therefore represented as vfio0.  The user can verify

its existence::



	$ ls -l /dev/vfio/devices/vfio0

	crw------- 1 root root 511, 0 Feb 16 01:22 /dev/vfio/devices/vfio0

	$ cat /sys/bus/pci/devices/0000:6a:01.0/vfio-dev/vfio0/dev

	511:0

	$ ls -l /dev/char/511\:0

	lrwxrwxrwx 1 root root 21 Feb 16 01:22 /dev/char/511:0 -> ../vfio/devices/vfio0



Then provide the user with access to the device if unprivileged

operation is desired::



	$ chown user:user /dev/vfio/devices/vfio0



Finally the user could get cdev fd by::



	cdev_fd = open("/dev/vfio/devices/vfio0", O_RDWR);



An opened cdev_fd doesn't give the user any permission of accessing

the device except binding the cdev_fd to an iommufd.  After that point

then the device is fully accessible including attaching it to an

IOMMUFD IOAS/HWPT to enable userspace DMA::



	struct vfio_device_bind_iommufd bind = {

		.argsz = sizeof(bind),

		.flags = 0,

	};

	struct iommu_ioas_alloc alloc_data  = {

		.size = sizeof(alloc_data),

		.flags = 0,

	};

	struct vfio_device_attach_iommufd_pt attach_data = {

		.argsz = sizeof(attach_data),

		.flags = 0,

	};

	struct iommu_ioas_map map = {

		.size = sizeof(map),

		.flags = IOMMU_IOAS_MAP_READABLE |

			 IOMMU_IOAS_MAP_WRITEABLE |

			 IOMMU_IOAS_MAP_FIXED_IOVA,

		.__reserved = 0,

	};



	iommufd = open("/dev/iommu", O_RDWR);



	bind.iommufd = iommufd;

	ioctl(cdev_fd, VFIO_DEVICE_BIND_IOMMUFD, &bind);



	ioctl(iommufd, IOMMU_IOAS_ALLOC, &alloc_data);

	attach_data.pt_id = alloc_data.out_ioas_id;

	ioctl(cdev_fd, VFIO_DEVICE_ATTACH_IOMMUFD_PT, &attach_data);



	/* Allocate some space and setup a DMA mapping */

	map.user_va = (int64_t)mmap(0, 1024 * 1024, PROT_READ | PROT_WRITE,

				    MAP_PRIVATE | MAP_ANONYMOUS, 0, 0);

	map.iova = 0; /* 1MB starting at 0x0 from device view */

	map.length = 1024 * 1024;

	map.ioas_id = alloc_data.out_ioas_id;;



	ioctl(iommufd, IOMMU_IOAS_MAP, &map);



	/* Other device operations as stated in "VFIO Usage Example" */



VFIO User API

-------------------------------------------------------------------------------
@@ -279,6 +410,7 @@ similar to a file operations structure:: 
					struct iommufd_ctx *ictx, u32 *out_device_id);

		void	(*unbind_iommufd)(struct vfio_device *vdev);

		int	(*attach_ioas)(struct vfio_device *vdev, u32 *pt_id);

		void	(*detach_ioas)(struct vfio_device *vdev);

		int	(*open_device)(struct vfio_device *vdev);

		void	(*close_device)(struct vfio_device *vdev);

		ssize_t	(*read)(struct vfio_device *vdev, char __user *buf,
@@ -315,9 +447,10 @@ container_of(). 
	- The [un]bind_iommufd callbacks are issued when the device is bound to

	  and unbound from iommufd.



	- The attach_ioas callback is issued when the device is attached to an

	  IOAS managed by the bound iommufd. The attached IOAS is automatically

	  detached when the device is unbound from iommufd.

	- The [de]attach_ioas callback is issued when the device is attached to

	  and detached from an IOAS managed by the bound iommufd. However, the

	  attached IOAS can also be automatically detached when the device is

	  unbound from iommufd.



	- The read/write/mmap callbacks implement the device region access defined

	  by the device's own VFIO_DEVICE_GET_REGION_INFO ioctl.
@@ -564,3 +697,11 @@ This implementation has some specifics: 
				\-0d.1



	00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 90)



.. [5] Nested translation is an IOMMU feature which supports two stage

   address translations.  This improves the address translation efficiency

   in IOMMU virtualization.



.. [6] PASID stands for Process Address Space ID, introduced by PCI

   Express.  It is a prerequisite for Shared Virtual Addressing (SVA)

   and Scalable I/O Virtualization (Scalable IOV).

Documentation/virt/kvm/devices/vfio.rst

+31 −16
Original line number Diff line number Diff line
@@ -9,22 +9,34 @@ Device types supported: 
  - KVM_DEV_TYPE_VFIO



Only one VFIO instance may be created per VM.  The created device

tracks VFIO groups in use by the VM and features of those groups

important to the correctness and acceleration of the VM.  As groups

are enabled and disabled for use by the VM, KVM should be updated

about their presence.  When registered with KVM, a reference to the

VFIO-group is held by KVM.

tracks VFIO files (group or device) in use by the VM and features

of those groups/devices important to the correctness and acceleration

of the VM.  As groups/devices are enabled and disabled for use by the

VM, KVM should be updated about their presence.  When registered with

KVM, a reference to the VFIO file is held by KVM.



Groups:

  KVM_DEV_VFIO_GROUP



KVM_DEV_VFIO_GROUP attributes:

  KVM_DEV_VFIO_GROUP_ADD: Add a VFIO group to VFIO-KVM device tracking

	kvm_device_attr.addr points to an int32_t file descriptor

	for the VFIO group.

  KVM_DEV_VFIO_GROUP_DEL: Remove a VFIO group from VFIO-KVM device tracking

	kvm_device_attr.addr points to an int32_t file descriptor

	for the VFIO group.

  KVM_DEV_VFIO_FILE

	alias: KVM_DEV_VFIO_GROUP



KVM_DEV_VFIO_FILE attributes:

  KVM_DEV_VFIO_FILE_ADD: Add a VFIO file (group/device) to VFIO-KVM device

	tracking



	kvm_device_attr.addr points to an int32_t file descriptor for the

	VFIO file.



  KVM_DEV_VFIO_FILE_DEL: Remove a VFIO file (group/device) from VFIO-KVM

	device tracking



	kvm_device_attr.addr points to an int32_t file descriptor for the

	VFIO file.



KVM_DEV_VFIO_GROUP (legacy kvm device group restricted to the handling of VFIO group fd):

  KVM_DEV_VFIO_GROUP_ADD: same as KVM_DEV_VFIO_FILE_ADD for group fd only



  KVM_DEV_VFIO_GROUP_DEL: same as KVM_DEV_VFIO_FILE_DEL for group fd only



  KVM_DEV_VFIO_GROUP_SET_SPAPR_TCE: attaches a guest visible TCE table

	allocated by sPAPR KVM.

	kvm_device_attr.addr points to a struct::
@@ -40,7 +52,10 @@ KVM_DEV_VFIO_GROUP attributes: 
	- @tablefd is a file descriptor for a TCE table allocated via

	  KVM_CREATE_SPAPR_TCE.



The GROUP_ADD operation above should be invoked prior to accessing the

The FILE/GROUP_ADD operation above should be invoked prior to accessing the

device file descriptor via VFIO_GROUP_GET_DEVICE_FD in order to support

drivers which require a kvm pointer to be set in their .open_device()

callback.
 
callback.  It is the same for device file descriptor via character device

open which gets device access via VFIO_DEVICE_BIND_IOMMUFD.  For such file

descriptors, FILE_ADD should be invoked before VFIO_DEVICE_BIND_IOMMUFD

to support the drivers mentioned in prior sentence as well.

drivers/gpu/drm/i915/gvt/kvmgt.c

+1 −0
Original line number Diff line number Diff line
@@ -1474,6 +1474,7 @@ static const struct vfio_device_ops intel_vgpu_dev_ops = { 
	.bind_iommufd	= vfio_iommufd_emulated_bind,

	.unbind_iommufd = vfio_iommufd_emulated_unbind,

	.attach_ioas	= vfio_iommufd_emulated_attach_ioas,

	.detach_ioas	= vfio_iommufd_emulated_detach_ioas,

};



static int intel_vgpu_probe(struct mdev_device *mdev)

drivers/iommu/iommufd/Kconfig

+2 −2
Original line number Diff line number Diff line
@@ -14,8 +14,8 @@ config IOMMUFD 
if IOMMUFD

config IOMMUFD_VFIO_CONTAINER

	bool "IOMMUFD provides the VFIO container /dev/vfio/vfio"

	depends on VFIO && !VFIO_CONTAINER

	default VFIO && !VFIO_CONTAINER

	depends on VFIO_GROUP && !VFIO_CONTAINER

	default VFIO_GROUP && !VFIO_CONTAINER

	help

	  IOMMUFD will provide /dev/vfio/vfio instead of VFIO. This relies on

	  IOMMUFD providing compatibility emulation to give the same ioctls.

drivers/iommu/iommufd/device.c

+111 −5
Original line number Diff line number Diff line
@@ -98,6 +98,36 @@ struct iommufd_device *iommufd_device_bind(struct iommufd_ctx *ictx, 
}

EXPORT_SYMBOL_NS_GPL(iommufd_device_bind, IOMMUFD);



/**

 * iommufd_ctx_has_group - True if any device within the group is bound

 *                         to the ictx

 * @ictx: iommufd file descriptor

 * @group: Pointer to a physical iommu_group struct

 *

 * True if any device within the group has been bound to this ictx, ex. via

 * iommufd_device_bind(), therefore implying ictx ownership of the group.

 */

bool iommufd_ctx_has_group(struct iommufd_ctx *ictx, struct iommu_group *group)

{

	struct iommufd_object *obj;

	unsigned long index;



	if (!ictx || !group)

		return false;



	xa_lock(&ictx->objects);

	xa_for_each(&ictx->objects, index, obj) {

		if (obj->type == IOMMUFD_OBJ_DEVICE &&

		    container_of(obj, struct iommufd_device, obj)->group == group) {

			xa_unlock(&ictx->objects);

			return true;

		}

	}

	xa_unlock(&ictx->objects);

	return false;

}

EXPORT_SYMBOL_NS_GPL(iommufd_ctx_has_group, IOMMUFD);



/**

 * iommufd_device_unbind - Undo iommufd_device_bind()

 * @idev: Device returned by iommufd_device_bind()
@@ -116,6 +146,18 @@ void iommufd_device_unbind(struct iommufd_device *idev) 
}

EXPORT_SYMBOL_NS_GPL(iommufd_device_unbind, IOMMUFD);



struct iommufd_ctx *iommufd_device_to_ictx(struct iommufd_device *idev)

{

	return idev->ictx;

}

EXPORT_SYMBOL_NS_GPL(iommufd_device_to_ictx, IOMMUFD);



u32 iommufd_device_to_id(struct iommufd_device *idev)

{

	return idev->obj.id;

}

EXPORT_SYMBOL_NS_GPL(iommufd_device_to_id, IOMMUFD);



static int iommufd_device_setup_msi(struct iommufd_device *idev,

				    struct iommufd_hw_pagetable *hwpt,

				    phys_addr_t sw_msi_start)
@@ -444,6 +486,7 @@ iommufd_access_create(struct iommufd_ctx *ictx, 
	iommufd_ctx_get(ictx);

	iommufd_object_finalize(ictx, &access->obj);

	*id = access->obj.id;

	mutex_init(&access->ioas_lock);

	return access;

}

EXPORT_SYMBOL_NS_GPL(iommufd_access_create, IOMMUFD);
@@ -463,26 +506,60 @@ void iommufd_access_destroy(struct iommufd_access *access) 
}

EXPORT_SYMBOL_NS_GPL(iommufd_access_destroy, IOMMUFD);



void iommufd_access_detach(struct iommufd_access *access)

{

	struct iommufd_ioas *cur_ioas = access->ioas;



	mutex_lock(&access->ioas_lock);

	if (WARN_ON(!access->ioas))

		goto out;

	/*

	 * Set ioas to NULL to block any further iommufd_access_pin_pages().

	 * iommufd_access_unpin_pages() can continue using access->ioas_unpin.

	 */

	access->ioas = NULL;



	if (access->ops->unmap) {

		mutex_unlock(&access->ioas_lock);

		access->ops->unmap(access->data, 0, ULONG_MAX);

		mutex_lock(&access->ioas_lock);

	}

	iopt_remove_access(&cur_ioas->iopt, access);

	refcount_dec(&cur_ioas->obj.users);

out:

	access->ioas_unpin = NULL;

	mutex_unlock(&access->ioas_lock);

}

EXPORT_SYMBOL_NS_GPL(iommufd_access_detach, IOMMUFD);



int iommufd_access_attach(struct iommufd_access *access, u32 ioas_id)

{

	struct iommufd_ioas *new_ioas;

	int rc = 0;



	if (access->ioas)

	mutex_lock(&access->ioas_lock);

	if (WARN_ON(access->ioas || access->ioas_unpin)) {

		mutex_unlock(&access->ioas_lock);

		return -EINVAL;

	}



	new_ioas = iommufd_get_ioas(access->ictx, ioas_id);

	if (IS_ERR(new_ioas))

	if (IS_ERR(new_ioas)) {

		mutex_unlock(&access->ioas_lock);

		return PTR_ERR(new_ioas);

	}



	rc = iopt_add_access(&new_ioas->iopt, access);

	if (rc) {

		mutex_unlock(&access->ioas_lock);

		iommufd_put_object(&new_ioas->obj);

		return rc;

	}

	iommufd_ref_to_users(&new_ioas->obj);



	access->ioas = new_ioas;

	access->ioas_unpin = new_ioas;

	mutex_unlock(&access->ioas_lock);

	return 0;

}

EXPORT_SYMBOL_NS_GPL(iommufd_access_attach, IOMMUFD);
@@ -537,8 +614,8 @@ void iommufd_access_notify_unmap(struct io_pagetable *iopt, unsigned long iova, 
void iommufd_access_unpin_pages(struct iommufd_access *access,

				unsigned long iova, unsigned long length)

{

	struct io_pagetable *iopt = &access->ioas->iopt;

	struct iopt_area_contig_iter iter;

	struct io_pagetable *iopt;

	unsigned long last_iova;

	struct iopt_area *area;
@@ -546,6 +623,17 @@ void iommufd_access_unpin_pages(struct iommufd_access *access, 
	    WARN_ON(check_add_overflow(iova, length - 1, &last_iova)))

		return;



	mutex_lock(&access->ioas_lock);

	/*

	 * The driver must be doing something wrong if it calls this before an

	 * iommufd_access_attach() or after an iommufd_access_detach().

	 */

	if (WARN_ON(!access->ioas_unpin)) {

		mutex_unlock(&access->ioas_lock);

		return;

	}

	iopt = &access->ioas_unpin->iopt;



	down_read(&iopt->iova_rwsem);

	iopt_for_each_contig_area(&iter, area, iopt, iova, last_iova)

		iopt_area_remove_access(
@@ -555,6 +643,7 @@ void iommufd_access_unpin_pages(struct iommufd_access *access, 
				min(last_iova, iopt_area_last_iova(area))));

	WARN_ON(!iopt_area_contig_done(&iter));

	up_read(&iopt->iova_rwsem);

	mutex_unlock(&access->ioas_lock);

}

EXPORT_SYMBOL_NS_GPL(iommufd_access_unpin_pages, IOMMUFD);
@@ -600,8 +689,8 @@ int iommufd_access_pin_pages(struct iommufd_access *access, unsigned long iova, 
			     unsigned long length, struct page **out_pages,

			     unsigned int flags)

{

	struct io_pagetable *iopt = &access->ioas->iopt;

	struct iopt_area_contig_iter iter;

	struct io_pagetable *iopt;

	unsigned long last_iova;

	struct iopt_area *area;

	int rc;
@@ -616,6 +705,13 @@ int iommufd_access_pin_pages(struct iommufd_access *access, unsigned long iova, 
	if (check_add_overflow(iova, length - 1, &last_iova))

		return -EOVERFLOW;



	mutex_lock(&access->ioas_lock);

	if (!access->ioas) {

		mutex_unlock(&access->ioas_lock);

		return -ENOENT;

	}

	iopt = &access->ioas->iopt;



	down_read(&iopt->iova_rwsem);

	iopt_for_each_contig_area(&iter, area, iopt, iova, last_iova) {

		unsigned long last = min(last_iova, iopt_area_last_iova(area));
@@ -646,6 +742,7 @@ int iommufd_access_pin_pages(struct iommufd_access *access, unsigned long iova, 
	}



	up_read(&iopt->iova_rwsem);

	mutex_unlock(&access->ioas_lock);

	return 0;



err_remove:
@@ -660,6 +757,7 @@ int iommufd_access_pin_pages(struct iommufd_access *access, unsigned long iova, 
						  iopt_area_last_iova(area))));

	}

	up_read(&iopt->iova_rwsem);

	mutex_unlock(&access->ioas_lock);

	return rc;

}

EXPORT_SYMBOL_NS_GPL(iommufd_access_pin_pages, IOMMUFD);
@@ -679,8 +777,8 @@ EXPORT_SYMBOL_NS_GPL(iommufd_access_pin_pages, IOMMUFD); 
int iommufd_access_rw(struct iommufd_access *access, unsigned long iova,

		      void *data, size_t length, unsigned int flags)

{

	struct io_pagetable *iopt = &access->ioas->iopt;

	struct iopt_area_contig_iter iter;

	struct io_pagetable *iopt;

	struct iopt_area *area;

	unsigned long last_iova;

	int rc;
@@ -690,6 +788,13 @@ int iommufd_access_rw(struct iommufd_access *access, unsigned long iova, 
	if (check_add_overflow(iova, length - 1, &last_iova))

		return -EOVERFLOW;



	mutex_lock(&access->ioas_lock);

	if (!access->ioas) {

		mutex_unlock(&access->ioas_lock);

		return -ENOENT;

	}

	iopt = &access->ioas->iopt;



	down_read(&iopt->iova_rwsem);

	iopt_for_each_contig_area(&iter, area, iopt, iova, last_iova) {

		unsigned long last = min(last_iova, iopt_area_last_iova(area));
@@ -716,6 +821,7 @@ int iommufd_access_rw(struct iommufd_access *access, unsigned long iova, 
		rc = -ENOENT;

err_out:

	up_read(&iopt->iova_rwsem);

	mutex_unlock(&access->ioas_lock);

	return rc;

}

EXPORT_SYMBOL_NS_GPL(iommufd_access_rw, IOMMUFD);