phb4: Enable PCI peer-to-peer

P9 supports PCI peer-to-peer: a PCI device can write directly to the
mmio space of another PCI device. It completely by-passes the CPU.

It requires some configuration on the PHBs involved:

1. on the initiating side, the address for the read/write operation is
in the mmio space of the target, i.e. well outside the range normally
allowed. So we disable range-checking on the TVT entry in bypass mode.

2. on the target side, we need to explicitly enable p2p by setting a
bit in a configuration register. It has the side-effect of reserving
an outbound (as seen from the CPU) store queue for p2p. Therefore we
only enable p2p on the PHBs using it, as we don't want to waste the
resource if we don't have to.

P9 supports p2p mmio writes. Reads are currently only supported if the
two devices are under the same PHB but that is expected to change in
the future, and it raises questions about intermediate switches
configuration, so we report an error for the time being.

The patch adds a new OPAL call to allow the OS to declare a p2p
(initiator, target) pair.

Signed-off-by: Frederic Barrat <fbarrat@linux.vnet.ibm.com>
Reviewed-by: Russell Currey <ruscur@russell.cc>
Signed-off-by: Stewart Smith <stewart@linux.vnet.ibm.com>

1 files changed, 39 insertions, 0 deletions

diff --git a/core/pci-opal.c b/core/pci-opal.c
index 5d58a88..b8aec94 100644
--- a/core/pci-opal.c
+++ b/core/pci-opal.c
@@ -977,3 +977,42 @@ static int64_t opal_pci_set_phb_capi_mode(uint64_t phb_id, uint64_t mode, uint64
 	return rc;
 }
 opal_call(OPAL_PCI_SET_PHB_CAPI_MODE, opal_pci_set_phb_capi_mode, 3);
+
+static int64_t opal_pci_set_p2p(uint64_t phbid_init, uint64_t phbid_target,
+				uint64_t desc, uint16_t pe_number)
+{
+	struct phb *phb_init = pci_get_phb(phbid_init);
+	struct phb *phb_target = pci_get_phb(phbid_target);
+
+	if (!phb_init || !phb_target)
+		return OPAL_PARAMETER;
+	/*
+	 * Having the 2 devices under the same PHB may require tuning
+	 * the configuration of intermediate switch(es), more easily
+	 * done from linux. And it shouldn't require a PHB config
+	 * change.
+	 * Return an error for the time being.
+	 */
+	if (phb_init == phb_target)
+		return OPAL_UNSUPPORTED;
+	if (!phb_init->ops->set_p2p || !phb_target->ops->set_p2p)
+		return OPAL_UNSUPPORTED;
+	/*
+	 * Loads would be supported on p9 if the 2 devices are under
+	 * the same PHB, but we ruled it out above.
+	 */
+	if (desc & OPAL_PCI_P2P_LOAD)
+		return OPAL_UNSUPPORTED;
+
+	phb_lock(phb_init);
+	phb_init->ops->set_p2p(phb_init, OPAL_PCI_P2P_INITIATOR, desc,
+			pe_number);
+	phb_unlock(phb_init);
+
+	phb_lock(phb_target);
+	phb_target->ops->set_p2p(phb_target, OPAL_PCI_P2P_TARGET, desc,
+				pe_number);
+	phb_unlock(phb_target);
+	return OPAL_SUCCESS;
+}
+opal_call(OPAL_PCI_SET_P2P, opal_pci_set_p2p, 4);