diff options
| author | Frederic Barrat <fbarrat@linux.ibm.com> | 2020-03-25 11:41:17 +0100 |
|---|---|---|
| committer | Oliver O'Halloran <oohall@gmail.com> | 2020-03-30 18:18:10 +1100 |
| commit | e876514b3773dcecc0b39317ca341d27db96d81c (patch) | |
| tree | ffc0ce4a55d76e0bb07c2897d797f534dc6e5319 /include | |
| parent | 3794167200e7169afc240516254ed0e601e76e72 (diff) | |
| download | skiboot-e876514b3773dcecc0b39317ca341d27db96d81c.zip skiboot-e876514b3773dcecc0b39317ca341d27db96d81c.tar.gz skiboot-e876514b3773dcecc0b39317ca341d27db96d81c.tar.bz2 | |
hw/phb4: Tune GPU direct performance on witherspoon in PCI mode
Good GPU direct performance on witherspoon, with a Mellanox adapter
on the shared slot, requires to reallocate some dma engines within
PEC2, "stealing" some from PHB4&5 and giving extras to PHB3. It's
currently done when using CAPI mode. But the same is true if the
adapter stays in PCI mode.
In preparation for upcoming versions of MOFED, which may not use CAPI
mode, this patch reallocates dma engines even in PCI mode for a series
of Mellanox adapters that can be used with GPU direct, on witherspoon
and on the shared slot only.
The loss of dma engines for PHB4&5 on witherspoon has not shown
problems in testing, as well as in current deployments where CAPI mode
is used.
Here is a comparison of the bandwidth numbers seen with the PHB in PCI
mode (no CAPI) with and without this patch. Variations on smaller
packet sizes can be attributed to jitter and are not that meaningful.
# OSU MPI-CUDA Bi-Directional Bandwidth Test v5.6.1
# Send Buffer on DEVICE (D) and Receive Buffer on DEVICE (D)
# Size Bandwidth (MB/s) Bandwidth (MB/s)
# with patch without patch
1 1.29 1.48
2 2.66 3.04
4 5.34 5.93
8 10.68 11.86
16 21.39 23.71
32 42.78 49.15
64 85.43 97.67
128 170.82 196.64
256 385.47 383.02
512 774.68 755.54
1024 1535.14 1495.30
2048 2599.31 2561.60
4096 5192.31 5092.47
8192 9930.30 9566.90
16384 18189.81 16803.42
32768 24671.48 21383.57
65536 28977.71 24104.50
131072 31110.55 25858.95
262144 32180.64 26470.61
524288 32842.23 26961.93
1048576 33184.87 27217.38
2097152 33342.67 27338.08
Signed-off-by: Frederic Barrat <fbarrat@linux.ibm.com>
Cc: skiboot-stable@lists.ozlabs.org # skiboot-op940.x
Reviewed-by: Andrew Donnellan <ajd@linux.ibm.com>
Signed-off-by: Oliver O'Halloran <oohall@gmail.com>
Diffstat (limited to 'include')
| -rw-r--r-- | include/phb4.h | 2 |
1 files changed, 2 insertions, 0 deletions
diff --git a/include/phb4.h b/include/phb4.h index 6d5fd51..abba2d9 100644 --- a/include/phb4.h +++ b/include/phb4.h @@ -257,4 +257,6 @@ static inline int phb4_get_opal_id(unsigned int chip_id, unsigned int index) return chip_id * PHB4_MAX_PHBS_PER_CHIP_P9P + index; } +void phb4_pec2_dma_engine_realloc(struct phb4 *p); + #endif /* __PHB4_H */ |