blob: 2dd877d5dc406ecb4106a6d2f5b6fbe8b3f44fb5 (
plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
|
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="X-UA-Compatible" content="IE=Edge" />
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<title>OPAL/Skiboot Nvlink Interface Documentation — skiboot 5133e1e
documentation</title>
<link rel="stylesheet" href="_static/classic.css" type="text/css" />
<link rel="stylesheet" href="_static/pygments.css" type="text/css" />
<script type="text/javascript" id="documentation_options" data-url_root="./" src="_static/documentation_options.js"></script>
<script type="text/javascript" src="_static/jquery.js"></script>
<script type="text/javascript" src="_static/underscore.js"></script>
<script type="text/javascript" src="_static/doctools.js"></script>
<script type="text/javascript" src="_static/language_data.js"></script>
<link rel="index" title="Index" href="genindex.html" />
<link rel="search" title="Search" href="search.html" />
<link rel="next" title="PCI" href="pci.html" />
<link rel="prev" title="Memory in skiboot" href="memory.html" />
</head><body>
<div class="related" role="navigation" aria-label="related navigation">
<h3>Navigation</h3>
<ul>
<li class="right" style="margin-right: 10px">
<a href="genindex.html" title="General Index"
accesskey="I">index</a></li>
<li class="right" >
<a href="pci.html" title="PCI"
accesskey="N">next</a> |</li>
<li class="right" >
<a href="memory.html" title="Memory in skiboot"
accesskey="P">previous</a> |</li>
<li class="nav-item nav-item-0"><a href="index.html">skiboot 5133e1e
documentation</a> »</li>
</ul>
</div>
<div class="document">
<div class="documentwrapper">
<div class="bodywrapper">
<div class="body" role="main">
<div class="section" id="opal-skiboot-nvlink-interface-documentation">
<span id="nvlink"></span><h1>OPAL/Skiboot Nvlink Interface Documentation<a class="headerlink" href="#opal-skiboot-nvlink-interface-documentation" title="Permalink to this headline">¶</a></h1>
<div class="section" id="overview">
<h2>Overview<a class="headerlink" href="#overview" title="Permalink to this headline">¶</a></h2>
<p>NV-Link is a high speed interconnect that is used in conjunction with
a PCI-E connection to create an interface between chips that provides
very high data bandwidth. The PCI-E connection is used as the control
path to initiate and report status of large data transfers. The data
transfers themselves are sent over the NV-Link.</p>
<p>On IBM Power systems the NV-Link hardware is similar to our standard
PCI hardware so to maximise code reuse the NV-Link is exposed as an
emulated PCI device through system firmware (OPAL/skiboot). Thus each
NV-Link capable device will appear as two devices on a system, the
real PCI-E device and at least one emulated PCI device used for the
NV-Link.</p>
<p>Presently the NV-Link is only capable of data transfers initiated by
the target, thus the emulated PCI device will only handle registers
for link initialisation, DMA transfers and error reporting (EEH).</p>
</div>
<div class="section" id="emulated-pci-devices">
<h2>Emulated PCI Devices<a class="headerlink" href="#emulated-pci-devices" title="Permalink to this headline">¶</a></h2>
<p>Each link will be exported as an emulated PCI device with a minimum of
two emulated PCI devices per GPU. Emulated PCI devices are grouped per
GPU.</p>
<p>The emulated PCI device will be exported as a standard PCI device by
the Linux kernel. It has a standard PCI configuration space to expose
necessary device parameters. The only functionality available is
related to the setup of DMA windows.</p>
<div class="section" id="configuration-space-parameters">
<h3>Configuration Space Parameters<a class="headerlink" href="#configuration-space-parameters" title="Permalink to this headline">¶</a></h3>
<table border="1" class="docutils">
<colgroup>
<col width="17%" />
<col width="26%" />
<col width="57%" />
</colgroup>
<thead valign="bottom">
<tr class="row-odd"><th class="head"> </th>
<th class="head"> </th>
<th class="head"> </th>
</tr>
</thead>
<tbody valign="top">
<tr class="row-even"><td>Vendor ID</td>
<td>0x1014</td>
<td>(IBM)</td>
</tr>
<tr class="row-odd"><td>Device ID</td>
<td>0x04ea</td>
<td> </td>
</tr>
<tr class="row-even"><td>Revision ID</td>
<td>0x00</td>
<td> </td>
</tr>
<tr class="row-odd"><td>Class</td>
<td>0x068000 / 0x068001</td>
<td>(Bridge Device Other, ProgIf = 0x0 / 0x1)</td>
</tr>
<tr class="row-even"><td>BAR0/1</td>
<td>TL/DL Registers</td>
<td> </td>
</tr>
<tr class="row-odd"><td>BAR2/3</td>
<td>GEN-ID Registers</td>
<td>(Only for rev-id = 0x1)</td>
</tr>
</tbody>
</table>
</div>
<div class="section" id="tl-dl-registers">
<h3>TL/DL Registers<a class="headerlink" href="#tl-dl-registers" title="Permalink to this headline">¶</a></h3>
<p>Each link has 128KB of TL/DL registers. These will always be mapped
to 64-bit BAR#0 of the emulated PCI device configuration space.</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">BAR</span><span class="c1">#0 + 128K +-----------+</span>
<span class="o">|</span> <span class="n">NTL</span> <span class="p">(</span><span class="mi">64</span><span class="n">K</span><span class="p">)</span> <span class="o">|</span>
<span class="n">BAR</span><span class="c1">#0 + 64K +-----------+</span>
<span class="o">|</span> <span class="n">DL</span> <span class="p">(</span><span class="mi">64</span><span class="n">K</span><span class="p">)</span> <span class="o">|</span>
<span class="n">BAR</span><span class="c1">#0 +-----------+</span>
</pre></div>
</div>
</div>
<div class="section" id="generation-registers-gen-id">
<h3>Generation Registers (GEN-ID)<a class="headerlink" href="#generation-registers-gen-id" title="Permalink to this headline">¶</a></h3>
<p>On POWER9 each link has 64K of generation ID registers for the relaxed
ordering mode syncronisation. Refer to the programming guide for
details of the register layout in this BAR.</p>
<p>Relaxed ordering mode will be disabled by default as it requires
device driver support. Device drivers will need to request relaxed
ordering mode through some yet to be designed mechanism.</p>
</div>
<div class="section" id="vendor-specific-capabilities">
<h3>Vendor Specific Capabilities<a class="headerlink" href="#vendor-specific-capabilities" title="Permalink to this headline">¶</a></h3>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="o">+-----------------+----------------+----------------+----------------+</span>
<span class="o">|</span> <span class="n">Version</span> <span class="p">(</span><span class="mh">0x02</span><span class="p">)</span> <span class="o">|</span> <span class="n">Cap</span> <span class="n">Length</span> <span class="o">|</span> <span class="n">Next</span> <span class="n">Cap</span> <span class="n">Ptr</span> <span class="o">|</span> <span class="n">Cap</span> <span class="n">ID</span> <span class="p">(</span><span class="mh">0x09</span><span class="p">)</span> <span class="o">|</span>
<span class="o">+-----------------+----------------+----------------+----------------+</span>
<span class="o">|</span> <span class="n">Procedure</span> <span class="n">Status</span> <span class="n">Register</span> <span class="o">|</span>
<span class="o">+--------------------------------------------------------------------+</span>
<span class="o">|</span> <span class="n">Procedure</span> <span class="n">Control</span> <span class="n">Register</span> <span class="o">|</span>
<span class="o">+---------------------------------------------------+----------------+</span>
<span class="o">|</span> <span class="n">Reserved</span> <span class="o">|</span> <span class="n">PCI</span> <span class="n">Dev</span> <span class="n">Flag</span> <span class="o">|</span> <span class="n">Link</span> <span class="n">Number</span> <span class="o">|</span>
<span class="o">+---------------------------------------------------+----------------+</span>
</pre></div>
</div>
<p>Version</p>
<blockquote>
<div>This refers to the version of the NPU config space. Used by device
drivers to determine which fields of the config space they can
expect to be available.</div></blockquote>
<p>Procedure Control Register</p>
<blockquote>
<div><p>Used to start hardware procedures.</p>
<p>Writes will start the corresponding procedure and set bit 31 in the
procedure status register. This register must not be written while
bit 31 is set in the status register. Performing a write while
another procudure is already in progress will abort that procedure.</p>
<p>Reads will return the in progress procedure or the last completed
procedure number depending on the procedure status field.</p>
<p>Procedure Numbers:</p>
<blockquote>
<div><ol class="arabic simple" start="0">
<li>Abort in-progress procedure</li>
<li>NOP</li>
<li>Unsupported procedure</li>
<li>Unsupported procedure</li>
<li>Naples PHY - RESET</li>
<li>Naples PHY - TX_ZCAL</li>
<li>Naples PHY - RX_DCCAL</li>
<li>Naples PHY - TX_RXCAL_ENABLE</li>
<li>Naples PHY - TX_RXCAL_DISABLE</li>
<li>Naples PHY - RX_TRAINING</li>
<li>Naples NPU - RESET</li>
<li>Naples PHY - PHY preterminate</li>
<li>Naples PHY - PHY terminated</li>
<li>Witherspoon TL credit validation</li>
</ol>
</div></blockquote>
<p>Procedure 5 (TX_ZCAL) should only be run once. System firmware will
ensure this so device drivers may call this procedure mutiple
times.</p>
</div></blockquote>
<p>Procedure Status Register</p>
<blockquote>
<div><p>The procedure status register is used to determine when execution
of the procedure number in the control register is complete and if
it completed successfully.</p>
<p>This register must be polled frequently to allow system firmware to
execute the procedures.</p>
<dl class="docutils">
<dt>Fields:</dt>
<dd>Bit 31 - Procedure in progress
Bit 30 - Procedure complete
Bit 3-0 - Procedure completion code</dd>
<dt>Procedure completion codes:</dt>
<dd>0 - Procedure completed successfully.
1 - Transient failure. Procedure should be rerun.
2 - Permanent failure. Procedure will never complete successfully.
3 - Procedure aborted.
4 - Unsupported procedure.</dd>
</dl>
</div></blockquote>
<p>PCI Device Flag</p>
<blockquote>
<div>Bit 0 - set if the GPU PCIe device associated with this nvlink was found.
bit 1 - set if the DL has been taken out of reset.</div></blockquote>
<p>Link Number</p>
<blockquote>
<div>Physical link number this emulated PCI device is associated
with. One of 0, 1, 4 or 5 (links 2 & 3 do not exist on Naples).</div></blockquote>
<p>Reserved</p>
<blockquote>
<div>These fields must be ignored and no value should be assumed.</div></blockquote>
</div>
<div class="section" id="interrupts">
<h3>Interrupts<a class="headerlink" href="#interrupts" title="Permalink to this headline">¶</a></h3>
<p>Each link has a single DL/TL interrupt assigned to it. These will be
exposed as an LSI via the emulated PCI device. There are 4 links
consuming 4 LSI interrupts. The 4 remaining interrupts supported by the
corresponding PHB will be routed to OS platform for the purpose of error
reporting.</p>
</div>
</div>
<div class="section" id="device-tree-bindings">
<h2>Device Tree Bindings<a class="headerlink" href="#device-tree-bindings" title="Permalink to this headline">¶</a></h2>
<p>See <a class="reference internal" href="device-tree/nvlink.html#device-tree-nvlink"><span class="std std-ref">Nvlink Device Tree Bindings</span></a></p>
</div>
</div>
</div>
</div>
</div>
<div class="sphinxsidebar" role="navigation" aria-label="main navigation">
<div class="sphinxsidebarwrapper">
<h3><a href="index.html">Table of Contents</a></h3>
<ul>
<li><a class="reference internal" href="#">OPAL/Skiboot Nvlink Interface Documentation</a><ul>
<li><a class="reference internal" href="#overview">Overview</a></li>
<li><a class="reference internal" href="#emulated-pci-devices">Emulated PCI Devices</a><ul>
<li><a class="reference internal" href="#configuration-space-parameters">Configuration Space Parameters</a></li>
<li><a class="reference internal" href="#tl-dl-registers">TL/DL Registers</a></li>
<li><a class="reference internal" href="#generation-registers-gen-id">Generation Registers (GEN-ID)</a></li>
<li><a class="reference internal" href="#vendor-specific-capabilities">Vendor Specific Capabilities</a></li>
<li><a class="reference internal" href="#interrupts">Interrupts</a></li>
</ul>
</li>
<li><a class="reference internal" href="#device-tree-bindings">Device Tree Bindings</a></li>
</ul>
</li>
</ul>
<h4>Previous topic</h4>
<p class="topless"><a href="memory.html"
title="previous chapter">Memory in skiboot</a></p>
<h4>Next topic</h4>
<p class="topless"><a href="pci.html"
title="next chapter">PCI</a></p>
<div role="note" aria-label="source link">
<h3>This Page</h3>
<ul class="this-page-menu">
<li><a href="_sources/nvlink.rst.txt"
rel="nofollow">Show Source</a></li>
</ul>
</div>
<div id="searchbox" style="display: none" role="search">
<h3>Quick search</h3>
<div class="searchformwrapper">
<form class="search" action="search.html" method="get">
<input type="text" name="q" />
<input type="submit" value="Go" />
<input type="hidden" name="check_keywords" value="yes" />
<input type="hidden" name="area" value="default" />
</form>
</div>
</div>
<script type="text/javascript">$('#searchbox').show(0);</script>
</div>
</div>
<div class="clearer"></div>
</div>
<div class="related" role="navigation" aria-label="related navigation">
<h3>Navigation</h3>
<ul>
<li class="right" style="margin-right: 10px">
<a href="genindex.html" title="General Index"
>index</a></li>
<li class="right" >
<a href="pci.html" title="PCI"
>next</a> |</li>
<li class="right" >
<a href="memory.html" title="Memory in skiboot"
>previous</a> |</li>
<li class="nav-item nav-item-0"><a href="index.html">skiboot 5133e1e
documentation</a> »</li>
</ul>
</div>
<div class="footer" role="contentinfo">
© Copyright 2016-2017, IBM, others.
Created using <a href="http://sphinx-doc.org/">Sphinx</a> 1.8.5.
</div>
</body>
</html>
|
