1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
|
.. _skiboot-6.3.2:
==============
skiboot-6.3.2
==============
skiboot 6.3.2 was released on Monday July 1st, 2019. It replaces
:ref:`skiboot-6.3.1` as the current stable release in the 6.3.x series.
It is recommended that 6.3.2 be used instead of 6.3.1 version due to the
bug fixes it contains.
Bug fixes included in this release are:
- npu2: Purge cache when resetting a GPU
After putting all a GPU's links in reset, do a cache purge in case we
have CPU cache lines belonging to the now-unaccessible GPU memory.
- npu2: Reset NVLinks when resetting a GPU
Resetting a V100 GPU brings its NVLinks down and if an NPU tries using
those, an HMI occurs. We were lucky not to observe this as the bare metal
does not normally reset a GPU and when passed through, GPUs are usually
before NPUs in QEMU command line or Libvirt XML and because of that NPUs
are naturally reset first. However simple change of the device order
brings HMIs.
This defines a bus control filter for a PCI slot with a GPU with NVLinks
so when the host system issues secondary bus reset to the slot, it resets
associated NVLinks.
- hw/phb4: Assert Link Disable bit after ETU init
The cursed RAID card in ozrom1 has a bug where it ignores PERST being
asserted. The PCIe Base spec is a little vague about what happens
while PERST is asserted, but it does clearly specify that when
PERST is de-asserted the Link Training and Status State Machine
(LTSSM) of a device should return to the initial state (Detect)
defined in the spec and the link training process should restart.
This bug was worked around in 9078f8268922 ("phb4: Delay training till
after PERST is deasserted") by setting the link disable bit at the
start of the FRESET process and clearing it after PERST was
de-asserted. Although this fixed the bug, the patch offered no
explaination of why the fix worked.
In b8b4c79d4419 ("hw/phb4: Factor out PERST control") the link disable
workaround was moved into phb4_assert_perst(). This is called
always in the CRESET case, but a following patch resulted in
assert_perst() not being called if phb4_freset() was entered following a
CRESET since p->skip_perst was set in the CRESET handler. This is bad
since a side-effect of the CRESET is that the Link Disable bit is
cleared.
This, combined with the RAID card ignoring PERST results in the PCIe
link being trained by the PHB while we're waiting out the 100ms
ETU reset time. If we hack skiboot to print a DLP trace after returning
from phb4_hw_init() we get: ::
PHB#0001[0:1]: Initialization complete
PHB#0001[0:1]: TRACE:0x0000102101000000 0ms presence GEN1:x16:polling
PHB#0001[0:1]: TRACE:0x0000001101000000 23ms GEN1:x16:detect
PHB#0001[0:1]: TRACE:0x0000102101000000 23ms presence GEN1:x16:polling
PHB#0001[0:1]: TRACE:0x0000183101000000 29ms training GEN1:x16:config
PHB#0001[0:1]: TRACE:0x00001c5881000000 30ms training GEN1:x08:recovery
PHB#0001[0:1]: TRACE:0x00001c5883000000 30ms training GEN3:x08:recovery
PHB#0001[0:1]: TRACE:0x0000144883000000 33ms presence GEN3:x08:L0
PHB#0001[0:1]: TRACE:0x0000154883000000 33ms trained GEN3:x08:L0
PHB#0001[0:1]: CRESET: wait_time = 100
PHB#0001[0:1]: FRESET: Starts
PHB#0001[0:1]: FRESET: Prepare for link down
PHB#0001[0:1]: FRESET: Assert skipped
PHB#0001[0:1]: FRESET: Deassert
PHB#0001[0:1]: TRACE:0x0000154883000000 0ms trained GEN3:x08:L0
PHB#0001[0:1]: TRACE: Reached target state
PHB#0001[0:1]: LINK: Start polling
PHB#0001[0:1]: LINK: Electrical link detected
PHB#0001[0:1]: LINK: Link is up
PHB#0001[0:1]: LINK: Went down waiting for stabilty
PHB#0001[0:1]: LINK: DLP train control: 0x0000105101000000
PHB#0001[0:1]: CRESET: Starts
What has happened here is that the link is trained to 8x Gen3 33ms after
we return from phb4_init_hw(), and before we've waitined to 100ms
that we normally wait after re-initialising the ETU. When we "deassert"
PERST later on in the FRESET handler the link in L0 (normal) state. At
this point we try to read from the Vendor/Device ID register to verify
that the link is stable and immediately get a PHB fence due to a PCIe
Completion Timeout. Skiboot attempts to recover by doing another CRESET,
but this will encounter the same issue.
This patch fixes the problem by setting the Link Disable bit (by calling
phb4_assert_perst()) immediately after we return from phb4_init_hw().
This prevents the link from being trained while PERST is asserted which
seems to avoid the Completion Timeout. With the patch applied we get: ::
PHB#0001[0:1]: Initialization complete
PHB#0001[0:1]: TRACE:0x0000102101000000 0ms presence GEN1:x16:polling
PHB#0001[0:1]: TRACE:0x0000001101000000 23ms GEN1:x16:detect
PHB#0001[0:1]: TRACE:0x0000102101000000 23ms presence GEN1:x16:polling
PHB#0001[0:1]: TRACE:0x0000909101000000 29ms presence GEN1:x16:disabled
PHB#0001[0:1]: CRESET: wait_time = 100
PHB#0001[0:1]: FRESET: Starts
PHB#0001[0:1]: FRESET: Prepare for link down
PHB#0001[0:1]: FRESET: Assert skipped
PHB#0001[0:1]: FRESET: Deassert
PHB#0001[0:1]: TRACE:0x0000001101000000 0ms GEN1:x16:detect
PHB#0001[0:1]: TRACE:0x0000102101000000 0ms presence GEN1:x16:polling
PHB#0001[0:1]: TRACE:0x0000001101000000 24ms GEN1:x16:detect
PHB#0001[0:1]: TRACE:0x0000102101000000 36ms presence GEN1:x16:polling
PHB#0001[0:1]: TRACE:0x0000183101000000 97ms training GEN1:x16:config
PHB#0001[0:1]: TRACE:0x00001c5881000000 97ms training GEN1:x08:recovery
PHB#0001[0:1]: TRACE:0x00001c5883000000 97ms training GEN3:x08:recovery
PHB#0001[0:1]: TRACE:0x0000144883000000 99ms presence GEN3:x08:L0
PHB#0001[0:1]: TRACE: Reached target state
PHB#0001[0:1]: LINK: Start polling
PHB#0001[0:1]: LINK: Electrical link detected
PHB#0001[0:1]: LINK: Link is up
PHB#0001[0:1]: LINK: Link is stable
PHB#0001[0:1]: LINK: Card [9005:028c] Optimal Retry:disabled
PHB#0001[0:1]: LINK: Speed Train:GEN3 PHB:GEN4 DEV:GEN3
PHB#0001[0:1]: LINK: Width Train:x08 PHB:x08 DEV:x08
PHB#0001[0:1]: LINK: RX Errors Now:0 Max:8 Lane:0x0000
- npu2: Reset PID wildcard and refcounter when mapped to LPID
Since 105d80f85b "npu2: Use unfiltered mode in XTS tables" we do not
register every PID in the XTS table so the table has one entry per LPID.
Then we added a reference counter to keep track of the entry use when
switching GPU between the host and guest systems (the "Fixes:" tag below).
The POWERNV platform setup creates such entries and references them
at the boot time when initializing IOMMUs and only removes it when
a GPU is passed through to a guest. This creates a problem as POWERNV
boots via kexec and no defererencing happens; the XTS table state remains
undefined. So when the host kernel boots, skiboot thinks there are valid
XTS entries and does not update the XTS table which breaks ATS.
This adds the reference counter and the XTS entry reset when a GPU is
assigned to LPID and we cannot rely on the kernel to clean that up.
- hw/phb4: Use read/write_reg in assert_perst
While the PHB is fenced we can't use the MMIO interface to access PHB
registers. While processing a complete reset we inject a PHB fence to
isolate the PHB from the rest of the system because the PHB won't
respond to MMIOs from the rest of the system while being reset.
We assert PERST after the fence has been erected which requires us to
use the XSCOM indirect interface to access the PHB registers rather than
the MMIO interface. Previously we did that when asserting PERST in the
CRESET path. However in b8b4c79d4419 ("hw/phb4: Factor out PERST
control"). This was re-written to use the raw in_be64() accessor. This
means that CRESET would not be asserted in the reset path. On some
Mellanox cards this would prevent them from re-loading their firmware
when the system was fast-reset.
This patch fixes the problem by replacing the raw {in|out}_be64()
accessors with the phb4_{read|write}_reg() functions.
- opal-prd: Fix prd message size issue
If prd messages size is insufficient then read_prd_msg() call fails with
below error. And caller is not reallocating sufficient buffer. Also its
hard to guess the size.
sample log::
Mar 28 03:31:43 zz24p1 opal-prd: FW: error reading from firmware: alloc 32 rc -1: Invalid argument
Mar 28 03:31:43 zz24p1 opal-prd: FW: error reading from firmware: alloc 32 rc -1: Invalid argument
Mar 28 03:31:43 zz24p1 opal-prd: FW: error reading from firmware: alloc 32 rc -1: Invalid argument
Lets use opal-msg-size device tree property to allocate memory
for prd message.
- npu2: Fix clearing the FIR bits
FIR registers are SCOM-only so they cannot be accesses with the indirect
write, and yet we use SCOM-based addresses for these; fix this.
- opal-gard: Account for ECC size when clearing partition
When 'opal-gard clear all' is run, it works by erasing the GUARD then
using blockevel_smart_write() to write nothing to the partition. This
second write call is needed because we rely on libflash to set the ECC
bits appropriately when the partition contained ECCed data.
The API for this is a little odd with the caller specifying how much
actual data to write, and libflash writing size + size/8 bytes
since there is one additional ECC byte for every eight bytes of data.
We currently do not account for the extra space consumed by the ECC data
in reset_partition() which is used to handle the 'clear all' command.
Which results in the paritition following the GUARD partition being
partially overwritten when the command is used. This patch fixes the
problem by reducing the length we would normally write by the number
of ECC bytes required.
- nvram: Flag dangerous NVRAM options
Most nvram options used by skiboot are just for debug or testing for
regressions. They should never be used long term.
We've hit a number of issues in testing and the field where nvram
options have been set "temporarily" but haven't been properly cleared
after, resulting in crashes or real bugs being masked.
This patch marks most nvram options used by skiboot as dangerous and
prints a chicken to remind users of the problem.
- devicetree: Don't set path to dtc in makefile
By setting the path we fail to build under buildroot which has it's own
set of host tools in PATH, but not at /usr/bin.
Keep the variable so it can be set if need be but default to whatever
'dtc' is in the users path.
|
