skiboot v6.2-rc1 release notesv6.2-rc1

Signed-off-by: Stewart Smith <stewart@linux.ibm.com>

1 files changed, 893 insertions, 0 deletions

diff --git a/doc/release-notes/skiboot-6.2-rc1.rst b/doc/release-notes/skiboot-6.2-rc1.rst
new file mode 100644
index 0000000..e930a4c
--- /dev/null
+++ b/doc/release-notes/skiboot-6.2-rc1.rst
@@ -0,0 +1,893 @@
+.. _skiboot-6.2-rc1:
+
+skiboot-6.2-rc1
+===============
+
+skiboot v6.2-rc1 was released on Monday November 19th 2018. It is the first
+release candidate of skiboot 6.2, which will become the new stable release
+of skiboot following the 6.1 release, first released July 11th 2018.
+
+Skiboot 6.2 will mark the basis for op-build v2.2.
+
+skiboot v6.2-rc1 contains all bug fixes as of :ref:`skiboot-6.0.13`,
+and :ref:`skiboot-5.4.10` (the currently maintained
+stable releases).
+
+For how the skiboot stable releases work, see :ref:`stable-rules` for details.
+
+This release has been a longer cycle than typical for a variety of reasons. It
+also contains a lot of cleanup work and minor bug fixes (much like skiboot 6.1
+did).
+
+Over skiboot 6.1, we have the following changes:
+
+General
+-------
+
+- cpu: Quieten OS endian switch messages
+
+  Users see these when loading an OS from Petitboot: ::
+
+     [  119.486794100,5] OPAL: Switch to big-endian OS
+     [  120.022302604,5] OPAL: Switch to little-endian OS
+
+  Which is expected and doesn't provide any information the user can act
+  on. Switch them to PR_INFO so they still appear in the log, but not on
+  the serial console.
+- Recognise signed VERSION partition
+
+  A few things need to change to support a signed VERSION partition:
+
+  - A signed VERSION partition will be 4K + SECURE_BOOT_HEADERS_SIZE (4K).
+  - The VERSION partition needs to be loaded after secure/trusted boot is
+    set up, and therefore after nvram_init().
+  - Added to the trustedboot resources array.
+
+  This also moves the ipmi_dt_add_bmc_info() call to after
+  flash_dt_add_fw_version() since it adds info to ibm,firmware-versions.
+- Run pollers in time_wait() when not booting
+
+  This only bit us hard with hiomap in one scenario.
+
+  Our OPAL API has been OPAL_POLL_EVENTS may be needed to make forward
+  progress on ongoing operations, and the internal to skiboot API has been
+  that time_wait() of a suitable time will run pollers (on at least one
+  CPU) to help ensure forward progress can be made.
+
+  In a perfect world, interrupts are used but they may: a) be disabled, or
+  b) the thing we're doing can't use interrupts because computers are
+  generally terrible.
+
+  Back in 3db397ea5892a (circa 2015), we changed skiboot so that we'd run
+  pollers only on the boot CPU, and not if we held any locks. This was to
+  reduce the chance of programming code that could deadlock, as well as to
+  ensure that we didn't just thrash all the cachelines for running pollers
+  all over a large system during boot, or hard spin on the same locks on
+  all secondary CPUs.
+
+  The problem arises if the OS we're booting makes an OPAL call early on,
+  with interrupts disabled, that requires a poller to run to make forward
+  progress. An example of this would be OPAL_WRITE_NVRAM early in Linux
+  boot (where Linux sets up the partitions it wants) - something that
+  occurs iff we've had to reformat NVRAM this boot (i.e. first boot or
+  corrupted NVRAM).
+
+  The hiomap implementation should arguably *not* rely on synchronous IPMI
+  messages, but this is a future improvement (as was for mbox before it).
+  The mbox-flash code solved this problem by spinning on check_timers().
+
+  More generically though, the approach of running the pollers when no
+  longer booting means we behave more in line with what the API is meant
+  to be, rather than have this odd case of "time_wait() for a condition
+  that could also be tripped by an interrupt works fine unless the OS is
+  up and running but hasn't set interrupts up yet".
+- ipmi: Reduce ipmi_queue_msg_sync() polling loop time to 10ms
+
+  On a plain boot, this reduces the time spent in OPAL by ~170ms on
+  p9dsu. This is due to hiomap (currently) using synchronous IPMI
+  messages.
+
+  It will also *significantly* reduce latency on runtime flash
+  operations for hiomap, as we'll spend typically 10-20ms in OPAL
+  rather than 100-200ms. It's not an ideal solution to that, but
+  it's a quick and obvious win for jitter.
+- core/device: NULL pointer dereference fix
+- core/flash: NULL pointer dereference fixes
+- core/cpu: Call memset with proper cpu_thread offset
+- libflash: Add ipmi-hiomap, and prefer it for PNOR access
+
+  ipmi-hiomap implements the PNOR access control protocol formerly known
+  as "the mbox protocol" but uses IPMI instead of the AST LPC mailbox as a
+  transport. As there is no-longer any mailbox involved in this alternate
+  implementation the old protocol name is quite misleading, and so it has
+  been renamed to "the hiomap protoocol" (Host I/O Mapping protocol). The
+  same commands and events are used though this client-side implementation
+  assumes v2 of the protocol is supported by the BMC.
+
+  The code is a heavily-reworked copy of the mbox-flash source and is
+  introduced this way to allow for the mbox implementation's eventual
+  removal.
+
+  mbox-flash should in theory be renamed to mbox-hiomap for consistency,
+  but as it is on life-support effective immediately we may as well just
+  remove it entirely when the time is right.
+- opal/hmi: Handle early HMIs on thread0 when secondaries are still in OPAL.
+
+  When primary thread receives a CORE level HMI for timer facility errors
+  while secondaries are still in OPAL, thread 0 ends up in rendez-vous
+  waiting for secondaries to get into hmi handling. This is because OPAL
+  runs with MSR(EE=0) and hence HMIs are delayed on secondary threads until
+  they are given to Linux OS. Fix this by adding a check for secondary
+  state and force them in hmi handling by queuing job on secondary threads.
+
+  I have tested this by injecting HDEC parity error very early during Linux
+  kernel boot. Recovery works fine for non-TB errors. But if TB is bad at
+  this very eary stage we already doomed.
+
+  Without this patch we see: ::
+
+    [  285.046347408,7] OPAL: Start CPU 0x0843 (PIR 0x0843) -> 0x000000000000a83c
+    [  285.051160609,7] OPAL: Start CPU 0x0844 (PIR 0x0844) -> 0x000000000000a83c
+    [  285.055359021,7] HMI: Received HMI interrupt: HMER = 0x0840000000000000
+    [  285.055361439,7] HMI: [Loc: U78D3.ND1.WZS004A-P1-C48]: P:8 C:17 T:0: TFMR(2e12002870e14000) Timer Facility Error
+    [  286.232183823,3] HMI: Rendez-vous stage 1 timeout, CPU 0x844 waiting for thread 1 (sptr=0000ccc1)
+    [  287.409002056,3] HMI: Rendez-vous stage 1 timeout, CPU 0x844 waiting for thread 2 (sptr=0000ccc1)
+    [  289.073820164,3] HMI: Rendez-vous stage 1 timeout, CPU 0x844 waiting for thread 3 (sptr=0000ccc1)
+    [  290.250638683,3] HMI: Rendez-vous stage 1 timeout, CPU 0x844 waiting for thread 1 (sptr=0000ccc2)
+    [  291.427456821,3] HMI: Rendez-vous stage 1 timeout, CPU 0x844 waiting for thread 2 (sptr=0000ccc2)
+    [  293.092274807,3] HMI: Rendez-vous stage 1 timeout, CPU 0x844 waiting for thread 3 (sptr=0000ccc2)
+    [  294.269092904,3] HMI: Rendez-vous stage 1 timeout, CPU 0x844 waiting for thread 1 (sptr=0000ccc3)
+    [  295.445910944,3] HMI: Rendez-vous stage 1 timeout, CPU 0x844 waiting for thread 2 (sptr=0000ccc3)
+    [  297.110728970,3] HMI: Rendez-vous stage 1 timeout, CPU 0x844 waiting for thread 3 (sptr=0000ccc3)
+
+  After this patch: ::
+
+    [  259.401719351,7] OPAL: Start CPU 0x0841 (PIR 0x0841) -> 0x000000000000a83c
+    [  259.406259572,7] OPAL: Start CPU 0x0842 (PIR 0x0842) -> 0x000000000000a83c
+    [  259.410615534,7] OPAL: Start CPU 0x0843 (PIR 0x0843) -> 0x000000000000a83c
+    [  259.415444519,7] OPAL: Start CPU 0x0844 (PIR 0x0844) -> 0x000000000000a83c
+    [  259.419641401,7] HMI: Received HMI interrupt: HMER = 0x0840000000000000
+    [  259.419644124,7] HMI: [Loc: U78D3.ND1.WZS004A-P1-C48]: P:8 C:17 T:0: TFMR(2e12002870e04000) Timer Facility Error
+    [  259.419650678,7] HMI: Sending hmi job to thread 1
+    [  259.419652744,7] HMI: Sending hmi job to thread 2
+    [  259.419653051,7] HMI: Received HMI interrupt: HMER = 0x0840000000000000
+    [  259.419654725,7] HMI: Sending hmi job to thread 3
+    [  259.419654916,7] HMI: Received HMI interrupt: HMER = 0x0840000000000000
+    [  259.419658025,7] HMI: Received HMI interrupt: HMER = 0x0840000000000000
+    [  259.419658406,7] HMI: [Loc: U78D3.ND1.WZS004A-P1-C48]: P:8 C:17 T:2: TFMR(2e12002870e04000) Timer Facility Error
+    [  259.419663095,7] HMI: [Loc: U78D3.ND1.WZS004A-P1-C48]: P:8 C:17 T:3: TFMR(2e12002870e04000) Timer Facility Error
+    [  259.419655234,7] HMI: [Loc: U78D3.ND1.WZS004A-P1-C48]: P:8 C:17 T:1: TFMR(2e12002870e04000) Timer Facility Error
+    [  259.425109779,7] OPAL: Start CPU 0x0845 (PIR 0x0845) -> 0x000000000000a83c
+    [  259.429870681,7] OPAL: Start CPU 0x0846 (PIR 0x0846) -> 0x000000000000a83c
+    [  259.434549250,7] OPAL: Start CPU 0x0847 (PIR 0x0847) -> 0x000000000000a83c
+
+- core/cpu: Fix memory allocation for job array
+
+  fixes: 7a3f307e core/cpu: parallelise global CPU register setting jobs
+
+  This bug would result in boot-hang on some configurations due to
+  cpu_wait_job() endlessly waiting for the last bogus jobs[cpu->pir] pointer.
+- i2c: Fix multiple-enqueue of the same request on NACK
+
+  i2c_request_send() will retry the request if the error is a NAK,
+  however it forgets to clear the "ud.done" flag. It will thus
+  loop again and try to re-enqueue the same request causing internal
+  request list corruption.
+- i2c: Ensure ordering between i2c_request_send() and completion
+
+  i2c_request_send loops waiting for a flag "uc.done" set by
+  the completion routine, and then look for a result code
+  also set by that same completion.
+
+  There is no synchronization, the completion can happen on another
+  processor, so we need to order the stores to uc and the reads
+  from uc so that uc.done is stored last and tested first using
+  memory barriers.
+- pci: Clarify power down logic
+
+  Currently pci_scan_bus() unconditionally calls pci_slot_set_power_state()
+  when it's finished scanning a bus. This is one of those things that
+  makes you go "WHAT?" when you first see it and frankly the skiboot PCI
+  code could do with less of that.
+
+Fast Reboot
+^^^^^^^^^^^
+
+- fast-reboot: parallel memory clearing
+
+  Arbitrarily pick 16GB as the unit of parallelism, and
+  split up clearing memory into jobs and schedule them
+  node-local to the memory (or on node 0 if we can't
+  work that out because it's the memory up to SKIBOOT_BASE)
+
+  This seems to cut at least ~40% time from memory zeroing on
+  fast-reboot on a 256GB Boston system.
+
+  For many systems, scanning PCI takes about as much time as
+  zeroing all of RAM, so we may as well do them at the same time
+  and cut a few seconds off the total fast reboot time.
+- fast-reboot: verify firmware "romem" checksum
+
+  This takes a checksum of skiboot memory after boot that should be
+  unchanged during OS operation, and verifies it before allowing a
+  fast reboot.
+
+  This is not read-only memory from skiboot's point of view, beause
+  it includes things like the opal branch table that gets populated
+  during boot.
+
+  This helps to improve the integrity of firmware against host and
+  runtime firmware memory scribble bugs.
+
+- core/fast-reboot: print the fast reboot disable reason
+
+  Once things start to go wrong, disable_fast_reboot can be called a
+  number of times, so make the first reason sticky, and also print it
+  to the console at disable time. This helps with making sense of
+  fast reboot disables.
+- Add fast-reboot property to /ibm,opal DT node
+
+  this means that if it's permanently disabled on boot, the test suite can
+  pick that up and not try a fast reboot test.
+
+Utilities
+---------
+
+- pflash: Add --skip option for reading
+
+  Add a --skip=N option to pflash to skip N number of bytes when reading.
+  This would allow users to print the VERSION partition without the STB
+  header by specifying the --skip=4096 argument, and it's a more generic
+  solution rather than making pflash depend on secure/trusted boot code.
+- xscom-utils: Rework getsram
+
+  Allow specifying a file on the command line to read OCC SRAM data into.
+  If no file is specified then we print it to stdout as text. This is a
+  bit inconsistent, but it retains compatibility with the existing tool.
+- xscom-utils/getsram: Make it work on P9
+
+  The XSCOM base address of the OCC control registers changed slightly
+  between P8 and P9. Fix this up and add a bit of PVR checking so we look
+  in the right place.
+- opal-prd: Fix opal-prd crash
+
+  Presently callback function from HBRT uses r11 to point to target function
+  pointer. r12 is garbage. This works fine when we compile with "-no-pie" option
+  (as we don't use r12 to calculate TOC).
+
+  As per ABIv2 : "r12 : Function entry address at global entry point"
+
+  With "-pie" compilation option, we have to set r12 to point to global function
+  entry point. So that we can calculate TOC properly.
+
+  Crash log without this patch: ::
+
+      opal-prd[2864]: unhandled signal 11 at 0000000000029320 nip 00000 00102012830 lr 0000000102016890 code 1
+
+
+Development and Debugging
+-------------------------
+
+- core/lock: Use try_lock_caller() in lock_caller() to capture owner
+
+  Otherwise we can get reports of core/lock.c owning the lock, which is
+  not helpful when tracking down ownership issues.
+- core/flash: Emit a warning if Skiboot version doesn't match
+
+  This means you'll get a warning that you've modified skiboot separately
+  to the rest of the PNOR image, which can be useful in determining what
+  firmware is actually running on a machine.
+- gcov: link in ctors* as newer GCC doesn't group them all
+
+  It seems that newer toolchains get us multiple ctors sections to link in
+  rather than just one. If we discard them (as we were doing), then we
+  don't have a working gcov build (and we get the "doesn't look sane"
+  warning on boot).
+- core/flash: Log return code when ffs_init() fails
+
+  Knowing the return code is at least better than not knowing the return
+  code.
+- gcov: Fix building with GCC8
+- travis/ci: rework Dockerfiles to produce build artifacts
+
+  ubuntu-latest was also missing clang, as ubuntu-latest is closer to
+  ubuntu 18.04 than 16.04
+- cpu: add cpu_queue_job_on_node()
+
+  Add a job scheduling API which will run the job on the requested
+  chip_id (or return failure).
+- opal-ci: Build old dtc version for fedora 28
+
+  There are patches that will go into dtc to fix the issues we hit, but
+  for the moment let's just build and use a slightly older version.
+- mem_region: Merge similar allocations when dumping
+
+  Currently we print one line for each allocation done at runtime when
+  dumping the memory allocations. We do a few thousand allocations at
+  boot so this can result in a huge amount of text being printed which
+  is a) slow to print, and b) Can result in the log buffer overflowing
+  which destroys otherwise useful information.
+
+  This patch adds a de-duplication to this memory allocation dump by
+  merging "similar" allocations (same location, same size) into one.
+
+  Unfortunately, the algorithm used to do the de-duplication is quadratic,
+  but considering we only dump the allocations in the event of a fatal
+  error I think this is acceptable. I also did some benchmarking and found
+  that on a ZZ it takes ~3ms to do a dump with 12k allocations. On a Zaius
+  it's slightly longer at about ~10ms for 10k allocs. However, the
+  difference there was due to the output being written to the UART.
+
+  This patch also bumps the log level to PR_NOTICE. PR_INFO messages are
+  suppressed at the default log level, which probably isn't something you
+  want considering we only dump the allocations when we run out of skiboot
+  heap space.
+- core/lock: fix timeout warning causing a deadlock false positive
+
+  If a lock waiter exceeds the warning timeout, it prints a message
+  while still registered as requesting the lock. Printing the message
+  can take locks, so if one is held when the owner of the original
+  lock tries to print a message, it will get a false positive deadlock
+  detection, which brings down the system.
+
+  This can easily be hit when there is a lot of HMI activity from a
+  KVM guest, where the timebase was not returned to host timebase
+  before calling the HMI handler.
+- hw/p8-i2c: Print the set error bits
+
+  This is purely to save me from having to look it up every time someone
+  gets an I2C error.
+- init: Fix starting stripped kernel
+
+  Currently if we try to run a raw/stripped binary kernel (ie. without
+  the elf header) we crash with: ::
+
+      [    0.008757768,5] INIT: Waiting for kernel...
+      [    0.008762937,5] INIT: platform wait for kernel load failed
+      [    0.008768171,5] INIT: Assuming kernel at 0x20000000
+      [    0.008779241,3] INIT: ELF header not found. Assuming raw binary.
+      [    0.017047348,5] INIT: Starting kernel at 0x0, fdt at 0x3044b230 14339 bytes
+      [    0.017054251,0] FATAL: Kernel is zeros, can't execute!
+      [    0.017059054,0] Assert fail: core/init.c:590:0
+      [    0.017065371,0] Aborting!
+
+  This is because we haven't set kernel_entry correctly in this path.
+  This fixes it.
+- cpu: Better output when waiting for a very long job
+
+  Instead of printing at the end if the job took more than 1s,
+  print in the loop every 30s along with a backtrace. This will
+  give us some output if the job is deadlocked.
+- lock: Fix interactions between lock dependency checker and stack checker
+
+  The lock dependency checker does a few nasty things that can cause
+  re-entrancy deadlocks in conjunction with the stack checker or
+  in fact other debug tests.
+
+  A lot of it revolves around taking a new lock (dl_lock) as part
+  of the locking process.
+
+  This tries to fix it by making sure we do not hit the stack
+  checker while holding dl_lock.
+
+  We achieve that in part by directly using the low-level __try_lock
+  and manually unlocking on the dl_lock, and making some functions
+  "nomcount".
+
+  In addition, we mark the dl_lock as being in the console path to
+  avoid deadlocks with the UART driver.
+
+  We move the enabling of the deadlock checker to a separate config
+  option from DEBUG_LOCKS as well, in case we chose to disable it
+  by default later on.
+- xscom-utils/adu_scoms.py: run 2to3 over it
+- clang: -Wno-error=ignored-attributes
+
+Mambo Platform
+^^^^^^^^^^^^^^
+
+- mambo: Merge PMEM_DISK and PMEM_VOLATILE code
+
+  PMEM_VOLATILE and PMEM_DISK can't be used together and are basically
+  copies of the same code.
+
+  This merges the two and allows them used together.  Same API is kept.
+- hw/chiptod: test QUIRK_NO_CHIPTOD in opal_resync_timebase
+
+  This allows some test coverage of deep stop states in Linux with
+  Mambo.
+- core/mem_region: mambo reserve kernel payload areas
+
+  Mambo image payloads get overwritten by the OS and by
+  fast reboot memory clearing because they have no region
+  defined. Add them, which allows fast reboot to work.
+
+Qemu platform
+^^^^^^^^^^^^^
+
+- nx: Don't abort on missing NX when using a QEMU machine
+
+  These don't have an NX node (and probably never will) as they
+  don't provide any coprocessor. However, the DARN instruction
+  works so this abort is unnecessary.
+
+POWER8 Platforms
+----------------
+- SBE-p8: Do all sbe timer update with xscom lock held
+
+  Without this, on some P8 platforms, we could (falsely) think the SBE timer
+  had stalled getting the dreaded "timer stuck" message.
+
+  The code was doing the mftb() to set the start of the timeout period while
+  *not* holding the lock, so the 1ms timeout started sometime when somebody
+  else had the xscom lock.
+
+  The simple solution is to just do the whole routine holding the xscom lock,
+  so do it that way.
+
+Vesnin Platform
+^^^^^^^^^^^^^^^
+- platforms/astbmc/vesnin: Send list of PCI devices to BMC through IPMI
+
+  Implements sending a list of installed PCI devices through IPMI protocol.
+  Each PCI device description is sent as a standalone IPMI message.
+  A list of devices can be gathered from separate messages using the
+  session identifier. The session Id is an incremental counter that is
+  updated at the start of synchronization session.
+
+
+POWER9 Platforms
+----------------
+
+- STOP API: API conditionally supports 255 SCOM restore entries for each quad.
+- hdata/i2c: Skip unknown device type
+
+  Do not add unknown I2C devices to device tree.
+- hdata/i2c: Add whitelisting for Host I2C devices
+
+  Many of the devices that we get information about through HDAT are for
+  use by firmware rather than the host operating system. This patch adds
+  a boolean flag to hdat_i2c_info structure that indicates whether devices
+  with a given purpose should be reserved for use inside of OPAL (or some
+  other firmware component, such as the OCC).
+- hdata/iohub: Fix Cumulus Hub ID number
+- opal/hmi: Wakeup the cpu before reading core_fir
+
+  When stop state 5 is enabled, reading the core_fir during an HMI can
+  result in a xscom read error with xscom_read() returning an
+  OPAL_XSCOM_PARTIAL_GOOD error code and core_fir value of all FFs. At
+  present this return error code is not handled in decode_core_fir()
+  hence the invalid core_fir value is sent to the kernel where it
+  interprets it as a FATAL hmi causing a system check-stop.
+
+  This can be prevented by forcing the core to wake-up using before
+  reading the core_fir. Hence this patch wraps the call to
+  read_core_fir() within calls to dctl_set_special_wakeup() and
+  dctl_clear_special_wakeup().
+- xive: Disable block tracker
+
+  Due to some HW errata, the block tracking facility (performance optimisation
+  for large systems) should be disabled on Nimbus chips. Disable it unconditionally
+  for now.
+- opal/hmi: Ignore debug trigger inject core FIR.
+
+  Core FIR[60] is a side effect of the work around for the CI Vector Load
+  issue in DD2.1. Usually this gets delivered as HMI with HMER[17] where
+  Linux already ignores it. But it looks like in some cases we may happen
+  to see CORE_FIR[60] while we are already in Malfunction Alert HMI
+  (HMER[0]) due to other reasons e.g. CAPI recovery or NPU xstop. If that
+  happens then just ignore it instead of crashing kernel as not recoverable.
+- hdata: Make sure reserved node name starts with "ibm, "
+
+  HDAT does not provide consistent label format for reserved memory label.
+  Few starts with "ibm," while few other starts with component name.
+- hdata: Fix dtc warnings
+
+  Fix dtc warnings related to mcbist node. ::
+
+    Warning (reg_format): "reg" property in /xscom@623fc00000000/mcbist@1 has invalid length (4 bytes) (#address-cells == 1, #size-cells == 1)
+    Warning (reg_format): "reg" property in /xscom@623fc00000000/mcbist@2 has invalid length (4 bytes) (#address-cells == 1, #size-cells == 1)
+    Warning (reg_format): "reg" property in /xscom@603fc00000000/mcbist@1 has invalid length (4 bytes) (#address-cells == 1, #size-cells == 1)
+    Warning (reg_format): "reg" property in /xscom@603fc00000000/mcbist@2 has invalid length (4 bytes) (#address-cells == 1, #size-cells == 1)
+
+  Ideally we should add proper xscom range here... but we are not getting that
+  information in HDAT today. Lets fix warning until we get proper data in HDAT.
+
+PHB4
+^^^^
+
+- phb4: Generate checkstop on AIB ECC corr/uncorr for DD2.0 parts
+
+  On DD2.0 parts, PCIe ECC protection is not warranted in the response
+  data path. Thus, for these parts, we need to flag any ECC errors
+  detected from the adjacent AIB RX Data path so the part can be
+  replaced.
+
+  This patch configures the FIRs so that we escalate these AIB ECC
+  errors to a checkstop so the parts can be replaced.
+- phb4: Reset pfir and nfir if new errors reported during ETU reset
+
+  During fast-reboot new PEC errors can be latched even after ETU-Reset
+  is asserted. This will result in values of variables nfir_cache and
+  pfir_cache to be out of sync.
+
+  During step-2 of CRESET nfir_cache and pfir_cache values are used to
+  bring the PHB out of reset state. However if these variables are out
+  as noted above of date the nfir/pfir registers are never reset
+  completely and ETU still remains frozen.
+
+  Hence this patch updates step-2 of phb4_creset to re-read the values of
+  nfir/pfir registers to check if any new errors were reported after
+  ETU-reset was asserted, report these new errors and reset the
+  nfir/pfir registers. This should bring the ETU out of reset
+  successfully.
+- phb4: Disable nodal scoped DMA accesses when PB pump mode is enabled
+
+  By default when a PCIe device issues a read request via the PHB it is first
+  issued with nodal scope. When accessing GPU memory the NPU does not know at the
+  time of response if the requested memory page is off node or not. Therefore
+  every read of GPU memory by a PHB is retried with larger scope which introduces
+  bandwidth and latency issues.
+
+  On smaller boxes which have pump mode enabled nodal and group scoped reads are
+  treated the same and both types of request are broadcast to one chip. Therefore
+  we can avoid the retry by disabling nodal scope on the PHB for these boxes. On
+  larger boxes nodal (single chip) and group (multiple chip) scoped reads are
+  treated differently. Therefore we avoid disabling nodal scope on large boxes
+  which have pump mode disabled to avoid all PHB requests being broadcast to
+  multiple chips.
+- phb4/capp: Only reset FIR bits that cause capp machine check
+
+  During CAPP recovery do_capp_recovery_scoms() will reset the CAPP Fir
+  register just after CAPP recovery is completed. This has an
+  unintentional side effect of preventing PRD from analyzing and
+  reporting this error. If PRD tries to read the CAPP FIR after opal has
+  already reset it, then it logs a critical error complaining "No active
+  error bits found".
+
+  To prevent this from happening we update do_capp_recovery_scoms() to
+  only reset fir bits that cause CAPP machine check (local xstop). This
+  is done by reading the CAPP Fir Action0/1 & Mask registers and
+  generating a mask which is then written on CAPP_FIR_CLEAR register.
+
+- phb4: Check for RX errors after link training
+
+  Some PHB4 PHYs can get stuck in a bad state where they are constantly
+  retraining the link. This happens transparently to skiboot and Linux
+  but will causes PCIe to be slow. Resetting the PHB4 clears the
+  problem.
+
+  We can detect this case by looking at the RX errors count where we
+  check for link stability. This patch does this by modifying the link
+  optimal code to check for RX errors. If errors are occurring we
+  retrain the link irrespective of the chip rev or card.
+
+  Normally when this problem occurs, the RX error count is maxed out at
+  255. When there is no problem, the count is 0. We chose 8 as the max
+  rx errors value to give us some margin for a few errors. There is also
+  a knob that can be used to set the error threshold for when we should
+  retrain the link. ie ::
+
+      nvram -p ibm,skiboot --update-config phb-rx-err-max=8
+
+- hw/phb4: Add a helper to dump the PELT-V
+
+  The "Partitionable Endpoint Lookup Table (Vector)" is used by the PHB
+  when processing EEH events. The PELT-V defines which PEs should be
+  additionally frozen in the event of an error being flagged on a
+  given PE. Knowing the state of the PELT-V is sometimes useful for
+  debugging PHB issues so this patch adds a helper to dump it.
+
+- hw/phb4: Print the PEs in the EEH dump in hex
+
+  Linux always displays the PE number in hexidecimal while skiboot
+  displays the PEST index (PE number) in decimal. This makes correlating
+  errors between Skiboot and Linux more annoying than it should be so
+  this patch makes Skiboot print the PEST number in hex.
+
+- phb4: Reallocate PEC2 DMA-Read engines to improve GPU-Direct bandwidth
+
+  We reallocate additional 16/8 DMA-Read engines allocated to stack0/1
+  on PEC2 respectively. This is needed to improve bandwidth available to
+  the Mellanox CX5 adapter when trying to read GPU memory (GPU-Direct).
+
+  If kernel cxl driver indicates a request to allocate maximum possible
+  DMA read engines when calling enable_capi_mode() and card is attached
+  to PEC2/stack0 slot then we assume its a Mellanox CX5 adapter. We then
+  allocate additional 16/8 extra DMA read engines to stack0 and stack1
+  respectively on PEC2. This is done by populating the
+  XPEC_PCI_PRDSTKOVR and XPEC_NEST_READ_STACK_OVERRIDE as suggested by
+  the h/w team.
+- phb4: Enable PHB MMIO-0/1 Bars only when mmio window exists
+
+  Presently phb4_probe_stack() will always enable PHB MMIO0/1 windows
+  even if they doesn't exist in phy_map. Hence we do some minor shuffling
+  in the phb4_probe_stack() so that MMIO-0/1 Bars are only enabled if
+  there corresponding MMIO window exists in the phy_map. In case phy_map
+  for an mmio window is '0' we set the corresponding BAR register to
+  '0'.
+- hw/phb4: Use local_alloc for phb4 structures
+
+  Struct phb4 is fairly heavyweight at 283664 bytes. On systems with
+  6x PHBs per socket this results in using 3.2MB of heap space the PHB
+  structures alone. This is a fairly large chunk of our 12MB heap and
+  on systems with particularly large PCIe topologies, or additional
+  PHBs we can fail to boot because we cannot allocate space for the
+  FDT blob.
+
+  This patch switches to using local_alloc() for the PHB structures
+  so they don't consume too large a portion of our 12MB heap space.
+- phb4: Fix typo in disable lane eq code
+
+  In this commit ::
+
+      commit 737c0ba3d72b8aab05a765a9fc111a48faac0f75
+      Author: Michael Neuling <mikey@neuling.org>
+      Date:   Thu Feb 22 10:52:18 2018 +1100
+      phb4: Disable lane eq when retrying some nvidia GEN3 devices
+
+  We made a typo and set PH2 twice. This fixes it.
+
+  It worked previously as if only phase 2 (PH2) is set it, skips phase 2
+  and phase 3 (PH3).
+- phb4: Don't probe a PHB if its garded
+
+  Presently phb4_probe_stack() causes an exception while trying to probe
+  a PHB if its garded. This causes skiboot to go into a reboot loop with
+  following exception log: ::
+
+     ***********************************************
+     Fatal MCE at 000000003006ecd4   .probe_phb4+0x570
+     CFAR : 00000000300b98a0
+     <snip>
+     Aborting!
+    CPU 0018 Backtrace:
+     S: 0000000031cc37e0 R: 000000003001a51c   ._abort+0x4c
+     S: 0000000031cc3860 R: 0000000030028170   .exception_entry+0x180
+     S: 0000000031cc3a40 R: 0000000000001f10 *
+     S: 0000000031cc3c20 R: 000000003006ecb0   .probe_phb4+0x54c
+     S: 0000000031cc3e30 R: 0000000030014ca4   .main_cpu_entry+0x5b0
+     S: 0000000031cc3f00 R: 0000000030002700   boot_entry+0x1b8
+
+  This is caused as phb4_probe_stack() will ignore all xscom read/write
+  errors to enable PHB Bars and then tries to perform an mmio to read
+  PHB Version registers that cause the fatal MCE.
+
+  We fix this by ignoring the PHB probe if the first xscom_write() to
+  populate the PHB Bar register fails, which indicates that there is
+  something wrong with the PHB.
+- phb4: Workaround PHB errata with CFG write UR/CA errors
+
+  If the PHB encounters a UR or CA status on a CFG write, it will
+  incorrectly freeze the wrong PE. Instead of using the PE# specified
+  in the CONFIG_ADDRESS register, it will use the PE# of whatever
+  MMIO occurred last.
+
+  Work around this disabling freeze on such errors
+- phb4: Handle allocation errors in phb4_eeh_dump_regs()
+
+  If the zalloc fails (and it can be a rather large allocation),
+  we will overwite memory at 0 instead of failing.
+- phb4: Don't try to access non-existent PEST entries
+
+  In a POWER9 chip, some PHB4s have 256 PEs, some have 512.
+
+  Currently, the diagnostics code retrieves 512 unconditionally,
+  which is wrong and causes us to incorrectly report bogus values
+  for the "high" PEs on the small PHBs.
+
+  Use the actual number of implemented PEs instead
+
+CAPI2
+^^^^^
+
+- phb4/capp: Use link width to allocate STQ engines to CAPP
+
+  Update phb4_init_capp_regs() to allocates STQ Engines to CAPP/PEC2
+  based on link width instead of always assuming it to x8.
+
+  Also re-factor the function slightly to evaluate the link-width only
+  once and cache it so that it can also be used to allocate DMA read
+  engines.
+- phb4/capp: Update DMA read engines set in APC_FSM_READ_MASK based on link-width
+
+  Commit 47c09cdfe7a3("phb4/capp: Calculate STQ/DMA read engines based
+  on link-width for PEC") update the CAPP init sequence by calculating
+  the needed STQ/DMA-read engines based on link width and populating it
+  in XPEC_NEST_CAPP_CNTL register. This however needs to be synchronized
+  with the value set in CAPP APC FSM Read Machine Mask Register.
+
+  Hence this patch update phb4_init_capp_regs() to calculate the link
+  width of the stack on PEC2 and populate the same values as previously
+  populated in PEC CAPP_CNTL register.
+- capp: Fix the capp recovery timeout comparison
+
+  The current capp recovery timeout control loop in
+  do_capp_recovery_scoms() uses a wrong comparison for return value of
+  tb_compare(). This may cause do_capp_recovery_scoms() to report an
+  timeout earlier than the 168ms stipulated time.
+
+  The patch fixes this by updating the loop timeout control branch in
+  do_capp_recovery_scoms() to use the correct enum tb_cmpval.
+- phb4: Disable 32-bit MSI in capi mode
+
+  If a capi device does a DMA write targeting an address lower than 4GB,
+  it does so through a 32-bit operation, per the PCI spec. In capi mode,
+  the first TVE entry is configured in bypass mode, so the address is
+  valid. But with any (bad) luck, the address could be 0xFFFFxxxx, thus
+  looking like a 32-bit MSI.
+
+  We currently enable both 32-bit and 64-bit MSIs, so the PHB will
+  interpret the DMA write as a MSI, which very likely results in an EEH
+  (MSI with a bad payload size).
+
+  We can fix it by disabling 32-bit MSI when switching the PHB to capi
+  mode. Capi devices are 64-bit.
+
+NVLINK2
+^^^^^^^
+- npu2: Add support for relaxed-ordering mode
+
+  Some device drivers support out of order access to GPU memory. This does
+  not affect the CPU view of memory but it does affect the GPU view of
+  memory. It should only be enabled if the GPU driver has requested it.
+
+  Add OPAL APIs allowing the driver to query relaxed ordering state or
+  request it to be set for a device. Current hardware only allows relaxed
+  ordering to be enabled per PCIe root port. So the code here doesn't
+  enable relaxed ordering until it has been explicitly requested for every
+  device on the port.
+- Add the other 7 ATSD registers to the device tree.
+- npu2/hw-procedures: Don't open code NPU2_NTL_MISC_CFG2_BRICK_ENABLE
+
+  Name this bit properly. There's a lot more cleanup like this to be done,
+  but I'm catching this one now as part of some related changes.
+- npu2/hw-procedures: Enable parity and credit overflow checks
+
+  Enable these error checking features by setting the appropriate bits in
+  our one-off initialization of each "NTL Misc Config 2" register.
+
+  The exception is NDL RX parity checking, which should be disabled during
+  the link training procedures.
+- npu2: Use correct kill type for TCE invalidation
+
+  kill_type is enum of OPAL_PCI_TCE_KILL_PAGES, OPAL_PCI_TCE_KILL_PE,
+  OPAL_PCI_TCE_KILL_ALL and phb4_tce_kill() gets it right but
+  npu2_tce_kill() uses OPAL_PCI_TCE_KILL which is an OPAL API token.
+
+  This fixes an obvious mistype.
+
+OpenCAPI
+^^^^^^^^
+
+- Support OpenCAPI on Witherspoon platform
+- npu2-opencapi: Enable presence detection on ZZ
+
+  Presence detection for opencapi adapters was broken for ZZ planars v3
+  and below. All ZZ systems currently used in the lab have had their
+  planar upgraded, so we can now remove the override we had to force
+  presence and activate presence detection. Which should improve boot
+  time.
+
+  Considering the state of opal support on ZZ, this is really only for
+  lab usage on BML. The opencapi enablement team has okay'd the
+  change. In the unlikely case somebody tries opencapi on an old ZZ, the
+  presence detection through i2c will show that no adapter is present
+  and skiboot won't try to access or train the link.
+- npu2-opencapi: Don't send commands to NPU when link is down
+
+  Even if an opencapi link is down, we currently always try to issue a
+  config read operation when probing for PCI devices, because of the
+  default scan map used for an opencapi PHB. The config operation fails,
+  as expected, but it can also raise a FIR bit and trigger an HMI.
+
+  For opencapi, there's no root device like for a "normal" PCI PHB, so
+  there's no reason to do the config operation. To fix it, we keep the
+  scan map blank by default, and only add a device once the link is
+  trained.
+- opal/hmi: Catch NPU2 HMIs for opencapi
+
+  HMIs for NPU2 are filtered with the 'compatible' string of the PHB, so
+  add opencapi to the mix.
+- occ: Wait if OCC GPU presence status not immediately available
+
+  It takes a few seconds for the OCC to set everything up in order to read
+  GPU presence. At present, we try to kick off OCC initialisation as early as
+  possible to maximise the time it has to read GPU presence.
+
+  Unfortunately sometimes that's not enough, so add a loop in
+  occ_get_gpu_presence() so that on the first time we try to get GPU presence
+  we keep trying for up to 2 seconds. Experimentally this seems to be
+  adequate.
+- hw/npu2-hw-procedures: Enable RX auto recal on OpenCAPI links
+
+  The RX_RC_ENABLE_AUTO_RECAL flag is required on OpenCAPI but not NVLink.
+
+  Traditionally, Hostboot sets this value according to the machine type.
+  However, now that Witherspoon supports both NVLink and OpenCAPI, it can't
+  tell whether or not a link is OpenCAPI.
+
+  So instead, set it in skiboot, where it will only be triggered after we've
+  done device detection and found an OpenCAPI device.
+- hw/npu2-opencapi: Fix setting of supported OpenCAPI templates
+
+  In opal_npu_tl_set(), we made a typo that means the OPAL_NPU_TL_SET call
+  may not clear the enable bits for templates that were previously enabled
+  but are now disabled.
+
+  Fix the typo so we clear NPU2_OTL_CONFIG1_TX_TEMP2_EN as well as
+  TEMP{1,3}_EN.
+
+Barreleye G2 and Zaius platforms
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+- zaius: Add a slot table
+- zaius: Add slots for the Barreleye G2 HDD rack
+
+  The Barreleye G2 is distinct from the Zaius in that it features a 24
+  Bay NVMe/SATA HDD rack. To provide meaningful slot names for each NVMe
+  device we need to define a slot table for the NVMe capable HDD bays.
+
+  Unfortunately this isn't straightforward because the PCIe path to the
+  NVMe devices isn't fixed. The PCIe topology is something like:
+  P9 -> HBA card -> 9797 switch -> 20x NVMe HDD slots
+
+  The 9797 switch is partitioned into two (or four) virtual switches which
+  allow multiple HBA cards to be used (e.g. one per socket). As a result
+  the exact BDFN of the ports will vary depending on how the system is
+  configured.
+
+  That said, the virtual switch configuration of the 9797 does not change
+  the device and function numbers of the switch downports. This means that
+  we can define a single slot table that maps switch ports to the NVMe bay
+  names.
+
+  Unfortunately we still need to guess which bus to use this table on, so
+  we assume that any switch downport we find with the PEX9797 VDID is part
+  of the 9797 that supports the HDD rack.
+
+FSP based platforms (firenze and ZZ)
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+- phb4/capp: Update the expected Eye-catcher for CAPP ucode lid
+
+  Currently on a FSP based P9 system load_capp_code() expects CAPP ucode
+  lid header to have eye-catcher magic of 'CAPPPSLL'. However skiboot
+  currently supports CAPP ucode only lids that have a eye-catcher magic
+  of 'CAPPLIDH'. This prevents skiboot from loading the ucode with this
+  error message: ::
+
+    CAPP: ucode header invalid
+
+  We fix this issue by updating load_capp_ucode() to use the eye-catcher
+  value of 'CAPPLIDH' instead of 'CAPPPSLL'.
+
+- FSP: Improve Reset/Reload log message
+
+  Below message is confusing. Lets make it clear.
+
+  FSP sends "R/R complete notification" whenever there is a dump. We use `flag`
+  to identify whether its its R/R completion -OR- just new dump notification. ::
+
+    [  483.406351956,6] FSP: SP says Reset/Reload complete
+    [  483.406354278,5] DUMP: FipS dump available. ID = 0x1a00001f [size: 6367640 bytes]
+    [  483.406355968,7]   A Reset/Reload was NOT done
+
+Witherspoon platform
+^^^^^^^^^^^^^^^^^^^^
+
+- platforms/astbmc/witherspoon: Implement OpenCAPI support
+
+  OpenCAPI on Witherspoon is slightly more involved than on Zaius and ZZ, due
+  to the OpenCAPI links using the SXM2 connectors that are used for NVLink
+  GPUs.
+
+  This patch adds the regular OpenCAPI platform information, and also a
+  Witherspoon-specific presence detection callback that uses the previously
+  added OCC GPU presence detection to figure out the device types plugged
+  into each SXM2 socket.
+
+  The SXM2 connectors are capable of carrying 2 OpenCAPI links, and future
+  OpenCAPI devices are expected to make use of this. However, we don't yet
+  support ganged links and the various implications that has for handling
+  things like device reset, so for now, we only enable 1 brick per device.