The absence of a probe implementation means that the extension is
always available. Remove the implementation for the PMU extension,
which does no checking, and indeed even has a comment saying it's
always available.

Signed-off-by: Andrew Jones <ajones@ventanamicro.com>
Reviewed-by: Anup Patel <anup@brainfault.org>

The vendor extension ID range is large, but at runtime at most
a single ID will be available. Narrow the range in the
register_extensions callback. After narrowing, we no longer
need to check that the extension ID is correct in the other
callbacks, as those callbacks will never be invoked with
anything other than the single ID.

Signed-off-by: Andrew Jones <ajones@ventanamicro.com>
Reviewed-by: Anup Patel <anup@brainfault.org>

Rather than registering all extensions on their behalf in
sbi_ecall_init(), introduce another extension callback and
invoke that instead. For now, implement each callback by
simply registering the extension, which means this patch
has no intended functional change. In later patches, extension
callbacks will be modified to choose when to register and to
possibly narrow the extension ID range prior to registering.
When an extension range needs to remove IDs, leaving gaps, then
multiple invocations of sbi_ecall_register_extension() may be
used. In summary, later patches for current extensions and the
introductions of future extensions will use the new callback to
ensure that only valid extension IDs from the initial range,
which are also available, will be registered.

Signed-off-by: Andrew Jones <ajones@ventanamicro.com>
Reviewed-by: Anup Patel <anup@brainfault.org>

No need to use #elif ladder when defining BITS_PER_LONG.

Signed-off-by: Xiang W <wxjstz@126.com>
Reviewed-by: Andrew Jones <ajones@ventanamicro.com>
Reviewed-by: Anup Patel <anup@brainfault.org>

Simplify sbi_ipi_process() by removing goto statement.

Signed-off-by: Xiang W <wxjstz@126.com>
Reviewed-by: Anup Patel <anup@brainfault.org>

We have redundant semicolon at quite a few places so let's remove it.

Signed-off-by: Xiang W <wxjstz@126.com>
Reviewed-by: Anup Patel <anup@brainfault.org>

Since sbi_pmu_init is called after sbi_console_init,
the sbi_printf can be called when sbi_pmu_init fails.

Signed-off-by: Tan En De <ende.tan@starfivetech.com>
Reviewed-by: Atish Patra <atishp@rivosinc.com>

The legacy console getchar SBI call returns character value in
the sbiret.error field so the "SBI_SUCCESS < ret" check in
sbi_ecall_handler() results in unwanted error prints for the
legacy console getchar SBI call. Let's suppress these unwanted
error prints.

Fixes: 67b2a408

 ("lib: sbi: sbi_ecall: Check the range of
SBI error")
Signed-off-by: Anup Patel <apatel@ventanamicro.com>

As per the SBI specification, we should "unconditionally select the first
counter from the set of counters specified by the counter_idx_base and
counter_idx_mask", so implement this behaviour.

Suggested-by: Atish Patra <atishp@atishpatra.org>
Signed-off-by: Alexandre Ghiti <alexghiti@rivosinc.com>
Reviewed-by: Atish Patra <atishp@rivosinc.com>

Those events are enabled by default and should not be reset afterwards
since when using SBI_PMU_CFG_FLAG_SKIP_MATCH, it leads to unaccessible
counters after the first use.

Signed-off-by: Alexandre Ghiti <alexghiti@rivosinc.com>
Reviewed-by: Atish Patra <atishp@rivosinc.com>

The codes currently skip the very first relocation entry, but later
reference the elements in the relocation entry using minus offsets.

Change to use positive offsets so that there is no need to skip the
first relocation entry.

Signed-off-by: Bin Meng <bmeng@tinylab.org>
Reviewed-by: Anup Patel <anup@brainfault.org>
Reviewed-by: Xiang W <wxjstz@126.com>

't5' already contains relocation type so don't bother reloading it.

Signed-off-by: Bin Meng <bmeng@tinylab.org>
Reviewed-by: Anup Patel <anup@brainfault.org>
Reviewed-by: Xiang W <wxjstz@126.com>

I/O Coherence Port (IOCP) provides an AXI interface for connecting
external non-caching masters, such as DMA controllers. The accesses
from IOCP are coherent with D-Caches and L2 Cache.

IOCP is a specification option and is disabled on the Renesas RZ/Five
SoC (which is based on Andes AX45MP core) due to this reason IP blocks
using DMA will fail.

As a workaround for SoCs with IOCP disabled CMO needs to be handled by
software. Firstly OpenSBI configures the memory region as
"Memory, Non-cacheable, Bufferable" and passes this region as a global
shared dma pool as a DT node. With DMA_GLOBAL_POOL enabled all DMA
allocations happen from this region and synchronization callbacks are
implemented to synchronize when doing DMA transactions.

SBI_EXT_ANDES_IOCP_SW_WORKAROUND checks if the IOCP errata should be
applied to handle cache management.

Signed-off-by: Lad Prabhakar <prabhakar.mahadev-lad.rj@bp.renesas.com>
Reviewed-by: Anup Patel <anup@brainfault.org>
Reviewed-by: Conor Dooley <conor.dooley@microchip.com>
Reviewed-by: Yu Chien Peter Lin <peterlin@andestech.com>

When tlb_fifo is full, it will wait and affect the ipi update to
other harts. This patch is optimized.

Signed-off-by: Xiang W <wxjstz@126.com>
Reviewed-by: Anup Patel <anup@brainfault.org>
Tested-by: Anup Patel <anup@brainfault.org>

tlb_process_count is only used when count=1, so refactor to
tlb_process_once and add the return value to be reused in
tlb_process

Signed-off-by: Xiang W <wxjstz@126.com>
Reviewed-by: Anup Patel <anup@brainfault.org>
Tested-by: Anup Patel <anup@brainfault.org>

The original sbi_ipi will be processed by hart by hart, after optimization,
send ipi first and finally wait together.

Signed-off-by: Xiang W <wxjstz@126.com>
Reviewed-by: Anup Patel <anup@brainfault.org>
Tested-by: Anup Patel <anup@brainfault.org>

Originally, the process and sync of sbi_tlb need to wait for each other.
Evasion by atomic addition and subtraction.

Signed-off-by: Xiang W <wxjstz@126.com>
Reviewed-by: Anup Patel <anup@brainfault.org>
Tested-by: Anup Patel <anup@brainfault.org>

fdt_reserved_memory_fixup() uses filtered_order[PMP_COUNT]. The index
must not reach PMP_COUNT.

Fixes: 199189bd

 ("lib: utils: Mark only the largest region as reserved in FDT")
Addresses-Coverity-ID: 1536994 ("Out-of-bounds write")
Signed-off-by: Heinrich Schuchardt <heinrich.schuchardt@canonical.com>
Reviewed-by: Xiang W <wxjstz@126.com>
Reviewed-by: Anup Patel <anup@brainfault.org>

When using `PLATFORM=generic` defaults, the kernel is loaded at
`FW_JUMP_ADDR`, and the FDT is loaded at `FW_JUMP_FDT_ADDR.

Therefore, the maximum kernel size before `FW_JUMP_FDT_ADDR` must
be increased is `$(( FW_JUMP_FDT_ADDR - FW_JUMP_ADDR ))`.

The example calculation assumes `rv64`, and is wrong to boot
(off by 0x200000). Fix it and update it for the general case.

Signed-off-by: Gabriel Somlo <gsomlo@gmail.com>
Reviewed-by: Andrew Jones <ajones@ventanamicro.com>
Reviewed-by: Xiang W <wxjstz@126.com>

The bits encoded in event_idx[19:16] indicate the event type, with
an offset of 16 instead of 20.

Fixes: 13d40f21

 ("lib: sbi: Add PMU support")
Signed-off-by: Yu Chien Peter Lin <peterlin@andestech.com>
Reviewed-by: Andrew Jones <ajones@ventanamicro.com>
Reviewed-by: Xiang W <wxjstz@126.com>

If CPPC device is registered by the platform, print its name.

Signed-off-by: Sunil V L <sunilvl@ventanamicro.com>
Reviewed-by: Andrew Jones <ajones@ventanamicro.com>
Reviewed-by: Anup Patel <anup@brainfault.org>

Implement SBI CPPC extension. This extension is only available when
OpenSBI platform provides a CPPC device to generic library.

Signed-off-by: Sunil V L <sunilvl@ventanamicro.com>
Reviewed-by: Andrew Jones <ajones@ventanamicro.com>
Reviewed-by: Anup Patel <anup@brainfault.org>

Add SBI CPPC extension related defines to the
SBI ecall interface header.

Signed-off-by: Sunil V L <sunilvl@ventanamicro.com>
Reviewed-by: Andrew Jones <ajones@ventanamicro.com>
Reviewed-by: Anup Patel <anup@brainfault.org>

If mip.SEIP bit is not cleared then on HiFive Unmatched board it causes
spurious external interrupts. This breaks the boot up of HiFive Unmatched
board. Hence it is required to bring the mip CSR to a known state during
hart init and avoid spurious interrupts.

Fixes: d9e7368 ("firmware: Not to clear all the MIP")
Signed-off-by: Mayuresh Chitale <mchitale@ventanamicro.com>
Reviewed-by: Anup Patel <anup@brainfault.org>

It possible that a platform supports hart hotplug (i.e. both hart_start
and hart_stop callbacks available) and all harts are start simultaneously
at platform boot-time. In this situation, the sbi_hsm_hart_start() will
call hsm_device_hart_start() for secondary harts at platform boot-time
which will fail because secondary harts were already started.

To fix above, we call hsm_device_hart_start() from sbi_hsm_hart_start()
only when entry_count is same as init_count for the secondary hart.

Signed-off-by: Anup Patel <apatel@ventanamicro.com>

We introduce sbi_entry_count() function which counts the number
of times a HART enters OpenSBI via cold-boot or warm-boot path.

Signed-off-by: Anup Patel <apatel@ventanamicro.com>

Because firmware is split into rw/rx segments, it cannot be recorded
by a root_fw_region. This problem is solved by adding a flag
fw_region_inited to sbi_domain.

Signed-off-by: Xiang W <wxjstz@126.com>
Reviewed-by: Himanshu Chauhan <hchauhan@ventanamicro.com>
Reviewed-by: Anup Patel <anup@brainfault.org>

Signed-off-by: Xiang W <wxjstz@126.com>
Reviewed-by: Anup Patel <anup@brainfault.org>

Since commit 50d4fde1

 ("lib: Remove redundant sbi_platform_ipi_clear()
calls"), the IPI sent from the boot hart in wake_coldboot_harts() is not
cleared in the secondary harts until they reach sbi_ipi_init(). However,
sbi_hsm_init() and sbi_hsm_hart_wait() are called earlier, so a secondary
hart might enter sbi_hsm_hart_wait() with an already pending IPI.

sbi_hsm_hart_wait() makes sure the hart leaves the loop only when it is
actually ready, so a pending unrelated IPI should not cause safety issues.
However, it might be inefficient on certain hardware, because it prevents
"wfi" from stalling the hart even if the hardware supports this, making the
hart needlessly spin in a "busy-wait" loop.

This behaviour can be observed, for example, in a QEMU VM (QEMU 7.2.0) with
"-machine virt" running a Linux guest. Inserting delays in
sbi_hsm_hart_start() allows reproducing the issue more reliably.

The comment in wait_for_coldboot() suggests that the initial IPI is needed
in the warm resume path, so let us clear it before init_warm_startup()
only.

To do this, sbi_ipi_raw_clear() was created similar to sbi_ipi_raw_send().

Signed-off-by: Evgenii Shatokhin <e.shatokhin@yadro.com>
Reviewed-by: Anup Patel <anup@brainfault.org>

When a boot hart executes sbi_hsm_hart_start() to start a secondary hart,
next_arg1, next_addr and next_mode for the latter are stored in the scratch
area after the state has been set to SBI_HSM_STATE_START_PENDING.

The secondary hart waits in the loop with wfi() in sbi_hsm_hart_wait() at
that time. However, "wfi" instruction is not guaranteed to wait for an
interrupt to be received by the hart, it is just a hint for the CPU.
According to RISC-V Privileged Architectures spec. v20211203, even an
implementation of "wfi" as "nop" is legal.

So, the secondary might leave the loop in sbi_hsm_hart_wait() as soon as
its state has been set to SBI_HSM_STATE_START_PENDING, even if it got no
IPI or it got an IPI unrelated to sbi_hsm_hart_start(). This could lead to
the following race condition when booting Linux, for example:

  Boot hart (#0)                        Secondary hart (#1)
  runs Linux startup code               waits in sbi_hsm_hart_wait()

  sbi_ecall(SBI_EXT_HSM,
            SBI_EXT_HSM_HART_START,
            ...)
  enters sbi_hsm_hart_start()
  sets state of hart #1 to START_PENDING
                                        leaves sbi_hsm_hart_wait()
                                        runs to the end of init_warmboot()
                                        returns to scratch->next_addr
                                        (next_addr can be garbage here)

  sets next_addr, etc. for hart #1
  (no good: hart #1 has already left)

  sends IPI to hart #1
  (no good either)

If this happens, the secondary hart jumps to a wrong next_addr at the end
of init_warmboot(), which leads to a system hang or crash.

To reproduce the issue more reliably, one could add a delay in
sbi_hsm_hart_start() after setting the hart's state but before sending
IPI to that hart:

    hstate = atomic_cmpxchg(&hdata->state, SBI_HSM_STATE_STOPPED,
                            SBI_HSM_STATE_START_PENDING);
    ...
  + sbi_timer_mdelay(10);
    init_count = sbi_init_count(hartid);
    rscratch->next_arg1 = arg1;
    rscratch->next_addr = saddr;

The issue can be reproduced, for example, in a QEMU VM with '-machine virt'
and 2 or more CPUs, with Linux as the guest OS.

This patch moves writing of next_arg1, next_addr and next_mode for the
secondary hart before setting its state to SBI_HSM_STATE_START_PENDING.

In theory, it is possible that two or more harts enter sbi_hsm_hart_start()
for the same target hart simultaneously. To make sure the current hart has
exclusive access to the scratch area of the target hart at that point, a
per-hart 'start_ticket' is used. It is initially 0. The current hart tries
to acquire the ticket first (set it to 1) at the beginning of
sbi_hsm_hart_start() and only proceeds if it has successfully acquired it.

The target hart reads next_addr, etc., and then the releases the ticket
(sets it to 0) before calling sbi_hart_switch_mode(). This way, even if
some other hart manages to enter sbi_hsm_hart_start() after the ticket has
been released but before the target hart jumps to next_addr, it will not
cause problems.

atomic_cmpxchg() already has "acquire" semantics, among other things, so
no additional barriers are needed in hsm_start_ticket_acquire(). No hart
can perform or observe the update of *rscratch before setting of
'start_ticket' to 1.

atomic_write() only imposes ordering of writes, so an explicit barrier is
needed in hsm_start_ticket_release() to ensure its "release" semantics.
This guarantees that reads of scratch->next_addr, etc., in
sbi_hsm_hart_start_finish() cannot happen after 'start_ticket' has been
released.

Signed-off-by: Evgenii Shatokhin <e.shatokhin@yadro.com>
Reviewed-by: Anup Patel <anup@brainfault.org>

Move them into sbi_hsm_hart_start_finish() and sbi_hsm_hart_resume_finish()
to make them easier to manage.

This will be used by subsequent patches.

Suggested-by: Anup Patel <anup@brainfault.org>
Signed-off-by: Evgenii Shatokhin <e.shatokhin@yadro.com>
Reviewed-by: Anup Patel <anup@brainfault.org>

Platform specific firmware event handler may leverage the hartid to program
per hart specific registers for a given counter.

Signed-off-by: Mayuresh Chitale <mchitale@ventanamicro.com>
Reviewed-by: Atish Patra <atishp@rivosinc.com>
Reviewed-by: Anup Patel <anup@brainfault.org>

Add fw_counter_write_value API for platform specific firmware events
which separates setting the counter's initial value from starting the
counter. This is required so that the fw_event_data array can be reused
to save the event data received.

Signed-off-by: Mayuresh Chitale <mchitale@ventanamicro.com>
Reviewed-by: Anup Patel <anup@brainfault.org>

For all platform specific firmware event operations use the dedicated
event code (0xFFFF) when matching against the input firmware event.
Furthermore save the real platform specific firmware event code received as
the event data for future use.

Signed-off-by: Mayuresh Chitale <mchitale@ventanamicro.com>
Reviewed-by: Anup Patel <anup@brainfault.org>

Update fw_event_validate_code, fw_counter_match_code and fw_counter_start
ops which used a 32 bit event code to use the 64 bit event data instead.

Signed-off-by: Mayuresh Chitale <mchitale@ventanamicro.com>
Reviewed-by: Anup Patel <anup@brainfault.org>

Rename and reuse fw_counter_value array to save both the counter values
for the SBI firmware events and event data for the SBI platform specific
firmware events.

Signed-off-by: Mayuresh Chitale <mchitale@ventanamicro.com>
Reviewed-by: Anup Patel <anup@brainfault.org>
Reviewed-by: Andrew Jones <ajones@ventanamicro.com>

We reserve space for SBI implementation specific custom firmware
events which can be used by M-mode firmwares and HS-mode hypervisors
for their own use. This reserved space is intentionally large to
ensure that SBI implementation has enough space to accommodate
platform specific firmware events as well.

Signed-off-by: Mayuresh Chitale <mchitale@ventanamicro.com>
Reviewed-by: Atish Patra <atishp@rivosinc.com>
Reviewed-by: Anup Patel <anup@brainfault.org>

To support 64 bit firmware counters on RV32 systems, we implement
sbi_pmu_counter_fw_read_hi() which returns the upper 32 bits of
the firmware counter value. On RV64 (or higher) systems, this
function will always return zero.

Signed-off-by: Mayuresh Chitale <mchitale@ventanamicro.com>
Reviewed-by: Atish Patra <atishp@rivosinc.com>
Reviewed-by: Anup Patel <anup@brainfault.org>

This patch adds a callback to fetch the number of bits implemented for a
custom firmware counter. If the callback fails or is not implemented then
width defaults to 63.

Signed-off-by: Mayuresh Chitale <mchitale@ventanamicro.com>
Reviewed-by: Atish Patra <atishp@rivosinc.com>
Reviewed-by: Anup Patel <anup@brainfault.org>

The Cadence driver does not use the RX byte status feature and hence can
be advertised to be compatible with cdns,uart-r1p8 as well.

Signed-off-by: Mayuresh Chitale <mchitale@ventanamicro.com>
Reviewed-by: Anup Patel <anup@brainfault.org>