gdb/amdgpu: Fix debugging multiple inferiors using the ROCm runtime

When debugging a multi-process application where a parent spawns
multiple child processes using the ROCm runtime, I see the following
assertion failure:

    ../../gdb/amd-dbgapi-target.c:1071: internal-error: process_one_event: Assertion `runtime_state == AMD_DBGAPI_RUNTIME_STATE_UNLOADED' failed.
    A problem internal to GDB has been detected,
    further debugging may prove unreliable.
    ----- Backtrace -----
    0x556e9a318540 gdb_internal_backtrace_1
            ../../gdb/bt-utils.c:122
    0x556e9a318540 _Z22gdb_internal_backtracev
            ../../gdb/bt-utils.c:168
    0x556e9a730224 internal_vproblem
            ../../gdb/utils.c:396
    0x556e9a7304e0 _Z15internal_verrorPKciS0_P13__va_list_tag
            ../../gdb/utils.c:476
    0x556e9a87aeb4 _Z18internal_error_locPKciS0_z
            ../../gdbsupport/errors.cc:58
    0x556e9a29f446 process_one_event
            ../../gdb/amd-dbgapi-target.c:1071
    0x556e9a29f446 process_event_queue
            ../../gdb/amd-dbgapi-target.c:1156
    0x556e9a29faf2 _ZN17amd_dbgapi_target4waitE6ptid_tP17target_waitstatus10enum_flagsI16target_wait_flagE
            ../../gdb/amd-dbgapi-target.c:1262
    0x556e9a6b0965 _Z11target_wait6ptid_tP17target_waitstatus10enum_flagsI16target_wait_flagE
            ../../gdb/target.c:2586
    0x556e9a4c221f do_target_wait_1
            ../../gdb/infrun.c:3876
    0x556e9a4d8489 operator()
            ../../gdb/infrun.c:3935
    0x556e9a4d8489 do_target_wait
            ../../gdb/infrun.c:3964
    0x556e9a4d8489 _Z20fetch_inferior_eventv
            ../../gdb/infrun.c:4365
    0x556e9a87b915 gdb_wait_for_event
            ../../gdbsupport/event-loop.cc:694
    0x556e9a87c3a9 gdb_wait_for_event
            ../../gdbsupport/event-loop.cc:593
    0x556e9a87c3a9 _Z16gdb_do_one_eventi
            ../../gdbsupport/event-loop.cc:217
    0x556e9a521689 start_event_loop
            ../../gdb/main.c:412
    0x556e9a521689 captured_command_loop
            ../../gdb/main.c:476
    0x556e9a523c04 captured_main
            ../../gdb/main.c:1320
    0x556e9a523c04 _Z8gdb_mainP18captured_main_args
            ../../gdb/main.c:1339
    0x556e9a24b1bf main
            ../../gdb/gdb.c:32
    ---------------------
    ../../gdb/amd-dbgapi-target.c:1071: internal-error: process_one_event: Assertion `runtime_state == AMD_DBGAPI_RUNTIME_STATE_UNLOADED' failed.
    A problem internal to GDB has been detected,

Before diving into why this error appears, let's explore how things are
expected to work in normal circumstances.  When a process being debugged
starts using the ROCm runtime, the following happens:

- The runtime registers itself to the driver.
- The driver creates a "runtime loaded" event and notifies the debugger
  that a new event is available by writing to a file descriptor which is
  registered in GDB's main event loop.
- GDB core calls the callback associated with this file descriptor
  (dbgapi_notifier_handler).  Because the amd-dbgapi-target is not
  pushed at this point, the handler pulls the "runtime loaded" event
  from the driver (this is the only event which can be available at this
  point) and eventually pushes the amd-dbgapi-target on the inferior's
  target stack.

In a nutshell, this is the expected AMDGPU runtime activation process.

From there, when new events are available regarding the GPU threads, the
same file descriptor is written to.  The callback sees that the
amd-dbgapi-target is pushed so marks the amd_dbgapi_async_event_handler.
This will later cause amd_dbgapi_target::wait to be called.  The wait
method pulls all the available events from the driver and handles them.
The wait method returns the information conveyed by the first event, the
other events are cached for later calls of the wait method.

Note that because we are under the wait method, we know that the
amd-dbgapi-target is pushed on the inferior target stack.  This implies
that the runtime activation event has been seen already.  As a
consequence, we cannot receive another event indicating that the runtime
gets activated.  This is what the failing assertion checks.

In the case when we have multiple inferiors however, there is a flaw in
what have been described above.  If one inferior (let's call it inferior
1) already has the amd-dbgapi-target pushed to its target stack and
another inferior (inferior 2) activates the ROCm runtime, here is what
can happen:

- The driver creates the runtime activation for inferior 2 and writes to
  the associated file descriptor.
- GDB has inferior 1 selected and calls target_wait for some reason.
- This prompts amd_dbgapi_target::wait to be called.  The method pulls
  all events from the driver, including the runtime activation event for
  inferior 2, leading to the assertion failure.

The fix for this problem is simple.  To avoid such problem, we need to
make sure that amd_dbgapi_target::wait only pulls events for the current
inferior from the driver.  This is what this patch implements.

This patch also includes a testcase which could fail before this patch.

This patch has been tested on a system with multiple GPUs which had more
chances to reproduce the original bug.  It has also been tested on top
of the downstream ROCgdb port which has more AMDGPU related tests.  The
testcase has been tested with `make check check-read1 check-readmore`.

Approved-By: Pedro Alves <pedro@palves.net>

1 files changed, 5 insertions, 3 deletions

diff --git a/gdb/amd-dbgapi-target.c b/gdb/amd-dbgapi-target.c
index 40f24b5..e90628c 100644
--- a/gdb/amd-dbgapi-target.c
+++ b/gdb/amd-dbgapi-target.c
@@ -148,7 +148,7 @@ struct amd_dbgapi_inferior_info
 };
 
 static amd_dbgapi_event_id_t process_event_queue
-  (amd_dbgapi_process_id_t process_id = AMD_DBGAPI_PROCESS_NONE,
+  (amd_dbgapi_process_id_t process_id,
    amd_dbgapi_event_kind_t until_event_kind = AMD_DBGAPI_EVENT_KIND_NONE);
 
 static const target_info amd_dbgapi_target_info = {
@@ -1255,8 +1255,10 @@ amd_dbgapi_target::wait (ptid_t ptid, struct target_waitstatus *ws,
   std::tie (event_ptid, gpu_waitstatus) = consume_one_event (ptid.pid ());
   if (event_ptid == minus_one_ptid)
     {
-      /* Drain the events from the amd_dbgapi and preserve the ordering.  */
-      process_event_queue ();
+      /* Drain the events for the current inferior from the amd_dbgapi and
+	 preserve the ordering.  */
+      auto info = get_amd_dbgapi_inferior_info (current_inferior ());
+      process_event_queue (info->process_id, AMD_DBGAPI_EVENT_KIND_NONE);
 
       std::tie (event_ptid, gpu_waitstatus) = consume_one_event (ptid.pid ());
       if (event_ptid == minus_one_ptid)