1. May 31, 2024
  2. May 24, 2024
  3. May 16, 2024
  4. May 15, 2024
    • Parshintsev Anatoly's avatar
      debug: fix sporadic failures of memory sampling tests · cb357771
      Parshintsev Anatoly authored
      Memory sampling tests fail sporadically for spike targets. A typical
      failure looks as follows (ROI from test log):
      
      ```
      ---------------------------------[ Message ]----------------------------------
      139670831 not less than 124104544
      --------------------------------[ Traceback ]---------------------------------
          ... SECTION IS SKIPPED FOR READABILITY ...
          raise TestFailed(f"{a!r} not less than {b!r}", comment)
      testlib.TestFailed
      ```
      
      Few observations:
      
      - 139670831 is 0x0853352f in hex, while 124104544 is 0x0765af60
      - Now, the assert which is failing corresponds to the following
        expression:
      
      ```
        assertLess(value, previous_value + tolerance)
      ```
      
      - tolerance is `0x500000`. (124104544 - 0x500000) is 0x0715af60
      
      If we look at the sampling output for such failing test, we'll see:
      
      ```
      ...
      0x1212340c5c: 0x0715af60
      timestamp after: 878087500
      timestamp before: 878088133
      0x1212340c5c: 0x0853352f
      ...
      ```
      
      The log above demonstrates the reason for the failure. Since memory
      sampling occures every poll (which by default happens approximately
      every 100ms) a value of the counter may exceed the threshold if the time
      between subsequent polls is increased (for whatever reason).
      
      In my opinion the failing assert can be safely removed, since the checks
      it perform are quite brittle and cannot be generalized. The assert
      violation is affected by CPU performance and sporadic delays between
      polls.
      
      For now, instead of assert removal we just avoid checks in-between
      memory sample bursts. This way we still can be certain that memory
      samples are frequent enough and hopefully this will avoid sporadic
      failures.
      cb357771
  5. May 14, 2024
    • Parshintsev Anatoly's avatar
      debug: workaround for sporadic failures of some tests due to unexpected data... · db7e451c
      Parshintsev Anatoly authored
      debug: workaround for sporadic failures of some tests due to unexpected data present in pexpect match
      
      Problem was observed on UnavailableMultiTest - this test was sporadically failing.
      When the failure was observed the log of the failing test looked as follows:
      
      ```
        File "/whatever/RISCVTests/debug/testlib.py", line 504, in <genexpr>
          if all(targets[hart.id]["State"] == "running" for hart in harts):
                 ~~~~~~~~~~~~~~~~^^^^^^^^^
      KeyError: 'State'
      ```
      
      Adding this modification to testlib.py
      
      ```
      --- a/debug/testlib.py
      +++ b/debug/testlib.py
      @@ -498,6 +498,10 @@ class Openocd:
               for line in lines[2:]:
                   if line.strip():
                       data.append(dict(zip(headers, line.split()[1:])))
      +        str_data = str(data)
      +        sys.stdout.flush()
      +        sys.stdout.write(f"parsed targets:\n{result}\n===\n{str_data}\n---\n")
      +        sys.stdout.flush()
               return data
      ```
      
      Allowed me to root cause the issue. Namely we have the following
      situation:
      
      ```
      parsed targets:
      Exception ignored in: <function _TemporaryFileCloser.__del__ at 0x7f2dee64d1c0>
      Traceback (most recent call last):
        File "/usr/local/lib/python3.11/tempfile.py", line 450, in __del__
          self.close()
        File "/usr/local/lib/python3.11/tempfile.py", line 446, in close
          unlink(self.name)
      FileNotFoundError: [Errno 2] No such file or directory: '/tmp/gdb@38873-8s6ud03x.log'
      ...
          TargetName         Type       Endian TapName            State
      --  ------------------ ---------- ------ ------------------ ------------
       0  riscv.cpu0         riscv      little riscv.cpu          running
       1* riscv.cpu1         riscv      little riscv.cpu          running
      ===
      [{'Exception': '"/usr/local/lib/python3.11/tempfile.py",', 'ignored': 'line', 'in:': '450,', ...
      ```
      
      The only reasonable (to me) explanation for the observed behavior is below.
      
      Here is how we connect to TCL-RPC server:
      
      ```
        self.openocd_cli = pexpect.spawn(f"nc localhost {self.tclrpc_port}")
        tty.setraw(self.openocd_cli.child_fd)
      ```
      
      Later we request target list by issuing "targets" command:
      
      ```
        self.command("targets")
      ```
      
      Internally, pexpect.spawn implemented as follows:
      
      - we fork the process
      - set up pty and then call execve
      - all these steps are written in python
      
      "Exception ignored" messages are result of exceptions thrown from
      finalizers of NamedTemporaryFile objects. When exception is thrown from
      the finalizer - python unconditionally prints a "warning" to stderr. It
      seems that these messages are polluting our output from "nc" since python
      GC can be invoked before the execve syscall.
      
      The workaround is just to make sure that execve was executed before we
      rely on the format of command output. To have such a guarantee we just
      issue a dummy "echo" command and check that we have a proper reply in the
      output stream.
      
      While this explanation looks convincing, the behavior above still looks
      strange, given that we have https://bugs.python.org/issue14548 which
      was resolved long ago.
      
      However, the proposed workaround fixes the issue.
      db7e451c
    • Anatoly Parshintsev's avatar
      Merge pull request #491 from en-sc/en-sc/warning-repeat-read · 084a6073
      Anatoly Parshintsev authored
      Remove old warning check in RepeatReadTest
      084a6073
  6. May 09, 2024
  7. May 04, 2024
  8. May 02, 2024
    • Marek Vrbka's avatar
      debug: Fix loading of empty exclude lists with comments · 203362f8
      Marek Vrbka authored
      This patch fixes the case when we are using an empty exception list (for example just a YAML file with comments but without any test items to skip).
      203362f8
    • Anatoly Parshintsev's avatar
      [debug tests] increase remotetimeout for all spike-based targets (#553) · 1dde0ef3
      Anatoly Parshintsev authored
      Spike simulator is very demanding to CPU resources. This causes debug
      tests to sporadically fail on slower machines. Increasing of gdb's
      `remotetimeout` should get rid of such failures, unless we run the
      testsuite on a potato.
      
      The only downside is that if OpenOCD is broken, tests can run longer.
      However, I think this is the sacrifice we can make, since execution time
      is not affected if everything works as expected.
      1dde0ef3
  9. Apr 17, 2024
  10. Apr 02, 2024
  11. Mar 20, 2024
  12. Mar 05, 2024
  13. Mar 02, 2024
  14. Mar 01, 2024
  15. Feb 29, 2024
  16. Feb 19, 2024
  17. Feb 04, 2024
  18. Feb 03, 2024
  19. Feb 02, 2024
  20. Jan 30, 2024
  21. Jan 23, 2024
  22. Nov 11, 2023