===========================================================================
RISC-V Torture Test Generator
===========================================================================
# Author: Yunsup Lee and Henry Cook
# Date: January 29th, 2012
# Version: (under version control)

This is the RISC-V torture test generator and framework. This repository
contains three sub-projects that build upon one another. The first, 
[generator], is used to create a single random torture test. The second,
[testrun], is used to run a particular test on particular simulators,
diffing the resulting signature with the ISA simulator and optionally
creating a derivative test subset that pinpoints the divergence. The third,
[overnight], wraps testrun, allowing tests to be run repeatedly for a given
duration or until a failure count.

Some basic use cases are illustrated here.

Make a single test:
% sbt generator/run
% cd output
% make
% fesvr -testrun test

Take an existing test and diff the signatures of ISA and C simulators:
% sbt 'testrun/run -a output/test.S -c ../riscv-rocket/emulator/emulator'

Generate a random test and diff the signatures of ISA and RTL simulators:
% sbt 'testrun/run -r ../riscv-rocket/vlsi/build/vcs-sim-rtl/sim'

Run tests for 30 minutes, email hcook when done, and save failures to dir:
% sbt 'overnight/run -m 30 -e hcook@eecs.berkeley.edu -p dir'

---------------------------------------------------------------------------
Overnight Overview
---------------------------------------------------------------------------

This framework utilizes both the test runner and test generator to perform
a long terms serach for failing test cases. It takes the following command
line arguments:

Usage: overnight/run [options] 
  -C <file> | --config <file>
        config file
  -p <dir> | --permdir <dir>
        dir to store failing tests
  -c <file> | --csim <file>
        C simulator
  -r <file> | --rtlsim <file>
        RTL simulator
  -e <address> | --email <address>
        email to report to
  -t <count> | --threshold <count>
        number of failures to trigger email
  -m <int> | --minutes <int>
        number of minutes to run tests

You can only generate tests with one instruction mix at a time (based on
the setting in the config file). It doesn't matter what simulator you use
with the -r and -c flags, they just determines the name used to describe
whose diff failed.

---------------------------------------------------------------------------
Testrun Overview
---------------------------------------------------------------------------

This utility compares the signatures generated by passing the -testsig flag
to the specified simulators. If it encounters a difference, it subdivides
the test into many subtests and searches for which exact program segment
reveals the failure. It takes the following command line arguments:

Usage: testrun/run [options] 
 -C <file> | --config <file>
        config file
  -a <file> | --asm <file>
        input ASM file
  -c <file> | --csim <file>
        C simulator
  -r <file> | --rtlsim <file>
        RTL simulator
  -s <boolean> | --seek <boolean>
        Seek for failing pseg
  -d <boolean> | --dump <boolean>
        Dump mismatched signatures

If you don't specify a asm file, a random one will be generated for you.
You can only generate tests with one instruction mix at a time (based on
the setting in the config file). It doesn't matter what simulator you use
with the -r and -c flags, they just determines the name used to describe
whose diff failed. By default, a failed diff will result in the subtest 
sweep occuring, but this search can be diasbled. Note that the pseg ID
reported is actually the pseg following the pseg containing the error.
You can optionally dump mistmatched signatures to the dir containing the 
asm file under test.

---------------------------------------------------------------------------
Generator Overview
---------------------------------------------------------------------------

To generate a random test, the torture test generator randomly generates
many test sequences from a set of test sequences that are written by hand,
performs a random register allocation for all test sequences, and finally
randomly interleaves instructions from these test sequences. To extend the
set of tests or coverage, the programmer needs to write new test sequences.
It takes the following command line arguments:

Usage: generator/run [options] 
  -o <filename> | --output <filename>
        output filename
  -C <file> | --config <file>
        config file

The following sections describe adding new functionality to the generator.

---------------------------------------------------------------------------
Test sequence example
---------------------------------------------------------------------------

Before we talk about how to write a test sequence, let's look at a very
simple example. The following example is a test sequence, which emits an
add instruction.

class SeqADD extends Seq
{
  val src1 = reg_read_any()
  val src2 = reg_read_any()
  val dest = reg_write(src1, src2)
  insts += ADD(dest, src1, src2)
}

As I hinted in the overview that the test generator will do register
allocation you don't write a string of instructions with architectural
registers. You request a virtual registers (i.e., registers that are yet
tied down to architectural registers) when you need them, save them in
scala values, and use them when you need to (e.g., in an instruction).

---------------------------------------------------------------------------
Types of virtual registers
---------------------------------------------------------------------------

 - Hidden (position dependent registers): Registers that will have
   different values when the code is positioned at a different address. An
   example is registers that hold addresses. Registers that are hidden should
   be excluded from the output signature.

 - Visible (position independent registers): Registers that are not hidden,
   therefore will have the same values when the code is positioned at
   a different address. These registers should be included as part of the
   output signature.

---------------------------------------------------------------------------
How to write a sequence
---------------------------------------------------------------------------

Use the following functions to request a register, and generate a string of
instructions (look at Inst.scala to see what instructions are available)
that uses these virtual registers, and add them to the insts array.

 - reg_read_zero(): returns register x0
 - reg_read_any(): returns any type of register (hidden or visible)
 - reg_read_visible(): returns a visible register
 - reg_write_ra(): returns register ra for write
 - reg_write_visible(): returns a visible register for write
 - reg_write_hidden(): returns a hidden register for write
 - reg_write(regs: Reg*): returns a register that matches the type of regs
   (if any reg in regs are hidden, the output type is hidden)

Note that the torture test framework is written in scala, you can use any
scala functionality to generate instructions. Look at SeqALU.scala,
SeqMem.scala, and SeqBranch.scala to get inspired.

---------------------------------------------------------------------------
Future TODO
---------------------------------------------------------------------------

 - generate statistics of a test to get a sense of coverage
   + statistics should include instruction count of each type
   + statistics should include register usage

 - complete floating point tests
   + add floating point memory move tests
   + improve floating point init randomization
   + add rounding modes tests

 - complete vector tests
   + better randomization
   + add SeqVOnly: Tests special vf-only instructions

 - code refactoring
   + consolidate RegPool logic
   + detect and suppress unallocatable sequences
