Improve analysis of racy testcases

This is an initial attempt to introduce some mechanisms to identify
racy testcases present in our testsuite.  As can be seen in previous
discussions, racy tests are really bothersome and cause our BuildBot
to pollute the gdb-testers mailing list with hundreds of
false-positives messages every month.  Hopefully, identifying these
racy tests in advance (and automatically) will contribute to the
reduction of noise traffic to gdb-testers, maybe to the point where we
will be able to send the failure messages directly to the authors of
the commits.

I spent some time trying to decide the best way to tackle this
problem, and decided that there is no silver bullet.  Racy tests are
tricky and it is difficult to catch them, so the best solution I could
find (for now?) is to run our testsuite a number of times in a row,
and then compare the results (i.e., the gdb.sum files generated during
each run).  The more times you run the tests, the more racy tests you
are likely to detect (at the expense of waiting longer and longer).
You can also run the tests in parallel, which makes things faster (and
contribute to catching more racy tests, because your machine will have
less resources for each test and some of them are likely to fail when
this happens).  I did some tests in my machine (8-core i7, 16GB RAM),
and running the whole GDB testsuite 5 times using -j6 took 23 minutes.
Not bad.

In order to run the racy test machinery, you need to specify the
RACY_ITER environment variable.  You will assign a number to this
variable, which represents the number of times you want to run the
tests.  So, for example, if you want to run the whole testsuite 3
times in parallel (using 2 cores), you will do:

  make check RACY_ITER=3 -j2

It is also possible to use the TESTS variable and specify which tests
you want to run:

  make check TEST='gdb.base/default.exp' RACY_ITER=3 -j2

And so on.  The output files will be put at the directory
gdb/testsuite/racy_outputs/.

After make invokes the necessary rules to run the tests, it finally
runs a Python script that will analyze the resulting gdb.sum files.
This Python script will read each file, and construct a series of sets
based on the results of the tests (one set for FAIL's, one for
PASS'es, one for KFAIL's, etc.).  It will then do some set operations
and come up with a list of unique, sorted testcases that are racy.
The algorithm behind this is:

  for state in PASS, FAIL, XFAIL, XPASS...; do
    if a test's state in every sumfile is $state; then
      it is not racy
    else
      it is racy

(The algorithm is actually a bit more complex than that, because it
takes into account other things in order to decide whether the test
should be ignored or not).

IOW, a test must have the same state in every sumfile.

After processing everything, the script prints the racy tests it could
identify on stdout.  I am redirecting this to a file named racy.sum.

Something else that I wasn't sure how to deal with was non-unique
messages in our testsuite.  I decided to do the same thing I do in our
BuildBot: include a unique identifier in the end of message, like:

  gdb.base/xyz.exp: non-unique message
  gdb.base/xyz.exp: non-unique message <<2>>

This means that you will have to be careful about them when you use
the racy.sum file.

I ran the script several times here, and it did a good job catching
some well-known racy tests.  Overall, I am satisfied with this
approach and I think it will be helpful to have it upstream'ed.  I
also intend to extend our BuildBot and create new, specialized
builders that will be responsible for detecting the racy tests every X
number of days.

2016-03-05  Sergio Durigan Junior  <sergiodj@redhat.com>

	* Makefile.in (DEFAULT_RACY_ITER): New variable.
	(CHECK_TARGET_TMP): Likewise.
	(check-single-racy): New rule.
	(check-parallel-racy): Likewise.
	(TEST_TARGETS): Adjust rule to account for RACY_ITER.
	(do-check-parallel-racy): New rule.
	(check-racy/%.exp): Likewise.
	* README (Racy testcases): New section.
	* analyze-racy-logs.py: New file.

1 files changed, 45 insertions, 0 deletions

diff --git a/gdb/testsuite/README b/gdb/testsuite/README
index 6b59027..043a8bd 100644
--- a/gdb/testsuite/README
+++ b/gdb/testsuite/README
@@ -50,6 +50,51 @@ to any non-empty value:
 If you want to use runtest directly instead of using the Makefile, see
 the description of GDB_PARALLEL below.
 
+Racy testcases
+**************
+
+Sometimes, new testcases are added to the testsuite that are not
+entirely deterministic, and can randomly pass or fail.  We call them
+"racy testcases", and they can be bothersome when one is comparing
+different testsuite runs.  In order to help identifying them, it is
+possible to run the tests several times in a row and ask the testsuite
+machinery to analyze the results.  To do that, you need to specify the
+RACY_ITER environment variable to make:
+
+	make check RACY_ITER=5 -j4
+
+The value assigned to RACY_ITER represents the number of times you
+wish to run the tests in sequence (in the example above, the entire
+testsuite will be executed 5 times in a row, in parallel).  It is also
+possible to check just a specific test:
+
+	make check TESTS='gdb.base/default.exp' RACY_ITER=3
+
+One can also decide to call the Makefile rules by hand inside the
+gdb/testsuite directory, e.g.:
+
+	make check-paralell-racy -j4
+
+In which case the value of the DEFAULT_RACY_ITER variable (inside
+gdb/testsuite/Makefile.in) will be used to determine how many
+iterations will be run.
+
+After running the tests, you shall see a file name 'racy.sum' in the
+gdb/testsuite directory.  You can also inspect the generated *.log and
+*.sum files by looking into the gdb/testsuite/racy_ouputs directory.
+
+If you already have *.sum files generated from previous testsuite runs
+and you would like to analyze them without having to run the testsuite
+again, you can also use the 'analyze-racy-logs.py' script directly.
+It is located in the gdb/testsuite/ directory, and it expects a list
+of two or more *.sum files to be provided as its argument.  For
+example:
+
+	./gdb/testsuite/analyze-racy-logs.py testsuite-01/gdb.sum \
+	  testsuite-02/gdb.sum testsuite-03/gdb.sum
+
+The script will output its analysis report to the standard output.
+
 Running the Performance Tests
 *****************************