introduce parallel mode

This introduces parallel mode for the test suite.

It doesn't fully work yet in the sense that if you do a fully parallel
run, you will encounter some file-name clashes, but this has to start
somewhere, and it seemed best to add some infrastructure now, so that
you can follow along and test subsequent patches if you care to.

This patch has two parts.

First, it checks for the GDB_PARALLEL variable.  If this is set (say,
on the runtest command line), then the test suite assumes "parallel
mode".  In this mode, files are put into a subdirectory named after
the test.  That is, for DIR/TEST.exp, the outputs are put into
./outputs/DIR/TEST/.

This first part has various follow-on changes coming in subsequent
patches.  This is why the code in this patch also makes "temp" and
"cache" directories.

Second, this adds an "inotify" mode.  If you have the inotifywait
command (part of inotify-tools), you can set the GDB_INOTIFY variable.
This will tell the test suite to watch for changes outside of the
allowed output directories.

This mode is useful for debugging the test suite, as it issues a
report whenever a possibly parallel-unsafe file open is done.

2013-08-13  Tom Tromey  <tromey@redhat.com>
	    Yao Qi  <yao@codesourcery.com>

	* lib/cache.exp (gdb_do_cache): Handle GDB_PARALLEL.
	* lib/gdb.exp: Handle GDB_PARALLEL.
	(default_gdb_version): Kill inotify_pid if it exists.
	(default_gdb_exit): Emit warning if the inotify log is not
	empty.
	(standard_output_file): Respect GDB_PARALLEL.
	(standard_temp_file): Likewise.
	(gdb_init): Start inotifywait if requested.

	* gdbint.texinfo (Testsuite): Use @table, not @itemize.
	Document GDB_PARALLEL and GDB_INOTIFY.

5 files changed, 145 insertions, 9 deletions

diff --git a/gdb/doc/ChangeLog b/gdb/doc/ChangeLog
index 04d3804..eaaccd5 100644
--- a/gdb/doc/ChangeLog
+++ b/gdb/doc/ChangeLog
@@ -1,3 +1,8 @@
+2013-08-13  Tom Tromey  <tromey@redhat.com>
+
+	* gdbint.texinfo (Testsuite): Use @table, not @itemize.
+	Document GDB_PARALLEL and GDB_INOTIFY.
+
 2013-08-09  Stan Shebs  <stan@codesourcery.com>
 
 	* LRS: Remove file, describes a long-abandoned live-range
diff --git a/gdb/doc/gdbint.texinfo b/gdb/doc/gdbint.texinfo
index 60805ad..b0f133f 100644
--- a/gdb/doc/gdbint.texinfo
+++ b/gdb/doc/gdbint.texinfo
@@ -7732,9 +7732,9 @@ UNRESOLVED: gdb.base/example.exp: This test script does not work on a remote hos
 
 Several variables exist to modify the behavior of the testsuite.
 
-@itemize @bullet
+@table @code
 
-@item @code{TRANSCRIPT}
+@item TRANSCRIPT
 
 Sometimes it is convenient to get a transcript of the commands which
 the testsuite sends to @value{GDBN}.  For example, if @value{GDBN}
@@ -7757,7 +7757,7 @@ make check RUNTESTFLAGS=TRANSCRIPT=y
 Note that the transcript is not always complete.  In particular, tests
 of completion can yield partial command lines.
 
-@item @code{GDB}
+@item GDB
 
 Sometimes one wishes to test a different @value{GDBN} than the one in the build
 directory.  For example, one may wish to run the testsuite on
@@ -7767,7 +7767,7 @@ directory.  For example, one may wish to run the testsuite on
 make check RUNTESTFLAGS=GDB=/usr/bin/gdb
 @end smallexample
 
-@item @code{GDBSERVER}
+@item GDBSERVER
 
 When testing a different @value{GDBN}, it is often useful to also test a
 different gdbserver.
@@ -7776,7 +7776,7 @@ different gdbserver.
 make check RUNTESTFLAGS="GDB=/usr/bin/gdb GDBSERVER=/usr/bin/gdbserver"
 @end smallexample
 
-@item @code{INTERNAL_GDBFLAGS}
+@item INTERNAL_GDBFLAGS
 
 When running the testsuite normally one doesn't want whatever is in
 @file{~/.gdbinit} to interfere with the tests, therefore the test harness
@@ -7803,7 +7803,38 @@ HOME=`pwd` runtest \
   INTERNAL_GDBFLAGS=-nw
 @end smallexample
 
-@end itemize
+@item GDB_PARALLEL
+
+When testing natively (that is, not with a remote host), the
+@value{GDBN} test suite can be run in a fully parallel mode.  In this
+mode, each @file{.exp} file can be run separately.  The test suite
+will ensure that all the temporary files created by the test suite do
+not clash, by putting them into separate directories.  This mode is
+primarily intended for use by the @file{Makefile}.
+
+To use this mode, set the @code{GDB_PARALLEL} on the @command{runtest}
+command line.  Before starting the tests, you must ensure that the
+directories @file{cache}, @file{outputs}, and @file{temp} in the test
+suite build directory are either empty or have been deleted.
+@file{cache} in particular is used to share data across invocations of
+@command{runtest}, and files there may affect the test results.  Note
+that the @file{Makefile} automatically does these deletions.
+
+@item GDB_INOTIFY
+
+For debugging parallel mode, it is handy to be able to see when a test
+case writes to a file outside of its designated output directory.
+
+If you have the @samp{inotify-tools} package installed, you can set
+the @code{GDB_INOTIFY} variable on the @command{runtest} command line.
+This will cause the test suite to watch for parallel-unsafe file
+creations and report them, both on @samp{stdout} and in the test suite
+@file{.log} file.
+
+This setting is only meaningful in conjunction with
+@code{GDB_PARALLEL}.
+
+@end table
 
 There are two ways to run the testsuite and pass additional parameters
 to DejaGnu.  The first is with @kbd{make check} and specifying the
diff --git a/gdb/testsuite/ChangeLog b/gdb/testsuite/ChangeLog
index 0b60da5..ac575b5 100644
--- a/gdb/testsuite/ChangeLog
+++ b/gdb/testsuite/ChangeLog
@@ -1,3 +1,15 @@
+2013-08-13  Tom Tromey  <tromey@redhat.com>
+	    Yao Qi  <yao@codesourcery.com>
+
+	* lib/cache.exp (gdb_do_cache): Handle GDB_PARALLEL.
+	* lib/gdb.exp: Handle GDB_PARALLEL.
+	(default_gdb_version): Kill inotify_pid if it exists.
+	(default_gdb_exit): Emit warning if the inotify log is not
+	empty.
+	(standard_output_file): Respect GDB_PARALLEL.
+	(standard_temp_file): Likewise.
+	(gdb_init): Start inotifywait if requested.
+
 2013-08-13  Andrew Burgess  <aburgess@broadcom.com>
 
 	* gdb.base/printcmds.exp (test_printf): Add test for printf of
diff --git a/gdb/testsuite/lib/cache.exp b/gdb/testsuite/lib/cache.exp
index 1de0dfd..e669ebf 100644
--- a/gdb/testsuite/lib/cache.exp
+++ b/gdb/testsuite/lib/cache.exp
@@ -21,6 +21,7 @@ array set gdb_data_cache {}
 
 proc gdb_do_cache {name} {
     global gdb_data_cache objdir
+    global GDB_PARALLEL
 
     # See if some other process wrote the cache file.  Cache value per
     # "board" to handle runs with multiple options
@@ -33,9 +34,29 @@ proc gdb_do_cache {name} {
 	return $gdb_data_cache($cache_name)
     }
 
+    if {[info exists GDB_PARALLEL]} {
+	set cache_filename [file join $objdir cache $cache_name]
+	if {[file exists $cache_filename]} {
+	    set fd [open $cache_filename]
+	    set gdb_data_cache($cache_name) [read -nonewline $fd]
+	    close $fd
+	    verbose "$name: returning '$gdb_data_cache($cache_name)' from file cache" 2
+	    return $gdb_data_cache($cache_name)
+	}
+    }
+
     set real_name gdb_real__$name
     set gdb_data_cache($cache_name) [uplevel 1 $real_name]
 
+    if {[info exists GDB_PARALLEL]} {
+	verbose "$name: returning '$gdb_data_cache($cache_name)' and writing file" 2
+	file mkdir [file dirname $cache_filename]
+	# Make sure to write the results file atomically.
+	set fd [open $cache_filename.[pid] w]
+	puts $fd $gdb_data_cache($cache_name)
+	close $fd
+	file rename -force -- $cache_filename.[pid] $cache_filename
+    }
     return $gdb_data_cache($cache_name)
 }
 
diff --git a/gdb/testsuite/lib/gdb.exp b/gdb/testsuite/lib/gdb.exp
index ceaaa41..a2f6a8f 100644
--- a/gdb/testsuite/lib/gdb.exp
+++ b/gdb/testsuite/lib/gdb.exp
@@ -112,6 +112,12 @@ proc default_gdb_version {} {
     global GDB
     global INTERNAL_GDBFLAGS GDBFLAGS
     global gdb_prompt
+    global inotify_pid
+
+    if {[info exists inotify_pid]} {
+	eval exec kill $inotify_pid
+    }
+
     set output [remote_exec host "$GDB $INTERNAL_GDBFLAGS --version"]
     set tmp [lindex $output 1]
     set version ""
@@ -1246,6 +1252,7 @@ proc default_gdb_exit {} {
     global INTERNAL_GDBFLAGS GDBFLAGS
     global verbose
     global gdb_spawn_id
+    global inotify_log_file
 
     gdb_stop_suppressing_tests
 
@@ -1255,6 +1262,20 @@ proc default_gdb_exit {} {
 
     verbose "Quitting $GDB $INTERNAL_GDBFLAGS $GDBFLAGS"
 
+    if {[info exists inotify_log_file] && [file exists $inotify_log_file]} {
+	set fd [open $inotify_log_file]
+	set data [read -nonewline $fd]
+	close $fd
+
+	if {[string compare $data ""] != 0} {
+	    warning "parallel-unsafe file creations noticed"
+
+	    # Clear the log.
+	    set fd [open $inotify_log_file w]
+	    close $fd
+	}
+    }
+
     if { [is_remote host] && [board_info host exists fileid] } {
 	send_gdb "quit\n"
 	gdb_expect 10 {
@@ -3349,15 +3370,27 @@ proc default_gdb_init { args } {
 # the directory is returned.
 
 proc standard_output_file {basename} {
-    global objdir subdir
+    global objdir subdir gdb_test_file_name GDB_PARALLEL
 
-    return [file join $objdir $subdir $basename]
+    if {[info exists GDB_PARALLEL]} {
+	set dir [file join $objdir outputs $subdir $gdb_test_file_name]
+	file mkdir $dir
+	return [file join $dir $basename]
+    } else {
+	return [file join $objdir $subdir $basename]
+    }
 }
 
 # Return the name of a file in our standard temporary directory.
 
 proc standard_temp_file {basename} {
-    return $basename
+    global objdir GDB_PARALLEL
+
+    if {[info exists GDB_PARALLEL]} {
+	return [file join $objdir temp $basename]
+    } else {
+	return $basename
+    }
 }
 
 # Set 'testfile', 'srcfile', and 'binfile'.
@@ -3463,6 +3496,31 @@ proc gdb_init { args } {
     global timeout
     set timeout $gdb_test_timeout
 
+    # If GDB_INOTIFY is given, check for writes to '.'.  This is a
+    # debugging tool to help confirm that the test suite is
+    # parallel-safe.  You need "inotifywait" from the
+    # inotify-tools package to use this.
+    global GDB_INOTIFY inotify_pid
+    if {[info exists GDB_INOTIFY] && ![info exists inotify_pid]} {
+	global outdir tool inotify_log_file
+
+	set exclusions {outputs temp gdb[.](log|sum) cache}
+	set exclusion_re ([join $exclusions |])
+
+	set inotify_log_file [standard_temp_file inotify.out]
+	set inotify_pid [exec inotifywait -r -m -e move,create,delete . \
+			     --exclude $exclusion_re \
+			     |& tee -a $outdir/$tool.log $inotify_log_file &]
+
+	# Wait for the watches; hopefully this is long enough.
+	sleep 2
+
+	# Clear the log so that we don't emit a warning the first time
+	# we check it.
+	set fd [open $inotify_log_file w]
+	close $fd
+    }
+
     # Block writes to all banned variables, and invocation of all
     # banned procedures...
     global banned_variables
@@ -4258,6 +4316,15 @@ if {[info exists TRANSCRIPT]} {
   }
 }
 
+# If GDB_PARALLEL exists, then set up the parallel-mode directories.
+if {[info exists GDB_PARALLEL]} {
+    if {[is_remote host]} {
+	unset GDB_PARALLEL
+    } else {
+	file mkdir outputs temp cache
+    }
+}
+
 proc core_find {binfile {deletefiles {}} {arg ""}} {
     global objdir subdir