Fix non-stop regressions caused by "breakpoints always-inserted off" changes

Commit a25a5a45 (Fix "breakpoint always-inserted off"; remove
"breakpoint always-inserted auto") regressed non-stop remote
debugging.

This was exposed by mi-nsintrall.exp intermittently failing with a
spurious SIGTRAP.

The problem is that when debugging with "target remote", new threads
the target has spawned but have never reported a stop aren't visible
to GDB until it explicitly resyncs its thread list with the target's.

For example, in a program like this:

 int
 main (void)
 {
   pthread_t child_thread;
   pthread_create (&child_thread, NULL, child_function, NULL);
   return 0;  <<<< set breakpoint here
 }

If the user sets a breakpoint at the "return" statement, and runs the
program, when that breakpoint hit is reported, GDB is only aware of
the main thread.  So if we base the decision to remove or insert
breakpoints from the target based on whether all the threads we know
about are stopped, we'll miss that child_thread is running, and thus
we'll remove breakpoints from the target, even through they should
still remain inserted, otherwise child_thread will miss them.

The break-while-running.exp test actually should also be exposing this
thread-list-out-of-synch problem.  That test sets a breakpoint while
the main thread is stopped, but other threads are running.  Because
other threads are running, the breakpoint is supposed to be inserted
immediately.  But, unless something forces a refetch of the thread
list, like, e.g., "info threads", GDB won't be aware of the other
threads that had been spawned by the main thread, and so won't insert
new or old breakpoints in the target.  And it turns out that the test
is exactly doing an explicit "info threads", masking out the
problem...  This commit adjust the test to exercise the case of not
issuing "info threads".  The test then fails without the GDB fix.

In the ni-nsintrall.exp case, what happens is that several threads hit
the same breakpoint, and when the first thread reports the stop,
because GDB wasn't aware other threads exist, all threads known to GDB
are found stopped, so GDB removes the breakpoints from the target.
The other threads follow up with SIGTRAPs too for that same
breakpoint, which has already been removed.  For the first few
threads, the moribund breakpoints machinery suppresses the SIGTRAPs,
but after a few events (precisely '3 * thread_count () + 1' at the
time the breakpoint was removed, see update_global_location_list), the
moribund breakpoint machinery is no longer aware of the removed
breakpoint, and the SIGTRAP is reported as a spurious stop.

The fix is naturally then to stop assuming that if no thread in the
list is executing, then the target is fully stopped.  We can't know
that until we fully sync the thread list.  Because updating the thread
list on every stop would be too much RSP traffic, I chose instead to
update it whenever we're about to present a stop to the user.

Actually updating the thread list at that point happens to be an item
I had added to the local/remote parity wiki page a while ago:

  Native GNU/Linux debugging adds new threads to the thread list as
  the program creates them "The [New Thread foo] messages". Remote
  debugging can't do that, and it's arguable whether we shouldn't even
  stop native debugging from doing that, as it hinders inferior
  performance. However, a related issue is that with remote targets
  (and gdbserver), even after the program stops, the user still needs
  to do "info threads" to pull an updated thread list. This, should
  most likely be addressed, so that GDB pulls the list itself, perhaps
  just before presenting a stop to the user.

With that in place, the need to delay "Program received signal FOO"
was actually caught by the manythreads.exp test.  Without that bit, I
was getting:

  [Thread 0x7ffff7f13700 (LWP 4499) exited]
  [New Thread 0x7ffff7f0b700 (LWP 4500)]
  ^C
  Program received signal SIGINT, Interrupt.
  [New Thread 0x7ffff7f03700 (LWP 4501)]           <<< new output
  [Switching to Thread 0x7ffff7f0b700 (LWP 4500)]
  __GI___nptl_death_event () at events.c:31
  31      {
  (gdb) FAIL: gdb.threads/manythreads.exp: stop threads 1

That is, I was now getting "New Thread" lines after the "Program
received signal" line, and the test doesn't expect them.  As the
number of new threads discovered before and after the "Program
received signal" output is unbounded, it's much nicer to defer
"Program received signal" until after synching the thread list, thus
close to the "switching to thread" output and "current frame/source"
info:

  [Thread 0x7ffff7863700 (LWP 7647) exited]
  ^C[New Thread 0x7ffff786b700 (LWP 7648)]

  Program received signal SIGINT, Interrupt.
  [Switching to Thread 0x7ffff7fc4740 (LWP 6243)]
  __GI___nptl_create_event () at events.c:25
  25      {
  (gdb) PASS: gdb.threads/manythreads.exp: stop threads 1

Tested on x86_64 Fedora 20, native and gdbserver.

gdb/
2014-10-02  Pedro Alves  <palves@redhat.com>

	* breakpoint.c (breakpoints_should_be_inserted_now): Use
	threads_are_executing.
	* breakpoint.h (breakpoints_should_be_inserted_now): Add
	describing comment.
	* gdbthread.h (threads_are_executing): Declare.
	(handle_signal_stop) <random signals>: Don't print about the
	signal here if stopping.
	(end_stepping_range): Don't notify observers here.
	(normal_stop): Update the thread list.  If stopped by a random
	signal or a stepping range ended, notify observers.
	* thread.c (threads_executing): New global.
	(init_thread_list): Clear 'threads_executing'.
	(set_executing): Set or clear 'threads_executing'.
	(threads_are_executing): New function.
	(update_threads_executing): New function.
	(update_thread_list): Use it.

gdb/testsuite/
2014-10-02  Pedro Alves  <palves@redhat.com>

	* gdb.threads/break-while-running.exp (test): Add new
	'update_thread_list' argument.  Skip "info threads" if false.
	(top level): Add new 'update_thread_list' axis.

1 files changed, 10 insertions, 0 deletions

diff --git a/gdb/breakpoint.h b/gdb/breakpoint.h
index e191c10..d65405f 100644
--- a/gdb/breakpoint.h
+++ b/gdb/breakpoint.h
@@ -1491,6 +1491,16 @@ extern void breakpoint_xfer_memory (gdb_byte *readbuf, gdb_byte *writebuf,
 				    const gdb_byte *writebuf_org,
 				    ULONGEST memaddr, LONGEST len);
 
+/* Return true if breakpoints should be inserted now.  That'll be the
+   case if either:
+
+    - the target has global breakpoints.
+
+    - "breakpoint always-inserted" is on, and the target has
+      execution.
+
+    - threads are executing.
+*/
 extern int breakpoints_should_be_inserted_now (void);
 
 /* Called each time new event from target is processed.