gdb/tui: disable tui mode when an assert triggers

When an assert triggers in tui mode the output is not great, the
internal backtrace that is generated is printed directly to the file
descriptor for gdb_stderr, and, as a result, does not currently format
itself correctly - the output uses only '\n' at the end of each line,
and so, when the terminal is in raw mode, the cursor does not return
to the start of each line after the '\n'.

This is mostly fixable, we could update bt-utils.c to use '\r\n'
instead of just '\n', and this would fix most of the problems.  The
one we can't easily fix is if/when GDB is built to use execinfo
instead of libbacktrace, in this case we use backtrace_symbols_fd to
print the symbols, and this function only uses '\n' as the line
terminator.  Fixing this would require switching to backtrace_symbols,
but that API uses malloc, which is something we're trying to
avoid (this code is called when GDB hits an error, so ideally we don't
want to rely on malloc).

However, the execinfo code is only used when libbacktrace is not
available (or the user specifically disables libbacktrace) so maybe we
can ignore that problem...

... but there is another problem.  When the backtrace is printed in
raw mode, it is possible that the backtrace fills the screen.  With
the terminal in raw mode we don't have the ability to scroll back,
which means we loose some of the backtrace, which isn't ideal.

In this commit I propose that we should disable tui mode whenever we
handle a fatal signal, or when we hit the internal error code
path (e.g. when an assert triggers).  With this done then we don't
need to update the bt-utils.c code, and the execinfo version of the
code (using backtrace_symbols_fd) works just fine.  We also get the
ability to scroll back to view the error message and all of the
backtrace, assuming the users terminal supports scrolling back.

The only downside I see with this change is if the tui_disable call
itself causes an error for some reason, or, if we handle a single at a
time when it is not safe to call tui_disable, in these cases the extra
tui_disable call might cause GDB to loose the original error.

However, I think (just from personal experience) that the above two
issues are pretty rare and the benefits from this change far out
weighs the possible drawbacks.

1 files changed, 6 insertions, 1 deletions

diff --git a/gdb/utils.c b/gdb/utils.c
index 734c5bf..95adbe5 100644
--- a/gdb/utils.c
+++ b/gdb/utils.c
@@ -29,7 +29,8 @@
 #endif /* HAVE_SYS_RESOURCE_H */
 
 #ifdef TUI
-#include "tui/tui.h"		/* For tui_get_command_dimension.   */
+/* For tui_get_command_dimension and tui_disable.   */
+#include "tui/tui.h"
 #endif
 
 #ifdef __GO32__
@@ -354,6 +355,10 @@ internal_vproblem (struct internal_problem *problem,
       }
   }
 
+#ifdef TUI
+  tui_disable ();
+#endif
+
   /* Create a string containing the full error/warning message.  Need
      to call query with this full string, as otherwize the reason
      (error/warning) and question become separated.  Format using a