Make ANSI terminal escape sequences work in TUI

PR tui/14126 notes that ANSI terminal escape sequences don't affect
the colors shown in the TUI.  A simple way to see this is to try the
extended-prompt example from the gdb manual.

Curses does not pass escape sequences through to the terminal.
Instead, it replaces non-printable characters with a visible
representation, for example "^[" for the ESC character.

This patch fixes the problem by adding a simple ANSI terminal sequence
parser to gdb.  These sequences are decoded and those that are
recognized are turned into the appropriate curses calls.

The curses approach to color handling is unusual and so there are some
oddities in the implementation.

Standard curses has no notion of the default colors of the terminal.
So, if you set the foreground color, it is not possible to reset it --
you have to pick some other color.  ncurses provides an extension to
handle this, so this patch updates configure and uses it when
available.

Second, in curses, colors always come in pairs: you cannot set just
the foreground.  This patch handles this by tracking actually-used
pairs of colors and keeping a table of these for reuse.

Third, there are a limited number of such pairs available.  In this
patch, if you try to use too many color combinations, gdb will just
ignore some color changes.

Finally, in addition to limiting the number of color pairs, curses
also limits the number of colors.  This means that, when using
extended 8- or 24-bit color sequences, it may be possible to exhaust
the curses color table.

I am very sour on the curses design now.

I do not know how to write a test for this, so I did not.

gdb/ChangeLog
2018-12-28  Tom Tromey  <tom@tromey.com>

	PR tui/14126:
	* tui/tui.c (tui_enable): Call start_color and
	use_default_colors.
	* tui/tui-io.c (struct color_pair): New.
	(color_pair_map, last_color_pair, last_style): New globals.
	(tui_setup_io): Clean up color map when shutting down.
	(curses_colors): New constant.
	(get_color_pair, apply_ansi_escape): New functions.
	(tui_write): Rewrite.
	(tui_puts_internal): New function, from tui_puts.  Add "height"
	parameter.
	(tui_puts): Use tui_puts_internal.
	(tui_redisplay_readline): Use tui_puts_internal.
	(_initialize_tui_io): New function.
	(color_map): New globals.
	(get_color): New function.
	* configure.ac: Check for use_default_colors.
	* config.in, configure: Rebuild.

6 files changed, 251 insertions, 18 deletions

diff --git a/gdb/ChangeLog b/gdb/ChangeLog
index c98d292..5d61d27 100644
--- a/gdb/ChangeLog
+++ b/gdb/ChangeLog
@@ -1,5 +1,26 @@
 2018-12-28  Tom Tromey  <tom@tromey.com>
 
+	PR tui/14126:
+	* tui/tui.c (tui_enable): Call start_color and
+	use_default_colors.
+	* tui/tui-io.c (struct color_pair): New.
+	(color_pair_map, last_color_pair, last_style): New globals.
+	(tui_setup_io): Clean up color map when shutting down.
+	(curses_colors): New constant.
+	(get_color_pair, apply_ansi_escape): New functions.
+	(tui_write): Rewrite.
+	(tui_puts_internal): New function, from tui_puts.  Add "height"
+	parameter.
+	(tui_puts): Use tui_puts_internal.
+	(tui_redisplay_readline): Use tui_puts_internal.
+	(_initialize_tui_io): New function.
+	(color_map): New globals.
+	(get_color): New function.
+	* configure.ac: Check for use_default_colors.
+	* config.in, configure: Rebuild.
+
+2018-12-28  Tom Tromey  <tom@tromey.com>
+
 	* ui-out.h (enum class ui_out_style_kind) <ADDRESS>: New
 	constant.
 	* ui-out.c (ui_out::field_core_addr): Add styling.
diff --git a/gdb/config.in b/gdb/config.in
index deb9d4a..760db6b 100644
--- a/gdb/config.in
+++ b/gdb/config.in
@@ -543,6 +543,9 @@
 /* Define to 1 if you have the <unistd.h> header file. */
 #undef HAVE_UNISTD_H
 
+/* Define to 1 if you have the `use_default_colors' function. */
+#undef HAVE_USE_DEFAULT_COLORS
+
 /* Define to 1 if you have the `vfork' function. */
 #undef HAVE_VFORK
 
diff --git a/gdb/configure b/gdb/configure
index 833c92e..da701cd 100755
--- a/gdb/configure
+++ b/gdb/configure
@@ -13320,7 +13320,7 @@ for ac_func in getauxval getrusage getuid getgid \
 		sigaction sigprocmask sigsetmask socketpair \
 		ttrace wborder wresize setlocale iconvlist libiconvlist btowc \
 		setrlimit getrlimit posix_madvise waitpid \
-		ptrace64 sigaltstack setns
+		ptrace64 sigaltstack setns use_default_colors
 do :
   as_ac_var=`$as_echo "ac_cv_func_$ac_func" | $as_tr_sh`
 ac_fn_c_check_func "$LINENO" "$ac_func" "$as_ac_var"
diff --git a/gdb/configure.ac b/gdb/configure.ac
index 8bc8748..0025b1d 100644
--- a/gdb/configure.ac
+++ b/gdb/configure.ac
@@ -1359,7 +1359,7 @@ AC_CHECK_FUNCS([getauxval getrusage getuid getgid \
 		sigaction sigprocmask sigsetmask socketpair \
 		ttrace wborder wresize setlocale iconvlist libiconvlist btowc \
 		setrlimit getrlimit posix_madvise waitpid \
-		ptrace64 sigaltstack setns])
+		ptrace64 sigaltstack setns use_default_colors])
 AM_LANGINFO_CODESET
 GDB_AC_COMMON
 
diff --git a/gdb/tui/tui-io.c b/gdb/tui/tui-io.c
index 7823688..5a84d08 100644
--- a/gdb/tui/tui-io.c
+++ b/gdb/tui/tui-io.c
@@ -40,6 +40,7 @@
 #include "filestuff.h"
 #include "completer.h"
 #include "gdb_curses.h"
+#include <map>
 
 /* This redefines CTRL if it is not already defined, so it must come
    after terminal state releated include files like <term.h> and
@@ -188,6 +189,171 @@ tui_putc (char c)
   update_cmdwin_start_line ();
 }
 
+/* This maps colors to their corresponding color index.  */
+
+static std::map<ui_file_style::color, int> color_map;
+
+/* This holds a pair of colors and is used to track the mapping
+   between a color pair index and the actual colors.  */
+
+struct color_pair
+{
+  int fg;
+  int bg;
+
+  bool operator< (const color_pair &o) const
+  {
+    return fg < o.fg || (fg == o.fg && bg < o.bg);
+  }
+};
+
+/* This maps pairs of colors to their corresponding color pair
+   index.  */
+
+static std::map<color_pair, int> color_pair_map;
+
+/* This is indexed by ANSI color offset from the base color, and holds
+   the corresponding curses color constant.  */
+
+static const int curses_colors[] = {
+  COLOR_BLACK,
+  COLOR_RED,
+  COLOR_GREEN,
+  COLOR_YELLOW,
+  COLOR_BLUE,
+  COLOR_MAGENTA,
+  COLOR_CYAN,
+  COLOR_WHITE
+};
+
+/* Given a color, find its index.  */
+
+static bool
+get_color (const ui_file_style::color &color, int *result)
+{
+  if (color.is_none ())
+    *result = -1;
+  else if (color.is_basic ())
+    *result = curses_colors[color.get_value ()];
+  else
+    {
+      auto it = color_map.find (color);
+      if (it == color_map.end ())
+	{
+	  /* The first 8 colors are standard.  */
+	  int next = color_map.size () + 8;
+	  if (next >= COLORS)
+	    return false;
+	  uint8_t rgb[3];
+	  color.get_rgb (rgb);
+	  /* We store RGB as 0..255, but curses wants 0..1000.  */
+	  if (init_color (next, rgb[0] * 1000 / 255, rgb[1] * 1000 / 255,
+			  rgb[2] * 1000 / 255) == ERR)
+	    return false;
+	  color_map[color] = next;
+	  *result = next;
+	}
+      else
+	*result = it->second;
+    }
+  return true;
+}
+
+/* The most recently emitted color pair.  */
+
+static int last_color_pair = -1;
+
+/* The most recently applied style.  */
+
+static ui_file_style last_style;
+
+/* Given two colors, return their color pair index; making a new one
+   if necessary.  */
+
+static int
+get_color_pair (int fg, int bg)
+{
+  color_pair c = { fg, bg };
+  auto it = color_pair_map.find (c);
+  if (it == color_pair_map.end ())
+    {
+      /* Color pair 0 is our default color, so new colors start at
+	 1.  */
+      int next = color_pair_map.size () + 1;
+      /* Curses has a limited number of available color pairs.  Fall
+	 back to the default if we've used too many.  */
+      if (next >= COLOR_PAIRS)
+	return 0;
+      init_pair (next, fg, bg);
+      color_pair_map[c] = next;
+      return next;
+    }
+  return it->second;
+}
+
+/* Apply an ANSI escape sequence from BUF to W.  BUF must start with
+   the ESC character.  If BUF does not start with an ANSI escape,
+   return 0.  Otherwise, apply the sequence if it is recognized, or
+   simply ignore it if not.  In this case, the number of bytes read
+   from BUF is returned.  */
+
+static size_t
+apply_ansi_escape (WINDOW *w, const char *buf)
+{
+  ui_file_style style = last_style;
+  size_t n_read;
+
+  if (!style.parse (buf, &n_read))
+    return n_read;
+
+  /* Reset.  */
+  wattron (w, A_NORMAL);
+  wattroff (w, A_BOLD);
+  wattroff (w, A_DIM);
+  wattroff (w, A_REVERSE);
+  if (last_color_pair != -1)
+    wattroff (w, COLOR_PAIR (last_color_pair));
+  wattron (w, COLOR_PAIR (0));
+
+  const ui_file_style::color &fg = style.get_foreground ();
+  const ui_file_style::color &bg = style.get_background ();
+  if (!fg.is_none () || !bg.is_none ())
+    {
+      int fgi, bgi;
+      if (get_color (fg, &fgi) && get_color (bg, &bgi))
+	{
+	  int pair = get_color_pair (fgi, bgi);
+	  if (last_color_pair != -1)
+	    wattroff (w, COLOR_PAIR (last_color_pair));
+	  wattron (w, COLOR_PAIR (pair));
+	  last_color_pair = pair;
+	}
+    }
+
+  switch (style.get_intensity ())
+    {
+    case ui_file_style::NORMAL:
+      break;
+
+    case ui_file_style::BOLD:
+      wattron (w, A_BOLD);
+      break;
+
+    case ui_file_style::DIM:
+      wattron (w, A_DIM);
+      break;
+
+    default:
+      gdb_assert_not_reached ("invalid intensity");
+    }
+
+  if (style.is_reverse ())
+    wattron (w, A_REVERSE);
+
+  last_style = style;
+  return n_read;
+}
+
 /* Print LENGTH characters from the buffer pointed to by BUF to the
    curses command window.  The output is buffered.  It is up to the
    caller to refresh the screen if necessary.  */
@@ -195,10 +361,47 @@ tui_putc (char c)
 void
 tui_write (const char *buf, size_t length)
 {
+  /* We need this to be \0-terminated for the regexp matching.  */
+  std::string copy (buf, length);
+  tui_puts (copy.c_str ());
+}
+
+static void
+tui_puts_internal (const char *string, int *height)
+{
   WINDOW *w = TUI_CMD_WIN->generic.handle;
+  char c;
+  int prev_col = 0;
 
-  for (size_t i = 0; i < length; i++)
-    do_tui_putc (w, buf[i]);
+  while ((c = *string++) != 0)
+    {
+      if (c == '\1' || c == '\2')
+	{
+	  /* Ignore these, they are readline escape-marking
+	     sequences.  */
+	}
+      else
+	{
+	  if (c == '\033')
+	    {
+	      size_t bytes_read = apply_ansi_escape (w, string - 1);
+	      if (bytes_read > 0)
+		{
+		  string = string + bytes_read - 1;
+		  continue;
+		}
+	    }
+	  do_tui_putc (w, c);
+
+	  if (height != nullptr)
+	    {
+	      int col = getcurx (w);
+	      if (col <= prev_col)
+		++*height;
+	      prev_col = col;
+	    }
+	}
+    }
   update_cmdwin_start_line ();
 }
 
@@ -209,12 +412,7 @@ tui_write (const char *buf, size_t length)
 void
 tui_puts (const char *string)
 {
-  WINDOW *w = TUI_CMD_WIN->generic.handle;
-  char c;
-
-  while ((c = *string++) != 0)
-    do_tui_putc (w, c);
-  update_cmdwin_start_line ();
+  tui_puts_internal (string, nullptr);
 }
 
 /* Readline callback.
@@ -254,14 +452,10 @@ tui_redisplay_readline (void)
   wmove (w, start_line, 0);
   prev_col = 0;
   height = 1;
-  for (in = 0; prompt && prompt[in]; in++)
-    {
-      waddch (w, prompt[in]);
-      col = getcurx (w);
-      if (col <= prev_col)
-        height++;
-      prev_col = col;
-    }
+  if (prompt != nullptr)
+    tui_puts_internal (prompt, &height);
+
+  prev_col = getcurx (w);
   for (in = 0; in <= rl_end; in++)
     {
       unsigned char c;
@@ -537,6 +731,12 @@ tui_setup_io (int mode)
 
       /* Save tty for SIGCONT.  */
       savetty ();
+
+      /* Clean up color information.  */
+      last_color_pair = -1;
+      last_style = ui_file_style ();
+      color_map.clear ();
+      color_pair_map.clear ();
     }
 }
 
diff --git a/gdb/tui/tui.c b/gdb/tui/tui.c
index 50cad22..2299824 100644
--- a/gdb/tui/tui.c
+++ b/gdb/tui/tui.c
@@ -437,6 +437,15 @@ tui_enable (void)
 		 gdb_getenv_term ());
 	}
       w = stdscr;
+      if (has_colors ())
+	{
+#ifdef HAVE_USE_DEFAULT_COLORS
+	  /* Ncurses extension to help with resetting to the default
+	     color.  */
+	  use_default_colors ();
+#endif
+	  start_color ();
+	}
 
       /* Check required terminal capabilities.  The MinGW port of
 	 ncurses does have them, but doesn't expose them through "cup".  */