pretty-print: reimplement pp_format with a new struct pp_token

The following patch rewrites the internals of pp_format.

A pretty_printer's output_buffer maintains a stack of chunk_info
instances, each one responsible for handling a call to pp_format, where
having a stack allows us to support re-entrant calls to pp_format on the
same pretty_printer.

Previously a chunk_info merely stored buffers of accumulated text
per unformatted run and per formatted argument.

This led to various special-casing for handling:

- urlifiers, needing class quoting_info to handle awkard cases where
  the run of quoted text could be split between stages 1 and 2
  of formatting

- dumpfiles, where the optinfo machinery could lead to objects being
  stashed during formatting for later replay to JSON optimization
  records

- in the C++ frontend, the format codes %H and %I can't be processed
  until we've seen both, leading to awkward code to manipulate the
  text buffers

Further, supporting URLs in messages in SARIF output (PR other/116419)
would add additional manipulations of text buffers, since our internal
pp_begin_url API gives the URL at the beginning of the wrapped text,
whereas SARIF's format for embedded URLs has the URL *after* the wrapped
text.  Also when handling "%@" we wouldn't necessarily know the URL of
an event ID until later, requiring further nasty special-case
manipulation of text buffers.

This patch rewrites pretty-print formatting by introducing a new
intermediate representation during formatting: pp_token and
pp_token_list.  Rather than simply accumulating a buffer of "char" in
the chunk_obstack during formatting, we now also accumulate a
pp_token_list, a doubly-linked list of pp_token, which can be:
- text buffers
- begin/end colorization
- begin/end quote
- begin/end URL
- "custom data" tokens

Working at the level of tokens rather than just text buffers allows the
various awkward special cases above to be replaced with uniform logic.
For example, all "urlification" is now done in phase 3 of formatting,
in one place, by looking for [..., BEGIN_QUOTE, TEXT, END_QUOTE, ...]
and injecting BEGIN_URL and END_URL wrapper tokens when the urlifier
has a URL for TEXT.  Doing so greatly simplifies the urlifier code,
allowing the removal of class quoting_info.

The tokens and token lists are allocated on the chunk_obstack, and so
there's no additional heap activity required, with the memory reclaimed
when the chunk_obstack is freed after phase 3 of formatting.

New kinds of pp_token can be added as needed to support output formats.
For example, the followup patch adds a token for "%@" for events IDs, to
better support SARIF output.

No functional change intended.

gcc/c/ChangeLog:
	* c-objc-common.cc (c_tree_printer): Convert final param from
	const char ** to pp_token_list &.

gcc/cp/ChangeLog:
	* error.cc: Include "make-unique.h".
	(deferred_printed_type::m_buffer_ptr): Replace with...
	(deferred_printed_type::m_printed_text): ...this and...
	(deferred_printed_type::m_token_list): ...this.
	(deferred_printed_type::deferred_printed_type): Update ctors for
	above changes.
	(deferred_printed_type::set_text_for_token_list): New.
	(append_formatted_chunk): Pass chunk_obstack to
	append_formatted_chunk.
	(add_quotes): Delete.
	(cxx_format_postprocessor::handle): Reimplement to call
	deferred_printed_type::set_text_for_token_list, rather than store
	buffer pointers.
	(defer_phase_2_of_type_diff): Replace param "buffer_ptr"
	with "formatted_token_list".  Reimplement by storing
	a pointer to formatted_token_list so that the postprocessor can
	put its text there.
	(cp_printer): Convert param "buffer_ptr" to
	"formatted_token_list".  Update calls to
	defer_phase_2_of_type_diff accordingly.

gcc/ChangeLog:
	* diagnostic.cc (diagnostic_context::report_diagnostic): Don't
	pass m_urlifier to pp_format, as urlification now happens in
	phase 3.
	* dump-context.h (class dump_pretty_printer): Update leading
	comment.
	(dump_pretty_printer::emit_items): Drop decl.
	(dump_pretty_printer::set_optinfo): New.
	(class dump_pretty_printer::stashed_item): Delete class.
	(class dump_pretty_printer::custom_token_printer): New class.
	(dump_pretty_printer::format_decoder_cb): Convert param from
	const char ** to pp_token_list &.
	(dump_pretty_printer::decode_format): Likewise.
	(dump_pretty_printer::stash_item): Likewise.
	(dump_pretty_printer::emit_any_pending_textual_chunks): Drop decl.
	(dump_pretty_printer::m_stashed_items): Delete field.
	(dump_pretty_printer::m_token_printer): New member data.
	* dumpfile.cc (struct wrapped_optinfo_item): New.
	(dump_pretty_printer::dump_pretty_printer): Update for dropping
	of field m_stashed_items and new field m_token_printer.
	(dump_pretty_printer::emit_items): Delete; we now use
	pp_output_formatted_text..
	(dump_pretty_printer::emit_any_pending_textual_chunks): Delete.
	(dump_pretty_printer::stash_item): Convert param from
	const char ** to pp_token_list &.
	(dump_pretty_printer::format_decoder_cb): Likewise.
	(dump_pretty_printer::decode_format): Likewise.
	(dump_pretty_printer::custom_token_printer::print_tokens): New.
	(dump_pretty_printer::custom_token_printer::emit_any_pending_textual_chunks):
	New.
	(dump_context::dump_printf_va): Call set_optinfo on the
	dump_pretty_printer.  Replace call to emit_items with a call to
	pp_output_formatted_text.
	* opt-problem.cc (opt_problem::opt_problem): Replace call to
	emit_items with call to set_optinfo and call to
	pp_output_formatted_text.
	* pretty-print-format-impl.h (struct pp_token): New.
	(struct pp_token_text): New.
	(is_a_helper <pp_token_text *>::test): New.
	(is_a_helper <const pp_token_text *>::test): New.
	(struct pp_token_begin_color): New.
	(is_a_helper <pp_token_begin_color *>::test): New.
	(is_a_helper <const pp_token_begin_color *>::test): New.
	(struct pp_token_end_color): New.
	(struct pp_token_begin_quote): New.
	(struct pp_token_end_quote): New.
	(struct pp_token_begin_url): New.
	(is_a_helper <pp_token_begin_url*>::test): New.
	(is_a_helper <const pp_token_begin_url*>::test): New.
	(struct pp_token_end_url): New.
	(struct pp_token_custom_data): New.
	(is_a_helper <pp_token_custom_data *>::test): New.
	(is_a_helper <const pp_token_custom_data *>::test): New.
	(class pp_token_list): New.
	(chunk_info::get_args): Drop.
	(chunk_info::get_quoting_info): Drop.
	(chunk_info::get_token_lists): New accessor.
	(chunk_info::append_formatted_chunk): Add obstack & param.
	(chunk_info::dump): New decls.
	(chunk_info::m_args): Convert element type from const char * to
	pp_token_list *.  Rewrite/update comment.
	(chunk_info::m_quotes): Drop field.
	* pretty-print-markup.h (class pp_token_list): New forward decl.
	(pp_markup::context::context): Drop urlifier param; add
	formatted_token_list param.
	(pp_markup::context::push_back_any_text): New decl.
	(pp_markup::context::m_urlifier): Drop field.
	(pp_markup::context::m_formatted_token_list): New field.
	* pretty-print-urlifier.h: Update comment.
	* pretty-print.cc: Define INCLUDE_MEMORY.  Include
	"make-unique.h".
	(default_token_printer): New forward decl.
	(obstack_append_string): Delete.
	(urlify_quoted_string): Delete.
	(pp_token::pp_token): New.
	(pp_token::dump): New.
	(allocate_object): New.
	(class quoting_info): Delete.
	(pp_token::operator new): New.
	(pp_token::operator delete): New.
	(pp_token_list::operator new): New.
	(pp_token_list::operator delete): New.
	(pp_token_list::pp_token_list): New.
	(pp_token_list::~pp_token_list): New.
	(pp_token_list::push_back_text): New.
	(pp_token_list::push_back): New.
	(pp_token_list::push_back_list): New.
	(pp_token_list::pop_front): New.
	(pp_token_list::remove_token): New.
	(pp_token_list::insert_after): New.
	(pp_token_list::replace_custom_tokens): New.
	(pp_token_list::merge_consecutive_text_tokens): New.
	(pp_token_list::apply_urlifier): New.
	(pp_token_list::dump): New.
	(chunk_info::append_formatted_chunk): Add obstack & param and use
	it to reimplement in terms of token lists.
	(chunk_info::pop_from_output_buffer): Drop m_quotes.
	(chunk_info::on_begin_quote): Delete.
	(chunk_info::dump): New.
	(chunk_info::on_end_quote): Delete.
	(push_back_any_text): New.
	(pretty_printer::format): Drop "urlifier" param and quoting_info
	logic.  Convert "formatters" and "args" from const ** to
	pp_token_list **.  Reimplement so that rather than just
	accumulating a text buffer in the chunk_obstack for each arg,
	instead also accumulate a pp_token_list and pp_tokens for each
	arg.
	(auto_obstack::operator obstack &): New.
	(quoting_info::handle_phase_3): Delete.
	(pp_output_formatted_text): Reimplement in terms of manipulations
	of pp_token_lists, rather than char buffers.  Call
	default_token_printer, or m_token_printer's print_tokens vfunc.
	(default_token_printer): New.
	(pretty_printer::pretty_printer): Initialize m_token_printer in
	both ctors.
	(pp_markup::context::begin_quote): Reimplement to use token list.
	(pp_markup::context::end_quote): Likewise.
	(pp_markup::context::begin_highlight_color): Likewise.
	(pp_markup::context::end_highlight_color): Likewise.
	(pp_markup::context::push_back_any_text): New.
	(selftest::test_merge_consecutive_text_tokens): New.
	(selftest::test_custom_tokens_1): New.
	(selftest::test_custom_tokens_2): New.
	(selftest::pp_printf_with_urlifier): Drop "urlifier" param from
	call to pp_format.
	(selftest::test_urlification): Add test of the example from
	pretty-print-format-impl.h.
	(selftest::pretty_print_cc_tests): Call the new selftest
	functions.
	* pretty-print.h (class quoting_info): Drop forward decl.
	(class pp_token_list): New forward decl.
	(printer_fn): Convert final param from const char ** to
	pp_token_list &.
	(class token_printer): New.
	(class pretty_printer): Add pp_output_formatted_text as friend.
	(pretty_printer::set_token_printer): New.
	(pretty_printer::format): Drop urlifier param as this now happens
	in phase 3.
	(pretty_printer::m_format_decoder): Update comment.
	(pretty_printer::m_token_printer): New field.
	(pp_format): Drop urlifier param.
	* tree-diagnostic.cc (default_tree_printer): Convert final param
	from const char ** to pp_token_list &.
	* tree-diagnostic.h: Likewise for decl.

gcc/fortran/ChangeLog:
	* error.cc (gfc_format_decoder): Convert final param from
	const char **buffer_ptr to pp_token_list &formatted_token_list,
	and update call to default_tree_printer accordingly.

Signed-off-by: David Malcolm <dmalcolm@redhat.com>

1 files changed, 57 insertions, 48 deletions

diff --git a/gcc/cp/error.cc b/gcc/cp/error.cc
index 3cc0dd1..420fad2 100644
--- a/gcc/cp/error.cc
+++ b/gcc/cp/error.cc
@@ -39,6 +39,7 @@ along with GCC; see the file COPYING3.  If not see
 #include "cp-name-hint.h"
 #include "attribs.h"
 #include "pretty-print-format-impl.h"
+#include "make-unique.h"
 
 #define pp_separate_with_comma(PP) pp_cxx_separate_with (PP, ',')
 #define pp_separate_with_semicolon(PP) pp_cxx_separate_with (PP, ';')
@@ -110,7 +111,7 @@ static void cp_print_error_function (diagnostic_context *,
 				     const diagnostic_info *);
 
 static bool cp_printer (pretty_printer *, text_info *, const char *,
-			int, bool, bool, bool, bool *, const char **);
+			int, bool, bool, bool, bool *, pp_token_list &);
 
 /* Color names for highlighting "%qH" vs "%qI" values,
    and ranges corresponding to them.  */
@@ -124,22 +125,50 @@ class deferred_printed_type
 {
 public:
   deferred_printed_type ()
-  : m_tree (NULL_TREE), m_buffer_ptr (NULL), m_verbose (false), m_quote (false)
+  : m_tree (NULL_TREE),
+    m_printed_text (),
+    m_token_list (nullptr),
+    m_verbose (false), m_quote (false)
   {}
 
-  deferred_printed_type (tree type, const char **buffer_ptr, bool verbose,
+  deferred_printed_type (tree type,
+			 pp_token_list &token_list,
+			 bool verbose,
 			 bool quote)
-  : m_tree (type), m_buffer_ptr (buffer_ptr), m_verbose (verbose),
+  : m_tree (type),
+    m_printed_text (),
+    m_token_list (&token_list),
+    m_verbose (verbose),
     m_quote (quote)
   {
     gcc_assert (type);
-    gcc_assert (buffer_ptr);
+  }
+
+  void set_text_for_token_list (const char *text, bool quote)
+  {
+    /* Replace the contents of m_token_list with a text token for TEXT,
+       possibly wrapped by BEGIN_QUOTE/END_QUOTE (if QUOTE is true).
+       This allows us to ignore any {BEGIN,END}_QUOTE tokens added
+       by %qH and %qI, and instead use the quoting from type_to_string,
+       and its logic for "aka".  */
+    while (m_token_list->m_first)
+      m_token_list->pop_front ();
+
+    if (quote)
+      m_token_list->push_back<pp_token_begin_quote> ();
+
+    // TEXT is gc-allocated, so we can borrow it
+    m_token_list->push_back_text (label_text::borrow (text));
+
+    if (quote)
+      m_token_list->push_back<pp_token_end_quote> ();
   }
 
   /* The tree is not GTY-marked: they are only non-NULL within a
      call to pp_format.  */
   tree m_tree;
-  const char **m_buffer_ptr;
+  label_text m_printed_text;
+  pp_token_list *m_token_list;
   bool m_verbose;
   bool m_quote;
 };
@@ -4402,26 +4431,7 @@ append_formatted_chunk (pretty_printer *pp, const char *content)
 {
   output_buffer *buffer = pp_buffer (pp);
   chunk_info *chunk_array = buffer->cur_chunk_array;
-  chunk_array->append_formatted_chunk (content);
-}
-
-/* Create a copy of CONTENT, with quotes added, and,
-   potentially, with colorization.
-   No escaped is performed on CONTENT.
-   The result is in a GC-allocated buffer. */
-
-static const char *
-add_quotes (const char *content, bool show_color)
-{
-  pretty_printer tmp_pp;
-  pp_show_color (&tmp_pp) = show_color;
-
-  /* We have to use "%<%s%>" rather than "%qs" here in order to avoid
-     quoting colorization bytes within the results and using either
-     pp_quote or pp_begin_quote doesn't work the same.  */
-  pp_printf (&tmp_pp, "%<%s%>", content);
-
-  return pp_ggc_formatted_text (&tmp_pp);
+  chunk_array->append_formatted_chunk (buffer->chunk_obstack, content);
 }
 
 #if __GNUC__ >= 10
@@ -4429,8 +4439,8 @@ add_quotes (const char *content, bool show_color)
 #endif
 
 /* If we had %H and %I, and hence deferred printing them,
-   print them now, storing the result into the chunk_info
-   for pp_format.  Quote them if 'q' was provided.
+   print them now, storing the result into custom_token_value
+   for the custom pp_token.  Quote them if 'q' was provided.
    Also print the difference in tree form, adding it as
    an additional chunk.  */
 
@@ -4448,13 +4458,13 @@ cxx_format_postprocessor::handle (pretty_printer *pp)
 	= show_highlight_colors ? highlight_colors::percent_i : nullptr;
       /* Avoid reentrancy issues by working with a copy of
 	 m_type_a and m_type_b, resetting them now.  */
-      deferred_printed_type type_a = m_type_a;
-      deferred_printed_type type_b = m_type_b;
+      deferred_printed_type type_a = std::move (m_type_a);
+      deferred_printed_type type_b = std::move (m_type_b);
       m_type_a = deferred_printed_type ();
       m_type_b = deferred_printed_type ();
 
-      gcc_assert (type_a.m_buffer_ptr);
-      gcc_assert (type_b.m_buffer_ptr);
+      gcc_assert (type_a.m_token_list);
+      gcc_assert (type_b.m_token_list);
 
       bool show_color = pp_show_color (pp);
 
@@ -4495,13 +4505,8 @@ cxx_format_postprocessor::handle (pretty_printer *pp)
 					percent_i);
 	}
 
-      if (type_a.m_quote)
-	type_a_text = add_quotes (type_a_text, show_color);
-      *type_a.m_buffer_ptr = type_a_text;
-
-       if (type_b.m_quote)
-	type_b_text = add_quotes (type_b_text, show_color);
-      *type_b.m_buffer_ptr = type_b_text;
+      type_a.set_text_for_token_list (type_a_text, type_a.m_quote);
+      type_b.set_text_for_token_list (type_b_text, type_b.m_quote);
    }
 }
 
@@ -4526,9 +4531,12 @@ cxx_format_postprocessor::handle (pretty_printer *pp)
    pretty_printer's m_format_postprocessor hook.
 
    This is called in phase 2 of pp_format, when it is accumulating
-   a series of formatted chunks.  We stash the location of the chunk
-   we're meant to have written to, so that we can write to it in the
-   m_format_postprocessor hook.
+   a series of pp_token lists.  Since we have to interact with the
+   fiddly quoting logic for "aka", we store the pp_token_list *
+   and in the m_format_postprocessor hook we generate text for the type
+   (possibly with quotes and colors), then replace all tokens in that token list
+   (such as [BEGIN_QUOTE, END_QUOTE]) with a text token containing the
+   freshly generated text.
 
    We also need to stash whether a 'q' prefix was provided (the QUOTE
    param)  so that we can add the quotes when writing out the delayed
@@ -4536,12 +4544,13 @@ cxx_format_postprocessor::handle (pretty_printer *pp)
 
 static void
 defer_phase_2_of_type_diff (deferred_printed_type *deferred,
-			    tree type, const char **buffer_ptr,
+			    tree type,
+			    pp_token_list &formatted_token_list,
 			    bool verbose, bool quote)
 {
   gcc_assert (deferred->m_tree == NULL_TREE);
-  gcc_assert (deferred->m_buffer_ptr == NULL);
-  *deferred = deferred_printed_type (type, buffer_ptr, verbose, quote);
+  *deferred = deferred_printed_type (type, formatted_token_list,
+				     verbose, quote);
 }
 
 /* Implementation of pp_markup::element_quoted_type::print_type
@@ -4578,7 +4587,7 @@ pp_markup::element_quoted_type::print_type (pp_markup::context &ctxt)
 static bool
 cp_printer (pretty_printer *pp, text_info *text, const char *spec,
 	    int precision, bool wide, bool set_locus, bool verbose,
-	    bool *quoted, const char **buffer_ptr)
+	    bool *quoted, pp_token_list &formatted_token_list)
 {
   gcc_assert (pp_format_postprocessor (pp));
   cxx_format_postprocessor *postprocessor
@@ -4618,11 +4627,11 @@ cp_printer (pretty_printer *pp, text_info *text, const char *spec,
     case 'F': result = fndecl_to_string (next_tree, verbose);	break;
     case 'H':
       defer_phase_2_of_type_diff (&postprocessor->m_type_a, next_tree,
-				  buffer_ptr, verbose, *quoted);
+				  formatted_token_list, verbose, *quoted);
       return true;
     case 'I':
       defer_phase_2_of_type_diff (&postprocessor->m_type_b, next_tree,
-				  buffer_ptr, verbose, *quoted);
+				  formatted_token_list, verbose, *quoted);
       return true;
     case 'L': result = language_to_string (next_lang);		break;
     case 'O': result = op_to_string (false, next_tcode);	break;