pretty-print: reimplement pp_format with a new struct pp_token

The following patch rewrites the internals of pp_format.

A pretty_printer's output_buffer maintains a stack of chunk_info
instances, each one responsible for handling a call to pp_format, where
having a stack allows us to support re-entrant calls to pp_format on the
same pretty_printer.

Previously a chunk_info merely stored buffers of accumulated text
per unformatted run and per formatted argument.

This led to various special-casing for handling:

- urlifiers, needing class quoting_info to handle awkard cases where
  the run of quoted text could be split between stages 1 and 2
  of formatting

- dumpfiles, where the optinfo machinery could lead to objects being
  stashed during formatting for later replay to JSON optimization
  records

- in the C++ frontend, the format codes %H and %I can't be processed
  until we've seen both, leading to awkward code to manipulate the
  text buffers

Further, supporting URLs in messages in SARIF output (PR other/116419)
would add additional manipulations of text buffers, since our internal
pp_begin_url API gives the URL at the beginning of the wrapped text,
whereas SARIF's format for embedded URLs has the URL *after* the wrapped
text.  Also when handling "%@" we wouldn't necessarily know the URL of
an event ID until later, requiring further nasty special-case
manipulation of text buffers.

This patch rewrites pretty-print formatting by introducing a new
intermediate representation during formatting: pp_token and
pp_token_list.  Rather than simply accumulating a buffer of "char" in
the chunk_obstack during formatting, we now also accumulate a
pp_token_list, a doubly-linked list of pp_token, which can be:
- text buffers
- begin/end colorization
- begin/end quote
- begin/end URL
- "custom data" tokens

Working at the level of tokens rather than just text buffers allows the
various awkward special cases above to be replaced with uniform logic.
For example, all "urlification" is now done in phase 3 of formatting,
in one place, by looking for [..., BEGIN_QUOTE, TEXT, END_QUOTE, ...]
and injecting BEGIN_URL and END_URL wrapper tokens when the urlifier
has a URL for TEXT.  Doing so greatly simplifies the urlifier code,
allowing the removal of class quoting_info.

The tokens and token lists are allocated on the chunk_obstack, and so
there's no additional heap activity required, with the memory reclaimed
when the chunk_obstack is freed after phase 3 of formatting.

New kinds of pp_token can be added as needed to support output formats.
For example, the followup patch adds a token for "%@" for events IDs, to
better support SARIF output.

No functional change intended.

gcc/c/ChangeLog:
	* c-objc-common.cc (c_tree_printer): Convert final param from
	const char ** to pp_token_list &.

gcc/cp/ChangeLog:
	* error.cc: Include "make-unique.h".
	(deferred_printed_type::m_buffer_ptr): Replace with...
	(deferred_printed_type::m_printed_text): ...this and...
	(deferred_printed_type::m_token_list): ...this.
	(deferred_printed_type::deferred_printed_type): Update ctors for
	above changes.
	(deferred_printed_type::set_text_for_token_list): New.
	(append_formatted_chunk): Pass chunk_obstack to
	append_formatted_chunk.
	(add_quotes): Delete.
	(cxx_format_postprocessor::handle): Reimplement to call
	deferred_printed_type::set_text_for_token_list, rather than store
	buffer pointers.
	(defer_phase_2_of_type_diff): Replace param "buffer_ptr"
	with "formatted_token_list".  Reimplement by storing
	a pointer to formatted_token_list so that the postprocessor can
	put its text there.
	(cp_printer): Convert param "buffer_ptr" to
	"formatted_token_list".  Update calls to
	defer_phase_2_of_type_diff accordingly.

gcc/ChangeLog:
	* diagnostic.cc (diagnostic_context::report_diagnostic): Don't
	pass m_urlifier to pp_format, as urlification now happens in
	phase 3.
	* dump-context.h (class dump_pretty_printer): Update leading
	comment.
	(dump_pretty_printer::emit_items): Drop decl.
	(dump_pretty_printer::set_optinfo): New.
	(class dump_pretty_printer::stashed_item): Delete class.
	(class dump_pretty_printer::custom_token_printer): New class.
	(dump_pretty_printer::format_decoder_cb): Convert param from
	const char ** to pp_token_list &.
	(dump_pretty_printer::decode_format): Likewise.
	(dump_pretty_printer::stash_item): Likewise.
	(dump_pretty_printer::emit_any_pending_textual_chunks): Drop decl.
	(dump_pretty_printer::m_stashed_items): Delete field.
	(dump_pretty_printer::m_token_printer): New member data.
	* dumpfile.cc (struct wrapped_optinfo_item): New.
	(dump_pretty_printer::dump_pretty_printer): Update for dropping
	of field m_stashed_items and new field m_token_printer.
	(dump_pretty_printer::emit_items): Delete; we now use
	pp_output_formatted_text..
	(dump_pretty_printer::emit_any_pending_textual_chunks): Delete.
	(dump_pretty_printer::stash_item): Convert param from
	const char ** to pp_token_list &.
	(dump_pretty_printer::format_decoder_cb): Likewise.
	(dump_pretty_printer::decode_format): Likewise.
	(dump_pretty_printer::custom_token_printer::print_tokens): New.
	(dump_pretty_printer::custom_token_printer::emit_any_pending_textual_chunks):
	New.
	(dump_context::dump_printf_va): Call set_optinfo on the
	dump_pretty_printer.  Replace call to emit_items with a call to
	pp_output_formatted_text.
	* opt-problem.cc (opt_problem::opt_problem): Replace call to
	emit_items with call to set_optinfo and call to
	pp_output_formatted_text.
	* pretty-print-format-impl.h (struct pp_token): New.
	(struct pp_token_text): New.
	(is_a_helper <pp_token_text *>::test): New.
	(is_a_helper <const pp_token_text *>::test): New.
	(struct pp_token_begin_color): New.
	(is_a_helper <pp_token_begin_color *>::test): New.
	(is_a_helper <const pp_token_begin_color *>::test): New.
	(struct pp_token_end_color): New.
	(struct pp_token_begin_quote): New.
	(struct pp_token_end_quote): New.
	(struct pp_token_begin_url): New.
	(is_a_helper <pp_token_begin_url*>::test): New.
	(is_a_helper <const pp_token_begin_url*>::test): New.
	(struct pp_token_end_url): New.
	(struct pp_token_custom_data): New.
	(is_a_helper <pp_token_custom_data *>::test): New.
	(is_a_helper <const pp_token_custom_data *>::test): New.
	(class pp_token_list): New.
	(chunk_info::get_args): Drop.
	(chunk_info::get_quoting_info): Drop.
	(chunk_info::get_token_lists): New accessor.
	(chunk_info::append_formatted_chunk): Add obstack & param.
	(chunk_info::dump): New decls.
	(chunk_info::m_args): Convert element type from const char * to
	pp_token_list *.  Rewrite/update comment.
	(chunk_info::m_quotes): Drop field.
	* pretty-print-markup.h (class pp_token_list): New forward decl.
	(pp_markup::context::context): Drop urlifier param; add
	formatted_token_list param.
	(pp_markup::context::push_back_any_text): New decl.
	(pp_markup::context::m_urlifier): Drop field.
	(pp_markup::context::m_formatted_token_list): New field.
	* pretty-print-urlifier.h: Update comment.
	* pretty-print.cc: Define INCLUDE_MEMORY.  Include
	"make-unique.h".
	(default_token_printer): New forward decl.
	(obstack_append_string): Delete.
	(urlify_quoted_string): Delete.
	(pp_token::pp_token): New.
	(pp_token::dump): New.
	(allocate_object): New.
	(class quoting_info): Delete.
	(pp_token::operator new): New.
	(pp_token::operator delete): New.
	(pp_token_list::operator new): New.
	(pp_token_list::operator delete): New.
	(pp_token_list::pp_token_list): New.
	(pp_token_list::~pp_token_list): New.
	(pp_token_list::push_back_text): New.
	(pp_token_list::push_back): New.
	(pp_token_list::push_back_list): New.
	(pp_token_list::pop_front): New.
	(pp_token_list::remove_token): New.
	(pp_token_list::insert_after): New.
	(pp_token_list::replace_custom_tokens): New.
	(pp_token_list::merge_consecutive_text_tokens): New.
	(pp_token_list::apply_urlifier): New.
	(pp_token_list::dump): New.
	(chunk_info::append_formatted_chunk): Add obstack & param and use
	it to reimplement in terms of token lists.
	(chunk_info::pop_from_output_buffer): Drop m_quotes.
	(chunk_info::on_begin_quote): Delete.
	(chunk_info::dump): New.
	(chunk_info::on_end_quote): Delete.
	(push_back_any_text): New.
	(pretty_printer::format): Drop "urlifier" param and quoting_info
	logic.  Convert "formatters" and "args" from const ** to
	pp_token_list **.  Reimplement so that rather than just
	accumulating a text buffer in the chunk_obstack for each arg,
	instead also accumulate a pp_token_list and pp_tokens for each
	arg.
	(auto_obstack::operator obstack &): New.
	(quoting_info::handle_phase_3): Delete.
	(pp_output_formatted_text): Reimplement in terms of manipulations
	of pp_token_lists, rather than char buffers.  Call
	default_token_printer, or m_token_printer's print_tokens vfunc.
	(default_token_printer): New.
	(pretty_printer::pretty_printer): Initialize m_token_printer in
	both ctors.
	(pp_markup::context::begin_quote): Reimplement to use token list.
	(pp_markup::context::end_quote): Likewise.
	(pp_markup::context::begin_highlight_color): Likewise.
	(pp_markup::context::end_highlight_color): Likewise.
	(pp_markup::context::push_back_any_text): New.
	(selftest::test_merge_consecutive_text_tokens): New.
	(selftest::test_custom_tokens_1): New.
	(selftest::test_custom_tokens_2): New.
	(selftest::pp_printf_with_urlifier): Drop "urlifier" param from
	call to pp_format.
	(selftest::test_urlification): Add test of the example from
	pretty-print-format-impl.h.
	(selftest::pretty_print_cc_tests): Call the new selftest
	functions.
	* pretty-print.h (class quoting_info): Drop forward decl.
	(class pp_token_list): New forward decl.
	(printer_fn): Convert final param from const char ** to
	pp_token_list &.
	(class token_printer): New.
	(class pretty_printer): Add pp_output_formatted_text as friend.
	(pretty_printer::set_token_printer): New.
	(pretty_printer::format): Drop urlifier param as this now happens
	in phase 3.
	(pretty_printer::m_format_decoder): Update comment.
	(pretty_printer::m_token_printer): New field.
	(pp_format): Drop urlifier param.
	* tree-diagnostic.cc (default_tree_printer): Convert final param
	from const char ** to pp_token_list &.
	* tree-diagnostic.h: Likewise for decl.

gcc/fortran/ChangeLog:
	* error.cc (gfc_format_decoder): Convert final param from
	const char **buffer_ptr to pp_token_list &formatted_token_list,
	and update call to default_tree_printer accordingly.

Signed-off-by: David Malcolm <dmalcolm@redhat.com>

1 files changed, 22 insertions, 18 deletions

diff --git a/gcc/dump-context.h b/gcc/dump-context.h
index 5992956..e90c4ee 100644
--- a/gcc/dump-context.h
+++ b/gcc/dump-context.h
@@ -154,48 +154,52 @@ class dump_context
 };
 
 /* A subclass of pretty_printer for implementing dump_context::dump_printf_va.
-   In particular, the formatted chunks are captured as optinfo_item instances,
-   thus retaining metadata about the entities being dumped (e.g. source
-   locations), rather than just as plain text.  */
+   In particular, the formatted chunks are captured as optinfo_item instances
+   as pp_token_custom_data, thus retaining metadata about the entities being
+   dumped (e.g. source locations), rather than just as plain text.
+   These custom items are retained through to the end of stage 3 of formatted
+   printing; the printer uses a custom token_printer subclass to emit them to
+   the active optinfo (if any).  */
 
 class dump_pretty_printer : public pretty_printer
 {
 public:
   dump_pretty_printer (dump_context *context, dump_flags_t dump_kind);
 
-  void emit_items (optinfo *dest);
+  void set_optinfo (optinfo *info) { m_token_printer.m_optinfo = info; }
 
 private:
-  /* Information on an optinfo_item that was generated during phase 2 of
-     formatting.  */
-  class stashed_item
+  struct custom_token_printer : public token_printer
   {
-  public:
-    stashed_item (const char **buffer_ptr_, optinfo_item *item_)
-      : buffer_ptr (buffer_ptr_), item (item_) {}
-    const char **buffer_ptr;
-    optinfo_item *item;
+    custom_token_printer (dump_pretty_printer &dump_pp)
+    : m_dump_pp (dump_pp),
+      m_optinfo (nullptr)
+    {}
+    void print_tokens (pretty_printer *pp,
+		       const pp_token_list &tokens) final override;
+    void emit_any_pending_textual_chunks ();
+
+    dump_pretty_printer &m_dump_pp;
+    optinfo *m_optinfo;
   };
 
   static bool format_decoder_cb (pretty_printer *pp, text_info *text,
 				 const char *spec, int /*precision*/,
 				 bool /*wide*/, bool /*set_locus*/,
 				 bool /*verbose*/, bool */*quoted*/,
-				 const char **buffer_ptr);
+				 pp_token_list &formatted_tok_list);
 
   bool decode_format (text_info *text, const char *spec,
-		      const char **buffer_ptr);
+		      pp_token_list &formatted_tok_list);
 
-  void stash_item (const char **buffer_ptr,
+  void stash_item (pp_token_list &formatted_tok_list,
 		   std::unique_ptr<optinfo_item> item);
 
-  void emit_any_pending_textual_chunks (optinfo *dest);
-
   void emit_item (std::unique_ptr<optinfo_item> item, optinfo *dest);
 
   dump_context *m_context;
   dump_flags_t m_dump_kind;
-  auto_vec<stashed_item> m_stashed_items;
+  custom_token_printer m_token_printer;
 };
 
 /* An RAII-style class for use in debug dumpers for temporarily using a