c-family: Implement pragma_lex () for preprocess-only mode

In order to support processing #pragma in preprocess-only mode (-E or
-save-temps for gcc/g++), we need a way to obtain the #pragma tokens from
libcpp. In full compilation modes, this is accomplished by calling
pragma_lex (), which is a symbol that must be exported by the frontend, and
which is currently implemented for C and C++. Neither of those frontends
initializes its parser machinery in preprocess-only mode, and consequently
pragma_lex () does not work in this case.

Address that by adding a new function c_init_preprocess () for the frontends
to implement, which arranges for pragma_lex () to work in preprocess-only
mode, and adjusting pragma_lex () accordingly.

In preprocess-only mode, the preprocessor is accustomed to controlling the
interaction with libcpp, and it only knows about tokens that it has called
into libcpp itself to obtain. Since it still needs to see the tokens
obtained by pragma_lex () so that they can be streamed to the output, also
adjust c_lex_with_flags () and related functions in c-family/c-lex.cc to
inform the preprocessor about any tokens it won't be aware of.

Currently, there is one place where we are already supporting #pragma in
preprocess-only mode, namely the handling of `#pragma GCC diagnostic'.  That
was done by directly interfacing with libcpp, rather than making use of
pragma_lex (). Now that pragma_lex () works, that code is no longer
necessary; remove it.

gcc/c-family/ChangeLog:

	* c-common.h (c_init_preprocess): Declare new function.
	* c-opts.cc (c_common_init): Call it.
	* c-lex.cc (cb_def_pragma): Add a comment.
	(get_token): New function wrapping cpp_get_token.
	(c_lex_with_flags): Use the new wrapper function to support
	obtaining tokens in preprocess_only mode.
	(lex_string): Likewise.
	* c-pragma.cc (pragma_diagnostic_lex_normal): Rename to...
	(pragma_diagnostic_lex): ...this.
	(pragma_diagnostic_lex_pp): Remove.
	(handle_pragma_diagnostic_impl): Call pragma_diagnostic_lex () in
	all modes.
	(c_pp_invoke_early_pragma_handler): Adapt to support pragma_lex ()
	usage.
	* c-pragma.h (pragma_lex_discard_to_eol): Declare.

gcc/c/ChangeLog:

	* c-parser.cc (pragma_lex_discard_to_eol): New function.
	(c_init_preprocess): New function.

gcc/cp/ChangeLog:

	* parser.cc (c_init_preprocess): New function.
	(maybe_read_tokens_for_pragma_lex): New function.
	(pragma_lex): Support preprocess-only mode.
	(pragma_lex_discard_to_eol): New function.

1 files changed, 45 insertions, 0 deletions

diff --git a/gcc/cp/parser.cc b/gcc/cp/parser.cc
index b1d2e14..2e24586 100644
--- a/gcc/cp/parser.cc
+++ b/gcc/cp/parser.cc
@@ -765,6 +765,15 @@ cp_lexer_new_main (void)
   return lexer;
 }
 
+/* Create a lexer and parser to be used during preprocess-only mode.
+   This will be filled with tokens to parse when needed by pragma_lex ().  */
+void
+c_init_preprocess ()
+{
+  gcc_assert (!the_parser);
+  the_parser = cp_parser_new (cp_lexer_alloc ());
+}
+
 /* Create a new lexer whose token stream is primed with the tokens in
    CACHE.  When these tokens are exhausted, no new tokens will be read.  */
 
@@ -49694,11 +49703,37 @@ cp_parser_pragma (cp_parser *parser, enum pragma_context context, bool *if_p)
   return ret;
 }
 
+/* Helper for pragma_lex in preprocess-only mode; in this mode, we have not
+   populated the lexer with any tokens (the tokens rather being read by
+   c-ppoutput.c's machinery), so we need to read enough tokens now to handle
+   a pragma.  */
+static void
+maybe_read_tokens_for_pragma_lex ()
+{
+  const auto lexer = the_parser->lexer;
+  if (!lexer->buffer->is_empty ())
+    return;
+
+  /* Read the rest of the tokens comprising the pragma line.  */
+  cp_token *tok;
+  do
+    {
+      tok = vec_safe_push (lexer->buffer, cp_token ());
+      cp_lexer_get_preprocessor_token (C_LEX_STRING_NO_JOIN, tok);
+      gcc_assert (tok->type != CPP_EOF);
+    } while (tok->type != CPP_PRAGMA_EOL);
+  lexer->next_token = lexer->buffer->address ();
+  lexer->last_token = lexer->next_token + lexer->buffer->length () - 1;
+}
+
 /* The interface the pragma parsers have to the lexer.  */
 
 enum cpp_ttype
 pragma_lex (tree *value, location_t *loc)
 {
+  if (flag_preprocess_only)
+    maybe_read_tokens_for_pragma_lex ();
+
   cp_token *tok = cp_lexer_peek_token (the_parser->lexer);
   enum cpp_ttype ret = tok->type;
 
@@ -49721,6 +49756,16 @@ pragma_lex (tree *value, location_t *loc)
   return ret;
 }
 
+void
+pragma_lex_discard_to_eol ()
+{
+  /* We have already read all the tokens, so we just need to discard
+     them here.  */
+  const auto lexer = the_parser->lexer;
+  lexer->next_token = lexer->last_token;
+  lexer->buffer->truncate (0);
+}
+
 
 /* External interface.  */