c-family: Implement pragma_lex () for preprocess-only mode

In order to support processing #pragma in preprocess-only mode (-E or
-save-temps for gcc/g++), we need a way to obtain the #pragma tokens from
libcpp. In full compilation modes, this is accomplished by calling
pragma_lex (), which is a symbol that must be exported by the frontend, and
which is currently implemented for C and C++. Neither of those frontends
initializes its parser machinery in preprocess-only mode, and consequently
pragma_lex () does not work in this case.

Address that by adding a new function c_init_preprocess () for the frontends
to implement, which arranges for pragma_lex () to work in preprocess-only
mode, and adjusting pragma_lex () accordingly.

In preprocess-only mode, the preprocessor is accustomed to controlling the
interaction with libcpp, and it only knows about tokens that it has called
into libcpp itself to obtain. Since it still needs to see the tokens
obtained by pragma_lex () so that they can be streamed to the output, also
adjust c_lex_with_flags () and related functions in c-family/c-lex.cc to
inform the preprocessor about any tokens it won't be aware of.

Currently, there is one place where we are already supporting #pragma in
preprocess-only mode, namely the handling of `#pragma GCC diagnostic'.  That
was done by directly interfacing with libcpp, rather than making use of
pragma_lex (). Now that pragma_lex () works, that code is no longer
necessary; remove it.

gcc/c-family/ChangeLog:

	* c-common.h (c_init_preprocess): Declare new function.
	* c-opts.cc (c_common_init): Call it.
	* c-lex.cc (cb_def_pragma): Add a comment.
	(get_token): New function wrapping cpp_get_token.
	(c_lex_with_flags): Use the new wrapper function to support
	obtaining tokens in preprocess_only mode.
	(lex_string): Likewise.
	* c-pragma.cc (pragma_diagnostic_lex_normal): Rename to...
	(pragma_diagnostic_lex): ...this.
	(pragma_diagnostic_lex_pp): Remove.
	(handle_pragma_diagnostic_impl): Call pragma_diagnostic_lex () in
	all modes.
	(c_pp_invoke_early_pragma_handler): Adapt to support pragma_lex ()
	usage.
	* c-pragma.h (pragma_lex_discard_to_eol): Declare.

gcc/c/ChangeLog:

	* c-parser.cc (pragma_lex_discard_to_eol): New function.
	(c_init_preprocess): New function.

gcc/cp/ChangeLog:

	* parser.cc (c_init_preprocess): New function.
	(maybe_read_tokens_for_pragma_lex): New function.
	(pragma_lex): Support preprocess-only mode.
	(pragma_lex_discard_to_eol): New function.

1 files changed, 21 insertions, 0 deletions

diff --git a/gcc/c/c-parser.cc b/gcc/c/c-parser.cc
index 80920b3..cf82b03 100644
--- a/gcc/c/c-parser.cc
+++ b/gcc/c/c-parser.cc
@@ -13376,6 +13376,18 @@ pragma_lex (tree *value, location_t *loc)
   return ret;
 }
 
+void
+pragma_lex_discard_to_eol ()
+{
+  cpp_ttype type;
+  do
+    {
+      type = c_parser_peek_token (the_parser)->type;
+      gcc_assert (type != CPP_EOF);
+      c_parser_consume_token (the_parser);
+    } while (type != CPP_PRAGMA_EOL);
+}
+
 static void
 c_parser_pragma_pch_preprocess (c_parser *parser)
 {
@@ -24761,6 +24773,15 @@ c_parse_file (void)
   the_parser = NULL;
 }
 
+void
+c_init_preprocess (void)
+{
+  /* Create a parser for use by pragma_lex during preprocessing.  */
+  the_parser = ggc_alloc<c_parser> ();
+  memset (the_parser, 0, sizeof (c_parser));
+  the_parser->tokens = &the_parser->tokens_buf[0];
+}
+
 /* Parse the body of a function declaration marked with "__RTL".
 
    The RTL parser works on the level of characters read from a