libcpp: Support extended characters for #pragma {push,pop}_macro [PR109704]

The implementation of #pragma push_macro and #pragma pop_macro has to date
made use of an ad-hoc function, _cpp_lex_identifier(), which lexes an
identifier out of a string. When support was added for extended characters
in identifiers ($, UCNs, or UTF-8), that support was added only for the
"normal" way of lexing identifiers out of a cpp_buffer (_cpp_lex_direct) and
not for the ad-hoc way. Consequently, extended identifiers are not usable
with these pragmas.

The logic for lexing identifiers has become more complicated than it was
when _cpp_lex_identifier() was written -- it now handles things like \N{}
escapes in C++, for instance -- and it no longer seems practical to maintain
a redundant code path for lexing identifiers. Address the issue by changing
the implementation of #pragma {push,pop}_macro to lex identifiers in the
expected way, i.e. by pushing a cpp_buffer and lexing the identifier from
there.

The existing implementation has some quirks because of the ad-hoc parsing
logic. For example:

 #pragma push_macro("X ")
 ...
 #pragma pop_macro("X")

will not restore macro X (note the extra space in the first string). However:

 #pragma push_macro("X ")
 ...
 #pragma pop_macro("X ")

actually does sucessfully restore "X". This is because the key for looking
up the saved macro on the push stack is the original string passed, so the
string passed to pop_macro needs to match it exactly. It is not that easy to
reproduce this logic in the world of extended characters, given that for
example it should be valid to pass a UCN to push_macro, and the
corresponding UTF-8 to pop_macro. Given that this aspect of the existing
behavior seems unintentional and has no tests (and does not match other
implementations), I opted to make the new logic more straightforward. The
string passed needs to lex to one token, which must be a valid identifier,
or else no action is taken and no error is generated. Any diagnostics
encountered during lexing (e.g., due to a UTF-8 character not permitted to
appear in an identifier) are also suppressed.

It could be nice (for GCC 15) to also add a warning if a pop_macro does not
match a previous push_macro.

libcpp/ChangeLog:

	PR preprocessor/109704
	* include/cpplib.h (class cpp_auto_suppress_diagnostics): New class.
	* errors.cc
	(cpp_auto_suppress_diagnostics::cpp_auto_suppress_diagnostics): New
	function.
	(cpp_auto_suppress_diagnostics::~cpp_auto_suppress_diagnostics): New
	function.
	* charset.cc (noop_diagnostic_cb): Remove.
	(cpp_interpret_string_ranges): Refactor diagnostic suppression logic
	into new class cpp_auto_suppress_diagnostics.
	(count_source_chars): Likewise.
	* directives.cc (cpp_pop_definition): Add cpp_hashnode argument.
	(lex_identifier_from_string): New static helper function.
	(push_pop_macro_common): Refactor common logic from
	do_pragma_push_macro and do_pragma_pop_macro; use
	lex_identifier_from_string instead of _cpp_lex_identifier.
	(do_pragma_push_macro): Reimplement using push_pop_macro_common.
	(do_pragma_pop_macro): Likewise.
	* internal.h (_cpp_lex_identifier): Remove.
	* lex.cc (lex_identifier_intern): Remove.
	(_cpp_lex_identifier): Remove.

gcc/testsuite/ChangeLog:

	PR preprocessor/109704
	* c-c++-common/cpp/pragma-push-pop-utf8.c: New test.
	* g++.dg/pch/pushpop-2.C: New test.
	* g++.dg/pch/pushpop-2.Hs: New test.
	* gcc.dg/pch/pushpop-2.c: New test.
	* gcc.dg/pch/pushpop-2.hs: New test.

5 files changed, 257 insertions, 0 deletions

diff --git a/gcc/testsuite/c-c++-common/cpp/pragma-push-pop-utf8.c b/gcc/testsuite/c-c++-common/cpp/pragma-push-pop-utf8.c
new file mode 100644
index 0000000..8634481
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/cpp/pragma-push-pop-utf8.c
@@ -0,0 +1,203 @@
+/* { dg-do preprocess } */
+/* { dg-options "-std=c11 -pedantic" { target c } } */
+/* { dg-options "-std=c++11 -pedantic" { target c++ } } */
+/* { dg-additional-options "-Wall" } */
+
+/* PR preprocessor/109704 */
+
+/* Verify basic operations for different extended identifiers...  */
+
+/* ...dollar sign.  */
+#define $x 1
+#pragma push_macro("$x")
+#undef $x
+#define $x 0
+#pragma pop_macro("$x")
+#if !$x
+#error $x
+#endif
+#define $x 1
+_Pragma("push_macro(\"$x\")")
+#undef $x
+#define $x 0
+_Pragma("pop_macro(\"$x\")")
+#if !$x
+#error $x
+#endif
+#define x$ 1
+#pragma push_macro("x$")
+#undef x$
+#define x$ 0
+#pragma pop_macro("x$")
+#if !x$
+#error x$
+#endif
+#define x$ 1
+_Pragma("push_macro(\"x$\")")
+#undef x$
+#define x$ 0
+_Pragma("pop_macro(\"x$\")")
+#if !x$
+#error x$
+#endif
+
+/* ...UCN.  */
+#define \u03B1x 1
+#pragma push_macro("\u03B1x")
+#undef \u03B1x
+#define \u03B1x 0
+#pragma pop_macro("\u03B1x")
+#if !\u03B1x
+#error \u03B1x
+#endif
+#define \u03B1x 1
+_Pragma("push_macro(\"\\u03B1x\")")
+#undef \u03B1x
+#define \u03B1x 0
+_Pragma("pop_macro(\"\\u03B1x\")")
+#if !\u03B1x
+#error \u03B1x
+#endif
+#define x\u03B1 1
+#pragma push_macro("x\u03B1")
+#undef x\u03B1
+#define x\u03B1 0
+#pragma pop_macro("x\u03B1")
+#if !x\u03B1
+#error x\u03B1
+#endif
+#define x\u03B1 1
+_Pragma("push_macro(\"x\\u03B1\")")
+#undef x\u03B1
+#define x\u03B1 0
+_Pragma("pop_macro(\"x\\u03B1\")")
+#if !x\u03B1
+#error x\u03B1
+#endif
+
+/* ...UTF-8.  */
+#define πx 1
+#pragma push_macro("πx")
+#undef πx
+#define πx 0
+#pragma pop_macro("πx")
+#if !πx
+#error πx
+#endif
+#define πx 1
+_Pragma("push_macro(\"πx\")")
+#undef πx
+#define πx 0
+_Pragma("pop_macro(\"πx\")")
+#if !πx
+#error πx
+#endif
+#define xπ 1
+#pragma push_macro("xπ")
+#undef xπ
+#define xπ 0
+#pragma pop_macro("xπ")
+#if !xπ
+#error xπ
+#endif
+#define xπ 1
+_Pragma("push_macro(\"xπ\")")
+#undef xπ
+#define xπ 0
+_Pragma("pop_macro(\"xπ\")")
+#if !xπ
+#error xπ
+#endif
+
+/* Verify UCN and UTF-8 can be intermixed.  */
+#define ħ_0 1
+#pragma push_macro("ħ_0")
+#undef ħ_0
+#define ħ_0 0
+#if ħ_0
+#error ħ_0 ħ_0 \U00000127_0
+#endif
+#pragma pop_macro("\U00000127_0")
+#if !ħ_0
+#error ħ_0 ħ_0 \U00000127_0
+#endif
+#define ħ_1 1
+#pragma push_macro("\U00000127_1")
+#undef ħ_1
+#define ħ_1 0
+#if ħ_1
+#error ħ_1 \U00000127_1 ħ_1
+#endif
+#pragma pop_macro("ħ_1")
+#if !ħ_1
+#error ħ_1 \U00000127_1 ħ_1
+#endif
+#define ħ_2 1
+#pragma push_macro("\U00000127_2")
+#undef ħ_2
+#define ħ_2 0
+#if ħ_2
+#error ħ_2 \U00000127_2 \U00000127_2
+#endif
+#pragma pop_macro("\U00000127_2")
+#if !ħ_2
+#error ħ_2 \U00000127_2 \U00000127_2
+#endif
+#define \U00000127_3 1
+#pragma push_macro("ħ_3")
+#undef \U00000127_3
+#define \U00000127_3 0
+#if \U00000127_3
+#error \U00000127_3 ħ_3 ħ_3
+#endif
+#pragma pop_macro("ħ_3")
+#if !\U00000127_3
+#error \U00000127_3 ħ_3 ħ_3
+#endif
+#define \U00000127_4 1
+#pragma push_macro("ħ_4")
+#undef \U00000127_4
+#define \U00000127_4 0
+#if \U00000127_4
+#error \U00000127_4 ħ_4 \U00000127_4
+#endif
+#pragma pop_macro("\U00000127_4")
+#if !\U00000127_4
+#error \U00000127_4 ħ_4 \U00000127_4
+#endif
+#define \U00000127_5 1
+#pragma push_macro("\U00000127_5")
+#undef \U00000127_5
+#define \U00000127_5 0
+#if \U00000127_5
+#error \U00000127_5 \U00000127_5 ħ_5
+#endif
+#pragma pop_macro("ħ_5")
+#if !\U00000127_5
+#error \U00000127_5 \U00000127_5 ħ_5
+#endif
+
+/* Verify invalid input produces no diagnostics.  */
+#pragma push_macro("") /* { dg-bogus "." } */
+#pragma push_macro("\u") /* { dg-bogus "." } */
+#pragma push_macro("\u0000") /* { dg-bogus "." } */
+#pragma push_macro("not a single identifier") /* { dg-bogus "." } */
+#pragma push_macro("invalid╬character") /* { dg-bogus "." } */
+#pragma push_macro("\u0300invalid_start") /* { dg-bogus "." } */
+#pragma push_macro("#include <cstdlib>") /* { dg-bogus "." } */
+
+/* Verify end-of-line diagnostics for valid and invalid input.  */
+#pragma push_macro("ö") oops /* { dg-warning "extra tokens" } */
+#pragma push_macro("") oops /* { dg-warning "extra tokens" } */
+#pragma push_macro("\u") oops /* { dg-warning "extra tokens" } */
+#pragma push_macro("\u0000") oops /* { dg-warning "extra tokens" } */
+#pragma push_macro("not a single identifier") oops /* { dg-warning "extra tokens" } */
+#pragma push_macro("invalid╬character") oops /* { dg-warning "extra tokens" } */
+#pragma push_macro("\u0300invalid_start") oops /* { dg-warning "extra tokens" } */
+#pragma push_macro("#include <cstdlib>") oops /* { dg-warning "extra tokens" } */
+
+/* Verify expected diagnostics.  */
+#pragma push_macro() /* { dg-error {invalid '#pragma push_macro'} } */
+#pragma pop_macro() /* { dg-error {invalid '#pragma pop_macro'} } */
+_Pragma("push_macro(0)") /* { dg-error {invalid '#pragma push_macro'} } */
+_Pragma("pop_macro(\"oops\"") /* { dg-error {invalid '#pragma pop_macro'} } */
diff --git a/gcc/testsuite/g++.dg/pch/pushpop-2.C b/gcc/testsuite/g++.dg/pch/pushpop-2.C
new file mode 100644
index 0000000..84886ae
--- /dev/null
+++ b/gcc/testsuite/g++.dg/pch/pushpop-2.C
@@ -0,0 +1,18 @@
+/* { dg-options -std=c++11 } */
+#include "pushpop-2.Hs"
+
+#if π != 4
+#error π != 4
+#endif
+#pragma pop_macro("\u03C0")
+#if π != 3
+#error π != 3
+#endif
+
+#if \u03B1 != 6
+#error α != 6
+#endif
+_Pragma("pop_macro(\"\\u03B1\")")
+#if α != 5
+#error α != 5
+#endif
diff --git a/gcc/testsuite/g++.dg/pch/pushpop-2.Hs b/gcc/testsuite/g++.dg/pch/pushpop-2.Hs
new file mode 100644
index 0000000..797139a
--- /dev/null
+++ b/gcc/testsuite/g++.dg/pch/pushpop-2.Hs
@@ -0,0 +1,9 @@
+#define π 3
+#pragma push_macro ("π")
+#undef π
+#define π 4
+
+#define \u03B1 5
+#pragma push_macro ("α")
+#undef α
+#define α 6
diff --git a/gcc/testsuite/gcc.dg/pch/pushpop-2.c b/gcc/testsuite/gcc.dg/pch/pushpop-2.c
new file mode 100644
index 0000000..61b8430
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/pch/pushpop-2.c
@@ -0,0 +1,18 @@
+/* { dg-options -std=c11 } */
+#include "pushpop-2.hs"
+
+#if π != 4
+#error π != 4
+#endif
+#pragma pop_macro("\u03C0")
+#if π != 3
+#error π != 3
+#endif
+
+#if \u03B1 != 6
+#error α != 6
+#endif
+_Pragma("pop_macro(\"\\u03B1\")")
+#if α != 5
+#error α != 5
+#endif
diff --git a/gcc/testsuite/gcc.dg/pch/pushpop-2.hs b/gcc/testsuite/gcc.dg/pch/pushpop-2.hs
new file mode 100644
index 0000000..797139a
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/pch/pushpop-2.hs
@@ -0,0 +1,9 @@
+#define π 3
+#pragma push_macro ("π")
+#undef π
+#define π 4
+
+#define \u03B1 5
+#pragma push_macro ("α")
+#undef α
+#define α 6