gccrs: macros: Perform macro expansion in a fixed-point fashion.

This commit changes our macro expansion system from an eager and recursive
macro expansion to a fixed-point like system. Instead of, when seeing
a macro invocation, expanding it and all of the macros within it, we
now perform multiple passes of expansion on the entire crate.

This, however, leads to a problem. Rust macros are expanded lazily, but
Rust builtin macros should be expanded eagerly. Due to this, we must
work around the lazy expansion in builtin macros and perform eager
expansion for each pass of the fixed-point, before finally expanding
the builtin when there are no longer any inner macro invocations.

To perform proper macro scoping, the ENR now keeps track of the current
scope (`current_scope` member) and resolves macros accordingly.

This is done through the use of the `scoped` method, which creates a new
scope, runs a specified lambda and then exits the scope. This prevents
pushing/popping errors that we've seen happen already in similar
contexts.

We might think about generalizing it to other classes, providing a
`Scoped<EntryFn, ExitFn>` class or similar

gcc/rust/ChangeLog:

	* ast/rust-macro.cc: New file.
	* Make-lang.in: Add `rust-macro.o` object
	* ast/rust-ast-fragment.cc (Fragment::Fragment): Change API around
	the construction of AST fragments.
	(Fragment::operator=): Correct `Fragment::operator=` to take into
	account the fragment tokens.
	(Fragment::create_error): Use new constructor.
	(Fragment::complete): Remove in favor of new constructor.
	(Fragment::unexpanded): Remove as that Fragment type is no longer used
	or possible.
	(Fragment::get_tokens): Add helper to access a fragment's tokens.
	* ast/rust-ast-fragment.h (enum class): Remove `FragmentKind::Unused`
	* ast/rust-ast.cc (MacroInvocation::as_string): Display
	builtin macro invocations properly.
	* ast/rust-ast.h: Fix `DelimTokenTree` class copy constructors and
	handling of its token vector.
	* ast/rust-macro.h (class MacroMatcher): Format.
	(class MetaItemSeq): Likewise.
	(builtin_macro_from_string): Get a `BuiltinMacroKind` from a given
	string, i.e the name of the macro (`assert!`, `cfg!` and so on).
	* expand/rust-attribute-visitor.cc (AttrVisitor::visit): Do not expand
	macros recursively anymore.
	(AttrVisitor::maybe_expand_expr): Likewise.
	(AttrVisitor::maybe_expand_type): Likewise.
	* expand/rust-attribute-visitor.h: Likewise, and remove
	`expand_macro_fragment_recursively` function.
	* expand/rust-macro-builtins.cc (make_token): Add shorthand for
	returning `std::unique_ptr<AST::Token>`s.
	(make_macro_invocation): Add shorthand for returning fragments
	containing builtin macro invocations.
	(try_expand_macro_expression): Do not expand macros recursively.
	(try_expand_single_string_literal): Likewise.
	(try_expand_many_expr): Likewise.
	(parse_single_string_literal): Error out more appropriately.
	(MacroBuiltin::compile_error_handler): Add explanation for eager
	invocation
	(MacroBuiltin::file_handler): Return the proper tokens associated with
	macro invocation, and builtin macros in the case of necessary eager
	expansion.
	(MacroBuiltin::column_handler): Likewise.
	(MacroBuiltin::include_bytes_handler): Likewise.
	(MacroBuiltin::include_str_handler): Likewise.
	(MacroBuiltin::concat_handler): Likewise.
	(MacroBuiltin::env_handler): Likewise.
	(MacroBuiltin::cfg_handler): Likewise.
	(MacroBuiltin::include_handler): Likewise.
	(MacroBuiltin::line_handler): Likewise.
	* expand/rust-macro-expand.cc (MacroExpander::expand_eager_invocations):
	Add function to expand eager invocations *once* in the fixed point
	pipeline.
	(MacroExpander::expand_invoc): Call into `expand_eager_invocations` for
	builtin macro invocations.
	(MacroExpander::expand_crate): Use new `AttrVisitor` API.
	(parse_many): Return tokens in `AST::Fragment`.
	(transcribe_expression): Likewise.
	(transcribe_type): Likewise.
	* expand/rust-macro-expand.h (struct MacroExpander): Add `has_changed`
	flag for fixed point checking.
	* resolve/rust-early-name-resolver.cc (EarlyNameResolver::EarlyNameResolver):
	Keep track of the current macro scope.
	(EarlyNameResolver::go): Use `scoped` API.
	(EarlyNameResolver::visit): Likewise.
	* resolve/rust-early-name-resolver.h: Add `scoped` API.
	* rust-session-manager.cc (Session::expansion): Perform macro expansion
	in a fixed-point fashion.

gcc/testsuite/ChangeLog:

	* rust/compile/macro17.rs: Fix testsuite for new recursion errors.
	* rust/compile/macro44.rs: Fix invalid testcase assertions.
	* rust/compile/builtin_macro_recurse.rs: Fix invalid test.
	* rust/compile/builtin_macro_recurse2.rs: New test.
	* rust/compile/macro46.rs: New test.

2 files changed, 158 insertions, 34 deletions

diff --git a/gcc/rust/resolve/rust-early-name-resolver.cc b/gcc/rust/resolve/rust-early-name-resolver.cc
index 8100564..60d6c8a 100644
--- a/gcc/rust/resolve/rust-early-name-resolver.cc
+++ b/gcc/rust/resolve/rust-early-name-resolver.cc
@@ -24,20 +24,17 @@ namespace Rust {
 namespace Resolver {
 
 EarlyNameResolver::EarlyNameResolver ()
-  : resolver (*Resolver::get ()), mappings (*Analysis::Mappings::get ())
+  : current_scope (UNKNOWN_NODEID), resolver (*Resolver::get ()),
+    mappings (*Analysis::Mappings::get ())
 {}
 
 void
 EarlyNameResolver::go (AST::Crate &crate)
 {
-  // FIXME: Is that valid? Why is the regular name resolution doing
-  // mappings->get_next_node_id()?
-  resolver.get_macro_scope ().push (crate.get_node_id ());
-
-  for (auto &item : crate.items)
-    item->accept_vis (*this);
-
-  // FIXME: Should we pop the macro scope?
+  scoped (crate.get_node_id (), [&crate, this] () {
+    for (auto &item : crate.items)
+      item->accept_vis (*this);
+  });
 }
 
 void
@@ -335,11 +332,13 @@ EarlyNameResolver::visit (AST::ClosureExprInner &expr)
 void
 EarlyNameResolver::visit (AST::BlockExpr &expr)
 {
-  for (auto &stmt : expr.get_statements ())
-    stmt->accept_vis (*this);
+  scoped (expr.get_node_id (), [&expr, this] () {
+    for (auto &stmt : expr.get_statements ())
+      stmt->accept_vis (*this);
 
-  if (expr.has_tail_expr ())
-    expr.get_tail_expr ()->accept_vis (*this);
+    if (expr.has_tail_expr ())
+      expr.get_tail_expr ()->accept_vis (*this);
+  });
 }
 
 void
@@ -434,8 +433,11 @@ EarlyNameResolver::visit (AST::WhileLetLoopExpr &expr)
 void
 EarlyNameResolver::visit (AST::ForLoopExpr &expr)
 {
-  expr.get_iterator_expr ()->accept_vis (*this);
-  expr.get_loop_block ()->accept_vis (*this);
+  scoped (expr.get_node_id (), [&expr, this] () {
+    expr.get_pattern ()->accept_vis (*this);
+    expr.get_iterator_expr ()->accept_vis (*this);
+    expr.get_loop_block ()->accept_vis (*this);
+  });
 }
 
 void
@@ -473,7 +475,9 @@ void
 EarlyNameResolver::visit (AST::IfLetExpr &expr)
 {
   expr.get_value_expr ()->accept_vis (*this);
-  expr.get_if_block ()->accept_vis (*this);
+
+  scoped (expr.get_node_id (),
+	  [&expr, this] () { expr.get_if_block ()->accept_vis (*this); });
 }
 
 void
@@ -504,16 +508,21 @@ void
 EarlyNameResolver::visit (AST::MatchExpr &expr)
 {
   expr.get_scrutinee_expr ()->accept_vis (*this);
-  for (auto &match_arm : expr.get_match_cases ())
-    {
-      if (match_arm.get_arm ().has_match_arm_guard ())
-	match_arm.get_arm ().get_guard_expr ()->accept_vis (*this);
 
-      for (auto &pattern : match_arm.get_arm ().get_patterns ())
-	pattern->accept_vis (*this);
+  scoped (expr.get_node_id (), [&expr, this] () {
+    for (auto &arm : expr.get_match_cases ())
+      {
+	scoped (arm.get_node_id (), [&arm, this] () {
+	  if (arm.get_arm ().has_match_arm_guard ())
+	    arm.get_arm ().get_guard_expr ()->accept_vis (*this);
 
-      match_arm.get_expr ()->accept_vis (*this);
-    }
+	  for (auto &pattern : arm.get_arm ().get_patterns ())
+	    pattern->accept_vis (*this);
+
+	  arm.get_expr ()->accept_vis (*this);
+	});
+      }
+  });
 }
 
 void
@@ -571,8 +580,10 @@ EarlyNameResolver::visit (AST::Method &method)
 void
 EarlyNameResolver::visit (AST::Module &module)
 {
-  for (auto &item : module.get_items ())
-    item->accept_vis (*this);
+  scoped (module.get_node_id (), [&module, this] () {
+    for (auto &item : module.get_items ())
+      item->accept_vis (*this);
+  });
 }
 
 void
@@ -722,8 +733,13 @@ EarlyNameResolver::visit (AST::TraitItemType &)
 void
 EarlyNameResolver::visit (AST::Trait &trait)
 {
-  for (auto &item : trait.get_trait_items ())
-    item->accept_vis (*this);
+  for (auto &generic : trait.get_generic_params ())
+    generic->accept_vis (*this);
+
+  scoped (trait.get_node_id (), [&trait, this] () {
+    for (auto &item : trait.get_trait_items ())
+      item->accept_vis (*this);
+  });
 }
 
 void
@@ -734,8 +750,10 @@ EarlyNameResolver::visit (AST::InherentImpl &impl)
   for (auto &generic : impl.get_generic_params ())
     generic->accept_vis (*this);
 
-  for (auto &item : impl.get_impl_items ())
-    item->accept_vis (*this);
+  scoped (impl.get_node_id (), [&impl, this] () {
+    for (auto &item : impl.get_impl_items ())
+      item->accept_vis (*this);
+  });
 }
 
 void
@@ -746,8 +764,10 @@ EarlyNameResolver::visit (AST::TraitImpl &impl)
   for (auto &generic : impl.get_generic_params ())
     generic->accept_vis (*this);
 
-  for (auto &item : impl.get_impl_items ())
-    item->accept_vis (*this);
+  scoped (impl.get_node_id (), [&impl, this] () {
+    for (auto &item : impl.get_impl_items ())
+      item->accept_vis (*this);
+  });
 }
 
 void
@@ -772,8 +792,10 @@ EarlyNameResolver::visit (AST::ExternalFunctionItem &item)
 void
 EarlyNameResolver::visit (AST::ExternBlock &block)
 {
-  for (auto &item : block.get_extern_items ())
-    item->accept_vis (*this);
+  scoped (block.get_node_id (), [&block, this] () {
+    for (auto &item : block.get_extern_items ())
+      item->accept_vis (*this);
+  });
 }
 
 void
@@ -795,6 +817,15 @@ EarlyNameResolver::visit (AST::MacroRulesDefinition &rules_def)
 				      rules_def.get_rule_name ());
   resolver.get_macro_scope ().insert (path, rules_def.get_node_id (),
 				      rules_def.get_locus ());
+
+  /* Since the EarlyNameResolver runs multiple time (fixed point algorithm)
+   * we could be inserting the same macro def over and over again until we
+   * implement some optimizations */
+  // FIXME: ARTHUR: Remove that lookup and add proper optimizations instead
+  AST::MacroRulesDefinition *tmp = nullptr;
+  if (mappings.lookup_macro_def (rules_def.get_node_id (), &tmp))
+    return;
+
   mappings.insert_macro_def (&rules_def);
   rust_debug_loc (rules_def.get_locus (), "inserting macro def: [%s]",
 		  path.get ().c_str ());
@@ -806,6 +837,10 @@ EarlyNameResolver::visit (AST::MacroInvocation &invoc)
   auto &invoc_data = invoc.get_invoc_data ();
   auto has_semicolon = invoc.has_semicolon ();
 
+  if (invoc.get_kind () == AST::MacroInvocation::InvocKind::Builtin)
+    for (auto &pending_invoc : invoc.get_pending_eager_invocations ())
+      pending_invoc->accept_vis (*this);
+
   // ??
   // switch on type of macro:
   //  - '!' syntax macro (inner switch)
@@ -847,6 +882,30 @@ EarlyNameResolver::visit (AST::MacroInvocation &invoc)
   bool ok = mappings.lookup_macro_def (resolved_node, &rules_def);
   rust_assert (ok);
 
+  auto &outer_attrs = rules_def->get_outer_attrs ();
+  bool is_builtin
+    = std::any_of (outer_attrs.begin (), outer_attrs.end (),
+		   [] (AST::Attribute attr) {
+		     return attr.get_path () == "rustc_builtin_macro";
+		   });
+
+  if (is_builtin)
+    {
+      auto builtin_kind
+	= AST::builtin_macro_from_string (rules_def->get_rule_name ());
+      invoc.map_to_builtin (builtin_kind);
+    }
+
+  auto attributes = rules_def->get_outer_attrs ();
+
+  /* Since the EarlyNameResolver runs multiple time (fixed point algorithm)
+   * we could be inserting the same macro def over and over again until we
+   * implement some optimizations */
+  // FIXME: ARTHUR: Remove that lookup and add proper optimizations instead
+  AST::MacroRulesDefinition *tmp_def = nullptr;
+  if (mappings.lookup_macro_invocation (invoc, &tmp_def))
+    return;
+
   mappings.insert_macro_invocation (invoc, rules_def);
 }
 
diff --git a/gcc/rust/resolve/rust-early-name-resolver.h b/gcc/rust/resolve/rust-early-name-resolver.h
index c53ab9f..c74c452 100644
--- a/gcc/rust/resolve/rust-early-name-resolver.h
+++ b/gcc/rust/resolve/rust-early-name-resolver.h
@@ -35,6 +35,71 @@ public:
   void go (AST::Crate &crate);
 
 private:
+  /**
+   * Execute a lambda within a scope. This is equivalent to calling
+   * `enter_scope` before your code and `exit_scope` after. This ensures
+   * no errors can be committed
+   */
+  void scoped (NodeId scope_id, std::function<void ()> fn)
+  {
+    auto old_scope = current_scope;
+    current_scope = scope_id;
+    resolver.get_macro_scope ().push (scope_id);
+    resolver.push_new_macro_rib (resolver.get_macro_scope ().peek ());
+
+    fn ();
+
+    resolver.get_macro_scope ().pop ();
+    current_scope = old_scope;
+  }
+
+  /**
+   * The "scope" we are currently in.
+   *
+   * This involves lexical scopes:
+   *
+   * ```rust
+   * // current_scope = crate_id;
+   * macro_rules! foo { () => {} )
+   *
+   * {
+   *     // current_scope = current_block_id;
+   *     macro_rules! foo { () => { something!(); } }
+   * }
+   * // current_scope = crate_id;
+   * ```
+   *
+   * as well as any sort of scope-like structure that might impact import name
+   * resolution or macro name resolution:
+   *
+   * ```rust
+   * macro_rules! foo {
+   *     () => { fn empty() {} }
+   * }
+   *
+   *
+   * trait Foo {
+   *     fn foo() {
+   *         fn inner_foo() {
+   *             macro_rules! foo { () => {} )
+   *
+   *             foo!();
+   *         }
+   *
+   *         foo!();
+   *     }
+   *
+   *     foo!();
+   * }
+   *
+   * foo!();
+   * ```
+   */
+  NodeId current_scope;
+
+  /* The crate's scope */
+  NodeId crate_scope;
+
   Resolver &resolver;
   Analysis::Mappings &mappings;