c++: implement P2564, consteval needs to propagate up [PR107687]

This patch implements P2564, described at <wg21.link/p2564>, whereby
certain functions are promoted to consteval.  For example:

  consteval int id(int i) { return i; }

  template <typename T>
  constexpr int f(T t)
  {
    return t + id(t); // id causes f<int> to be promoted to consteval
  }

  void g(int i)
  {
    f (3);
  }

now compiles.  Previously the code was ill-formed: we would complain
that 't' in 'f' is not a constant expression.  Since 'f' is now
consteval, it means that the call to id(t) is in an immediate context,
so doesn't have to produce a constant -- this is how we allow consteval
functions composition.  But making 'f<int>' consteval also means that
the call to 'f' in 'g' must yield a constant; failure to do so results
in an error.  I made the effort to have cc1plus explain to us what's
going on.  For example, calling f(i) produces this neat diagnostic:

w.C:11:11: error: call to consteval function 'f<int>(i)' is not a constant expression
   11 |         f (i);
      |         ~~^~~
w.C:11:11: error: 'i' is not a constant expression
w.C:6:22: note: 'constexpr int f(T) [with T = int]' was promoted to an immediate function because its body contains an immediate-escalating expression 'id(t)'
    6 |         return t + id(t); // id causes f<int> to be promoted to consteval
      |                    ~~^~~

which hopefully makes it clear what's going on.

Implementing this proposal has been tricky.  One problem was delayed
instantiation: instantiating a function can set off a domino effect
where one call promotes a function to consteval but that then means
that another function should also be promoted, etc.

In v1, I addressed the delayed instantiation problem by instantiating
trees early, so that we can escalate functions right away.  That caused
a number of problems, and in certain cases, like consteval-prop3.C, it
can't work, because we need to wait till EOF to see the definition of
the function anyway.  Overeager instantiation tends to cause diagnostic
problems too.

In v2, I attempted to move the escalation to the gimplifier, at which
point all templates have been instantiated.  That attempt flopped,
however, because once we've gimplified a function, its body is discarded
and as a consequence, you can no longer evaluate a call to that function
which is required for escalating, which needs to decide if a call is
a constant expression or not.

Therefore, we have to perform the escalation before gimplifying, but
after instantiate_pending_templates.  That's not easy because we have
no way to walk all the trees.  In the v2 patch, I use two vectors: one
to store function decls that may become consteval, and another to
remember references to immediate-escalating functions.  Unfortunately
the latter must also stash functions that call immediate-escalating
functions.  Consider:

  int g(int i)
  {
    f<int>(i); // f is immediate-escalating
  }

where g itself is not immediate-escalating, but we have to make sure
that if f gets promoted to consteval, we give an error.

A new option, -fno-immediate-escalation, is provided to suppress
escalating functions.

v2 also adds a new flag, DECL_ESCALATION_CHECKED_P, so that we don't
escalate a function multiple times, and so that we can distinguish between
explicitly consteval functions and functions that have been promoted
to consteval.

In v3, I removed one of the new vectors and changed the other one
to a hash set.  This version also contains numerous cleanups.

v4 merges find_escalating_expr_r into cp_fold_immediate_r.  It also
adds a new optimization in cp_fold_function.

v5 greatly simplifies the code.

v6 simplifies the code further and removes an ff_ flag.

v7 removes maybe_promote_function_to_consteval and further simplifies
cp_fold_immediate_r logic.

v8 removes maybe_store_immediate_escalating_fn.

	PR c++/107687
	PR c++/110997

gcc/c-family/ChangeLog:

	* c-cppbuiltin.cc (c_cpp_builtins): Update __cpp_consteval.
	* c-opts.cc (c_common_post_options): Pre-C++20, unset
	flag_immediate_escalation.
	* c.opt (fimmediate-escalation): New option.

gcc/cp/ChangeLog:

	* call.cc (in_immediate_context): No longer static.
	* constexpr.cc (cxx_eval_call_expression): Adjust assert.
	* cp-gimplify.cc (deferred_escalating_exprs): New vec.
	(remember_escalating_expr): New.
	(enum fold_flags): Remove ff_fold_immediate.
	(immediate_escalating_function_p): New.
	(unchecked_immediate_escalating_function_p): New.
	(promote_function_to_consteval): New.
	(cp_fold_immediate): Move above.  Return non-null if any errors were
	emitted.
	(maybe_explain_promoted_consteval): New.
	(cp_gimplify_expr) <case CALL_EXPR>: Assert we've handled all
	immediate invocations.
	(taking_address_of_imm_fn_error): New.
	(cp_fold_immediate_r): Merge ADDR_EXPR and PTRMEM_CST cases.  Implement
	P2564 - promoting functions to consteval.
	<case CALL_EXPR>: Implement P2564 - promoting functions to consteval.
	(cp_fold_r): If an expression turns into a CALL_EXPR after cp_fold,
	call cp_fold_immediate_r on the CALL_EXPR.
	(cp_fold_function): Set DECL_ESCALATION_CHECKED_P if
	deferred_escalating_exprs does not contain current_function_decl.
	(process_and_check_pending_immediate_escalating_fns): New.
	* cp-tree.h (struct lang_decl_fn): Add escalated_p bit-field.
	(DECL_ESCALATION_CHECKED_P): New.
	(immediate_invocation_p): Declare.
	(process_pending_immediate_escalating_fns): Likewise.
	* decl2.cc (c_parse_final_cleanups): Set at_eof to 2 after all
	templates have been instantiated; and to 3 at the end of the function.
	Call process_pending_immediate_escalating_fns.
	* error.cc (dump_template_bindings): Check at_eof against an updated
	value.
	* module.cc (trees_out::lang_decl_bools): Stream escalated_p.
	(trees_in::lang_decl_bools): Likewise.
	* pt.cc (push_tinst_level_loc): Set at_eof to 3, not 2.
	* typeck.cc (cp_build_addr_expr_1): Don't check
	DECL_IMMEDIATE_FUNCTION_P.

gcc/ChangeLog:

	* doc/invoke.texi: Document -fno-immediate-escalation.

libstdc++-v3/ChangeLog:

	* testsuite/18_support/comparisons/categories/zero_neg.cc: Add
	dg-prune-output.
	* testsuite/std/format/string_neg.cc: Add dg-error.

gcc/testsuite/ChangeLog:

	* g++.dg/cpp23/consteval-if10.C: Remove dg-error.
	* g++.dg/cpp23/consteval-if2.C: Likewise.
	* g++.dg/cpp23/feat-cxx2b.C: Adjust expected value of __cpp_consteval.
	* g++.dg/cpp26/feat-cxx26.C: Likewise.
	* g++.dg/cpp2a/consteval-memfn1.C: Add dg-error.
	* g++.dg/cpp2a/consteval11.C: Likewise.
	* g++.dg/cpp2a/consteval3.C: Adjust dg-error.
	* g++.dg/cpp2a/consteval34.C: Add dg-error.
	* g++.dg/cpp2a/consteval36.C: Likewise.
	* g++.dg/cpp2a/consteval9.C: Likewise.
	* g++.dg/cpp2a/feat-cxx2a.C: Adjust expected value of __cpp_consteval.
	* g++.dg/cpp2a/spaceship-synth9.C: Adjust dg-error.
	* g++.dg/cpp2a/consteval-prop1.C: New test.
	* g++.dg/cpp2a/consteval-prop10.C: New test.
	* g++.dg/cpp2a/consteval-prop11.C: New test.
	* g++.dg/cpp2a/consteval-prop12.C: New test.
	* g++.dg/cpp2a/consteval-prop13.C: New test.
	* g++.dg/cpp2a/consteval-prop14.C: New test.
	* g++.dg/cpp2a/consteval-prop15.C: New test.
	* g++.dg/cpp2a/consteval-prop16.C: New test.
	* g++.dg/cpp2a/consteval-prop17.C: New test.
	* g++.dg/cpp2a/consteval-prop18.C: New test.
	* g++.dg/cpp2a/consteval-prop19.C: New test.
	* g++.dg/cpp2a/consteval-prop20.C: New test.
	* g++.dg/cpp2a/consteval-prop2.C: New test.
	* g++.dg/cpp2a/consteval-prop3.C: New test.
	* g++.dg/cpp2a/consteval-prop4.C: New test.
	* g++.dg/cpp2a/consteval-prop5.C: New test.
	* g++.dg/cpp2a/consteval-prop6.C: New test.
	* g++.dg/cpp2a/consteval-prop7.C: New test.
	* g++.dg/cpp2a/consteval-prop8.C: New test.
	* g++.dg/cpp2a/consteval-prop9.C: New test.

9 files changed, 321 insertions, 82 deletions

diff --git a/gcc/cp/call.cc b/gcc/cp/call.cc
index ae0decd..c7efc5b 100644
--- a/gcc/cp/call.cc
+++ b/gcc/cp/call.cc
@@ -9742,7 +9742,7 @@ in_immediate_context ()
 /* Return true if a call to FN with number of arguments NARGS
    is an immediate invocation.  */
 
-static bool
+bool
 immediate_invocation_p (tree fn)
 {
   return (TREE_CODE (fn) == FUNCTION_DECL
diff --git a/gcc/cp/constexpr.cc b/gcc/cp/constexpr.cc
index 96c6166..58187a4 100644
--- a/gcc/cp/constexpr.cc
+++ b/gcc/cp/constexpr.cc
@@ -3128,11 +3128,11 @@ cxx_eval_call_expression (const constexpr_ctx *ctx, tree t,
 	/* OK */;
       else if (!DECL_SAVED_TREE (fun))
 	{
-	  /* When at_eof >= 2, cgraph has started throwing away
+	  /* When at_eof >= 3, cgraph has started throwing away
 	     DECL_SAVED_TREE, so fail quietly.  FIXME we get here because of
 	     late code generation for VEC_INIT_EXPR, which needs to be
 	     completely reconsidered.  */
-	  gcc_assert (at_eof >= 2 && ctx->quiet);
+	  gcc_assert (at_eof >= 3 && ctx->quiet);
 	  *non_constant_p = true;
 	}
       else if (tree copy = get_fundef_copy (new_call.fundef))
diff --git a/gcc/cp/cp-gimplify.cc b/gcc/cp/cp-gimplify.cc
index 795c811..5abb91b 100644
--- a/gcc/cp/cp-gimplify.cc
+++ b/gcc/cp/cp-gimplify.cc
@@ -43,6 +43,21 @@ along with GCC; see the file COPYING3.  If not see
 #include "omp-general.h"
 #include "opts.h"
 
+/* Keep track of forward references to immediate-escalating functions in
+   case they become consteval.  This vector contains ADDR_EXPRs and
+   PTRMEM_CSTs; it also stores FUNCTION_DECLs that had an escalating
+   function call in them, to check that they can be evaluated to a constant,
+   and immediate-escalating functions that may become consteval.  */
+static GTY(()) hash_set<tree> *deferred_escalating_exprs;
+
+static void
+remember_escalating_expr (tree t)
+{
+  if (!deferred_escalating_exprs)
+    deferred_escalating_exprs = hash_set<tree>::create_ggc (37);
+  deferred_escalating_exprs->add (t);
+}
+
 /* Flags for cp_fold and cp_fold_r.  */
 
 enum fold_flags {
@@ -53,8 +68,6 @@ enum fold_flags {
      definitely not in a manifestly constant-evaluated
      context.  */
   ff_mce_false = 1 << 1,
-  /* Whether we're being called from cp_fold_immediate.  */
-  ff_fold_immediate = 1 << 2,
 };
 
 using fold_flags_t = int;
@@ -72,6 +85,7 @@ static tree cp_genericize_r (tree *, int *, void *);
 static tree cp_fold_r (tree *, int *, void *);
 static void cp_genericize_tree (tree*, bool);
 static tree cp_fold (tree, fold_flags_t);
+static tree cp_fold_immediate_r (tree *, int *, void *);
 
 /* Genericize a TRY_BLOCK.  */
 
@@ -428,6 +442,104 @@ lvalue_has_side_effects (tree e)
     return TREE_SIDE_EFFECTS (e);
 }
 
+/* Return true if FN is an immediate-escalating function.  */
+
+static bool
+immediate_escalating_function_p (tree fn)
+{
+  if (!fn || !flag_immediate_escalation)
+    return false;
+
+  gcc_checking_assert (TREE_CODE (fn) == FUNCTION_DECL);
+
+  if (DECL_IMMEDIATE_FUNCTION_P (fn))
+    return false;
+
+  /* An immediate-escalating function is
+      -- the call operator of a lambda that is not declared with the consteval
+	 specifier  */
+  if (LAMBDA_FUNCTION_P (fn))
+    return true;
+  /* -- a defaulted special member function that is not declared with the
+	consteval specifier  */
+  special_function_kind sfk = special_memfn_p (fn);
+  if (sfk != sfk_none && DECL_DEFAULTED_FN (fn))
+    return true;
+  /* -- a function that results from the instantiation of a templated entity
+	defined with the constexpr specifier.  */
+  return is_instantiation_of_constexpr (fn);
+}
+
+/* Return true if FN is an immediate-escalating function that has not been
+   checked for escalating expressions..  */
+
+static bool
+unchecked_immediate_escalating_function_p (tree fn)
+{
+  return (immediate_escalating_function_p (fn)
+	  && !DECL_ESCALATION_CHECKED_P (fn));
+}
+
+/* Promote FN to an immediate function, including its clones.  */
+
+static void
+promote_function_to_consteval (tree fn)
+{
+  SET_DECL_IMMEDIATE_FUNCTION_P (fn);
+  DECL_ESCALATION_CHECKED_P (fn) = true;
+  tree clone;
+  FOR_EACH_CLONE (clone, fn)
+    {
+      SET_DECL_IMMEDIATE_FUNCTION_P (clone);
+      DECL_ESCALATION_CHECKED_P (clone) = true;
+    }
+}
+
+/* A wrapper around cp_fold_immediate_r.  Return a non-null tree if
+   we found a non-constant immediate function, or taking the address
+   of an immediate function.  */
+
+tree
+cp_fold_immediate (tree *tp, mce_value manifestly_const_eval,
+		   tree decl /*= current_function_decl*/)
+{
+  if (cxx_dialect <= cxx17)
+    return NULL_TREE;
+
+  temp_override<tree> cfd (current_function_decl, decl);
+
+  fold_flags_t flags = ff_none;
+  if (manifestly_const_eval == mce_false)
+    flags |= ff_mce_false;
+
+  cp_fold_data data (flags);
+  int save_errorcount = errorcount;
+  tree r = cp_walk_tree_without_duplicates (tp, cp_fold_immediate_r, &data);
+  if (errorcount > save_errorcount)
+    return integer_one_node;
+  return r;
+}
+
+/* Maybe say that FN (a function decl with DECL_IMMEDIATE_FUNCTION_P set)
+   was initially not an immediate function, but was promoted to one because
+   its body contained an immediate-escalating expression or conversion.  */
+
+static void
+maybe_explain_promoted_consteval (location_t loc, tree fn)
+{
+  if (DECL_ESCALATION_CHECKED_P (fn))
+    {
+      /* See if we can figure out what made the function consteval.  */
+      tree x = cp_fold_immediate (&DECL_SAVED_TREE (fn), mce_unknown, NULL_TREE);
+      if (x)
+	inform (cp_expr_loc_or_loc (x, loc),
+		"%qD was promoted to an immediate function because its "
+		"body contains an immediate-escalating expression %qE", fn, x);
+      else
+	inform (loc, "%qD was promoted to an immediate function", fn);
+    }
+}
+
 /* Gimplify *EXPR_P as rvalue into an expression that can't be modified
    by expressions with side-effects in other operands.  */
 
@@ -746,7 +858,9 @@ cp_gimplify_expr (tree *expr_p, gimple_seq *pre_p, gimple_seq *post_p)
       if (ret != GS_ERROR)
 	{
 	  tree decl = cp_get_callee_fndecl_nofold (*expr_p);
-	  if (decl && fndecl_built_in_p (decl, BUILT_IN_FRONTEND))
+	  if (!decl)
+	    break;
+	  if (fndecl_built_in_p (decl, BUILT_IN_FRONTEND))
 	    switch (DECL_FE_FUNCTION_CODE (decl))
 	      {
 	      case CP_BUILT_IN_IS_CONSTANT_EVALUATED:
@@ -771,10 +885,12 @@ cp_gimplify_expr (tree *expr_p, gimple_seq *pre_p, gimple_seq *post_p)
 	      default:
 		break;
 	      }
-	  else if (decl
-		   && fndecl_built_in_p (decl, BUILT_IN_CLZG, BUILT_IN_CTZG))
+	  else if (fndecl_built_in_p (decl, BUILT_IN_CLZG, BUILT_IN_CTZG))
 	    ret = (enum gimplify_status) c_gimplify_expr (expr_p, pre_p,
 							  post_p);
+	  else
+	    /* All consteval functions should have been processed by now.  */
+	    gcc_checking_assert (!immediate_invocation_p (decl));
 	}
       break;
 
@@ -1035,6 +1151,20 @@ struct cp_genericize_data
   bool handle_invisiref_parm_p;
 };
 
+/* Emit an error about taking the address of an immediate function.
+   EXPR is the whole expression; DECL is the immediate function.  */
+
+static void
+taking_address_of_imm_fn_error (tree expr, tree decl)
+{
+  auto_diagnostic_group d;
+  const location_t loc = (TREE_CODE (expr) == PTRMEM_CST
+			  ? PTRMEM_CST_LOCATION (expr)
+			  : EXPR_LOCATION (expr));
+  error_at (loc, "taking address of an immediate function %qD", decl);
+  maybe_explain_promoted_consteval (loc, decl);
+}
+
 /* A subroutine of cp_fold_r to handle immediate functions.  */
 
 static tree
@@ -1045,90 +1175,128 @@ cp_fold_immediate_r (tree *stmt_p, int *walk_subtrees, void *data_)
   /* The purpose of this is not to emit errors for mce_unknown.  */
   const tsubst_flags_t complain = (data->flags & ff_mce_false
 				   ? tf_error : tf_none);
+  const tree_code code = TREE_CODE (stmt);
 
   /* No need to look into types or unevaluated operands.
      NB: This affects cp_fold_r as well.  */
-  if (TYPE_P (stmt) || unevaluated_p (TREE_CODE (stmt)))
+  if (TYPE_P (stmt) || unevaluated_p (code) || in_immediate_context ())
     {
       *walk_subtrees = 0;
       return NULL_TREE;
     }
 
-  switch (TREE_CODE (stmt))
-    {
-    case PTRMEM_CST:
-      if (TREE_CODE (PTRMEM_CST_MEMBER (stmt)) == FUNCTION_DECL
-	  && DECL_IMMEDIATE_FUNCTION_P (PTRMEM_CST_MEMBER (stmt)))
-	{
-	  if (!data->pset.add (stmt) && (complain & tf_error))
-	    {
-	      error_at (PTRMEM_CST_LOCATION (stmt),
-			"taking address of an immediate function %qD",
-			PTRMEM_CST_MEMBER (stmt));
-	      *stmt_p = build_zero_cst (TREE_TYPE (stmt));
-	    }
-	  return error_mark_node;
-	}
-      break;
+  tree decl = NULL_TREE;
+  bool call_p = false;
 
-    /* Expand immediate invocations.  */
+  /* We are looking for &fn or fn().  */
+  switch (code)
+    {
     case CALL_EXPR:
     case AGGR_INIT_EXPR:
       if (tree fn = cp_get_callee (stmt))
 	if (TREE_CODE (fn) != ADDR_EXPR || ADDR_EXPR_DENOTES_CALL_P (fn))
-	  if (tree fndecl = cp_get_fndecl_from_callee (fn, /*fold*/false))
-	    if (DECL_IMMEDIATE_FUNCTION_P (fndecl))
-	      {
-		stmt = cxx_constant_value (stmt, complain);
-		if (stmt == error_mark_node)
-		  {
-		    if (complain & tf_error)
-		      *stmt_p = error_mark_node;
-		    return error_mark_node;
-		  }
-		*stmt_p = stmt;
-	      }
+	  decl = cp_get_fndecl_from_callee (fn, /*fold*/false);
+      call_p = true;
+      break;
+    case PTRMEM_CST:
+      decl = PTRMEM_CST_MEMBER (stmt);
       break;
-
     case ADDR_EXPR:
-      if (TREE_CODE (TREE_OPERAND (stmt, 0)) == FUNCTION_DECL
-	  && DECL_IMMEDIATE_FUNCTION_P (TREE_OPERAND (stmt, 0))
-	  && !ADDR_EXPR_DENOTES_CALL_P (stmt))
-	{
-	  if (complain & tf_error)
-	    {
-	      error_at (EXPR_LOCATION (stmt),
-			"taking address of an immediate function %qD",
-			TREE_OPERAND (stmt, 0));
-	      *stmt_p = build_zero_cst (TREE_TYPE (stmt));
-	    }
-	  return error_mark_node;
-	}
+      if (!ADDR_EXPR_DENOTES_CALL_P (stmt))
+	decl = TREE_OPERAND (stmt, 0);
       break;
-
     default:
-      break;
+      return NULL_TREE;
     }
 
-  return NULL_TREE;
-}
+  if (!decl || TREE_CODE (decl) != FUNCTION_DECL)
+    return NULL_TREE;
 
-/* A wrapper around cp_fold_immediate_r.  Return true if we found
-   a non-constant immediate function, or taking the address of an
-   immediate function.  */
+  /* Fully escalate once all templates have been instantiated.  What we're
+     calling is not a consteval function but it may become one.  This
+     requires recursing; DECL may be promoted to consteval because it
+     contains an escalating expression E, but E itself may have to be
+     promoted first, etc.  */
+  if (at_eof > 1 && unchecked_immediate_escalating_function_p (decl))
+    {
+      /* Set before the actual walk to avoid endless recursion.  */
+      DECL_ESCALATION_CHECKED_P (decl) = true;
+      /* We're only looking for the first escalating expression.  Let us not
+	 walk more trees than necessary, hence mce_unknown.  */
+      cp_fold_immediate (&DECL_SAVED_TREE (decl), mce_unknown, decl);
+    }
 
-bool
-cp_fold_immediate (tree *tp, mce_value manifestly_const_eval)
-{
-  if (cxx_dialect <= cxx17)
-    return false;
+  /* [expr.const]p16 "An expression or conversion is immediate-escalating if
+     it is not initially in an immediate function context and it is either
+     -- an immediate invocation that is not a constant expression and is not
+     a subexpression of an immediate invocation."
 
-  fold_flags_t flags = ff_fold_immediate;
-  if (manifestly_const_eval == mce_false)
-    flags |= ff_mce_false;
+     If we are in an immediate-escalating function, the immediate-escalating
+     expression or conversion makes it an immediate function.  So STMT does
+     not need to produce a constant expression.  */
+  if (DECL_IMMEDIATE_FUNCTION_P (decl))
+    {
+      tree e = cxx_constant_value (stmt, tf_none);
+      if (e == error_mark_node)
+	{
+	  /* This takes care of, e.g.,
+	      template <typename T>
+	      constexpr int f(T t)
+	      {
+		return id(t);
+	      }
+	    where id (consteval) causes f<int> to be promoted.  */
+	  if (immediate_escalating_function_p (current_function_decl))
+	    promote_function_to_consteval (current_function_decl);
+	  else if (complain & tf_error)
+	    {
+	      if (call_p)
+		{
+		  auto_diagnostic_group d;
+		  location_t loc = cp_expr_loc_or_input_loc (stmt);
+		  error_at (loc, "call to consteval function %qE is "
+			    "not a constant expression", stmt);
+		  /* Explain why it's not a constant expression.  */
+		  *stmt_p = cxx_constant_value (stmt, complain);
+		  maybe_explain_promoted_consteval (loc, decl);
+		}
+	      else if (!data->pset.add (stmt))
+		{
+		  taking_address_of_imm_fn_error (stmt, decl);
+		  *stmt_p = build_zero_cst (TREE_TYPE (stmt));
+		}
+	      /* If we're giving hard errors, continue the walk rather than
+		 bailing out after the first error.  */
+	      return NULL_TREE;
+	    }
+	  *walk_subtrees = 0;
+	  return stmt;
+	}
+      /* We've evaluated the consteval function call.  */
+      if (call_p)
+	*stmt_p = e;
+    }
+  /* We've encountered a function call that may turn out to be consteval
+     later.  Store its caller so that we can ensure that the call is
+     a constant expression.  */
+  else if (unchecked_immediate_escalating_function_p (decl))
+    {
+      /* Make sure we're not inserting new elements while walking
+	 the deferred_escalating_exprs hash table; if we are, it's
+	 likely that a function wasn't properly marked checked for
+	 i-e expressions.  */
+      gcc_checking_assert (at_eof <= 1);
+      if (current_function_decl)
+	remember_escalating_expr (current_function_decl);
+      /* auto p = &f<int>; in the global scope won't be ensconced in
+	 a function we could store for later at this point.  (If there's
+	 no c_f_d at this point and we're dealing with a call, we should
+	 see the call when cp_fold_function __static_i_and_d.)  */
+      else if (!call_p)
+	remember_escalating_expr (stmt);
+    }
 
-  cp_fold_data data (flags);
-  return !!cp_walk_tree_without_duplicates (tp, cp_fold_immediate_r, &data);
+  return NULL_TREE;
 }
 
 /* Perform any pre-gimplification folding of C++ front end trees to
@@ -1178,11 +1346,19 @@ cp_fold_r (tree *stmt_p, int *walk_subtrees, void *data_)
 	  *walk_subtrees = 0;
 	  /* Don't return yet, still need the cp_fold below.  */
 	}
-      cp_fold_immediate_r (stmt_p, walk_subtrees, data);
+      else
+	cp_fold_immediate_r (stmt_p, walk_subtrees, data);
     }
 
   *stmt_p = stmt = cp_fold (*stmt_p, data->flags);
 
+  /* For certain trees, like +foo(), the cp_fold above will remove the +,
+     and the subsequent tree walk would go straight down to the CALL_EXPR's
+     operands, meaning that cp_fold_immediate_r would never see the
+     CALL_EXPR.  Ew :(.  */
+  if (TREE_CODE (stmt) == CALL_EXPR && code != CALL_EXPR)
+    cp_fold_immediate_r (stmt_p, walk_subtrees, data);
+
   if (data->pset.add (stmt))
     {
       /* Don't walk subtrees of stmts we've already walked once, otherwise
@@ -1304,6 +1480,44 @@ cp_fold_function (tree fndecl)
      pass ff_mce_false.  */
   cp_fold_data data (ff_genericize | ff_mce_false);
   cp_walk_tree (&DECL_SAVED_TREE (fndecl), cp_fold_r, &data, NULL);
+
+  /* This is merely an optimization: if FNDECL has no i-e expressions,
+     we'll not save c_f_d, and we can safely say that FNDECL will not
+     be promoted to consteval.  */
+  if (deferred_escalating_exprs
+      && !deferred_escalating_exprs->contains (current_function_decl))
+    DECL_ESCALATION_CHECKED_P (fndecl) = true;
+}
+
+/* We've stashed immediate-escalating functions.  Now see if they indeed
+   ought to be promoted to consteval.  */
+
+void
+process_and_check_pending_immediate_escalating_fns ()
+{
+  /* This will be null for -fno-immediate-escalation.  */
+  if (!deferred_escalating_exprs)
+    return;
+
+  for (auto e : *deferred_escalating_exprs)
+    if (TREE_CODE (e) == FUNCTION_DECL && !DECL_ESCALATION_CHECKED_P (e))
+      cp_fold_immediate (&DECL_SAVED_TREE (e), mce_false, e);
+
+  /* We've escalated every function that could have been promoted to
+     consteval.  Check that we are not taking the address of a consteval
+     function.  */
+  for (auto e : *deferred_escalating_exprs)
+    {
+      if (TREE_CODE (e) == FUNCTION_DECL)
+	continue;
+      tree decl = (TREE_CODE (e) == PTRMEM_CST
+		   ? PTRMEM_CST_MEMBER (e)
+		   : TREE_OPERAND (e, 0));
+      if (DECL_IMMEDIATE_FUNCTION_P (decl))
+	taking_address_of_imm_fn_error (e, decl);
+    }
+
+  deferred_escalating_exprs = nullptr;
 }
 
 /* Turn SPACESHIP_EXPR EXPR into GENERIC.  */
diff --git a/gcc/cp/cp-tree.h b/gcc/cp/cp-tree.h
index 6b3ce9d..fef8081 100644
--- a/gcc/cp/cp-tree.h
+++ b/gcc/cp/cp-tree.h
@@ -2946,8 +2946,9 @@ struct GTY(()) lang_decl_fn {
   unsigned maybe_deleted : 1;
   unsigned coroutine_p : 1;
   unsigned implicit_constexpr : 1;
+  unsigned escalated_p : 1;
 
-  unsigned spare : 9;
+  unsigned spare : 8;
 
   /* 32-bits padding on 64-bit host.  */
 
@@ -3399,6 +3400,14 @@ struct GTY(()) lang_decl {
 #define DECL_MAYBE_DELETED(NODE) \
   (LANG_DECL_FN_CHECK (NODE)->maybe_deleted)
 
+/* Nonzero for FUNCTION_DECL means that this function's body has been
+   checked for immediate-escalating expressions and maybe promoted.  It
+   does *not* mean the function is consteval.  It must not be set in
+   a function that was marked consteval by the user, so that we can
+   distinguish between explicitly consteval functions and promoted consteval
+   functions.  */
+#define DECL_ESCALATION_CHECKED_P(NODE) (LANG_DECL_FN_CHECK (NODE)->escalated_p)
+
 /* True (in a FUNCTION_DECL) if NODE is a virtual function that is an
    invalid overrider for a function from a base class.  Once we have
    complained about an invalid overrider we avoid complaining about it
@@ -5882,7 +5891,8 @@ extern GTY(()) vec<tree, va_gc> *keyed_classes;
 
 
 /* Nonzero if we're done parsing and into end-of-file activities.
-   Two if we're done with front-end processing.  */
+   2 if all templates have been instantiated.
+   3 if we're done with front-end processing.  */
 
 extern int at_eof;
 
@@ -6774,6 +6784,7 @@ extern tree perform_direct_initialization_if_possible (tree, tree, bool,
 extern vec<tree,va_gc> *resolve_args (vec<tree,va_gc>*, tsubst_flags_t);
 extern tree in_charge_arg_for_name		(tree);
 extern bool in_immediate_context		();
+extern bool immediate_invocation_p		(tree);
 extern tree build_cxx_call			(tree, int, tree *,
 						 tsubst_flags_t,
 						 tree = NULL_TREE);
@@ -8415,7 +8426,9 @@ extern tree process_stmt_assume_attribute	(tree, tree, location_t);
 extern bool simple_empty_class_p		(tree, tree, tree_code);
 extern tree fold_builtin_source_location	(const_tree);
 extern tree get_source_location_impl_type	();
-extern bool cp_fold_immediate			(tree *, mce_value);
+extern tree cp_fold_immediate			(tree *, mce_value,
+						 tree = current_function_decl);
+extern void process_and_check_pending_immediate_escalating_fns ();
 
 /* in name-lookup.cc */
 extern tree strip_using_decl                    (tree);
diff --git a/gcc/cp/decl2.cc b/gcc/cp/decl2.cc
index 9e666e5..bee8487 100644
--- a/gcc/cp/decl2.cc
+++ b/gcc/cp/decl2.cc
@@ -169,7 +169,9 @@ typedef hash_map<unsigned/*Priority*/, tree/*List*/,
    one for init.  The fini table is only ever used when !cxa_atexit.  */
 static GTY(()) priority_map_t *static_init_fini_fns[2];
 
-/* Nonzero if we're done parsing and into end-of-file activities.  */
+/* Nonzero if we're done parsing and into end-of-file activities.
+   2 if all templates have been instantiated.
+   3 if we're done with front-end processing.  */
 
 int at_eof;
 
@@ -4987,6 +4989,7 @@ c_parse_final_cleanups (void)
   tree decl;
 
   locus_at_end_of_parsing = input_location;
+  /* We're done parsing.  */
   at_eof = 1;
 
   /* Bad parse errors.  Just forget about it.  */
@@ -5252,6 +5255,9 @@ c_parse_final_cleanups (void)
 	reconsider = true;
     }
 
+  /* All templates have been instantiated.  */
+  at_eof = 2;
+
   void *module_cookie = finish_module_processing (parse_in);
 
   lower_var_init ();
@@ -5294,7 +5300,11 @@ c_parse_final_cleanups (void)
   if (static_init_fini_fns[true])
     for (auto iter : *static_init_fini_fns[true])
       iter.second = nreverse (iter.second);
-  
+
+  /* Now we've instantiated all templates.  Now we can escalate the functions
+     we squirreled away earlier.  */
+  process_and_check_pending_immediate_escalating_fns ();
+
   /* Then, do the Objective-C stuff.  This is where all the
      Objective-C module stuff gets generated (symtab,
      class/protocol/selector lists etc).  This must be done after C++
@@ -5376,7 +5386,7 @@ c_parse_final_cleanups (void)
   timevar_start (TV_PHASE_PARSING);
 
   /* Indicate that we're done with front end processing.  */
-  at_eof = 2;
+  at_eof = 3;
 }
 
 /* Perform any post compilation-proper cleanups for the C++ front-end.
diff --git a/gcc/cp/error.cc b/gcc/cp/error.cc
index 785909c..3b1b5de 100644
--- a/gcc/cp/error.cc
+++ b/gcc/cp/error.cc
@@ -478,7 +478,7 @@ dump_template_bindings (cxx_pretty_printer *pp, tree parms, tree args,
 
   /* Don't try to do this once cgraph starts throwing away front-end
      information.  */
-  if (at_eof >= 2)
+  if (at_eof >= 3)
     return;
 
   FOR_EACH_VEC_SAFE_ELT (typenames, i, t)
diff --git a/gcc/cp/module.cc b/gcc/cp/module.cc
index 33fcf39..1b57fbe 100644
--- a/gcc/cp/module.cc
+++ b/gcc/cp/module.cc
@@ -5683,6 +5683,8 @@ trees_out::lang_decl_bools (tree t)
       WB (lang->u.fn.has_dependent_explicit_spec_p);
       WB (lang->u.fn.immediate_fn_p);
       WB (lang->u.fn.maybe_deleted);
+      WB (lang->u.fn.escalated_p);
+      /* We do not stream lang->u.fn.implicit_constexpr.  */
       goto lds_min;
 
     case lds_decomp:  /* lang_decl_decomp.  */
@@ -5751,6 +5753,8 @@ trees_in::lang_decl_bools (tree t)
       RB (lang->u.fn.has_dependent_explicit_spec_p);
       RB (lang->u.fn.immediate_fn_p);
       RB (lang->u.fn.maybe_deleted);
+      RB (lang->u.fn.escalated_p);
+      /* We do not stream lang->u.fn.implicit_constexpr.  */
       goto lds_min;
 
     case lds_decomp:  /* lang_decl_decomp.  */
diff --git a/gcc/cp/pt.cc b/gcc/cp/pt.cc
index 7265201..924a209 100644
--- a/gcc/cp/pt.cc
+++ b/gcc/cp/pt.cc
@@ -11107,7 +11107,7 @@ push_tinst_level_loc (tree tldcl, tree targs, location_t loc)
   if (tinst_depth >= max_tinst_depth)
     {
       /* Tell error.cc not to try to instantiate any templates.  */
-      at_eof = 2;
+      at_eof = 3;
       fatal_error (input_location,
 		   "template instantiation depth exceeds maximum of %d"
 		   " (use %<-ftemplate-depth=%> to increase the maximum)",
diff --git a/gcc/cp/typeck.cc b/gcc/cp/typeck.cc
index bf8ffaa..8e4cfae 100644
--- a/gcc/cp/typeck.cc
+++ b/gcc/cp/typeck.cc
@@ -7269,11 +7269,9 @@ cp_build_addr_expr_1 (tree arg, bool strict_lvalue, tsubst_flags_t complain)
 			      complain);
     }
 
-  /* For addresses of immediate functions ensure we have EXPR_LOCATION
-     set for possible later diagnostics.  */
+  /* Ensure we have EXPR_LOCATION set for possible later diagnostics.  */
   if (TREE_CODE (val) == ADDR_EXPR
-      && TREE_CODE (TREE_OPERAND (val, 0)) == FUNCTION_DECL
-      && DECL_IMMEDIATE_FUNCTION_P (TREE_OPERAND (val, 0)))
+      && TREE_CODE (TREE_OPERAND (val, 0)) == FUNCTION_DECL)
     SET_EXPR_LOCATION (val, input_location);
 
   return val;