tailr: Punt on tail recursions that would break musttail [PR119493]

While working on the previous tailc patch, I've noticed the following
problem.
The testcase below fails, because we decide to tail recursion optimize
the call, but tail recursion (as documented in tree-tailcall.cc) needs to
add some result multiplication and/or addition if any tail recursion uses
accumulator, which is added right before the return.
So, if there are musttail non-recurive calls in the function, successful
tail recursion optimization will mean we'll later error on the musttail
calls.  musttail recursive calls are ok, those would be tail recursion
optimized.

So, the following patch punts on all tail recursion optimizations if it
needs accumulators (add and/or mult) if there is at least one non-recursive
musttail call.

2025-04-01  Jakub Jelinek  <jakub@redhat.com>

	PR tree-optimization/119493
	* tree-tailcall.cc (tree_optimize_tail_calls_1): Ignore tail recursion
	candidates which need accumulators if there is at least one musttail
	non-recursive call.

	* gcc.dg/pr119493-2.c: New test.

2 files changed, 54 insertions, 0 deletions

diff --git a/gcc/testsuite/gcc.dg/pr119493-2.c b/gcc/testsuite/gcc.dg/pr119493-2.c
new file mode 100644
index 0000000..951529f
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/pr119493-2.c
@@ -0,0 +1,22 @@
+/* PR tree-optimization/119493 */
+/* { dg-do compile { target musttail } } */
+/* { dg-options "-O2 -fdump-tree-tailr1-details" } */
+/* { dg-final { scan-tree-dump-times "tail recursion with accumulation mixed with musttail non-recursive call" 2 "tailr1" } } */
+
+[[gnu::noipa]] int
+bar (int x, int y)
+{
+  return x + y;
+}
+
+[[gnu::noinline, gnu::noclone]] int
+foo (int x, int y)
+{
+  if (x < 10)
+    [[gnu::musttail]] return bar (x, y);
+  if (y & 2)
+    return foo (x - 1, y) * 2;
+  if (y & 1)
+    [[gnu::musttail]] return foo (x - 1, y);
+  return foo (x - 1, y) * 3;
+}
diff --git a/gcc/tree-tailcall.cc b/gcc/tree-tailcall.cc
index 8ea1c8b..bc053ae 100644
--- a/gcc/tree-tailcall.cc
+++ b/gcc/tree-tailcall.cc
@@ -1345,6 +1345,38 @@ tree_optimize_tail_calls_1 (bool opt_tailcalls, bool only_musttail,
       live_vars = NULL;
     }
 
+  if (cfun->has_musttail)
+    {
+      /* We can't mix non-recursive must tail calls with tail recursive
+	 calls which require accumulators, because in that case we have to
+	 emit code in between the musttail calls and return, which prevent
+	 calling them as tail calls.  So, in that case give up on the
+	 tail recursion.  */
+      for (act = tailcalls; act; act = act->next)
+	if (!act->tail_recursion)
+	  {
+	    gcall *call = as_a <gcall *> (gsi_stmt (act->call_gsi));
+	    if (gimple_call_must_tail_p (call))
+	      break;
+	  }
+      if (act)
+	for (struct tailcall **p = &tailcalls; *p; )
+	  {
+	    if ((*p)->tail_recursion && ((*p)->add || (*p)->mult))
+	      {
+		struct tailcall *a = *p;
+		*p = (*p)->next;
+		gcall *call = as_a <gcall *> (gsi_stmt (a->call_gsi));
+		maybe_error_musttail (call,
+				      _("tail recursion with accumulation "
+					"mixed with musttail "
+					"non-recursive call"), diag_musttail);
+		free (a);
+	      }
+	    else
+	      p = &(*p)->next;
+	  }
+    }
   /* Construct the phi nodes and accumulators if necessary.  */
   a_acc = m_acc = NULL_TREE;
   for (act = tailcalls; act; act = act->next)