ipa: fix handling of multiple speculations (PR93318)

This patch started as work to resole Richard's comment on quadratic lookups
in resolve_speculation. While doing it I however noticed multiple problems
in the new speuclative call code which made the patch quite big. In
particular:
 1) Before applying speculation we consider only targets with at lest
    probability 1/2.
    If profile is sane at most two targets can have probability greater or
    equal to 1/2. So the new multi-target speculation code got enabled only
    in very special scenario when there ae precisely two target with precise
    probability 1/2 (which is tested by the single testcase).

    As a conseuqence the multiple target logic got minimal test coverage and
    this made us to miss several ICEs.
 2) Profile updating in profile merging, tree-inline and indirect call
    expansion was wrong which led to inconsistent profiles (as already seen
    on the testcase).
 3) Code responsible to turn speculative call to direct call was broken for
    anything with more than one target.
 4) There were multiple cases where call_site_hash went out of sync which
    eventually leads to an ICE..
 5) Some code expects that all speculative call targets forms a sequence in
    the callee linked list but there is no code to maintain that invariant
    nor a verifier.
Fixing this it became obvious that the current API of speculative_call_info is
not useful because it really builds on fact tht there are precisely three
components (direct call, ref and indirect call) in every speculative call
sequence.  I ended up replacing it with iterator API for direct call
(first_speculative_call_target, next_speculative_call_target) and accessors for
the other coponents updating comment in cgraph.h.

Finally I made the work with call site hash more effetive by updating edge
manipulation to keep them in sequence. So first one can be looked up from the
hash and then they can be iterated by callee.

There are other things that can be improved (for example the speculation should
start with most common target first), but I will try to keep that for next
stage1. This patch is mostly about getting rid of ICE and profile corruption
which is a regression from GCC 9.

gcc/ChangeLog:

	PR lto/93318
	* cgraph.c (cgraph_add_edge_to_call_site_hash): Update call site
	hash only when edge is first within the sequence.
	(cgraph_edge::set_call_stmt): Update handling of speculative calls.
	(symbol_table::create_edge): Do not set target_prob.
	(cgraph_edge::remove_caller): Watch for speculative calls when updating
	the call site hash.
	(cgraph_edge::make_speculative): Drop target_prob parameter.
	(cgraph_edge::speculative_call_info): Remove.
	(cgraph_edge::first_speculative_call_target): New member function.
	(update_call_stmt_hash_for_removing_direct_edge): New function.
	(cgraph_edge::resolve_speculation): Rewrite to new API.
	(cgraph_edge::speculative_call_for_target): New member function.
	(cgraph_edge::make_direct): Rewrite to new API; fix handling of
	multiple speculation targets.
	(cgraph_edge::redirect_call_stmt_to_callee): Likewise; fix updating
	of profile.
	(verify_speculative_call): Verify that targets form an interval.
	* cgraph.h (cgraph_edge::speculative_call_info): Remove.
	(cgraph_edge::first_speculative_call_target): New member function.
	(cgraph_edge::next_speculative_call_target): New member function.
	(cgraph_edge::speculative_call_target_ref): New member function.
	(cgraph_edge;:speculative_call_indirect_edge): New member funtion.
	(cgraph_edge): Remove target_prob.
	* cgraphclones.c (cgraph_node::set_call_stmt_including_clones):
	Fix handling of speculative calls.
	* ipa-devirt.c (ipa_devirt): Fix handling of speculative cals.
	* ipa-fnsummary.c (analyze_function_body): Likewise.
	* ipa-inline.c (speculation_useful_p): Use new speculative call API.
	* ipa-profile.c (dump_histogram): Fix formating.
	(ipa_profile_generate_summary): Watch for overflows.
	(ipa_profile): Do not require probablity to be 1/2; update to new API.
	* ipa-prop.c (ipa_make_edge_direct_to_target): Update to new API.
	(update_indirect_edges_after_inlining): Update to new API.
	* ipa-utils.c (ipa_merge_profiles): Rewrite merging of speculative call
	profiles.
	* profile-count.h: (profile_probability::adjusted): New.
	* tree-inline.c (copy_bb): Update to new speculative call API; fix
	updating of profile.
	* value-prof.c (gimple_ic_transform): Rename to ...
	(dump_ic_profile): ... this one; update dumping.
	(stream_in_histogram_value): Fix formating.
	(gimple_value_profile_transformations): Update.

gcc/testsuite/ChangeLog:

	* g++.dg/tree-prof/indir-call-prof.C: Update template.
	* gcc.dg/tree-prof/crossmodule-indircall-1.c: Add more targets.
	* gcc.dg/tree-prof/crossmodule-indircall-1a.c: Add more targets.
	* gcc.dg/tree-prof/indir-call-prof.c: Update template.

1 files changed, 50 insertions, 41 deletions

diff --git a/gcc/tree-inline.c b/gcc/tree-inline.c
index 2197769..5b0050a 100644
--- a/gcc/tree-inline.c
+++ b/gcc/tree-inline.c
@@ -2181,47 +2181,51 @@ copy_bb (copy_body_data *id, basic_block bb,
 		  if (edge)
 		    {
 		      struct cgraph_edge *old_edge = edge;
-		      profile_count old_cnt = edge->count;
-		      edge = edge->clone (id->dst_node, call_stmt,
-					  gimple_uid (stmt),
-					  num, den,
-					  true);
-
-		      /* A speculative call is consist of edges - indirect edge
-			 and direct edges (one indirect edeg may has multiple
-			 direct edges).  Duplicate the whole thing and
-			 distribute frequencies accordingly.  */
+
+		      /* A speculative call is consist of multiple
+			 edges - indirect edge and one or more direct edges
+			 Duplicate the whole thing and distribute frequencies
+			 accordingly.  */
 		      if (edge->speculative)
 			{
-			  struct cgraph_edge *direct, *indirect;
-			  struct ipa_ref *ref;
-
-			  gcc_assert (!edge->indirect_unknown_callee);
-			  old_edge->speculative_call_info (direct, indirect, ref);
-			  while (old_edge->next_callee
-				 && old_edge->next_callee->speculative
-				 && indirect->num_speculative_call_targets_p ()
-				      > 1)
+			  int n = 0;
+			  profile_count direct_cnt
+				 = profile_count::zero ();
+
+			  /* First figure out the distribution of counts
+			     so we can re-scale BB profile accordingly.  */
+			  for (cgraph_edge *e = old_edge; e;
+			       e = e->next_speculative_call_target ())
+			    direct_cnt = direct_cnt + e->count;
+
+			  cgraph_edge *indirect
+				 = old_edge->speculative_call_indirect_edge ();
+			  profile_count indir_cnt = indirect->count;
+
+			  /* Next iterate all direct edges, clone it and its
+			     corresponding reference and update profile.  */
+			  for (cgraph_edge *e = old_edge;
+			       e;
+			       e = e->next_speculative_call_target ())
 			    {
-			      id->dst_node->clone_reference (ref, stmt);
-
-			      edge = old_edge->next_callee;
-			      edge = edge->clone (id->dst_node, call_stmt,
-						  gimple_uid (stmt), num, den,
-						  true);
-			      old_edge = old_edge->next_callee;
-			      gcc_assert (!edge->indirect_unknown_callee);
-
-			      /* If the indirect edge has multiple speculative
-				 calls, iterate through all direct calls
-				 associated to the speculative call and clone
-				 all related direct edges before cloning the
-				 related indirect edge.  */
-			      old_edge->speculative_call_info (direct, indirect,
-							       ref);
+			      profile_count cnt = e->count;
+
+			      id->dst_node->clone_reference
+				 (e->speculative_call_target_ref (), stmt);
+			      edge = e->clone (id->dst_node, call_stmt,
+					       gimple_uid (stmt), num, den,
+					       true);
+			      profile_probability prob
+				 = cnt.probability_in (direct_cnt
+						       + indir_cnt);
+			      edge->count
+				 = copy_basic_block->count.apply_probability
+					 (prob);
+			      n++;
 			    }
-
-			  profile_count indir_cnt = indirect->count;
+			  gcc_checking_assert
+				 (indirect->num_speculative_call_targets_p ()
+				  == n);
 
 			  /* Duplicate the indirect edge after all direct edges
 			     cloned.  */
@@ -2231,14 +2235,19 @@ copy_bb (copy_body_data *id, basic_block bb,
 						      true);
 
 			  profile_probability prob
-			     = indir_cnt.probability_in (old_cnt + indir_cnt);
+			     = indir_cnt.probability_in (direct_cnt
+							 + indir_cnt);
 			  indirect->count
 			     = copy_basic_block->count.apply_probability (prob);
-			  edge->count = copy_basic_block->count - indirect->count;
-			  id->dst_node->clone_reference (ref, stmt);
 			}
 		      else
-			edge->count = copy_basic_block->count;
+			{
+			  edge = edge->clone (id->dst_node, call_stmt,
+					      gimple_uid (stmt),
+					      num, den,
+					      true);
+			  edge->count = copy_basic_block->count;
+			}
 		    }
 		  break;