SLP vectorize multiple BBs at once

This work from Martin Liska was motivated by gcc.dg/vect/bb-slp-22.c
which shows how poorly we currently BB vectorize code like

  a0 = in[0] + 23;
  a1 = in[1] + 142;
  a2 = in[2] + 2;
  a3 = in[3] + 31;

  if (x > y)
    {
      b[0] = a0;
      b[1] = a1;
      b[2] = a2;
      b[3] = a3;
    }
  else
    {
      out[0] = a0 * (x + 1);
      out[1] = a1 * (y + 1);
      out[2] = a2 * (x + 1);
      out[3] = a3 * (y + 1);
    }

namely by vectorizing the stores but not the common load (and add)
they are feeded with.

Thus with the following patch we change the BB vectorizer from
operating on a single basic-block at a time to consider somewhat
larger regions (but not the whole function yet because of issues
with vector size iteration).

I took the opportunity to remove the fancy region iterations again
now that we operate on BB granularity and in the end need to visit
PHI nodes as well.

2020-10-08  Martin Liska  <mliska@suse.cz>
	    Richard Biener  <rguenther@suse.de>

	* tree-vectorizer.h (_bb_vec_info::const_iterator): Remove.
	(_bb_vec_info::const_reverse_iterator): Likewise.
	(_bb_vec_info::region_stmts): Likewise.
	(_bb_vec_info::reverse_region_stmts): Likewise.
	(_bb_vec_info::_bb_vec_info): Adjust.
	(_bb_vec_info::bb): Remove.
	(_bb_vec_info::region_begin): Remove.
	(_bb_vec_info::region_end): Remove.
	(_bb_vec_info::bbs): New vector of BBs.
	(vect_slp_function): Declare.
	* tree-vect-patterns.c (vect_determine_precisions): Use
	regular stmt iteration.
	(vect_pattern_recog): Likewise.
	* tree-vect-slp.c: Include cfganal.h, tree-eh.h and tree-cfg.h.
	(vect_build_slp_tree_1): Properly refuse to vectorize
	volatile and throwing stmts.
	(vect_build_slp_tree_2): Pass group-size down to
	get_vectype_for_scalar_type.
	(_bb_vec_info::_bb_vec_info): Use regular stmt iteration,
	adjust for changed region specification.
	(_bb_vec_info::~_bb_vec_info): Likewise.
	(vect_slp_check_for_constructors): Likewise.
	(vect_slp_region): Likewise.
	(vect_slp_bbs): New worker operating on a vector of BBs.
	(vect_slp_bb): Wrap it.
	(vect_slp_function): New function splitting the function
	into multi-BB regions.
	(vect_create_constant_vectors): Handle the case of inserting
	after a throwing def.
	(vect_schedule_slp_instance): Adjust.
	* tree-vectorizer.c (vec_info::remove_stmt): Simplify again.
	(vec_info::insert_seq_on_entry): Adjust.
	(pass_slp_vectorize::execute): Also init PHIs.  Call
	vect_slp_function.

	* gcc.dg/vect/bb-slp-22.c: Adjust.
	* gfortran.dg/pr68627.f: Likewise.

1 files changed, 11 insertions, 8 deletions

diff --git a/gcc/tree-vectorizer.c b/gcc/tree-vectorizer.c
index 626f0ce..02da755 100644
--- a/gcc/tree-vectorizer.c
+++ b/gcc/tree-vectorizer.c
@@ -605,11 +605,7 @@ vec_info::remove_stmt (stmt_vec_info stmt_info)
   set_vinfo_for_stmt (stmt_info->stmt, NULL);
   unlink_stmt_vdef (stmt_info->stmt);
   gimple_stmt_iterator si = gsi_for_stmt (stmt_info->stmt);
-  gimple_stmt_iterator *psi = &si;
-  if (bb_vec_info bb_vinfo = dyn_cast <bb_vec_info> (this))
-    if (gsi_stmt (bb_vinfo->region_begin) == stmt_info->stmt)
-      psi = &bb_vinfo->region_begin;
-  gsi_remove (psi, true);
+  gsi_remove (&si, true);
   release_defs (stmt_info->stmt);
   free_stmt_vec_info (stmt_info);
 }
@@ -653,7 +649,8 @@ vec_info::insert_seq_on_entry (stmt_vec_info context, gimple_seq seq)
   else
     {
       bb_vec_info bb_vinfo = as_a <bb_vec_info> (this);
-      gimple_stmt_iterator gsi_region_begin = bb_vinfo->region_begin;
+      gimple_stmt_iterator gsi_region_begin
+	= gsi_after_labels (bb_vinfo->bbs[0]);
       gsi_insert_seq_before (&gsi_region_begin, seq, GSI_SAME_STMT);
     }
 }
@@ -1416,6 +1413,13 @@ pass_slp_vectorize::execute (function *fun)
   /* Mark all stmts as not belonging to the current region and unvisited.  */
   FOR_EACH_BB_FN (bb, fun)
     {
+      for (gphi_iterator gsi = gsi_start_phis (bb); !gsi_end_p (gsi);
+	   gsi_next (&gsi))
+	{
+	  gphi *stmt = gsi.phi ();
+	  gimple_set_uid (stmt, -1);
+	  gimple_set_visited (stmt, false);
+	}
       for (gimple_stmt_iterator gsi = gsi_start_bb (bb); !gsi_end_p (gsi);
 	   gsi_next (&gsi))
 	{
@@ -1425,8 +1429,7 @@ pass_slp_vectorize::execute (function *fun)
 	}
     }
 
-  FOR_EACH_BB_FN (bb, fun)
-    vect_slp_bb (bb);
+  vect_slp_function (fun);
 
   if (!in_loop_pipeline)
     {