Makefile.in (omp-low.o): Depend on $(TARGET_H).

	* Makefile.in (omp-low.o): Depend on $(TARGET_H).
	* cfgloop.h (struct loop): Add safelen, force_vect, simduid.
	* function.h (struct function): Add has_force_vect_loops and
	has_simduid_loops.
	* gimple-pretty-print.c (dump_gimple_omp_for): Handle
	GF_OMP_FOR_KIND*.
	* gimple.c (gimple_build_omp_critical): Add KIND argument and
	handle it.
	* gimple.def: Update CLAUSES comments.
	* gimple.h (enum gf_mask): Add GF_OMP_FOR_KIND_{FOR,SIMD}.
	(gimple_build_omp_for): Add argument to prototype.
	(gimple_omp_for_kind): New.
	(gimple_omp_for_set_kind): New.
	* gimplify.c (enum gimplify_omp_var_data): Add GOVD_LINEAR to
	GOVD_DATA_SHARE_CLASS.
	(enum omp_region_type): Add ORT_SIMD.
	(gimple_add_tmp_var): Handle ORT_SIMD.
	(gimplify_var_or_parm_decl): Same.
	(is_gimple_stmt): Same.
	(omp_firstprivatize_variable): Same.
	(omp_add_variable): Only use splay_tree_insert if lookup failed.
	(omp_notice_variable): Handle ORT_SIMD.
	(omp_is_private): Add SIMD argument and handle it as well as
	ORT_SIMD.
	(omp_check_private): Handle ORT_SIMD.
	(gimplify_scan_omp_clauses): Handle OMP_CLAUSE_LINEAR and
	OMP_CLAUSE_SAFELEN.
	(gimplify_adjust_omp_clauses_1): Handle GOVD_LINEAR.
	Handle OMP_CLAUSE_LASTPRIVATE.
	(gimplify_adjust_omp_clauses): Handle OMP_CLAUSE_LINEAR and
	OMP_CLAUSE_SAFELEN.
	(gimplify_omp_for): Handle OMP_SIMD and OMP_CLAUSE_LINEAR.
	(gimplify_expr): Handle OMP_SIMD.
	* internal-fn.c (expand_GOMP_SIMD_LANE): New.
	(expand_GOMP_SIMD_VF): New.
	(expand_GOMP_SIMD_LAST_LANE): New.
	* internal-fn.def (GOMP_SIMD_LANE): New.
	(GOMP_SIMD_VF): New.
	(GOMP_SIMD_LAST_LANE): New.
	* omp-low.c: Include target.h.
	(extract_omp_for_data): Handle OMP_SIMD, OMP_CLAUSE_LINEAR,
	OMP_CLAUSE_SAFELEN.
	(check_omp_nesting_restrictions): Same.
	(omp_max_vf): New.
	(lower_rec_simd_input_clauses): New.
	(lower_rec_input_clauses): Handle OMP_SIMD, GF_OMP_FOR_KIND_SIMD,
	OMP_CLAUSE_LINEAR.
	(lower_lastprivate_clauses): Handle OMP_CLAUSE_LINEAR,
	GF_OMP_FOR_KIND_SIMD, OMP_SIMD.
	(expand_omp_build_assign): New.
	(expand_omp_for_init_counts): New.
	(expand_omp_for_init_vars): New.
	(extract_omp_for_update_vars): New.
	(expand_omp_for_generic): Use expand_omp_for_{init,update}_vars
	and rewrite accordingly.
	(expand_omp_simd): New.
	(expand_omp_for): Use expand_omp_simd.
	(lower_omp_for_lastprivate): Unshare vinit when appropriate.
	(lower_omp_for): Do not lower the body.
	* tree-data-ref (get_references_in_stmt): Allow IFN_GOMP_SIMD_LANE
	in their own loops.
	* tree-flow.h (find_omp_clause): Remove prototype.
	* tree-if-conv.c (main_tree_if_conversion): Run if doing if
	conversion, forcing vectorization of the loop, or if
	flag_tree_vectorize.
	(gate_tree_if_conversion): Similarly.
	* tree-inline.c (remap_gimple_stmt): Pass for kind argument to
	gimple_build_omp_for.
	(copy_cfg_body): set has_force_vect_loops and has_simduid_loops.
	* tree-parloops (create_parallel_loop): Pass kind argument to
	gimple_build_omp_for.
	* tree-pretty-print.c (dump_omp_clause): Add cases for
	OMP_CLAUSE_UNIFORM, OMP_CLAUSE_LINEAR, OMP_CLAUSE_SAFELEN,
	OMP_CLAUSE__SIMDUID_.
	(dump_generic_node): Handle OMP_SIMD.
	* tree-ssa-ccp.c (likely_value): Handle IFN_GOMP_SIMD*.
	* tree-ssa-loop-ivcanon.c (tree_unroll_loops_completely_1): Do not
	unroll OMP_SIMD loops here.
	* tree-ssa-loop.c (gate_tree_vectorize): Run if
	has_force_vect_loops.
	* tree-vect-data-refs.c (vect_analyze_data_ref_dependence): Handle
	loop->safelen
	(vect_analyze_data_refs): Handle simd loops.
	* tree-vect-loop.c (vectorizable_live_operation): Handle
	IFN_GOMP_SIMD*.
	* tree-vect-stmts.c (vectorizable_call): Handle
	IFN_GOMP_SIMD_LANE.
	(vectorizable_store): Handle STMT_VINFO_SIMD_LANE_ACCESS_P.
	(vectorizable_load): Same.
	* tree-vectorizer.c: Include hash-table.h and
	tree-ssa-propagate.h.
	(struct simduid_to_vf): New.
	(simduid_to_vf::hash): New.
	(simduid_to-vf::equal): New.
	(struct simd_array_to_simduid): New.
	(simd_array_to_simduid::hash): New.
	(simd_array_to_simduid::equal): New.
	(adjust_simduid_builtins): New.
	(struct note_simd_array_uses_struct): New.
	(note_simd_array_uses_cb): New.
	(note_simd_array_uses): New.
	(vectorize_loops): Handle simd hints and adjust simd builtins
	accordingly.
	* tree-vectorizer.h (struct _stmt_vec_info): Add
	simd_lane_access_p field.
	(STMT_VINFO_SIMD_LANE_ACCESS_P): New macro.
	* tree.c (omp_clause_num_ops): Add entries for OMP_CLAUSE_LINEAR,
	OMP_CLAUSE_SAFELEN, OMP_CLAUSE__SIMDUID_, OMP_CLAUSE_UNIFORM.
	(omp_clause_code_name): Same.
	(walk_tree_1): Handle OMP_CLAUSE_UNIFORM, OMP_CLAUSE_SAFELEN,
	OMP_CLAUSE__SIMDUID_, OMP_CLAUSE_LINEAR.
	* tree.def (OMP_SIMD): New entry.
	* tree.h (enum omp_clause_code): Add entries for
	OMP_CLAUSE_LINEAR, OMP_CLAUSE_UNIFORM, OMP_CLAUSE_SAFELEN,
	OMP_CLAUSE__SIMDUID_.
	(OMP_CLAUSE_DECL): Adjust range for new clauses.
	(OMP_CLAUSE_LINEAR_NO_COPYIN): New.
	(OMP_CLAUSE_LINEAR_NO_COPYOUT): New.
	(OMP_CLAUSE_LINEAR_STEP): New.
	(OMP_CLAUSE_SAFELEN_EXPR): New.
	(OMP_CLAUSE__SIMDUID__DECL): New.
	(find_omp_clause): New prototype.
cp/
	* cp-tree.h (CP_OMP_CLAUSE_INFO): Adjust range for new clauses.

Co-Authored-By: Aldy Hernandez <aldyh@redhat.com>

From-SVN: r202029

1 files changed, 89 insertions, 15 deletions

diff --git a/gcc/tree-vect-data-refs.c b/gcc/tree-vect-data-refs.c
index 4e99747..3f37510 100644
--- a/gcc/tree-vect-data-refs.c
+++ b/gcc/tree-vect-data-refs.c
@@ -255,6 +255,15 @@ vect_analyze_data_ref_dependence (struct data_dependence_relation *ddr,
   /* Unknown data dependence.  */
   if (DDR_ARE_DEPENDENT (ddr) == chrec_dont_know)
     {
+      /* If user asserted safelen consecutive iterations can be
+	 executed concurrently, assume independence.  */
+      if (loop->safelen >= 2)
+	{
+	  if (loop->safelen < *max_vf)
+	    *max_vf = loop->safelen;
+	  return false;
+	}
+
       if (STMT_VINFO_GATHER_P (stmtinfo_a)
 	  || STMT_VINFO_GATHER_P (stmtinfo_b))
 	{
@@ -291,6 +300,15 @@ vect_analyze_data_ref_dependence (struct data_dependence_relation *ddr,
   /* Known data dependence.  */
   if (DDR_NUM_DIST_VECTS (ddr) == 0)
     {
+      /* If user asserted safelen consecutive iterations can be
+	 executed concurrently, assume independence.  */
+      if (loop->safelen >= 2)
+	{
+	  if (loop->safelen < *max_vf)
+	    *max_vf = loop->safelen;
+	  return false;
+	}
+
       if (STMT_VINFO_GATHER_P (stmtinfo_a)
 	  || STMT_VINFO_GATHER_P (stmtinfo_b))
 	{
@@ -2930,6 +2948,7 @@ vect_analyze_data_refs (loop_vec_info loop_vinfo,
       stmt_vec_info stmt_info;
       tree base, offset, init;
       bool gather = false;
+      bool simd_lane_access = false;
       int vf;
 
 again:
@@ -2961,12 +2980,17 @@ again:
       if (!DR_BASE_ADDRESS (dr) || !DR_OFFSET (dr) || !DR_INIT (dr)
 	  || !DR_STEP (dr))
         {
-	  /* If target supports vector gather loads, see if they can't
-	     be used.  */
-	  if (loop_vinfo
-	      && DR_IS_READ (dr)
+	  bool maybe_gather
+	    = DR_IS_READ (dr)
 	      && !TREE_THIS_VOLATILE (DR_REF (dr))
-	      && targetm.vectorize.builtin_gather != NULL
+	      && targetm.vectorize.builtin_gather != NULL;
+	  bool maybe_simd_lane_access
+	    = loop_vinfo && loop->simduid;
+
+	  /* If target supports vector gather loads, or if this might be
+	     a SIMD lane access, see if they can't be used.  */
+	  if (loop_vinfo
+	      && (maybe_gather || maybe_simd_lane_access)
 	      && !nested_in_vect_loop_p (loop, stmt))
 	    {
 	      struct data_reference *newdr
@@ -2979,14 +3003,59 @@ again:
 		  && DR_STEP (newdr)
 		  && integer_zerop (DR_STEP (newdr)))
 		{
-		  dr = newdr;
-		  gather = true;
+		  if (maybe_simd_lane_access)
+		    {
+		      tree off = DR_OFFSET (newdr);
+		      STRIP_NOPS (off);
+		      if (TREE_CODE (DR_INIT (newdr)) == INTEGER_CST
+			  && TREE_CODE (off) == MULT_EXPR
+			  && host_integerp (TREE_OPERAND (off, 1), 1))
+			{
+			  tree step = TREE_OPERAND (off, 1);
+			  off = TREE_OPERAND (off, 0);
+			  STRIP_NOPS (off);
+			  if (CONVERT_EXPR_P (off)
+			      && TYPE_PRECISION (TREE_TYPE (TREE_OPERAND (off,
+									  0)))
+				 < TYPE_PRECISION (TREE_TYPE (off)))
+			    off = TREE_OPERAND (off, 0);
+			  if (TREE_CODE (off) == SSA_NAME)
+			    {
+			      gimple def = SSA_NAME_DEF_STMT (off);
+			      tree reft = TREE_TYPE (DR_REF (newdr));
+			      if (gimple_call_internal_p (def)
+				  && gimple_call_internal_fn (def)
+				  == IFN_GOMP_SIMD_LANE)
+				{
+				  tree arg = gimple_call_arg (def, 0);
+				  gcc_assert (TREE_CODE (arg) == SSA_NAME);
+				  arg = SSA_NAME_VAR (arg);
+				  if (arg == loop->simduid
+				      /* For now.  */
+				      && tree_int_cst_equal
+					   (TYPE_SIZE_UNIT (reft),
+					    step))
+				    {
+				      DR_OFFSET (newdr) = ssize_int (0);
+				      DR_STEP (newdr) = step;
+				      dr = newdr;
+				      simd_lane_access = true;
+				    }
+				}
+			    }
+			}
+		    }
+		  if (!simd_lane_access && maybe_gather)
+		    {
+		      dr = newdr;
+		      gather = true;
+		    }
 		}
-	      else
+	      if (!gather && !simd_lane_access)
 		free_data_ref (newdr);
 	    }
 
-	  if (!gather)
+	  if (!gather && !simd_lane_access)
 	    {
 	      if (dump_enabled_p ())
 		{
@@ -3013,7 +3082,7 @@ again:
           if (bb_vinfo)
 	    break;
 
-	  if (gather)
+	  if (gather || simd_lane_access)
 	    free_data_ref (dr);
 	  return false;
         }
@@ -3046,7 +3115,7 @@ again:
           if (bb_vinfo)
 	    break;
 
-	  if (gather)
+	  if (gather || simd_lane_access)
 	    free_data_ref (dr);
           return false;
         }
@@ -3065,7 +3134,7 @@ again:
           if (bb_vinfo)
 	    break;
 
-	  if (gather)
+	  if (gather || simd_lane_access)
 	    free_data_ref (dr);
           return false;
 	}
@@ -3086,7 +3155,7 @@ again:
 	  if (bb_vinfo)
 	    break;
 
-	  if (gather)
+	  if (gather || simd_lane_access)
 	    free_data_ref (dr);
 	  return false;
 	}
@@ -3221,12 +3290,17 @@ again:
           if (bb_vinfo)
 	    break;
 
-	  if (gather)
+	  if (gather || simd_lane_access)
 	    free_data_ref (dr);
           return false;
         }
 
       STMT_VINFO_DATA_REF (stmt_info) = dr;
+      if (simd_lane_access)
+	{
+	  STMT_VINFO_SIMD_LANE_ACCESS_P (stmt_info) = true;
+	  datarefs[i] = dr;
+	}
 
       /* Set vectype for STMT.  */
       scalar_type = TREE_TYPE (DR_REF (dr));
@@ -3247,7 +3321,7 @@ again:
           if (bb_vinfo)
 	    break;
 
-	  if (gather)
+	  if (gather || simd_lane_access)
 	    {
 	      STMT_VINFO_DATA_REF (stmt_info) = NULL;
 	      free_data_ref (dr);