vect: Make reduction code handle calls

This patch extends the reduction code to handle calls.  So far
it's a structural change only; a later patch adds support for
specific function reductions.

Most of the patch consists of using code_helper and gimple_match_op
to describe the reduction operations.  The other main change is that
vectorizable_call now needs to handle fully-predicated reductions.

There are some new functions that are provided for ABI completeness
and aren't currently used:

  first_commutative_argument
  commutative_ternary_op_p
  1- and 3-argument forms of gimple_build

gcc/
	* builtins.h (associated_internal_fn): Declare overload that
	takes a (combined_cfn, return type) pair.
	* builtins.c (associated_internal_fn): Split new overload out
	of original fndecl version.  Also provide an overload that takes
	a (combined_cfn, return type) pair.
	* internal-fn.h (commutative_binary_fn_p): Declare.
	(commutative_ternary_fn_p): Likewise.
	(associative_binary_fn_p): Likewise.
	* internal-fn.c (commutative_binary_fn_p, commutative_ternary_fn_p):
	New functions, split out from...
	(first_commutative_argument): ...here.
	(associative_binary_fn_p): New function.
	* gimple-match.h (code_helper): Add a constructor that takes
	internal functions.
	(commutative_binary_op_p): Declare.
	(commutative_ternary_op_p): Likewise.
	(first_commutative_argument): Likewise.
	(associative_binary_op_p): Likewise.
	(canonicalize_code): Likewise.
	(directly_supported_p): Likewise.
	(get_conditional_internal_fn): Likewise.
	(gimple_build): New overloads that takes a code_helper.
	* gimple-fold.c (gimple_build): Likewise.
	* gimple-match-head.c (commutative_binary_op_p): New function.
	(commutative_ternary_op_p): Likewise.
	(first_commutative_argument): Likewise.
	(associative_binary_op_p): Likewise.
	(canonicalize_code): Likewise.
	(directly_supported_p): Likewise.
	(get_conditional_internal_fn): Likewise.
	* tree-vectorizer.h: Include gimple-match.h.
	(neutral_op_for_reduction): Take a code_helper instead of a tree_code.
	(needs_fold_left_reduction_p): Likewise.
	(reduction_fn_for_scalar_code): Likewise.
	(vect_can_vectorize_without_simd_p): Declare a nNew overload that takes
	a code_helper.
	* tree-vect-loop.c: Include case-cfn-macros.h.
	(fold_left_reduction_fn): Take a code_helper instead of a tree_code.
	(reduction_fn_for_scalar_code): Likewise.
	(neutral_op_for_reduction): Likewise.
	(needs_fold_left_reduction_p): Likewise.
	(use_mask_by_cond_expr_p): Likewise.
	(build_vect_cond_expr): Likewise.
	(vect_create_partial_epilog): Likewise.  Use gimple_build rather
	than gimple_build_assign.
	(check_reduction_path): Handle calls and operate on code_helpers
	rather than tree_codes.
	(vect_is_simple_reduction): Likewise.
	(vect_model_reduction_cost): Likewise.
	(vect_find_reusable_accumulator): Likewise.
	(vect_create_epilog_for_reduction): Likewise.
	(vect_transform_cycle_phi): Likewise.
	(vectorizable_reduction): Likewise.  Make more use of
	lane_reduc_code_p.
	(vect_transform_reduction): Use gimple_extract_op but expect
	a tree_code for now.
	(vect_can_vectorize_without_simd_p): New overload that takes
	a code_helper.
	* tree-vect-stmts.c (vectorizable_call): Handle reductions in
	fully-masked loops.
	* tree-vect-patterns.c (vect_mark_pattern_stmts): Use
	gimple_extract_op when updating STMT_VINFO_REDUC_IDX.

1 files changed, 107 insertions, 0 deletions

diff --git a/gcc/gimple-match-head.c b/gcc/gimple-match-head.c
index 73d9c5c..c481a62 100644
--- a/gcc/gimple-match-head.c
+++ b/gcc/gimple-match-head.c
@@ -1267,3 +1267,110 @@ optimize_successive_divisions_p (tree divisor, tree inner_div)
     }
   return true;
 }
+
+/* Return a canonical form for CODE when operating on TYPE.  The idea
+   is to remove redundant ways of representing the same operation so
+   that code_helpers can be hashed and compared for equality.
+
+   The only current canonicalization is to replace built-in functions
+   with internal functions, in cases where internal-fn.def defines
+   such an internal function.
+
+   Note that the new code_helper cannot necessarily be used in place of
+   the original code_helper.  For example, the new code_helper might be
+   an internal function that the target does not support.  */
+
+code_helper
+canonicalize_code (code_helper code, tree type)
+{
+  if (code.is_fn_code ())
+    return associated_internal_fn (combined_fn (code), type);
+  return code;
+}
+
+/* Return true if CODE is a binary operation and if CODE is commutative when
+   operating on type TYPE.  */
+
+bool
+commutative_binary_op_p (code_helper code, tree type)
+{
+  if (code.is_tree_code ())
+    return commutative_tree_code (tree_code (code));
+  auto cfn = combined_fn (code);
+  return commutative_binary_fn_p (associated_internal_fn (cfn, type));
+}
+
+/* Return true if CODE represents a ternary operation and if the first two
+   operands are commutative when CODE is operating on TYPE.  */
+
+bool
+commutative_ternary_op_p (code_helper code, tree type)
+{
+  if (code.is_tree_code ())
+    return commutative_ternary_tree_code (tree_code (code));
+  auto cfn = combined_fn (code);
+  return commutative_ternary_fn_p (associated_internal_fn (cfn, type));
+}
+
+/* If CODE is commutative in two consecutive operands, return the
+   index of the first, otherwise return -1.  */
+
+int
+first_commutative_argument (code_helper code, tree type)
+{
+  if (code.is_tree_code ())
+    {
+      auto tcode = tree_code (code);
+      if (commutative_tree_code (tcode)
+	  || commutative_ternary_tree_code (tcode))
+	return 0;
+      return -1;
+    }
+  auto cfn = combined_fn (code);
+  return first_commutative_argument (associated_internal_fn (cfn, type));
+}
+
+/* Return true if CODE is a binary operation that is associative when
+   operating on type TYPE.  */
+
+bool
+associative_binary_op_p (code_helper code, tree type)
+{
+  if (code.is_tree_code ())
+    return associative_tree_code (tree_code (code));
+  auto cfn = combined_fn (code);
+  return associative_binary_fn_p (associated_internal_fn (cfn, type));
+}
+
+/* Return true if the target directly supports operation CODE on type TYPE.
+   QUERY_TYPE acts as for optab_for_tree_code.  */
+
+bool
+directly_supported_p (code_helper code, tree type, optab_subtype query_type)
+{
+  if (code.is_tree_code ())
+    {
+      direct_optab optab = optab_for_tree_code (tree_code (code), type,
+						query_type);
+      return (optab != unknown_optab
+	      && optab_handler (optab, TYPE_MODE (type)) != CODE_FOR_nothing);
+    }
+  gcc_assert (query_type == optab_default
+	      || (query_type == optab_vector && VECTOR_TYPE_P (type))
+	      || (query_type == optab_scalar && !VECTOR_TYPE_P (type)));
+  internal_fn ifn = associated_internal_fn (combined_fn (code), type);
+  return (direct_internal_fn_p (ifn)
+	  && direct_internal_fn_supported_p (ifn, type, OPTIMIZE_FOR_SPEED));
+}
+
+/* A wrapper around the internal-fn.c versions of get_conditional_internal_fn
+   for a code_helper CODE operating on type TYPE.  */
+
+internal_fn
+get_conditional_internal_fn (code_helper code, tree type)
+{
+  if (code.is_tree_code ())
+    return get_conditional_internal_fn (tree_code (code));
+  auto cfn = combined_fn (code);
+  return get_conditional_internal_fn (associated_internal_fn (cfn, type));
+}