diff options
author | Richard Biener <rguenther@suse.de> | 2024-03-01 09:29:32 +0100 |
---|---|---|
committer | Richard Biener <rguenther@suse.de> | 2024-05-10 15:04:48 +0200 |
commit | 52d4691294c84793b301ad3cc24e277b8c7efe0b (patch) | |
tree | 88d7d72b2e90cc936c95f1aa749d787c34940e08 /gcc/tree-vect-loop.cc | |
parent | af64af69c3cc85dbe00c520651a54850bf5cadc1 (diff) | |
download | gcc-52d4691294c84793b301ad3cc24e277b8c7efe0b.zip gcc-52d4691294c84793b301ad3cc24e277b8c7efe0b.tar.gz gcc-52d4691294c84793b301ad3cc24e277b8c7efe0b.tar.bz2 |
Allow patterns in SLP reductions
The following removes the over-broad rejection of patterns for SLP
reductions which is done by removing them from LOOP_VINFO_REDUCTIONS
during pattern detection. That's also insufficient in case the
pattern only appears on the reduction path. Instead this implements
the proper correctness check in vectorizable_reduction and guides
SLP discovery to heuristically avoid forming later invalid groups.
I also couldn't find any testcase that FAILs when allowing the SLP
reductions to form so I've added one.
I came across this for single-lane SLP reductions with the all-SLP
work where we rely on patterns to properly vectorize COND_EXPR
reductions.
* tree-vect-patterns.cc (vect_pattern_recog_1): Do not
remove reductions involving patterns.
* tree-vect-loop.cc (vectorizable_reduction): Reject SLP
reduction groups with multiple lane-reducing reductions.
* tree-vect-slp.cc (vect_analyze_slp_instance): When discovering
SLP reduction groups avoid including lane-reducing ones.
* gcc.dg/vect/vect-reduc-sad-9.c: New testcase.
Diffstat (limited to 'gcc/tree-vect-loop.cc')
-rw-r--r-- | gcc/tree-vect-loop.cc | 15 |
1 files changed, 15 insertions, 0 deletions
diff --git a/gcc/tree-vect-loop.cc b/gcc/tree-vect-loop.cc index 704df7b..361aec0 100644 --- a/gcc/tree-vect-loop.cc +++ b/gcc/tree-vect-loop.cc @@ -7667,6 +7667,21 @@ vectorizable_reduction (loop_vec_info loop_vinfo, return false; } + /* Lane-reducing ops also never can be used in a SLP reduction group + since we'll mix lanes belonging to different reductions. But it's + OK to use them in a reduction chain or when the reduction group + has just one element. */ + if (lane_reduc_code_p + && slp_node + && !REDUC_GROUP_FIRST_ELEMENT (stmt_info) + && SLP_TREE_LANES (slp_node) > 1) + { + if (dump_enabled_p ()) + dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location, + "lane-reducing reduction in reduction group.\n"); + return false; + } + /* All uses but the last are expected to be defined in the loop. The last use is the reduction variable. In case of nested cycle this assumption is not true: we use reduc_index to record the index of the |