reassoc: Fix up optimize_range_tests_to_bit_test [PR118915]

The following testcase is miscompiled due to a bug in
optimize_range_tests_to_bit_test.  It is trying to optimize
check for a in [-34,-34] or [-26,-26] or [-6,-6] or [-4,inf] ranges.
Another reassoc optimization folds the the test for the first
two ranges into (a + 34U) & ~8U in [0U,0U] range, and extract_bit_test_mask
actually has code to virtually undo it and treat that again as test
for a being -34 or -26.  The problem is that optimize_range_tests_to_bit_test
remembers in the type variable TREE_TYPE (ranges[i].exp); from the first
range.  If extract_bit_test_mask doesn't do that virtual undoing of the
BIT_AND_EXPR handling, that is just fine, the returned exp is ranges[i].exp.
But if the first range is BIT_AND_EXPR, the type could be different, the
BIT_AND_EXPR form has the optional cast to corresponding unsigned type
in order to avoid introducing UB.  Now, type was used to fill in the
max value if ranges[j].high was missing in subsequently tested range,
and so in this particular testcase the [-4,inf] range which was
signed int and so [-4,INT_MAX] was treated as [-4,UINT_MAX] instead.
And we were subtracting values of 2 different types and trying to make
sense out of that.

The following patch fixes this by using the type of the low bound
(which is always non-NULL) for the max value of the high bound instead.

2025-02-24  Jakub Jelinek  <jakub@redhat.com>

	PR tree-optimization/118915
	* tree-ssa-reassoc.cc (optimize_range_tests_to_bit_test): For
	highj == NULL_TREE use TYPE_MAX_VALUE (TREE_TYPE (lowj)) rather
	than TYPE_MAX_VALUE (type).

	* gcc.c-torture/execute/pr118915.c: New test.

2 files changed, 23 insertions, 1 deletions

diff --git a/gcc/testsuite/gcc.c-torture/execute/pr118915.c b/gcc/testsuite/gcc.c-torture/execute/pr118915.c
new file mode 100644
index 0000000..2e0cb71
--- /dev/null
+++ b/gcc/testsuite/gcc.c-torture/execute/pr118915.c
@@ -0,0 +1,22 @@
+/* PR tree-optimization/118915 */
+
+int a;
+
+int
+foo (int c, int d, int e, int f)
+{
+  if (!d || !e)
+    return -22;
+  if (c > 16)
+    return -22;
+  if (!f)
+    return -22;
+  return 2;
+}
+
+int
+main ()
+{
+  if (foo (a + 21, a + 6, a + 34, a + 26) != -22)
+    __builtin_abort ();
+}
diff --git a/gcc/tree-ssa-reassoc.cc b/gcc/tree-ssa-reassoc.cc
index 4a0dfa7..4017eea 100644
--- a/gcc/tree-ssa-reassoc.cc
+++ b/gcc/tree-ssa-reassoc.cc
@@ -3362,7 +3362,7 @@ optimize_range_tests_to_bit_test (enum tree_code opcode, int first, int length,
 	    continue;
 	  highj = ranges[j].high;
 	  if (highj == NULL_TREE)
-	    highj = TYPE_MAX_VALUE (type);
+	    highj = TYPE_MAX_VALUE (TREE_TYPE (lowj));
 	  wide_int mask2;
 	  exp2 = extract_bit_test_mask (ranges[j].exp, prec, lowi, lowj,
 					highj, &mask2, NULL);