PR111778, PowerPC: Do not depend on an undefined shift

I was building a cross compiler to PowerPC on my x86_86 workstation with the
latest version of GCC on October 11th.  I could not build the compiler on the
x86_64 system as it died in building libgcc.  I looked into it, and I
discovered the compiler was recursing until it ran out of stack space.  If I
build a native compiler with the same sources on a PowerPC system, it builds
fine.

I traced this down to a change made around October 10th:

| commit 8f1a70a4fbcc6441c70da60d4ef6db1e5635e18a (HEAD)
| Author: Jiufu Guo <guojiufu@linux.ibm.com>
| Date:   Tue Jan 10 20:52:33 2023 +0800
|
|   rs6000: build constant via li/lis;rldicl/rldicr
|
|   If a constant is possible left/right cleaned on a rotated value from
|   a negative value of "li/lis".  Then, using "li/lis ; rldicl/rldicr"
|   to build the constant.

The code was doing a -1 << 64 which is undefined behavior because different
machines produce different results.  On the x86_64 system, (-1 << 64) produces
-1 while on a PowerPC 64-bit system, (-1 << 64) produces 0.  The x86_64 then
recurses until the stack runs out of space.

If I apply this patch, the compiler builds fine on both x86_64 as a PowerPC
crosss compiler and on a native PowerPC system.

2023-10-12  Michael Meissner  <meissner@linux.ibm.com>

gcc/

	PR target/111778
	* config/rs6000/rs6000.cc (can_be_built_by_li_lis_and_rldicl): Protect
	code from shifts that are undefined.
	(can_be_built_by_li_lis_and_rldicr): Likewise.
	(can_be_built_by_li_and_rldic): Protect code from shifts that
	undefined.  Also replace uses of 1ULL with HOST_WIDE_INT_1U.

1 files changed, 26 insertions, 3 deletions

diff --git a/gcc/config/rs6000/rs6000.cc b/gcc/config/rs6000/rs6000.cc
index 2828f01..cc24dd5 100644
--- a/gcc/config/rs6000/rs6000.cc
+++ b/gcc/config/rs6000/rs6000.cc
@@ -10370,6 +10370,11 @@ can_be_built_by_li_lis_and_rldicl (HOST_WIDE_INT c, int *shift,
   /* Leading zeros may be cleaned by rldicl with a mask.  Change leading zeros
      to ones and then recheck it.  */
   int lz = clz_hwi (c);
+
+  /* If lz == 0, the left shift is undefined.  */
+  if (!lz)
+    return false;
+
   HOST_WIDE_INT unmask_c
     = c | (HOST_WIDE_INT_M1U << (HOST_BITS_PER_WIDE_INT - lz));
   int n;
@@ -10398,6 +10403,11 @@ can_be_built_by_li_lis_and_rldicr (HOST_WIDE_INT c, int *shift,
   /* Tailing zeros may be cleaned by rldicr with a mask.  Change tailing zeros
      to ones and then recheck it.  */
   int tz = ctz_hwi (c);
+
+  /* If tz == HOST_BITS_PER_WIDE_INT, the left shift is undefined.  */
+  if (tz >= HOST_BITS_PER_WIDE_INT)
+    return false;
+
   HOST_WIDE_INT unmask_c = c | ((HOST_WIDE_INT_1U << tz) - 1);
   int n;
   if (can_be_rotated_to_lowbits (~unmask_c, 15, &n)
@@ -10428,8 +10438,15 @@ can_be_built_by_li_and_rldic (HOST_WIDE_INT c, int *shift, HOST_WIDE_INT *mask)
      right bits are shifted as 0's, and left 1's(and x's) are cleaned.  */
   int tz = ctz_hwi (c);
   int lz = clz_hwi (c);
+
+  /* If lz == HOST_BITS_PER_WIDE_INT, the left shift is undefined.  */
+  if (lz >= HOST_BITS_PER_WIDE_INT)
+    return false;
+
   int middle_ones = clz_hwi (~(c << lz));
-  if (tz + lz + middle_ones >= ones)
+  if (tz + lz + middle_ones >= ones
+      && (tz - lz) < HOST_BITS_PER_WIDE_INT
+      && tz < HOST_BITS_PER_WIDE_INT)
     {
       *mask = ((1LL << (HOST_BITS_PER_WIDE_INT - tz - lz)) - 1LL) << tz;
       *shift = tz;
@@ -10440,7 +10457,8 @@ can_be_built_by_li_and_rldic (HOST_WIDE_INT c, int *shift, HOST_WIDE_INT *mask)
   int leading_ones = clz_hwi (~c);
   int tailing_ones = ctz_hwi (~c);
   int middle_zeros = ctz_hwi (c >> tailing_ones);
-  if (leading_ones + tailing_ones + middle_zeros >= ones)
+  if (leading_ones + tailing_ones + middle_zeros >= ones
+      && middle_zeros < HOST_BITS_PER_WIDE_INT)
     {
       *mask = ~(((1ULL << middle_zeros) - 1ULL) << tailing_ones);
       *shift = tailing_ones + middle_zeros;
@@ -10450,10 +10468,15 @@ can_be_built_by_li_and_rldic (HOST_WIDE_INT c, int *shift, HOST_WIDE_INT *mask)
   /* xx1..1xx: --> xx0..01..1xx: some 1's(following x's) are cleaned. */
   /* Get the position for the first bit of successive 1.
      The 24th bit would be in successive 0 or 1.  */
-  HOST_WIDE_INT low_mask = (1LL << 24) - 1LL;
+  HOST_WIDE_INT low_mask = (HOST_WIDE_INT_1U << 24) - HOST_WIDE_INT_1U;
   int pos_first_1 = ((c & (low_mask + 1)) == 0)
 		      ? clz_hwi (c & low_mask)
 		      : HOST_BITS_PER_WIDE_INT - ctz_hwi (~(c | low_mask));
+
+  /* Make sure the left and right shifts are defined.  */
+  if (!IN_RANGE (pos_first_1, 1, HOST_BITS_PER_WIDE_INT-1))
+    return false;
+
   middle_ones = clz_hwi (~c << pos_first_1);
   middle_zeros = ctz_hwi (c >> (HOST_BITS_PER_WIDE_INT - pos_first_1));
   if (pos_first_1 < HOST_BITS_PER_WIDE_INT