c-family: Fix up -Wsign-compare BIT_NOT_EXPR handling [PR107465]

The following patch fixes multiple bugs in warn_for_sign_compare related to
the BIT_NOT_EXPR related warnings.
My understanding is that what those 3 warnings are meant to warn (since 1995
apparently) is the case where we have BIT_NOT_EXPR of a zero-extended
value, so in result_type the value is something like:
0b11111111XXXXXXXX (e.g. ~ of a 8->16 bit zero extension)
0b000000000000000011111111XXXXXXXX (e.g. ~ of a 8->16 bit zero extension
then zero extended to 32 bits)
0b111111111111111111111111XXXXXXXX (e.g. ~ of a 8->16 bit zero extension
then sign extended to 32 bits)
and the intention of the warning is to warn when this is compared against
something that has some 0 bits at the place where the above has guaranteed
1 bits, either ensured through comparison against constant where we know
the bits exactly, or through zero extension from some narrower type where
again we know at least some upper bits are zero extended.
The bugs in the warning code are:
1) misunderstanding of the {,c_common_}get_narrower APIs - the unsignedp
   it sets is only meaningful if the function actually returns something
   narrower (in that case it says whether the narrower value is then
   sign (0) or zero (1) extended to the originally passed value.
   Though op0 or op1 at this point might be already narrower than
   result_type, and if the function doesn't return anything narrower,
   it all depends on whether the passed in op{0,1} had TYPE_UNSIGNED
   type or not
2) the code didn't check at all whether the BIT_NOT_EXPR operand
   was actually zero extended (i.e. that it was narrower and unsignedp
   was set to 1 for it), all it did is check that unsignedp from the
   call was 1.  But that isn't well defined thing, if the argument
   is returned as is, the function sets unsignedp to 0, but if there
   is e.g. a useless cast to the same or compatible type in between,
   it can return 1 if the cast is unsigned; now, if BIT_NOT_EXPR
   operand is not zero extended, we know nothing at all about any bits
   in the operand containing BIT_NOT_EXPR, so there is nothing to warn
   about
3) the code was actually testing both operands after calling
   c_common_get_narrower on them and on the one with BIT_NOT_EXPR
   again for constants; I think that is just wrong in case the BIT_NOT_EXPR
   operand wouldn't be fully folded, the warning makes sense only if the
   other operand not having BIT_NOT_EXPR in it is constant
4) as can be seen from the above bit pattern examples, the upper bits above
   (in the patch arg0) aren't always all 1s, there could be some zero extension
   above it and from it one would have 0s, so that needs to be taken into
   account for the choice which constant bits to test for being always set
   otherwise warning is emitted, or for the zero extension guaranteed zero
   bits
5) the patch also simplifies the handling, we only do it if one but not
   both operands are BIT_NOT_EXPR after first {,c_common_}get_narrower,
   so we can just use std::swap to ensure it is the first one
6) the code compared bits against HOST_BITS_PER_LONG, which made sense
   back in 1995 when the values were stored into long, but now that they
   are HOST_WIDE_INT should test HOST_BITS_PER_WIDE_INT (or we could rewrite
   the stuff to wide_int, not done in the patch)

2023-03-04  Jakub Jelinek  <jakub@redhat.com>

	PR c/107465
	* c-warn.cc (warn_for_sign_compare): If c_common_get_narrower
	doesn't return a narrower result, use TYPE_UNSIGNED to set unsignedp0
	and unsignedp1.  For the one BIT_NOT_EXPR case vs. one without,
	only check for constant in the non-BIT_NOT_EXPR operand, use std::swap
	to simplify the code, only warn if BIT_NOT_EXPR operand is extended
	from narrower unsigned, fix up computation of mask for the constant
	cases and for unsigned other operand case handle differently
	BIT_NOT_EXPR result being sign vs. zero extended.

	* c-c++-common/Wsign-compare-2.c: New test.
	* c-c++-common/pr107465.c: New test.

2 files changed, 128 insertions, 0 deletions

diff --git a/gcc/testsuite/c-c++-common/Wsign-compare-2.c b/gcc/testsuite/c-c++-common/Wsign-compare-2.c
new file mode 100644
index 0000000..23933d0
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/Wsign-compare-2.c
@@ -0,0 +1,106 @@
+/* { dg-do compile { target { ilp32 || lp64 } } } */
+/* { dg-options "-Wsign-compare" } */
+
+int
+f1 (unsigned char x)
+{
+  return (unsigned short) (~(unsigned short) x) == 0;		/* { dg-warning "promoted bitwise complement of an unsigned value is always nonzero" "" { target c } } */
+}
+
+int
+f2 (unsigned char x)
+{
+  return (unsigned short) (~(unsigned short) x) == 5;		/* { dg-warning "comparison of promoted bitwise complement of an unsigned value with constant" "" { target c }  } */
+}
+
+int
+f3 (unsigned char x)
+{
+  return (unsigned int) (~(unsigned short) x) == 0xffff0005U;	/* { dg-warning "comparison of promoted bitwise complement of an unsigned value with constant" } */
+}
+
+int
+f4 (unsigned char x)
+{
+  return (unsigned int) (~(unsigned short) x) == 0xffff0005ULL;	/* { dg-warning "comparison of promoted bitwise complement of an unsigned value with constant" } */
+}
+
+int
+f5 (unsigned char x)
+{
+  return (unsigned int) (~(unsigned short) x) == 0xffffff05U;	/* { dg-bogus "comparison of promoted bitwise complement of an unsigned value with constant" } */
+}
+
+int
+f6 (unsigned char x)
+{
+  return (unsigned int) (~(unsigned short) x) == 0xffffff05ULL;	/* { dg-bogus "comparison of promoted bitwise complement of an unsigned value with constant" } */
+}
+
+int
+f7 (unsigned char x)
+{
+  return (unsigned long long) (~(unsigned short) x) == 0xffffffffffffff05ULL;	/* { dg-bogus "comparison of promoted bitwise complement of an unsigned value with constant" } */
+}
+
+typedef unsigned short T;
+
+int
+f8 (T x)
+{
+  return (unsigned short) (~(unsigned short) x) == 0;		/* { dg-bogus "promoted bitwise complement of an unsigned value is always nonzero" } */
+}
+
+int
+f9 (T x)
+{
+  return (unsigned short) (~(unsigned short) x) == 5;		/* { dg-bogus "comparison of promoted bitwise complement of an unsigned value with constant" } */
+}
+
+int
+f10 (T x, unsigned char y)
+{
+  return (unsigned short) (~(unsigned short) x) == y;		/* { dg-bogus "comparison of promoted bitwise complement of an unsigned value with unsigned" } */
+}
+
+int
+f11 (T x, unsigned char y)
+{
+  return (unsigned short) (~(unsigned short) x) == y;		/* { dg-bogus "comparison of promoted bitwise complement of an unsigned value with unsigned" } */
+}
+
+int
+f12 (unsigned char x, unsigned char y)
+{
+  return (unsigned short) (~(unsigned short) x) == y;		/* { dg-warning "comparison of promoted bitwise complement of an unsigned value with unsigned" "" { target c } } */
+}
+
+int
+f13 (unsigned char x, unsigned char y)
+{
+  return (unsigned short) (~(unsigned short) x) == y;		/* { dg-warning "comparison of promoted bitwise complement of an unsigned value with unsigned" "" { target c } } */
+}
+
+int
+f14 (unsigned char x, unsigned int y)
+{
+  return (unsigned long long) (~x) == y;			/* { dg-warning "comparison of promoted bitwise complement of an unsigned value with unsigned" } */
+}
+
+int
+f15 (unsigned short x, unsigned int y)
+{
+  return (long long) (~x) == y;					/* { dg-warning "comparison of promoted bitwise complement of an unsigned value with unsigned" } */
+}
+
+int
+f16 (unsigned char x, unsigned short y)
+{
+  return (unsigned short) (~(unsigned short) x) == y;		/* { dg-bogus "comparison of promoted bitwise complement of an unsigned value with unsigned" } */
+}
+
+int
+f17 (unsigned char x, unsigned short y)
+{
+  return (unsigned short) (~(unsigned short) x) == y;		/* { dg-bogus "comparison of promoted bitwise complement of an unsigned value with unsigned" } */
+}
diff --git a/gcc/testsuite/c-c++-common/pr107465.c b/gcc/testsuite/c-c++-common/pr107465.c
new file mode 100644
index 0000000..7c9ff24
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/pr107465.c
@@ -0,0 +1,22 @@
+/* PR c/107465 */
+/* { dg-do compile } */
+/* { dg-options "-Wsign-compare" } */
+
+void baz (void);
+typedef unsigned short T;
+
+#if __SIZEOF_SHORT__ * __CHAR_BIT__ == 16
+void
+foo (unsigned short x)
+{
+  if (!(x ^ 0xFFFF))
+    baz ();
+}
+
+void
+bar (T x)
+{
+  if (!(x ^ 0xFFFF))	/* { dg-bogus "promoted bitwise complement of an unsigned value is always nonzero" } */
+    baz ();
+}
+#endif