wide-int: Fix estimation of buffer sizes for wide_int printing [PR111800]

As mentioned in the PR, my estimations on needed buffer size for wide_int
and especially widest_int printing were incorrect, I've used get_len ()
in the estimations, but that is true only for !wi::neg_p (x) values.
Under the hood, we have 3 ways to print numbers.
print_decs which if
  if ((wi.get_precision () <= HOST_BITS_PER_WIDE_INT)
      || (wi.get_len () == 1))
uses sprintf which always fits into WIDE_INT_PRINT_BUFFER_SIZE (positive or
negative) and otherwise uses print_hex,
print_decu which if
  if ((wi.get_precision () <= HOST_BITS_PER_WIDE_INT)
      || (wi.get_len () == 1 && !wi::neg_p (wi)))
uses sprintf which always fits into WIDE_INT_PRINT_BUFFER_SIZE (positive
only) and print_hex, which doesn't print most significant limbs which are
zero and the first limb which is non-zero prints such that redundant 0
hex digits aren't printed, while all limbs below that are printed with
"%016" PRIx64.  For wi::neg_p (x) values, the first limb of the precision
is always non-zero, so we print all the limbs for the precision.
So, the current estimations are accurate if !wi::neg_p (x), or when
print_decs will be used and x.get_len () == 1, otherwise we need to use
estimation based on get_precision () rather than get_len ().

I've introduced new inlines print_{dec{,s,u},hex}_buf_size which compute the
needed buffer length in bytes and return true if WIDE_INT_PRINT_BUFFER_SIZE
isn't sufficient and caller should XALLOCAVEC the buffer.

2023-10-15  Jakub Jelinek  <jakub@redhat.com>

	PR tree-optimization/111800
gcc/
	* wide-int-print.h (print_dec_buf_size, print_decs_buf_size,
	print_decu_buf_size, print_hex_buf_size): New inline functions.
	* wide-int.cc (assert_deceq): Use print_dec_buf_size.
	(assert_hexeq): Use print_hex_buf_size.
	* wide-int-print.cc (print_decs): Use print_decs_buf_size.
	(print_decu): Use print_decu_buf_size.
	(print_hex): Use print_hex_buf_size.
	(pp_wide_int_large): Use print_dec_buf_size.
	* value-range.cc (irange_bitmask::dump): Use print_hex_buf_size.
	* value-range-pretty-print.cc (vrange_printer::print_irange_bitmasks):
	Likewise.
	* tree-ssa-loop-niter.cc (do_warn_aggressive_loop_optimizations): Use
	print_dec_buf_size.  Use TYPE_SIGN macro in print_dec call argument.
gcc/c-family/
	* c-warn.cc (match_case_to_enum_1): Assert w.get_precision ()
	is smaller or equal to WIDE_INT_MAX_INL_PRECISION rather than
	w.get_len () is smaller or equal to WIDE_INT_MAX_INL_ELTS.

1 files changed, 4 insertions, 5 deletions

diff --git a/gcc/value-range.cc b/gcc/value-range.cc
index 333245e..f507ec5 100644
--- a/gcc/value-range.cc
+++ b/gcc/value-range.cc
@@ -251,11 +251,10 @@ irange_bitmask::dump (FILE *file) const
   pp_needs_newline (&buffer) = true;
   buffer.buffer->stream = file;
   pp_string (&buffer, "MASK ");
-  unsigned len_mask = m_mask.get_len ();
-  unsigned len_val = m_value.get_len ();
-  unsigned len = MAX (len_mask, len_val);
-  if (len > WIDE_INT_MAX_INL_ELTS)
-    p = XALLOCAVEC (char, len * HOST_BITS_PER_WIDE_INT / 4 + 4);
+  unsigned len_mask, len_val;
+  if (print_hex_buf_size (m_mask, &len_mask)
+      | print_hex_buf_size (m_value, &len_val))
+    p = XALLOCAVEC (char, MAX (len_mask, len_val));
   else
     p = buf;
   print_hex (m_mask, p);