gmp-utils: protect gdb_mpz exports against out-of-range values

The gdb_mpz class currently provides a couple of methods which
essentially export an mpz_t value into either a buffer, or an integral
type. The export is based on using the mpz_export function which
we discovered can be a bit treacherous if used without caution.

In particular, the initial motivation for this patch was to catch
situations where the mpz_t value was so large that it would not fit
in the destination area. mpz_export does not know the size of
the buffer, and therefore can happily write past the end of our buffer.

While designing a solution to the above problem, I also discovered
that we also needed to be careful when exporting signed numbers.
In particular, numbers which are larger than the maximum value
for a given signed type size, but no so large as to fit in the
*unsigned* version with the same size, would end up being exported
incorrectly. This is related to the fact that mpz_export ignores
the sign of the value being exportd, and assumes an unsigned export.
Thus, for such large values, the appears as if mpz_export is able
to fit our value into our buffer, but in fact, it does not.

Also, I noticed that gdb_mpz::write wasn't taking its unsigned_p
parameter, which was a hole.

For all these reasons, a new low-level private method called
"safe_export" has been added to class gdb_mpz, whose goal is
to perform all necessary checks and manipulations for a safe
and correct export. As a bonus, this method allows us to factorize
the handling of negative value exports.

The gdb_mpz::as_integer and gdb_mpz::write methods are then simplified
to take advantage of this new safe_export method.

gdb/ChangeLog:

        * gmp-utils.h (gdb_mpz::safe_export): New private method.
        (gdb_mpz::as_integer): Reimplement using gdb_mpz::safe_export.
        * gmp-utils.c (gdb_mpz::write): Rewrite using gdb_mpz::safe_export.
        (gdb_mpz::safe_export): New method.
        * unittests/gmp-utils-selftests .c (gdb_mpz_as_integer):
        Update function description.
        (check_as_integer_raises_out_of_range_error): New function.
        (gdb_mpz_as_integer_out_of_range): New function.
        (_initialize_gmp_utils_selftests): Register
        gdb_mpz_as_integer_out_of_range as a selftest.

1 files changed, 22 insertions, 18 deletions

diff --git a/gdb/gmp-utils.h b/gdb/gmp-utils.h
index 12e4f8e..dae62cf 100644
--- a/gdb/gmp-utils.h
+++ b/gdb/gmp-utils.h
@@ -121,6 +121,24 @@ private:
 
   /* Helper template for constructor and operator=.  */
   template<typename T> void set (T src);
+
+  /* Low-level function to export VAL into BUF as a number whose byte size
+     is the size of BUF.
+
+     If UNSIGNED_P is true, then export VAL into BUF as an unsigned value.
+     Otherwise, export it as a signed value.
+
+     The API is inspired from GMP's mpz_export, hence the naming and types
+     of the following parameter:
+       - ENDIAN should be:
+           . 1 for most significant byte first; or
+	   . -1 for least significant byte first; or
+	   . 0 for native endianness.
+
+    An error is raised if BUF is not large enough to contain the value
+    being exported.  */
+  void safe_export (gdb::array_view<gdb_byte> buf,
+		    int endian, bool unsigned_p) const;
 };
 
 /* A class to make it easier to use GMP's mpq_t values within GDB.  */
@@ -258,26 +276,12 @@ template<typename T>
 T
 gdb_mpz::as_integer () const
 {
-  /* Initialize RESULT, because mpz_export only write the minimum
-     number of bytes, including none if our value is zero!  */
-  T result = 0;
-
-  gdb_mpz exported_val (val);
-  if (std::is_signed<T>::value && mpz_cmp_ui (val, 0) < 0)
-    {
-      /* We want to use mpz_export to set the return value, but
-	 this function does not handle the sign. So give exported_val
-	 a value which is at the same time positive, and has the same
-	 bit representation as our negative value.  */
-      gdb_mpz neg_offset;
+  T result;
 
-      mpz_ui_pow_ui (neg_offset.val, 2, sizeof (T) * HOST_CHAR_BIT);
-      mpz_add (exported_val.val, exported_val.val, neg_offset.val);
-    }
+  this->safe_export ({(gdb_byte *) &result, sizeof (result)},
+		     0 /* endian (0 = native) */,
+		     !std::is_signed<T>::value /* unsigned_p */);
 
-  mpz_export (&result, NULL /* count */, -1 /* order */,
-	      sizeof (T) /* size */, 0 /* endian (0 = native) */,
-	      0 /* nails */, exported_val.val);
   return result;
 }