gmp-utils: protect gdb_mpz exports against out-of-range values

The gdb_mpz class currently provides a couple of methods which
essentially export an mpz_t value into either a buffer, or an integral
type. The export is based on using the mpz_export function which
we discovered can be a bit treacherous if used without caution.

In particular, the initial motivation for this patch was to catch
situations where the mpz_t value was so large that it would not fit
in the destination area. mpz_export does not know the size of
the buffer, and therefore can happily write past the end of our buffer.

While designing a solution to the above problem, I also discovered
that we also needed to be careful when exporting signed numbers.
In particular, numbers which are larger than the maximum value
for a given signed type size, but no so large as to fit in the
*unsigned* version with the same size, would end up being exported
incorrectly. This is related to the fact that mpz_export ignores
the sign of the value being exportd, and assumes an unsigned export.
Thus, for such large values, the appears as if mpz_export is able
to fit our value into our buffer, but in fact, it does not.

Also, I noticed that gdb_mpz::write wasn't taking its unsigned_p
parameter, which was a hole.

For all these reasons, a new low-level private method called
"safe_export" has been added to class gdb_mpz, whose goal is
to perform all necessary checks and manipulations for a safe
and correct export. As a bonus, this method allows us to factorize
the handling of negative value exports.

The gdb_mpz::as_integer and gdb_mpz::write methods are then simplified
to take advantage of this new safe_export method.

gdb/ChangeLog:

        * gmp-utils.h (gdb_mpz::safe_export): New private method.
        (gdb_mpz::as_integer): Reimplement using gdb_mpz::safe_export.
        * gmp-utils.c (gdb_mpz::write): Rewrite using gdb_mpz::safe_export.
        (gdb_mpz::safe_export): New method.
        * unittests/gmp-utils-selftests .c (gdb_mpz_as_integer):
        Update function description.
        (check_as_integer_raises_out_of_range_error): New function.
        (gdb_mpz_as_integer_out_of_range): New function.
        (_initialize_gmp_utils_selftests): Register
        gdb_mpz_as_integer_out_of_range as a selftest.

0 files changed, 0 insertions, 0 deletions