diff options
author | Joel Brobecker <brobecker@adacore.com> | 2020-11-24 06:34:57 +0400 |
---|---|---|
committer | Joel Brobecker <brobecker@adacore.com> | 2020-11-24 06:34:57 +0400 |
commit | 4fbb7ccebe1fdcbae762e8fed6af7a810c81f85c (patch) | |
tree | 052ac508401935e3e1c185ea2136dac5f84630c1 /opcodes/i386-reg.tbl | |
parent | fc23d4728dcc8f88b822a6ecb8236067bd68cf97 (diff) | |
download | gdb-4fbb7ccebe1fdcbae762e8fed6af7a810c81f85c.zip gdb-4fbb7ccebe1fdcbae762e8fed6af7a810c81f85c.tar.gz gdb-4fbb7ccebe1fdcbae762e8fed6af7a810c81f85c.tar.bz2 |
Fix stack smashing error during gdb_mpq_write_fixed_point selftest
When building GDB using Ubuntu 20.04's system libgmp and compiler,
running the "maintenance selftest" command triggers the following error:
| Running selftest gdb_mpq_write_fixed_point.
| *** stack smashing detected ***: terminated
| [1] 1092790 abort (core dumped) ./gdb gdb
This happens while trying to construct an mpq_t object (a rational)
from two integers representing the numerator and denominator.
In our test, the numerator is -8, and the denominator is 1.
The problem was that the rational was constructed using the wrong
function. This is what we were doing prior to this patch:
mpq_set_ui (v.val, numerator, denominator);
The 'u' in "ui" stands for *unsigned*, which is wrong because
numerator and denominator's type is "int".
As a result of the above, instead of getting a rational value of -8,
we get a rational with a very large positive value (gmp_printf
says "18446744073709551608").
From there, the test performs an operation which is expected to
write this value into a buffer which was not dimensioned to fit
such a number, thus leading GMP into a buffer overflow.
This was verified by applying the formula that GMP's documentation
gives for the required memory buffer size needed during export:
| When an application is allocating space itself the required size can
| be determined with a calculation like the following. Since
| mpz_sizeinbase always returns at least 1, count here will be at
| least one, which avoids any portability problems with malloc(0),
| though if z is zero no space at all is actually needed (or written).
|
| numb = 8*size - nail;
| count = (mpz_sizeinbase (z, 2) + numb-1) / numb;
| p = malloc (count * size);
With the very large number, mpz_sizeinbase returns 66 and thus
the malloc size becomes 16 bytes instead of the 8 we allocated.
This patch fixes the issue by using the correct "set" function.
gdb/ChangeLog:
* unittests/gmp-utils-selftests.c (write_fp_test): Use mpq_set_si
instead of mpq_set_ui to initialize our GMP rational.
Diffstat (limited to 'opcodes/i386-reg.tbl')
0 files changed, 0 insertions, 0 deletions