[gdb/tdep] Fix avx512 -m32 support in gdbserver

PR27257 reports a problem that can be reproduced as follows:
- use x86_64 machine with avx512 support
- compile a hello world with -m32 to a.out
- start a gdbserver session with a.out
- use gdb to connect to the gdbserver session

This makes us run into:
...
Listening on port 2346
Remote debugging from host ::1, port 34940
src/gdbserver/regcache.cc:257: \
  A problem internal to GDBserver has been detected.
Unknown register zmm16h requested
...

The problem is that i387_xsave_to_cache in gdbserver/i387-fp.cc can't find a
register zmm16h in the register cache.

To understand how this happens, first some background.

SSE has 16 128-bit wide xmm registers.

AVX extends the SSE registers set as follows:
- it extends the 16 existing 128-bit wide xmm registers to 256-bit wide ymm
  registers.

AVX512 extends the AVX register set as follows:
- it extends the 16 existing 256-bit wide ymm registers to 512-bit wide zmm
  registers.
- it adds 16 additional 512-bit wide zmm registers (with corresponding ymm and
  xmm subregisters added as well)

However, in 32-bit mode, there are only 8 xmm/ymm/zmm registers.

The problem we're running into is that gdbserver/i387-fp.cc uses these
constants to describe the size of the register file:
...
static const int num_avx512_zmmh_low_registers = 16;
static const int num_avx512_zmmh_high_registers = 16;
static const int num_avx512_ymmh_registers = 16;
static const int num_avx512_xmm_registers = 16;
...
which are all incorrect for the 32-bit case.

Fix this by replacing the constants with variables that have the appropriate
values in 64-bit and 32-bit mode.

Tested on x86_64-linux with native and unix/-m32.

0 files changed, 0 insertions, 0 deletions