gdb: Start supporting AVX instruction

This patch introduces the information needed to properly identify the
VEX prefix, used to signal an AVX and AVX2 instruction, and introduces
a helper function to handle all AVX instruction, instead of adding to
the 3000 line long recording function.

This new function will temporarily set the current thread as "not
executing" so that it can read from pseudo registers as we record, since
most AVX/AVX2 instructions would benefit from recording ymm registers.

The new helper also handles unsupported instructions so that the largest
part of the i386_process_record doesn't have to be shifted by 2 spaces,
which made an unreadably big patch file.

The only expected difference to the end user added by this patch is a
small change to the unsupported message. This patch also updates the
test gdb.reverse/step-precsave.exp, by recognizing the new output.

As a note for the future, we don't handle xmm16-31 and ymm16-31 because
those require the EVEX prefix, meaning avx512 support.

Approved-By: Tom Tromey <tom@tromey.com>

1 files changed, 1 insertions, 0 deletions

diff --git a/gdb/i386-tdep.h b/gdb/i386-tdep.h
index 82676c2..9df8611 100644
--- a/gdb/i386-tdep.h
+++ b/gdb/i386-tdep.h
@@ -325,6 +325,7 @@ enum record_i386_regnum
   X86_RECORD_ES_REGNUM,
   X86_RECORD_FS_REGNUM,
   X86_RECORD_GS_REGNUM,
+  X86_RECORD_XMM0_REGNUM,
 };
 
 #define I386_NUM_GREGS	16