gdb: correct endbr64 instruction handling in amd64_analyze_prologue

Compilers can put a sequence aligning the stack at the entry of a
function.  However with -fcf-protection enabled, "endbr64" is
generated before.  Current implementation of amd64 prologue analyzer
first checks for stack alignment and then for "endbr64", which is not
correct.  This behavior was introduced with patch "gdb: handle endbr64
instruction in amd64_analyze_prologue".  In case both are generated,
prologue will not be skipped.  This patch swaps the order so that
"endbr64" is checked first and adds a regression test.  i386-tdep
implementation also already had those checked in the correct order,
that is stack alignment is after endbr64.

Given such source compiled with gcc 11.4.0 via:
gcc -O0 main.c -o main

```
	#include <alloca.h>

	void
	foo (int id)
	{
	  volatile __attribute__ ((__aligned__ (64))) int a;
	  volatile char *p = (char *) alloca (id * 12);
	  p[2] = 'b';
	}

	int
	main (int argc, char **argv)
	{
	  foo (argc + 1);
	  return 1;
	}
```

we get such function entry for foo (generated with objdump -d):
```
0000000000001149 <foo>:
    1149:       f3 0f 1e fa             endbr64
    114d:       4c 8d 54 24 08          lea    0x8(%rsp),%r10
    1152:       48 83 e4 c0             and    $0xffffffffffffffc0,%rsp
    1156:       41 ff 72 f8             push   -0x8(%r10)
    115a:       55                      push   %rbp
    115b:       48 89 e5                mov    %rsp,%rbp
    115e:       41 52                   push   %r10
    1160:       48 81 ec a8 00 00 00    sub    $0xa8,%rsp
    1167:       89 7d 8c                mov    %edi,-0x74(%rbp)
...
```

The 3 instructions following endbr64 align the stack.  If we were to set
a breakpoint on foo, gdb would set it at function's entry:
```
(gdb) b foo
Breakpoint 1 at 0x1149
(gdb) r
...
Breakpoint 1, 0x0000555555555149 in foo ()
(gdb) disassemble
Dump of assembler code for function foo:
=> 0x0000555555555149 <+0>:     endbr64
   0x000055555555514d <+4>:     lea    0x8(%rsp),%r10
   0x0000555555555152 <+9>:     and    $0xffffffffffffffc0,%rsp
   0x0000555555555156 <+13>:    push   -0x8(%r10)
   0x000055555555515a <+17>:    push   %rbp
   0x000055555555515b <+18>:    mov    %rsp,%rbp
   0x000055555555515e <+21>:    push   %r10
   0x0000555555555160 <+23>:    sub    $0xa8,%rsp
   0x0000555555555167 <+30>:    mov    %edi,-0x74(%rbp)
...
```

With this patch fixing the order of checked instructions, gdb can
properly analyze the prologue:
```
(gdb) b foo
Breakpoint 1 at 0x115e
(gdb) r
...
Breakpoint 1, 0x000055555555515e in foo ()
(gdb) disassemble
Dump of assembler code for function foo:
   0x0000555555555149 <+0>:     endbr64
   0x000055555555514d <+4>:     lea    0x8(%rsp),%r10
   0x0000555555555152 <+9>:     and    $0xffffffffffffffc0,%rsp
   0x0000555555555156 <+13>:    push   -0x8(%r10)
   0x000055555555515a <+17>:    push   %rbp
   0x000055555555515b <+18>:    mov    %rsp,%rbp
=> 0x000055555555515e <+21>:    push   %r10
   0x0000555555555160 <+23>:    sub    $0xa8,%rsp
   0x0000555555555167 <+30>:    mov    %edi,-0x74(%rbp)
...
```

Approved-By: Andrew Burgess <aburgess@redhat.com>

0 files changed, 0 insertions, 0 deletions