Improve ptrace-error detection on Linux targets

In Fedora GDB, we carry the following patch:

  https://src.fedoraproject.org/rpms/gdb/blob/8ac06474ff1e2aa4920d14e0666b083eeaca8952/f/gdb-attach-fail-reasons-5of5.patch

Its purpose is to try to detect a specific scenario where SELinux's
'deny_ptrace' option is enabled, which prevents GDB from ptrace'ing in
order to debug the inferior (PTRACE_ATTACH and PTRACE_TRACEME will
fail with EACCES in this case).

I like the idea of improving error detection and providing more
information to the user (a simple "Permission denied" can be really
frustrating), but I don't fully agree with the way the patch was
implemented: it makes GDB link against libselinux only for the sake of
consulting the 'deny_ptrace' setting, and then prints a warning if
ptrace failed and this setting is on.

My first thought (and attempt) was to make GDB print a generic warning
when a ptrace error happened; this message would just point the user
to our documentation, where she could find more information about
possible causes for the error (and try to diagnose/fix the problem).
This proved to be too simple, and I was convinced that it is actually
a good idea to go the extra kilometre and try to pinpoint the specific
problem (or problems) preventing ptrace from working, as well as
provide useful suggestions on how the user can fix things.

Here is the patch I came up with.  It implements a new function,
'linux_ptrace_restricted_fail_reason', which does a few things to
check what's wrong with ptrace:

  - It dlopen's "libselinux.so.1" and checks if the "deny_ptrace"
    option is enabled.

  - It reads the contents of "/proc/sys/kernel/yama/ptrace_scope" and
    checks if it's different than 0.

For each of these checks, if it succeeds, the user will see a message
informing about the restriction in place, and how it can be disabled.
For example, if "deny_ptrace" is enabled, the user will see:

  # gdb /usr/bin/true
  ...
  Starting program: /usr/bin/true
  warning: Could not trace the inferior process.
  warning: ptrace: Permission denied
  The SELinux 'deny_ptrace' option is enabled and preventing GDB
  from using 'ptrace'.  You can disable it by executing (as root):

    setsebool deny_ptrace off

  If you are debugging the inferior remotely, the ptrace restriction(s) need
  to be disabled in the target system (e.g., where GDBserver is running).
  During startup program exited with code 127.
  (gdb)

In case "/proc/sys/kernel/yama/ptrace_scope" is > 0:

  # gdb /usr/bin/true
  ...
  Starting program: /usr/bin/true
  warning: Could not trace the inferior process.
  warning: ptrace: Operation not permitted
  The Linux kernel's Yama ptrace scope is in effect, which can prevent
  GDB from using 'ptrace'.  You can disable it by executing (as root):

    echo 0 > /proc/sys/kernel/yama/ptrace_scope

  If you are debugging the inferior remotely, the ptrace restriction(s) need
  to be disabled in the target system (e.g., where GDBserver is running).
  During startup program exited with code 127.
  (gdb)

If both restrictions are enabled, both messages will show up.

This works for gdbserver as well, and actually fixes a latent bug I
found: when ptrace is restricted, gdbserver would hang due to an
unchecked ptrace call:

  # gdbserver :9988 /usr/bin/true
  gdbserver: linux_ptrace_test_ret_to_nx: Cannot PTRACE_TRACEME: Operation not permitted
  gdbserver: linux_ptrace_test_ret_to_nx: status 256 is not WIFSTOPPED!
  gdbserver: linux_ptrace_test_ret_to_nx: failed to kill child pid 2668100 No such process
  [ Here you would have to issue a C-c ]

Now, you will see:

  # gdbserver :9988 /usr/bin/true
  gdbserver: linux_ptrace_test_ret_to_nx: Cannot PTRACE_TRACEME: Permission denied
  gdbserver: linux_ptrace_test_ret_to_nx: status 256 is not WIFSTOPPED!
  gdbserver: linux_ptrace_test_ret_to_nx: failed to kill child pid 2766868 No such process
  gdbserver: Could not trace the inferior process.
  gdbserver: ptrace: Permission denied
  The SELinux 'deny_ptrace' option is enabled and preventing GDB
  from using 'ptrace'.  You can disable it by executing (as root):

    setsebool deny_ptrace off

  If you are debugging the inferior remotely, the ptrace restriction(s) need
  to be disabled in the target system (e.g., where GDBserver is running).
  #

(I decided to keep all the other messages, even though I find them a
bit distracting).

If GDB can't determine the cause for the failure, it will still print
the generic error message which tells the user to check our
documentation:

  There might be restrictions preventing ptrace from working.  Please see
  the appendix "Linux kernel ptrace restrictions" in the GDB documentation
  for more details.
  If you are debugging the inferior remotely, the ptrace restriction(s) need
  to be disabled in the target system (e.g., where GDBserver is running).

This means that the patch expands our documentation and creates a new
appendix section named "Linux kernel ptrace restrictions", with
sub-sections for each possible restriction that might be in place.

Notice how, on every message, we instruct the user to "do the right
thing" if gdbserver is being used.  This is because if the user
started gdbserver *before* any ptrace restriction was in place, and
then, for some reason, one or more restrictions get enabled, then the
error message will be displayed both on gdbserver *and* on the
connected GDB.  Since the user will be piloting GDB, it's important to
explicitly say that the ptrace restrictions are enabled in the target,
where gdbserver is running.

The current list of possible restrictions is:

  - SELinux's 'deny_ptrace' option (detected).

  - YAMA's /proc/sys/kernel/yama/ptrace_scope setting (detected).

  - seccomp on Docker containers (I couldn't find how to detect).

It's important to mention that all of this is Linux-specific; as far
as I know, SELinux, YAMA and seccomp are Linux-only features.

I tested this patch locally, on my Fedora 30 machine (actually, a
Fedora Rawhide VM), but I'm not proposing a testcase for it because of
the difficulty of writing one.

WDYT?

gdb/doc/ChangeLog:
2019-09-26  Sergio Durigan Junior  <sergiodj@redhat.com>

	* gdb.texinfo (Linux kernel ptrace restrictions): New appendix
	section.

gdb/ChangeLog:
2019-09-26  Sergio Durigan Junior  <sergiodj@redhat.com>
	    Jan Kratochvil  <jan.kratochvil@redhat.com>
	    Pedro Alves  <palves@redhat.com>

	* gdbsupport/gdb-dlfcn.h (gdb_dlopen): Update comment and
	mention that the function throws an error.
	* inf-ptrace.c (default_inf_ptrace_me_fail_reason): New
	function.
	(inf_ptrace_me_fail_reason): New variable.
	(inf_ptrace_me): Update call to 'trace_start_error_with_name'.
	* inf-ptrace.h (inf_ptrace_me_fail_reason): New variable.
	* linux-nat.c (attach_proc_task_lwp_callback): Call
	'linux_ptrace_attach_fail_reason_lwp'.
	(linux_nat_target::attach): Update call to
	'linux_ptrace_attach_fail_reason'.
	(_initialize_linux_nat): Set 'inf_ptrace_me_fail_reason'.
	* nat/fork-inferior.c (trace_start_error_with_name): Add
	optional 'append' argument.
	* nat/fork-inferior.h (trace_start_error_with_name): Update
	prototype.
	* nat/linux-ptrace.c: Include "gdbsupport/gdb-dlfcn.h",
	"gdbsupport/filestuff.h" and "nat/fork-inferior.h".
	(selinux_ftype): New typedef.
	(linux_ptrace_restricted_fail_reason): New function.
	(linux_ptrace_attach_fail_reason_1): New function.
	(linux_ptrace_attach_fail_reason): Change first argument type
	from 'ptid_t' to 'pid_t'.  Call
	'linux_ptrace_attach_fail_reason_1' and
	'linux_ptrace_restricted_fail_reason'.
	(linux_ptrace_attach_fail_reason_lwp): New function.
	(linux_ptrace_me_fail_reason): New function.
	(errno_pipe): New variable.
	(linux_fork_to_function): Initialize pipe before forking.
	(linux_child_function): Deal with errno-passing from child.
	Handle ptrace error.
	(linux_check_child_ptrace_errno): New function.
	(linux_check_child_ptrace_errno): Call
	'linux_check_child_ptrace_errno'.
	* nat/linux-ptrace.h (linux_ptrace_attach_fail_reason): Update
	prototype.
	(linux_ptrace_attach_fail_reason_lwp): New prototype.
	(linux_ptrace_me_fail_reason): New prototype.
	* remote.c (extended_remote_target::attach): Handle error
	message passed by the server when attach fails.

gdb/gdbserver/ChangeLog:
2019-09-26  Sergio Durigan Junior  <sergiodj@redhat.com>
	    Pedro Alves  <palves@redhat.com>

	* linux-low.c (linux_ptrace_fun): Call
	'linux_ptrace_me_fail_reason'.
	(attach_proc_task_lwp_callback): Call
	'linux_ptrace_attach_fail_reason_lwp'.
	(linux_attach): Call 'linux_ptrace_attach_fail_reason'.
	* server.c (handle_v_attach): Use try..catch when calling
	'attach_inferior', and send an error message to the client
	when needed.
	* thread-db.c (attach_thread): Call
	'linux_ptrace_attach_fail_reason_lwp'.

2 files changed, 148 insertions, 0 deletions

diff --git a/gdb/doc/ChangeLog b/gdb/doc/ChangeLog
index 0a10fa3..003c8c2 100644
--- a/gdb/doc/ChangeLog
+++ b/gdb/doc/ChangeLog
@@ -1,3 +1,8 @@
+2019-09-26  Sergio Durigan Junior  <sergiodj@redhat.com>
+
+	* gdb.texinfo (Linux kernel ptrace restrictions): New appendix
+	section.
+
 2019-09-20  Ulrich Weigand  <uweigand@de.ibm.com>
 
 	* doc/gdb.texinfo (Remote Configuration): Remove documentation for
diff --git a/gdb/doc/gdb.texinfo b/gdb/doc/gdb.texinfo
index f2713c0..e7b5b18 100644
--- a/gdb/doc/gdb.texinfo
+++ b/gdb/doc/gdb.texinfo
@@ -182,6 +182,9 @@ software in general.  We will miss him.
                                 @value{GDBN}
 * Operating System Information:: Getting additional information from
                                  the operating system
+* Linux kernel ptrace restrictions::        Restrictions sometimes
+                                            imposed by the Linux
+                                            kernel on @code{ptrace}
 * Trace File Format::		GDB trace file format
 * Index Section Format::        .gdb_index section format
 * Man Pages::			Manual pages
@@ -44665,6 +44668,146 @@ should contain a comma-separated list of cores that this process
 is running on.  Target may provide additional columns,
 which @value{GDBN} currently ignores.
 
+@node Linux kernel ptrace restrictions
+@appendix Linux kernel @code{ptrace} restrictions
+@cindex linux kernel ptrace restrictions, attach
+
+The @code{ptrace} system call is used by @value{GDBN} and
+@code{gdbserver} on GNU/Linux to, among other things, attach to a new
+or existing inferior in order to start debugging it.  Due to security
+concerns, some distributions and vendors disable or severely restrict
+the ability to perform these operations, which can make @value{GDBN}
+or @code{gdbserver} malfunction.  In this section, we will expand on
+how this malfunction can manifest itself, and how to modify the
+system's settings in order to be able to use @value{GDBN} and
+@code{gdbserver} properly.
+
+@menu
+* The error message::                   The error message displayed when the
+                                        system prevents @value{GDBN}
+                                        or @code{gdbserver} from using
+                                        @code{ptrace}
+* SELinux's deny_ptrace::               SELinux and the @code{deny_ptrace} option
+* Yama's ptrace_scope::                 Yama and the @code{ptrace_scope} setting
+* Docker and seccomp::                  Docker and the @code{seccomp}
+                                        infrastructure
+@end menu
+
+@node The error message
+@appendixsection The error message
+
+When the system prevents @value{GDBN} or @code{gdbserver} from using
+the @code{ptrace} system call, you will likely see a descriptive error
+message explaining what is wrong and how to attempt to fix the
+problem.  For example, when SELinux's @code{deny_ptrace} option is
+enabled, you can see:
+
+@smallexample
+$ gdb program
+...
+(@value{GDBP}) run
+Starting program: program
+warning: Could not trace the inferior process.
+Error:
+warning: ptrace: Permission denied
+The SELinux 'deny_ptrace' option is enabled and preventing @value{GDBN}
+from using 'ptrace'.  You can disable it by executing (as root):
+
+  setsebool deny_ptrace off
+
+If you are debugging the inferior remotely, the instruction(s) above must
+be performed in the target system (e.g., where GDBserver is running).
+During startup program exited with code 127.
+(@value{GDBP})
+@end smallexample
+
+Sometimes, it may not be possible to acquire the necessary data to
+determine the root cause of the failure.  In this case, you will see a
+generic error message pointing you to this section:
+
+@smallexample
+$ gdb program
+...
+Starting program: program
+warning: Could not trace the inferior process.
+Error:
+warning: ptrace: Permission denied
+There might be restrictions preventing ptrace from working.  Please see
+the appendix "Linux kernel ptrace restrictions" in the GDB documentation
+for more details.
+During startup program exited with code 127.
+(@value{GDBP})
+@end smallexample
+
+@node SELinux's deny_ptrace
+@appendixsection SELinux's @code{deny_ptrace}
+@cindex SELinux
+@cindex deny_ptrace
+
+If you are using SELinux, you might want to check whether the
+@code{deny_ptrace} option is enabled by doing:
+
+@smallexample
+$ getsebool deny_ptrace
+deny_ptrace --> on
+@end smallexample
+
+If the option is enabled, you can disable it by doing, as root:
+
+@smallexample
+# setsebool deny_ptrace off
+@end smallexample
+
+The option will be disabled until the next reboot.  If you would like
+to disable it permanently, you can do (as root):
+
+@smallexample
+# setsebool -P deny_ptrace off
+@end smallexample
+
+@node Yama's ptrace_scope
+@appendixsection Yama's @code{ptrace_scope}
+@cindex yama, ptrace_scope
+
+If your system has Yama enabled, you might want to check whether the
+@code{ptrace_scope} setting is enabled by checking the value of
+@file{/proc/sys/kernel/yama/ptrace_scope}:
+
+@smallexample
+$ cat /proc/sys/kernel/yama/ptrace_scope
+0
+@end smallexample
+
+If you see anything other than @code{0}, @value{GDBN} or
+@code{gdbserver} can be affected by it.  You can temporarily disable
+the feature by doing, as root:
+
+@smallexample
+# sysctl kernel.yama.ptrace_scope=0
+kernel.yama.ptrace_scope = 0
+@end smallexample
+
+You can make this permanent by doing, as root:
+
+@smallexample
+# sysctl -w kernel.yama.ptrace_scope=0
+kernel.yama.ptrace_scope = 0
+@end smallexample
+
+@node Docker and seccomp
+@appendixsection Docker and @code{seccomp}
+@cindex docker, seccomp
+
+If you are using Docker (@uref{https://www.docker.com/}) containers,
+you will probably have to disable its @code{seccomp} protections in
+order to be able to use @value{GDBN} or @code{gdbserver}.  To do that,
+you can use the options @code{--cap-add=SYS_PTRACE --security-opt
+seccomp=unconfined} when invoking Docker:
+
+@smallexample
+$ docker run --cap-add=SYS_PTRACE --security-opt seccomp=unconfined
+@end smallexample
+
 @node Trace File Format
 @appendix Trace File Format
 @cindex trace file format