Unify Solaris procfs and largefile handling

GDB currently doesn't build on 32-bit Solaris:

* On Solaris 11.4/x86:

In file included from /usr/include/sys/procfs.h:26,
                 from /vol/src/gnu/gdb/hg/master/dist/gdb/i386-sol2-nat.c:24:
/usr/include/sys/old_procfs.h:31:2: error: #error "Cannot use procfs in the large file compilation environment"
 #error "Cannot use procfs in the large file compilation environment"
  ^~~~~

* On Solaris 11.3/x86 there are several more instances of this.

The interaction between procfs and large-file support historically has
been a royal mess on Solaris:

* There are two versions of the procfs interface:

** The old ioctl-based /proc, deprecated and not used any longer in
   either gdb or binutils.

** The `new' (introduced in Solaris 2.6, 1997) structured /proc.

* There are two headers one can possibly include:

** <procfs.h> which only provides the structured /proc, definining
   _STRUCTURED_PROC=1 and then including ...

** <sys/procfs.h> which defaults to _STRUCTURED_PROC=0, the ioctl-based
   /proc, but provides structured /proc if _STRUCTURED_PROC == 1.

* procfs and the large-file environment didn't go well together:

** Until Solaris 11.3, <sys/procfs.h> would always #error in 32-bit
   compilations when the large-file environment was active
   (_FILE_OFFSET_BITS == 64).

** In both Solaris 11.4 and Illumos, this restriction was lifted for
   structured /proc.

So one has to be careful always to define _STRUCTURED_PROC=1 when
testing for or using <sys/procfs.h> on Solaris.  As the errors above
show, this isn't always the case in binutils-gdb right now.

Also one may need to disable large-file support for 32-bit compilations
on Solaris.  config/largefile.m4 meant to do this by wrapping the
AC_SYS_LARGEFILE autoconf macro with appropriate checks, yielding
ACX_LARGEFILE.  Unfortunately the macro doesn't always succeed because
it neglects the _STRUCTURED_PROC part.

To make things even worse, since GCC 9 g++ predefines
_FILE_OFFSET_BITS=64 on Solaris.  So even if largefile.m4 deciced not to
enable large-file support, this has no effect, breaking the gdb build.

This patch addresses all this as follows:

* All tests for the <sys/procfs.h> header are made with
  _STRUCTURED_PROC=1, the definition going into the various config.h
  files instead of having to make them (and sometimes failing) in the
  affected sources.

* To cope with the g++ predefine of _FILE_OFFSET_BITS=64,
  -U_FILE_OFFSET_BITS is added to various *_CPPFLAGS variables.  It had
  been far easier to have just

  #undef _FILE_OFFSET_BITS

  in config.h, but unfortunately such a construct in config.in is
  commented by config.status irrespective of indentation and whitespace
  if large-file support is disabled.  I found no way around this and
  putting the #undef in several global headers for bfd, binutils, ld,
  and gdb seemed way more invasive.

* Last, the applicability check in largefile.m4 was modified only to
  disable largefile support if really needed.  To do so, it checks if
  <sys/procfs.h> compiles with _FILE_OFFSET_BITS=64 defined.  If it
  doesn't, the disabling only happens if gdb exists in-tree and isn't
  disabled, otherwise (building binutils from a tarball), there's no
  conflict.

  What initially confused me was the check for $plugins here, which
  originally caused the disabling not to take place.  Since AC_PLUGINGS
  does enable plugin support if <dlfcn.h> exists (which it does on
  Solaris), the disabling never happened.

  I could find no explanation why the linker plugin needs large-file
  support but thought it would be enough if gld and GCC's lto-plugin
  agreed on the _FILE_OFFSET_BITS value.  Unfortunately, that's not
  enough: lto-plugin uses the simple-object interface from libiberty,
  which includes off_t arguments.  So to fully disable large-file
  support would mean also disabling it in libiberty and its users: gcc
  and libstdc++-v3.  This seems highly undesirable, so I decided to
  disable the linker plugin instead if large-file support won't work.

The patch allows binutils+gdb to build on i386-pc-solaris2.11 (both
Solaris 11.3 and 11.4, using GCC 9.3.0 which is the worst case due to
predefined _FILE_OFFSET_BITS=64).  Also regtested on
amd64-pc-solaris2.11 (again on Solaris 11.3 and 11.4),
x86_64-pc-linux-gnu and i686-pc-linux-gnu.

	config:
	* largefile.m4 (ACX_LARGEFILE) <sparc-*-solaris*|i?86-*-solaris*>:
	Check for <sys/procfs.h> incompatilibity with large-file support
	on Solaris.
	Only disable large-file support and perhaps plugins if needed.
	Set, substitute LARGEFILE_CPPFLAGS if so.

	bfd:
	* bfd.m4 (BFD_SYS_PROCFS_H): New macro.
	(BFD_HAVE_SYS_PROCFS_TYPE): Require BFD_SYS_PROCFS_H.
	Don't define _STRUCTURED_PROC.
	(BFD_HAVE_SYS_PROCFS_TYPE_MEMBER): Likewise.
	* elf.c [HAVE_SYS_PROCFS_H] (_STRUCTURED_PROC): Don't define.
	* configure.ac: Use BFD_SYS_PROCFS_H to check for <sys/procfs.h>.
	* configure, config.in: Regenerate.
	* Makefile.am (AM_CPPFLAGS): Add LARGEFILE_CPPFLAGS.
	* Makefile.in, doc/Makefile.in: Regenerate.

	binutils:
	* Makefile.am (AM_CPPFLAGS): Add LARGEFILE_CPPFLAGS.
	* Makefile.in, doc/Makefile.in: Regenerate.
	* configure: Regenerate.

	gas:
	* Makefile.am (AM_CPPFLAGS): Add LARGEFILE_CPPFLAGS.
	* Makefile.in, doc/Makefile.in: Regenerate.
	* configure: Regenerate.

	gdb:
	* proc-api.c (_STRUCTURED_PROC): Don't define.
	* proc-events.c: Likewise.
	* proc-flags.c: Likewise.
	* proc-why.c: Likewise.
	* procfs.c: Likewise.

	* Makefile.in (INTERNAL_CPPFLAGS): Add LARGEFILE_CPPFLAGS.
	* configure, config.in: Regenerate.

	gdbserver:
	* configure, config.in: Regenerate.

	gdbsupport:
	* Makefile.am (AM_CPPFLAGS): Add LARGEFILE_CPPFLAGS.
	* common.m4 (GDB_AC_COMMON): Use BFD_SYS_PROCFS_H to check for
	<sys/procfs.h>.
	* Makefile.in: Regenerate.
	* configure, config.in: Regenerate.

	gnulib:
	* configure.ac: Run ACX_LARGEFILE before gl_EARLY.
	* configure: Regenerate.

	gprof:
	* Makefile.am (AM_CPPFLAGS): Add LARGEFILE_CPPFLAGS.
	* Makefile.in: Regenerate.
	* configure: Regenerate.

	ld:
	* Makefile.am (AM_CPPFLAGS): Add LARGEFILE_CPPFLAGS.
	* Makefile.in: Regenerate.
	* configure: Regenerate.

9 files changed, 99 insertions, 45 deletions

diff --git a/gdb/ChangeLog b/gdb/ChangeLog
index a61a573..7e1d5b9 100644
--- a/gdb/ChangeLog
+++ b/gdb/ChangeLog
@@ -1,3 +1,14 @@
+2020-07-30  Rainer Orth  <ro@CeBiTec.Uni-Bielefeld.DE>
+
+	* proc-api.c (_STRUCTURED_PROC): Don't define.
+	* proc-events.c: Likewise.
+	* proc-flags.c: Likewise.
+	* proc-why.c: Likewise.
+	* procfs.c: Likewise.
+
+	* Makefile.in (INTERNAL_CPPFLAGS): Add LARGEFILE_CPPFLAGS.
+	* configure, config.in: Regenerate.
+
 2020-07-30  Tom de Vries  <tdevries@suse.de>
 
 	PR build/26320
diff --git a/gdb/Makefile.in b/gdb/Makefile.in
index f9bde2b..0d6d813 100644
--- a/gdb/Makefile.in
+++ b/gdb/Makefile.in
@@ -589,7 +589,8 @@ CPPFLAGS = @CPPFLAGS@
 # are sometimes a little generic, we think that the risk of collision
 # with other header files is high.  If that happens, we try to mitigate
 # a bit the consequences by putting the Python includes last in the list.
-INTERNAL_CPPFLAGS = $(CPPFLAGS) @GUILE_CPPFLAGS@ @PYTHON_CPPFLAGS@
+INTERNAL_CPPFLAGS = $(CPPFLAGS) @GUILE_CPPFLAGS@ @PYTHON_CPPFLAGS@ \
+	@LARGEFILE_CPPFLAGS@
 
 # INTERNAL_CFLAGS is the aggregate of all other *CFLAGS macros.
 INTERNAL_CFLAGS_BASE = \
diff --git a/gdb/config.in b/gdb/config.in
index 1aeb860..9755458 100644
--- a/gdb/config.in
+++ b/gdb/config.in
@@ -799,6 +799,9 @@
 /* Define to 1 if you need to in order for `stat' and other things to work. */
 #undef _POSIX_SOURCE
 
+/* Use structured /proc on Solaris. */
+#undef _STRUCTURED_PROC
+
 /* Define to empty if `const' does not conform to ANSI C. */
 #undef const
 
diff --git a/gdb/configure b/gdb/configure
index 5ef85e8..e7811e8 100755
--- a/gdb/configure
+++ b/gdb/configure
@@ -795,6 +795,7 @@ HAVE_CXX11
 INSTALL_STRIP_PROGRAM
 STRIP
 install_sh
+LARGEFILE_CPPFLAGS
 target_os
 target_vendor
 target_cpu
@@ -4769,15 +4770,56 @@ fi
 
 
 case "${host}" in
-  sparc-*-solaris*|i[3-7]86-*-solaris*)
-    # On native 32bit sparc and ia32 solaris, large-file and procfs support
-    # are mutually exclusive; and without procfs support, the bfd/ elf module
-    # cannot provide certain routines such as elfcore_write_prpsinfo
-    # or elfcore_write_prstatus.  So unless the user explicitly requested
-    # large-file support through the --enable-largefile switch, disable
-    # large-file support in favor of procfs support.
-    test "${target}" = "${host}" -a "x$plugins" = xno \
-      && : ${enable_largefile="no"}
+  sparc-*-solaris*|i?86-*-solaris*)
+    # On native 32-bit Solaris/SPARC and x86, large-file and procfs support
+    # were mutually exclusive until Solaris 11.3.  Without procfs support,
+    # the bfd/ elf module cannot provide certain routines such as
+    # elfcore_write_prpsinfo or elfcore_write_prstatus.  So unless the user
+    # explicitly requested large-file support through the
+    # --enable-largefile switch, disable large-file support in favor of
+    # procfs support.
+    #
+    # Check if <sys/procfs.h> is incompatible with large-file support.
+    cat confdefs.h - <<_ACEOF >conftest.$ac_ext
+/* end confdefs.h.  */
+#define _FILE_OFFSET_BITS 64
+#define _STRUCTURED_PROC 1
+#include <sys/procfs.h>
+int
+main ()
+{
+
+  ;
+  return 0;
+}
+_ACEOF
+if ac_fn_c_try_compile "$LINENO"; then :
+  acx_cv_procfs_lfs=yes
+else
+  acx_cv_procfs_lfs=no
+fi
+rm -f core conftest.err conftest.$ac_objext conftest.$ac_ext
+    #
+    # Forcefully disable large-file support only if necessary, gdb is in
+    # tree and enabled.
+    if test "${target}" = "${host}" -a "$acx_cv_procfs_lfs" = no \
+         -a -d $srcdir/../gdb -a "$enable_gdb" != no; then
+      : ${enable_largefile="no"}
+      if test "$plugins" = yes; then
+	{ $as_echo "$as_me:${as_lineno-$LINENO}: WARNING:
+plugin support disabled; require large-file support which is incompatible with GDB." >&5
+$as_echo "$as_me: WARNING:
+plugin support disabled; require large-file support which is incompatible with GDB." >&2;}
+	plugins=no
+      fi
+    fi
+    #
+    # Explicitly undef _FILE_OFFSET_BITS if enable_largefile=no for the
+    # benefit of g++ 9+ which predefines it on Solaris.
+    if test "$enable_largefile" = no; then
+      LARGEFILE_CPPFLAGS="-U_FILE_OFFSET_BITS"
+
+    fi
     ;;
 esac
 
@@ -12995,7 +13037,7 @@ $as_echo "#define HAVE_LANGINFO_CODESET 1" >>confdefs.h
   fi
 
 
-  for ac_header in linux/perf_event.h locale.h memory.h signal.h 		   sys/resource.h sys/socket.h 		   sys/un.h sys/wait.h 		   thread_db.h wait.h 		   termios.h 		   dlfcn.h 		   linux/elf.h sys/procfs.h proc_service.h 		   poll.h sys/poll.h sys/select.h
+  for ac_header in linux/perf_event.h locale.h memory.h signal.h 		   sys/resource.h sys/socket.h 		   sys/un.h sys/wait.h 		   thread_db.h wait.h 		   termios.h 		   dlfcn.h 		   linux/elf.h proc_service.h 		   poll.h sys/poll.h sys/select.h
 do :
   as_ac_Header=`$as_echo "ac_cv_header_$ac_header" | $as_tr_sh`
 ac_fn_c_check_header_mongrel "$LINENO" "$ac_header" "$as_ac_Header" "$ac_includes_default"
@@ -14891,8 +14933,24 @@ fi
     fi
   fi
 
+
+$as_echo "#define _STRUCTURED_PROC 1" >>confdefs.h
+
+ for ac_header in sys/procfs.h
+do :
+  ac_fn_c_check_header_mongrel "$LINENO" "sys/procfs.h" "ac_cv_header_sys_procfs_h" "$ac_includes_default"
+if test "x$ac_cv_header_sys_procfs_h" = xyes; then :
+  cat >>confdefs.h <<_ACEOF
+#define HAVE_SYS_PROCFS_H 1
+_ACEOF
+
+fi
+
+done
+
   if test "$ac_cv_header_sys_procfs_h" = yes; then
-    { $as_echo "$as_me:${as_lineno-$LINENO}: checking for gregset_t in sys/procfs.h" >&5
+
+ { $as_echo "$as_me:${as_lineno-$LINENO}: checking for gregset_t in sys/procfs.h" >&5
 $as_echo_n "checking for gregset_t in sys/procfs.h... " >&6; }
  if ${bfd_cv_have_sys_procfs_type_gregset_t+:} false; then :
   $as_echo_n "(cached) " >&6
@@ -14901,8 +14959,6 @@ else
 /* end confdefs.h.  */
 
 #define _SYSCALL32
-/* Needed for new procfs interface on sparc-solaris.  */
-#define _STRUCTURED_PROC 1
 #include <sys/procfs.h>
 int
 main ()
@@ -14929,7 +14985,8 @@ $as_echo "#define HAVE_GREGSET_T 1" >>confdefs.h
  { $as_echo "$as_me:${as_lineno-$LINENO}: result: $bfd_cv_have_sys_procfs_type_gregset_t" >&5
 $as_echo "$bfd_cv_have_sys_procfs_type_gregset_t" >&6; }
 
-    { $as_echo "$as_me:${as_lineno-$LINENO}: checking for fpregset_t in sys/procfs.h" >&5
+
+ { $as_echo "$as_me:${as_lineno-$LINENO}: checking for fpregset_t in sys/procfs.h" >&5
 $as_echo_n "checking for fpregset_t in sys/procfs.h... " >&6; }
  if ${bfd_cv_have_sys_procfs_type_fpregset_t+:} false; then :
   $as_echo_n "(cached) " >&6
@@ -14938,8 +14995,6 @@ else
 /* end confdefs.h.  */
 
 #define _SYSCALL32
-/* Needed for new procfs interface on sparc-solaris.  */
-#define _STRUCTURED_PROC 1
 #include <sys/procfs.h>
 int
 main ()
@@ -14966,7 +15021,8 @@ $as_echo "#define HAVE_FPREGSET_T 1" >>confdefs.h
  { $as_echo "$as_me:${as_lineno-$LINENO}: result: $bfd_cv_have_sys_procfs_type_fpregset_t" >&5
 $as_echo "$bfd_cv_have_sys_procfs_type_fpregset_t" >&6; }
 
-    { $as_echo "$as_me:${as_lineno-$LINENO}: checking for prgregset_t in sys/procfs.h" >&5
+
+ { $as_echo "$as_me:${as_lineno-$LINENO}: checking for prgregset_t in sys/procfs.h" >&5
 $as_echo_n "checking for prgregset_t in sys/procfs.h... " >&6; }
  if ${bfd_cv_have_sys_procfs_type_prgregset_t+:} false; then :
   $as_echo_n "(cached) " >&6
@@ -14975,8 +15031,6 @@ else
 /* end confdefs.h.  */
 
 #define _SYSCALL32
-/* Needed for new procfs interface on sparc-solaris.  */
-#define _STRUCTURED_PROC 1
 #include <sys/procfs.h>
 int
 main ()
@@ -15003,7 +15057,8 @@ $as_echo "#define HAVE_PRGREGSET_T 1" >>confdefs.h
  { $as_echo "$as_me:${as_lineno-$LINENO}: result: $bfd_cv_have_sys_procfs_type_prgregset_t" >&5
 $as_echo "$bfd_cv_have_sys_procfs_type_prgregset_t" >&6; }
 
-    { $as_echo "$as_me:${as_lineno-$LINENO}: checking for prfpregset_t in sys/procfs.h" >&5
+
+ { $as_echo "$as_me:${as_lineno-$LINENO}: checking for prfpregset_t in sys/procfs.h" >&5
 $as_echo_n "checking for prfpregset_t in sys/procfs.h... " >&6; }
  if ${bfd_cv_have_sys_procfs_type_prfpregset_t+:} false; then :
   $as_echo_n "(cached) " >&6
@@ -15012,8 +15067,6 @@ else
 /* end confdefs.h.  */
 
 #define _SYSCALL32
-/* Needed for new procfs interface on sparc-solaris.  */
-#define _STRUCTURED_PROC 1
 #include <sys/procfs.h>
 int
 main ()
@@ -15040,7 +15093,8 @@ $as_echo "#define HAVE_PRFPREGSET_T 1" >>confdefs.h
  { $as_echo "$as_me:${as_lineno-$LINENO}: result: $bfd_cv_have_sys_procfs_type_prfpregset_t" >&5
 $as_echo "$bfd_cv_have_sys_procfs_type_prfpregset_t" >&6; }
 
-    { $as_echo "$as_me:${as_lineno-$LINENO}: checking for prgregset32_t in sys/procfs.h" >&5
+
+ { $as_echo "$as_me:${as_lineno-$LINENO}: checking for prgregset32_t in sys/procfs.h" >&5
 $as_echo_n "checking for prgregset32_t in sys/procfs.h... " >&6; }
  if ${bfd_cv_have_sys_procfs_type_prgregset32_t+:} false; then :
   $as_echo_n "(cached) " >&6
@@ -15049,8 +15103,6 @@ else
 /* end confdefs.h.  */
 
 #define _SYSCALL32
-/* Needed for new procfs interface on sparc-solaris.  */
-#define _STRUCTURED_PROC 1
 #include <sys/procfs.h>
 int
 main ()
@@ -15077,7 +15129,8 @@ $as_echo "#define HAVE_PRGREGSET32_T 1" >>confdefs.h
  { $as_echo "$as_me:${as_lineno-$LINENO}: result: $bfd_cv_have_sys_procfs_type_prgregset32_t" >&5
 $as_echo "$bfd_cv_have_sys_procfs_type_prgregset32_t" >&6; }
 
-    { $as_echo "$as_me:${as_lineno-$LINENO}: checking for lwpid_t in sys/procfs.h" >&5
+
+ { $as_echo "$as_me:${as_lineno-$LINENO}: checking for lwpid_t in sys/procfs.h" >&5
 $as_echo_n "checking for lwpid_t in sys/procfs.h... " >&6; }
  if ${bfd_cv_have_sys_procfs_type_lwpid_t+:} false; then :
   $as_echo_n "(cached) " >&6
@@ -15086,8 +15139,6 @@ else
 /* end confdefs.h.  */
 
 #define _SYSCALL32
-/* Needed for new procfs interface on sparc-solaris.  */
-#define _STRUCTURED_PROC 1
 #include <sys/procfs.h>
 int
 main ()
@@ -15114,7 +15165,8 @@ $as_echo "#define HAVE_LWPID_T 1" >>confdefs.h
  { $as_echo "$as_me:${as_lineno-$LINENO}: result: $bfd_cv_have_sys_procfs_type_lwpid_t" >&5
 $as_echo "$bfd_cv_have_sys_procfs_type_lwpid_t" >&6; }
 
-    { $as_echo "$as_me:${as_lineno-$LINENO}: checking for psaddr_t in sys/procfs.h" >&5
+
+ { $as_echo "$as_me:${as_lineno-$LINENO}: checking for psaddr_t in sys/procfs.h" >&5
 $as_echo_n "checking for psaddr_t in sys/procfs.h... " >&6; }
  if ${bfd_cv_have_sys_procfs_type_psaddr_t+:} false; then :
   $as_echo_n "(cached) " >&6
@@ -15123,8 +15175,6 @@ else
 /* end confdefs.h.  */
 
 #define _SYSCALL32
-/* Needed for new procfs interface on sparc-solaris.  */
-#define _STRUCTURED_PROC 1
 #include <sys/procfs.h>
 int
 main ()
@@ -15151,7 +15201,8 @@ $as_echo "#define HAVE_PSADDR_T 1" >>confdefs.h
  { $as_echo "$as_me:${as_lineno-$LINENO}: result: $bfd_cv_have_sys_procfs_type_psaddr_t" >&5
 $as_echo "$bfd_cv_have_sys_procfs_type_psaddr_t" >&6; }
 
-    { $as_echo "$as_me:${as_lineno-$LINENO}: checking for elf_fpregset_t in sys/procfs.h" >&5
+
+ { $as_echo "$as_me:${as_lineno-$LINENO}: checking for elf_fpregset_t in sys/procfs.h" >&5
 $as_echo_n "checking for elf_fpregset_t in sys/procfs.h... " >&6; }
  if ${bfd_cv_have_sys_procfs_type_elf_fpregset_t+:} false; then :
   $as_echo_n "(cached) " >&6
@@ -15160,8 +15211,6 @@ else
 /* end confdefs.h.  */
 
 #define _SYSCALL32
-/* Needed for new procfs interface on sparc-solaris.  */
-#define _STRUCTURED_PROC 1
 #include <sys/procfs.h>
 int
 main ()
diff --git a/gdb/proc-api.c b/gdb/proc-api.c
index d7ad503..2db4a19 100644
--- a/gdb/proc-api.c
+++ b/gdb/proc-api.c
@@ -28,8 +28,6 @@
 #include "gdbcmd.h"
 #include "completer.h"
 
-#define _STRUCTURED_PROC 1
-
 #include <sys/types.h>
 #include <sys/procfs.h>
 #include <sys/proc.h>	/* for struct proc */
diff --git a/gdb/proc-events.c b/gdb/proc-events.c
index e921c47..1f970b8 100644
--- a/gdb/proc-events.c
+++ b/gdb/proc-events.c
@@ -30,8 +30,6 @@
 
 #include "defs.h"
 
-#define _STRUCTURED_PROC 1
-
 #include <sys/types.h>
 #include <sys/procfs.h>
 #include <sys/syscall.h>
diff --git a/gdb/proc-flags.c b/gdb/proc-flags.c
index de263a4..e51ae08 100644
--- a/gdb/proc-flags.c
+++ b/gdb/proc-flags.c
@@ -27,8 +27,6 @@
 
 #include "defs.h"
 
-#define _STRUCTURED_PROC 1
-
 #include <sys/types.h>
 #include <sys/procfs.h>
 
diff --git a/gdb/proc-why.c b/gdb/proc-why.c
index 8d73538..88bed16 100644
--- a/gdb/proc-why.c
+++ b/gdb/proc-why.c
@@ -20,8 +20,6 @@
 
 #include "defs.h"
 
-#define _STRUCTURED_PROC 1
-
 #include <sys/types.h>
 #include <sys/procfs.h>
 
diff --git a/gdb/procfs.c b/gdb/procfs.c
index d3085a2..b628c2d 100644
--- a/gdb/procfs.c
+++ b/gdb/procfs.c
@@ -33,8 +33,6 @@
 #include "nat/fork-inferior.h"
 #include "gdbarch.h"
 
-#define _STRUCTURED_PROC 1	/* Should be done by configure script.  */
-
 #include <sys/procfs.h>
 #include <sys/fault.h>
 #include <sys/syscall.h>