gdb: check for multiple matching build-id files

Within the debug-file-directory GDB looks for the existence of a
.build-id directory.

Within the .build-id directory GDB looks for files with the form:

  .build-id/ff/4b4142d62b399499844924d53e33d4028380db.debug

which contain the debug information for the objfile with the build-id
ff4b4142d62b399499844924d53e33d4028380db.

There appear to be two strategies for populating the .build-id
directory.  Ubuntu takes the approach of placing the actual debug
information in this directory, so
4b4142d62b399499844924d53e33d4028380db.debug is an actual file
containing the debug information.

Fedora, RHEL, and SUSE take a slightly different approach, placing the
debug information elsewhere, and then creating symlinks in the
.build-id directory back to the original debug information file.  The
actual debug information is arranged in a mirror of the filesystem
within the debug directory, as an example, if the debug-file-directory
is /usr/lib/debug, then the debug information for /bin/foo can be
found in /usr/lib/debug/bin/foo.debug.

Where this gets interesting is that in some cases a package will
install a single binary with multiple names, in this case a single
binary will be install with either hard-links, or symlinks providing
the alternative names.

The debug information for these multiple binaries will then be placed
into the /usr/lib/debug/ tree, and again, links are created so a
single file can provide debug information for each of the names that
binary presents as.  An example file system might look like this (the
[link] could be symlinks, but are more likely hard-links):

  /bin/
    foo
    bar -> foo	[ HARD LINK ]
    baz -> foo	[ HARD LINK ]
  /usr/
    lib/
      debug/
        bin/
	  foo.debug
	  bar.debug -> foo.debug	[ HARD LINK ]
	  baz.debug -> foo.debug	[ HARD LINK ]

In the .build-id tree though we have a problem.  Do we have a single
entry that links to one of the .debug files?  This would work; a user
debugging any of the binaries will find the debug information based on
the build-id, and will get the correct information, after all the
.debug files are identical (same file linked together).  But there is
one problem with this approach.

Sometimes, for *reasons* it's possible that one or more the linked
binaries might get removed, along with its associated debug
information.  I'm honestly not 100% certain under what circumstances
this can happen, but what I observe is that sometime a single name for
a binary, and its corresponding .debug entry, can be missing.  If this
happens to be the entry that the .build-id link is pointing at, then
we have a problem.  The user can no longer find the debug information
based on the .build-id link.

The solution that Fedora, RHEL, & SUSE have adopted is to add multiple
entries in the .build-id tree, with each entry pointing to a different
name within the debug/ tree, a sequence number is added to the
build-id to distinguish the multiple entries.  Thus, we might end up
with a layout like this:

  /bin/
    foo
    bar -> foo	[ HARD LINK ]
    baz -> foo	[ HARD LINK ]
  /usr/
    lib/
      debug/
        bin/
	  foo.debug
	  bar.debug -> foo.debug	[ HARD LINK ]
	  baz.debug -> foo.debug	[ HARD LINK ]
      .build-id/
        a3/
          4b4142d62b399499844924d53e33d4028380db.debug -> ../../debug/bin/foo.debug	[ SYMLINK ]
          4b4142d62b399499844924d53e33d4028380db.1.debug -> ../../debug/bin/bar.debug	[ SYMLINK ]
          4b4142d62b399499844924d53e33d4028380db.2.debug -> ../../debug/bin/baz.debug	[ SYMLINK ]

With current master GDB, debug information will only ever be looked up
via the 4b4142d62b399499844924d53e33d4028380db.debug link.  But if
'foo' and its corresponding 'foo.debug' are ever removed, then master
GDB will fail to find the debug information.

Ubuntu seems to have a much better approach for debug information
handling; they place the debug information directly into the .build-id
tree, so there only ever needs to be a single entry for any one
build-id.  I wonder if/how they handle the case where multiple names
might share a single .debug file, if one of those names is then
uninstalled, how do they know the .debug file should be retained or
not ... but I assume that problem either doesn't exist or has been
solved.

Anyway, for a while Fedora has carried a patch that handles the
build-id sequence number logic.  What's presented here is inspired by
the Fedora patch, but has some changes to fix some issues.

I'm aware that this is a patch that applies to only some (probably a
minority) of distros.  However, the logic is contained to only a
single function in build-id.c, and isn't too complex, so I'm hoping
that there wont be too many objections.

For distros that don't have build-id sequence numbers there should be
no impact.  The sequence number approach still leaves the first file
without a sequence number, and this is the first file that GDB (after
this patch) checks for.  The new logic only kicks in if the
non-sequence numbered first file exists, but is a symlink to a non
existent file; in this case GDB checks for the sequence numbered files
instead.

Tests are included.

There is a small fix needed for gdb.base/sysroot-debug-lookup.exp,
after this commit GDB now treats a target: sysroot where the target
file system is local to GDB the same as if the sysroot had no target:
prefix.  The consequence of this is that GDB now resolves a symlink
back to the real filename in the sysroot-debug-lookup.exp test where
it didn't previously.  As this behaviour is inline with the case where
there is no target: prefix I think this is fine.

5 files changed, 380 insertions, 8 deletions

diff --git a/gdb/testsuite/gdb.base/build-id-seqno.c b/gdb/testsuite/gdb.base/build-id-seqno.c
new file mode 100644
index 0000000..e2119ba7
--- /dev/null
+++ b/gdb/testsuite/gdb.base/build-id-seqno.c
@@ -0,0 +1,22 @@
+/* This testcase is part of GDB, the GNU debugger.
+
+   Copyright 2024 Free Software Foundation, Inc.
+
+   This program is free software; you can redistribute it and/or modify
+   it under the terms of the GNU General Public License as published by
+   the Free Software Foundation; either version 3 of the License, or
+   (at your option) any later version.
+
+   This program is distributed in the hope that it will be useful,
+   but WITHOUT ANY WARRANTY; without even the implied warranty of
+   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+   GNU General Public License for more details.
+
+   You should have received a copy of the GNU General Public License
+   along with this program.  If not, see <http://www.gnu.org/licenses/>.  */
+
+int
+main (void)
+{
+  return 0;
+}
diff --git a/gdb/testsuite/gdb.base/build-id-seqno.exp b/gdb/testsuite/gdb.base/build-id-seqno.exp
new file mode 100644
index 0000000..fba8a8d
--- /dev/null
+++ b/gdb/testsuite/gdb.base/build-id-seqno.exp
@@ -0,0 +1,133 @@
+# This testcase is part of GDB, the GNU debugger.
+#
+# Copyright 2024 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+# Setup a .build-id/ based debug directory containing multiple entries
+# for the same build-id, with each entry given a different sequence
+# number.
+#
+# Ensure that GDB will scan over broken symlinks for the same build-id
+# (but different sequence number) to find later working symlinks.
+#
+# This test only places debug information on the host, so it is always
+# local to GDB.
+
+require {!is_remote host}
+
+standard_testfile
+
+if {[build_executable "failed to prepare" $testfile $srcfile] == -1} {
+    return -1
+}
+
+# Split out BINFILE.debug.  Remove debug from BINFILE.
+if {[gdb_gnu_strip_debug $binfile] != 0} {
+    return -1
+}
+
+# Get the '.build-id/xx/xxx...xxx' part of the filename.
+set build_id_filename [build_id_debug_filename_get $binfile]
+
+# Hide (rename) BINFILE.debug, this should ensure GDB can't find it
+# directly but needs to look for the build-id based file in the debug
+# directory.
+set hidden_debuginfo [standard_output_file "hidden_$testfile.debug"]
+remote_exec build "mv ${binfile}.debug $hidden_debuginfo"
+
+# A filename that doesn't exist.  Some symlinks will point at this
+# file.
+set missing_debuginfo [host_standard_output_file "missing_debuginfo"]
+
+# Create the debug directory, and the .build-id directory structure
+# within it.
+set debugdir [host_standard_output_file "debug"]
+remote_exec host "mkdir -p $debugdir/[file dirname $build_id_filename]"
+
+set host_hidden_debuginfo [gdb_remote_download host $hidden_debuginfo]
+remote_exec host "ln -fs $host_hidden_debuginfo $debugdir/$build_id_filename"
+
+# Start GDB and load global BINFILE.  If FIND_DEBUGINFO is true then
+# we expect GDB to find the debug information matching BINFILE,
+# otherwise, we expect GDB not to find the debug information.
+proc load_binfile_check_debug_is_found { find_debuginfo testname } {
+    with_test_prefix "$testname" {
+	clean_restart
+
+	gdb_test_no_output "set debug-file-directory $::debugdir" \
+	    "set debug-file-directory"
+
+	gdb_file_cmd $::binfile
+
+	if { $find_debuginfo } {
+	    gdb_assert { [regexp [string_to_regexp \
+				      "Reading symbols from $::hidden_debuginfo..."] \
+			      $::gdb_file_cmd_msg] } \
+		"debuginfo was read via build-id"
+	} else {
+	    gdb_assert { [regexp "\\(No debugging symbols found in \[^\r\n\]+/$::testfile\\)" \
+			      $::gdb_file_cmd_msg] } \
+	    }
+    }
+}
+
+# Return a copy of FILENAME, which should end '.debug', with NUMBER
+# added, e.g. add_seqno 1 "foo.debug" --> "foo.1.debug".
+proc add_seqno { number filename } {
+    return [regsub "\.debug\$" $filename ".${number}.debug"]
+}
+
+load_binfile_check_debug_is_found true \
+    "find debuginfo with a single build-id file"
+
+remote_exec host "ln -fs $host_hidden_debuginfo \
+			$debugdir/[add_seqno 1 $build_id_filename]"
+remote_exec host "ln -fs $host_hidden_debuginfo \
+			$debugdir/[add_seqno 2 $build_id_filename]"
+remote_exec host "ln -fs $host_hidden_debuginfo \
+			$debugdir/[add_seqno 3 $build_id_filename]"
+
+load_binfile_check_debug_is_found true \
+    "find debuginfo with 4 build-id files"
+
+remote_exec host "ln -fs $missing_debuginfo $debugdir/$build_id_filename"
+
+load_binfile_check_debug_is_found true \
+    "find debuginfo, first build-id file is bad"
+
+remote_exec host "ln -fs $missing_debuginfo \
+			$debugdir/[add_seqno 1 $build_id_filename]"
+remote_exec host "ln -fs $missing_debuginfo \
+			$debugdir/[add_seqno 3 $build_id_filename]"
+
+load_binfile_check_debug_is_found true  \
+    "find debuginfo, first 2 build-id files are bad"
+
+remote_exec host "ln -fs $missing_debuginfo \
+			$debugdir/[add_seqno 2 $build_id_filename]"
+
+load_binfile_check_debug_is_found false  \
+    "cannot find debuginfo, all build-id files are bad"
+
+remote_exec host "ln -fs $host_hidden_debuginfo \
+			$debugdir/[add_seqno 3 $build_id_filename]"
+
+load_binfile_check_debug_is_found true  \
+    "find debuginfo, last build-id file is good"
+
+remote_exec host "rm -f $debugdir/[add_seqno 1 $build_id_filename]"
+
+load_binfile_check_debug_is_found false  \
+    "cannot find debuginfo, file with seqno 1 is missing"
diff --git a/gdb/testsuite/gdb.base/sysroot-debug-lookup.exp b/gdb/testsuite/gdb.base/sysroot-debug-lookup.exp
index 5f17315..36f6519 100644
--- a/gdb/testsuite/gdb.base/sysroot-debug-lookup.exp
+++ b/gdb/testsuite/gdb.base/sysroot-debug-lookup.exp
@@ -78,9 +78,8 @@ proc_with_prefix lookup_via_build_id {} {
 	gdb_assert { $::gdb_file_cmd_debug_info eq "debug" } \
 	    "ensure debug information was found"
 
-	if { $sysroot_prefix eq "" } {
-	    set lookup_filename $debug_filename
-	} else {
+	if { $sysroot_prefix eq "target:"
+	     && [target_info gdb_protocol] == "extended-remote"} {
 	    # Only when using the extended-remote board will we have
 	    # started a remote target by this point.  In this case GDB
 	    # will see the 'target:' prefix as remote, and so the
@@ -89,11 +88,9 @@ proc_with_prefix lookup_via_build_id {} {
 	    # In all other cases we will still be using the default,
 	    # initial target, in which case GDB considers the
 	    # 'target:' prefix to indicate the local filesystem.
-	    if {[target_info gdb_protocol] == "extended-remote"} {
-		set lookup_filename $sysroot_prefix$debug_symlink
-	    } else {
-		set lookup_filename $debug_symlink
-	    }
+	    set lookup_filename $sysroot_prefix$debug_symlink
+	} else {
+	    set lookup_filename $debug_filename
 	}
 	set re [string_to_regexp "Reading symbols from $lookup_filename..."]
 	gdb_assert {[regexp $re $::gdb_file_cmd_msg]} \
diff --git a/gdb/testsuite/gdb.server/build-id-seqno.c b/gdb/testsuite/gdb.server/build-id-seqno.c
new file mode 100644
index 0000000..e2119ba7
--- /dev/null
+++ b/gdb/testsuite/gdb.server/build-id-seqno.c
@@ -0,0 +1,22 @@
+/* This testcase is part of GDB, the GNU debugger.
+
+   Copyright 2024 Free Software Foundation, Inc.
+
+   This program is free software; you can redistribute it and/or modify
+   it under the terms of the GNU General Public License as published by
+   the Free Software Foundation; either version 3 of the License, or
+   (at your option) any later version.
+
+   This program is distributed in the hope that it will be useful,
+   but WITHOUT ANY WARRANTY; without even the implied warranty of
+   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+   GNU General Public License for more details.
+
+   You should have received a copy of the GNU General Public License
+   along with this program.  If not, see <http://www.gnu.org/licenses/>.  */
+
+int
+main (void)
+{
+  return 0;
+}
diff --git a/gdb/testsuite/gdb.server/build-id-seqno.exp b/gdb/testsuite/gdb.server/build-id-seqno.exp
new file mode 100644
index 0000000..33ad7a1
--- /dev/null
+++ b/gdb/testsuite/gdb.server/build-id-seqno.exp
@@ -0,0 +1,198 @@
+# This testcase is part of GDB, the GNU debugger.
+#
+# Copyright 2024 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+# Setup a .build-id/ based debug directory containing multiple entries
+# for the same build-id, with each entry given a different sequence
+# number.
+#
+# Ensure that GDB will scan over broken symlinks for the same build-id
+# (but different sequence number) to find later working symlinks.
+#
+# This test places the build-id files within a directory next to where
+# gdbserver is started, and places a relative address in the
+# debug-file-directory, in this way we require GDB to find the debug
+# information via gdbserver.
+
+require {!is_remote host}
+
+load_lib gdbserver-support.exp
+
+standard_testfile
+
+if {[build_executable "failed to prepare" $testfile $srcfile] == -1} {
+    return -1
+}
+
+# Split out BINFILE.debug.  Remove debug from BINFILE.
+if {[gdb_gnu_strip_debug $binfile] != 0} {
+    return -1
+}
+
+# Get the '.build-id/xx/xxx...xxx' part of the filename.
+set build_id_filename [build_id_debug_filename_get $binfile]
+
+# Hide (rename) BINFILE.debug, this should ensure GDB can't find it
+# directly but needs to look for the build-id based file in the debug
+# directory.
+set hidden_debuginfo [standard_output_file "hidden_$testfile.debug"]
+remote_exec build "mv ${binfile}.debug $hidden_debuginfo"
+
+# A filename that doesn't exist.  Some symlinks will point at this
+# file.
+set missing_debuginfo "missing_debuginfo"
+
+# Helper called from gdb_finish when the 'target' is remote.  Ensure the
+# debug directory we create is deleted.
+proc cleanup_remote_target {} {
+    remote_exec target "rm -fr debug/"
+}
+
+if { ![is_remote target] } {
+    set gdbserver_dir [standard_output_file "gdbserver-dir"]/
+} else {
+    lappend gdb_finish_hooks cleanup_remote_target
+    set gdbserver_dir ""
+}
+
+# Copy files to the target (if needed).
+set target_binfile [gdb_remote_download target $binfile]
+set target_debuginfo [gdb_remote_download target $hidden_debuginfo]
+
+# Setup the debug information on the target.
+set debugdir "${gdbserver_dir}debug"
+remote_exec target \
+    "mkdir -p $debugdir/[file dirname $build_id_filename]"
+remote_exec target \
+    "ln -sf $target_debuginfo $debugdir/$build_id_filename"
+
+# Start GDB and load global BINFILE.  If DEBUGINFO_FILE is not the
+# empty string then this contains the '.build-id/xx/xxx....xxxx' part
+# of the filename which we expect GDB to read from the remote target.
+# If DEBUGINFO_FILE is the empty string then we don't expect GDB to
+# find any debug information.
+proc load_binfile_check_debug_is_found { debuginfo_file testname } {
+    with_test_prefix "$testname" {
+	with_timeout_factor 5 {
+	    # Probing for .build-id based debug files on remote
+	    # targets uses the vFile:stat packet by default, though
+	    # there is a work around that avoids this which can be
+	    # used if GDB is connected to an older gdbserver without
+	    # 'stat' support.
+	    #
+	    # Check the work around works by disabling use of the
+	    # vFile:stat packet.
+	    foreach_with_prefix stat_pkt {auto off} {
+		clean_restart
+
+		gdb_test_no_output "set debug-file-directory debug" \
+		    "set debug-file-directory"
+
+		gdb_test_no_output "set sysroot target:"
+
+		gdb_test "set remote hostio-stat-packet $stat_pkt"
+
+		# Make sure we're disconnected, in case we're testing with an
+		# extended-remote board, therefore already connected.
+		gdb_test "disconnect" ".*"
+
+		# Start gdbserver.  This needs to be done after starting GDB.  When
+		# gdbserver is running local to GDB, start gdbserver in a sub-directory,
+		# this prevents GDB from finding the debug information itself.
+		if { ![is_remote target] } {
+		    with_cwd $::gdbserver_dir {
+			set res [gdbserver_start "" $::target_binfile]
+		    }
+		} else {
+		    set res [gdbserver_start "" $::target_binfile]
+		}
+		set gdbserver_protocol [lindex $res 0]
+		set gdbserver_gdbport [lindex $res 1]
+
+		# Connect to gdbserver.  The output will be placed into the global
+		# GDB_TARGET_REMOTE_CMD_MSG, and we'll match against this below.
+		gdb_assert {[gdb_target_cmd $gdbserver_protocol $gdbserver_gdbport] == 0} \
+		    "connect to gdbserver"
+
+		if { $debuginfo_file ne "" } {
+		    gdb_assert { [regexp "Reading symbols from target:debug/[string_to_regexp $debuginfo_file]\\.\\.\\." \
+				      $::gdb_target_remote_cmd_msg] } \
+			"debuginfo was read via build-id"
+		    gdb_assert { [regexp "Reading debug/[string_to_regexp $debuginfo_file] from remote target\\.\\.\\." \
+				      $::gdb_target_remote_cmd_msg] } \
+			"debuginfo was read from remote target"
+		} else {
+		    gdb_assert { [regexp "\\(No debugging symbols found in \[^\r\n\]+/$::testfile\\)" \
+				      $::gdb_target_remote_cmd_msg] }
+		}
+	    }
+	}
+    }
+}
+
+# Return a copy of FILENAME, which should end '.debug', with NUMBER
+# added, e.g. add_seqno 1 "foo.debug" --> "foo.1.debug".
+proc add_seqno { number filename } {
+    return [regsub "\.debug\$" $filename ".${number}.debug"]
+}
+
+# Precompute sequence numbered build-id filenames.
+set build_id_1_filename [add_seqno 1 $build_id_filename]
+set build_id_2_filename [add_seqno 2 $build_id_filename]
+set build_id_3_filename [add_seqno 3 $build_id_filename]
+
+load_binfile_check_debug_is_found $build_id_filename \
+    "find debuginfo with a single build-id file"
+
+remote_exec target "ln -fs $target_debuginfo \
+			$debugdir/$build_id_1_filename"
+remote_exec target "ln -fs $target_debuginfo \
+			$debugdir/$build_id_2_filename"
+remote_exec target "ln -fs $target_debuginfo \
+			$debugdir/$build_id_3_filename"
+
+load_binfile_check_debug_is_found $build_id_filename \
+    "find debuginfo with 4 build-id files"
+
+remote_exec target "ln -fs $missing_debuginfo $debugdir/$build_id_filename"
+
+load_binfile_check_debug_is_found $build_id_1_filename \
+    "find debuginfo, first build-id file is bad"
+
+remote_exec target "ln -fs $missing_debuginfo \
+			$debugdir/$build_id_1_filename"
+remote_exec target "ln -fs $missing_debuginfo \
+			$debugdir/$build_id_3_filename"
+
+load_binfile_check_debug_is_found $build_id_2_filename  \
+    "find debuginfo, first 2 build-id files are bad"
+
+remote_exec target "ln -fs $missing_debuginfo \
+			$debugdir/$build_id_2_filename"
+
+load_binfile_check_debug_is_found ""  \
+    "cannot find debuginfo, all build-id files are bad"
+
+remote_exec target "ln -fs $target_debuginfo \
+			$debugdir/$build_id_3_filename"
+
+load_binfile_check_debug_is_found $build_id_3_filename  \
+    "find debuginfo, last build-id file is good"
+
+remote_exec target "rm -f $debugdir/$build_id_1_filename"
+
+load_binfile_check_debug_is_found ""  \
+    "cannot find debuginfo, file with seqno 1 is missing"