Fix BFD format matching for x86_64-w64-mingw32 -m32 LTO.

In <https://sourceware.org/ml/binutils/2015-12/msg00190.html> (commit
4a07dc81356ed8728e204e9aabeb256703c59aef), Kwok fixed a problem with
the template used for a dummy BFD for an IR file for LTO on MinGW,
where the input and output formats are not the same.

A problem, however, remains in the case of linking for
x86_64-w64-mingw32 -m32, where LTO linking reports an ambiguity
between the pe-i386 and pei-i386 formats.  An object (pe-i386) with
plugin data is being tested by the linker to see what formats match.
The default format initially set by the linker when
bfd_check_format_matches is called is pei-i386 (as that's the output
format from the linker script), which does not match, so the function
goes on to the loop over possible BFD vectors.  The pe-i386 vector
matches, as it should.  One other vector matches: the plugin vector.

bfd_check_format_matches tests a vector for matching by temporarily
modifying the BFD object to use that vector then using
_bfd_check_format on it.  So the BFD object is temporarily using
plugin_vec.  _bfd_check_format ends up using bfd_plugin_object_p which
uses plugin_object_p from ld which uses plugin_get_ir_dummy_bfd which
succeeds, having created a BFD based on link_info.output_bfd (because
srctemplate is the BFD temporarily using plugin_vec, even after Kwok's
patch link_info.output_bfd is all that's available to base the dummy
BFD on).  So we end up with a match from the plugin vector which uses
the pei-i386 vector even though the pei-i386 vector itself does not
match the input object.  (In the i686-mingw32 case, as opposed to this
multilib case, pe-i386 is the default BFD target, which would
short-circuit that logic.)

There are two cases of the linker handling inputs with a plugin: they
may be inputs that are also accepted by some non-plugin BFD format, as
here, or they may be a format that would not be recognized at all, as
with some tests in the ld testsuite.  In the former case, there is no
need for BFD to accept the objects using the plugin vector, as the
linker has its own logic to allow plugins to claim objects accepted by
some other BFD vector.  Thus, this patch arranges for the plugin
vector to have the lowest match priority, and for the priority from
that vector to be used in the relevant case (the attempted match to
the plugin vector results in TEMP pointing to the pei-i386 vector).

Tested for GCC and Binutils testsuites for x86_64-pc-linux-gnu, as
well as verifying that it fixes the observed LTO issue for
x86_64-w64-mingw32.

	* plugin.c (plugin_vec): Set match priority to 255.
	* format.c (bfd_check_format_matches) [BFD_SUPPORTS_PLUGINS]: When
	matching against the plugin vector, take priority from there not
	from TEMP.

3 files changed, 23 insertions, 3 deletions

diff --git a/bfd/ChangeLog b/bfd/ChangeLog
index 29aac39..58a9de4 100644
--- a/bfd/ChangeLog
+++ b/bfd/ChangeLog
@@ -1,3 +1,10 @@
+2016-02-16  Joseph Myers  <joseph@codesourcery.com>
+
+	* plugin.c (plugin_vec): Set match priority to 255.
+	* format.c (bfd_check_format_matches) [BFD_SUPPORTS_PLUGINS]: When
+	matching against the plugin vector, take priority from there not
+	from TEMP.
+
 2016-02-15  Nick Clifton  <nickc@redhat.com>
 
 	* elf-bfd.h (struct bfd_elf_special_section): Use unsigned values
diff --git a/bfd/format.c b/bfd/format.c
index aad4dc2..f34b1d4 100644
--- a/bfd/format.c
+++ b/bfd/format.c
@@ -203,6 +203,9 @@ bfd_boolean
 bfd_check_format_matches (bfd *abfd, bfd_format format, char ***matching)
 {
   extern const bfd_target binary_vec;
+#if BFD_SUPPORTS_PLUGINS
+  extern const bfd_target plugin_vec;
+#endif
   const bfd_target * const *target;
   const bfd_target **matching_vector = NULL;
   const bfd_target *save_targ, *right_targ, *ar_right_targ, *match_targ;
@@ -305,6 +308,16 @@ bfd_check_format_matches (bfd *abfd, bfd_format format, char ***matching)
       temp = BFD_SEND_FMT (abfd, _bfd_check_format, (abfd));
       if (temp)
 	{
+	  int match_priority = temp->match_priority;
+#if BFD_SUPPORTS_PLUGINS
+	  /* If this object can be handled by a plugin, give that the
+	     lowest priority; objects both handled by a plugin and
+	     with an underlying object format will be claimed
+	     separately by the plugin.  */
+	  if (*target == &plugin_vec)
+	    match_priority = (*target)->match_priority;
+#endif
+
 	  match_targ = temp;
 	  if (preserve.marker != NULL)
 	    bfd_preserve_finish (abfd, &preserve);
@@ -326,9 +339,9 @@ bfd_check_format_matches (bfd *abfd, bfd_format format, char ***matching)
 		matching_vector[match_count] = temp;
 	      match_count++;
 
-	      if (temp->match_priority < best_match)
+	      if (match_priority < best_match)
 		{
-		  best_match = temp->match_priority;
+		  best_match = match_priority;
 		  best_count = 0;
 		}
 	      best_count++;
diff --git a/bfd/plugin.c b/bfd/plugin.c
index b832e23..82a87d6 100644
--- a/bfd/plugin.c
+++ b/bfd/plugin.c
@@ -566,7 +566,7 @@ const bfd_target plugin_vec =
   0,				/* symbol_leading_char.  */
   '/',				/* ar_pad_char.  */
   15,				/* ar_max_namelen.  */
-  0,				/* match priority.  */
+  255,				/* match priority.  */
 
   bfd_getl64, bfd_getl_signed_64, bfd_putl64,
   bfd_getl32, bfd_getl_signed_32, bfd_putl32,