bfd, ld: add CTF section linking

This is quite complicated because the CTF section's contents depend on
the final contents of the symtab and strtab, because it has two sections
whose contents are shuffled to be in 1:1 correspondence with the symtab,
and an internal strtab that gets deduplicated against the ELF strtab
(with offsets adjusted to point into the ELF strtab instead).  It is
also compressed if large enough, so its size depends on its contents!

So we cannot construct it as early as most sections: we cannot even
*begin* construction until after the symtab and strtab are finalized.
Thankfully there is already one section treated similarly: compressed
debugging sections: the only differences are that compressed debugging
sections have extra handling to deal with their changing name if
compressed (CTF sections are always called ".ctf" for now, though we
have reserved ".ctf.*" against future use), and that compressed
debugging sections have previously-uncompressed content which has to be
stashed away for later compression, while CTF sections have no content
at all until we generate it (very late).

BFD also cannot do the link itself: libctf knows how to do it, and BFD
cannot call libctf directly because libctf already depends on bfd for
file I/O.  So we have to use a pair of callbacks, one, examine_strtab,
which allows a caller to examine the symtab and strtab after
finalization (called from elf_link_swap_symbols_out(), right before the
symtabs are written, and after the strtab has been finalized), and one
which actually does the emission (called emit_ctf simply because it is
grouped with a bunch of section-specific late-emission function calls at
the bottom of bfd_elf_final_link, and a section-specific name seems best
for that).  emit_ctf is actually called *twice*: once from lang_process
if the emulation suggests that this bfd target does not examine the
symtab or strtab, and once via a bfd callback if it does.  (This means
that non-ELF targets still get CTF emitted, even though the late CTF
emission stage is never called for them).

v2: merged with non-ELF support patch: slight commit message
    adjustments.
v3: do not spend time merging CTF, or crash, if the CTF section is
    explicitly discarded.  Do not try to merge or compress CTF unless
    linking.
v4: add CTF_COMPRESSION_THRESHOLD.  Annul the freed input ctf_file_t's
    after writeout: set SEC_IN_MEMORY on the output contents so a future
    bfd enhancement knows it could free it.  Add SEC_LINKER_CREATED |
    SEC_KEEP to avoid having to add .ctf to the linker script.  Drop
    now-unnecessary ldlang.h-level elf-bfd.h include and hackery around
    it.  Adapt to elf32.em->elf.em and elf-generic.em->ldelf*.c
    changes.
v5: fix tabdamage.  Drop #inclusions in .h files: include in .c files,
    .em files, and use struct forwards instead.  Use bfd_section_is_ctf
    inline function rather than SECTION_IS_CTF macro.  Move a few
    comments.

	* Makefile.def (dependencies): all-ld depends on all-libctf.
	* Makefile.in: Regenerated.

include/
	* bfdlink.h (elf_strtab_hash): New forward.
	(elf_sym_strtab): Likewise.
	(struct bfd_link_callbacks <examine_strtab>): New.
	(struct bfd_link_callbacks <emit_ctf>): Likewise.

bfd/
	* elf-bfd.h (bfd_section_is_ctf): New inline function.
	* elf.c (special_sections_c): Add ".ctf".
	(assign_file_positions_for_non_load_sections): Note that
	compressed debugging sections etc are not assigned here.  Treat
	CTF sections like SEC_ELF_COMPRESS sections when is_linker_output:
	sh_offset -1.
	(assign_file_positions_except_relocs): Likewise.
	(find_section_in_list): Note that debugging and CTF sections, as
	well as reloc sections, are assigned later.
	(_bfd_elf_assign_file_positions_for_non_load): CTF sections get
	their size and contents updated.
	(_bfd_elf_set_section_contents): Skip CTF sections: unlike
	compressed sections, they have no uncompressed content to copy at
	this stage.
	* elflink.c (elf_link_swap_symbols_out): Call the examine_strtab
	callback right before the strtab is written out.
	(bfd_elf_final_link): Don't cache the section contents of CTF
	sections: they are not populated yet.  Call the emit_ctf callback
	right at the end, after all the symbols and strings are flushed
	out.

ld/
	* ldlang.h: (struct lang_input_statement_struct): Add the_ctf.
	(struct elf_sym_strtab): Add forward.
	(struct elf_strtab_hash): Likewise.
	(ldlang_ctf_apply_strsym): Declare.
	(ldlang_write_ctf_late): Likewise.
	* ldemul.h (ldemul_emit_ctf_early): New.
	(ldemul_examine_strtab_for_ctf): Likewise.
	(ld_emulation_xfer_type) <emit_ctf_early>: Likewise.
	(ld_emulation_xfer_type) <examine_strtab_for_ctf>: Likewise.
	* ldemul.c (ldemul_emit_ctf_early): New.
	(ldemul_examine_strtab_for_ctf): Likewise.
	* ldlang.c: Include ctf-api.h.
	(CTF_COMPRESSION_THRESHOLD): New.
	(ctf_output): New. Initialized in...
	(ldlang_open_ctf): ... this new function.  Open all the CTF
	sections in the input files: mark them non-loaded and empty
	so as not to copy their contents to the output, but linker-created
	so the section gets created in the target.
	(ldlang_merge_ctf): New, merge types via ctf_link_add_ctf and
	ctf_link.
	(ldlang_ctf_apply_strsym): New, an examine_strtab callback: wrap
	ldemul_examine_strtab_for_ctf.
	(lang_write_ctf): New, write out the CTF section.
	(ldlang_write_ctf_late): New, late call via bfd's emit_ctf hook.
	(lang_process): Call ldlang_open_ctf, ldlang_merge_ctf, and
	lang_write_ctf.
	* ldmain.c (link_callbacks): Add ldlang_ctf_apply_strsym,
	ldlang_write_ctf_late.
	* emultempl/aix.em: Add ctf-api.h.
	* emultempl/armcoff.em: Likewise.
	* emultempl/beos.em: Likewise.
	* emultempl/elf.em: Likewise.
	* emultempl/generic.em: Likewise.
	* emultempl/linux.em: Likewise.
	* emultempl/msp430.em: Likewise.
	* emultempl/pe.em: Likewise.
	* emultempl/pep.em: Likewise.
	* emultempl/ticoff.em: Likewise.
	* emultempl/vanilla.em: Likewise.
	* ldcref.c: Likewise.
	* ldctor.c: Likewise.
	* ldelf.c: Likewise.
	* ldelfgen.c: Likewise.
	* ldemul.c: Likewise.
	* ldexp.c: Likewise.
	* ldfile.c: Likewise.
	* ldgram.c: Likewise.
	* ldlex.l: Likewise.
	* ldmain.c: Likewise.
	* ldmisc.c: Likewise.
	* ldver.c: Likewise.
	* ldwrite.c: Likewise.
	* lexsup.c: Likewise.
	* mri.c: Likewise.
	* pe-dll.c: Likewise.
	* plugin.c: Likewise.

	* ldelfgen.c (ldelf_emit_ctf_early): New.
	(ldelf_examine_strtab_for_ctf): tell libctf about the symtab and
	strtab.
	(struct ctf_strsym_iter_cb_arg): New, state to do so.
	(ldelf_ctf_strtab_iter_cb): New: tell libctf about
	each string in the strtab in turn.
	(ldelf_ctf_symbols_iter_cb): New, tell libctf
	about each symbol in the symtab in turn.
	* ldelfgen.h (struct elf_sym_strtab): Add forward.
	(struct elf_strtab_hash): Likewise.
	(struct ctf_file): Likewise.
	(ldelf_emit_ctf_early): Declare.
	(ldelf_examine_strtab_for_ctf): Likewise.
	* emultempl/elf-generic.em (LDEMUL_EMIT_CTF_EARLY): Set it.
	(LDEMUL_EXAMINE_STRTAB_FOR_CTF): Likewise.
	* emultempl/aix.em (ld_${EMULATION_NAME}_emulation): Add
	emit_ctf_early and examine_strtab_for_ctf, NULL by default.
	* emultempl/armcoff.em (ld_${EMULATION_NAME}_emulation): Likewise.
	* emultempl/beos.em (ld_${EMULATION_NAME}_emulation): Likewise.
	* emultempl/elf.em (ld_${EMULATION_NAME}_emulation): Likewise.
	* emultempl/generic.em (ld_${EMULATION_NAME}_emulation): Likewise.
	* emultempl/linux.em (ld_${EMULATION_NAME}_emulation): Likewise.
	* emultempl/msp430.em (ld_${EMULATION_NAME}_emulation): Likewise.
	* emultempl/pe.em (ld_${EMULATION_NAME}_emulation): Likewise.
	* emultempl/pep.em (ld_${EMULATION_NAME}_emulation): Likewise.
	* emultempl/ticoff.em (ld_${EMULATION_NAME}_emulation): Likewise.
	* emultempl/vanilla.em (ld_vanilla_emulation): Likewise.

	* Makefile.am: Pull in libctf (and zlib, a transitive requirement
	for compressed CTF section emission).  Pass it on to DejaGNU.
	* configure.ac: Add AM_ZLIB.
	* aclocal.m4: Added zlib.m4.
	* Makefile.in: Regenerated.
	* testsuite/ld-bootstrap/bootstrap.exp: Use it when relinking ld.

4 files changed, 84 insertions, 18 deletions

diff --git a/bfd/ChangeLog b/bfd/ChangeLog
index 420b13d..fcb645b 100644
--- a/bfd/ChangeLog
+++ b/bfd/ChangeLog
@@ -1,3 +1,26 @@
+2019-09-30  Nick Alcock  <nick.alcock@oracle.com>
+
+	* elf-bfd.h (bfd_section_is_ctf): New inline function.
+	* elf.c (special_sections_c): Add ".ctf".
+	(assign_file_positions_for_non_load_sections): Note that
+	compressed debugging sections etc are not assigned here.  Treat
+	CTF sections like SEC_ELF_COMPRESS sections when is_linker_output:
+	sh_offset -1.
+	(assign_file_positions_except_relocs): Likewise.
+	(find_section_in_list): Note that debugging and CTF sections, as
+	well as reloc sections, are assigned later.
+	(_bfd_elf_assign_file_positions_for_non_load): CTF sections get
+	their size and contents updated.
+	(_bfd_elf_set_section_contents): Skip CTF sections: unlike
+	compressed sections, they have no uncompressed content to copy at
+	this stage.
+	* elflink.c (elf_link_swap_symbols_out): Call the examine_strtab
+	callback right before the strtab is written out.
+	(bfd_elf_final_link): Don't cache the section contents of CTF
+	sections: they are not populated yet.  Call the emit_ctf callback
+	right at the end, after all the symbols and strings are flushed
+	out.
+
 2019-07-13  Nick Alcock  <nick.alcock@oracle.com>
 
 	* elf-strtab.c (_bfd_elf_strtab_len): New.
diff --git a/bfd/elf-bfd.h b/bfd/elf-bfd.h
index 4240d97..ccd2c35 100644
--- a/bfd/elf-bfd.h
+++ b/bfd/elf-bfd.h
@@ -2257,7 +2257,7 @@ extern bfd_size_type _bfd_elf_strtab_len
 extern bfd_size_type _bfd_elf_strtab_offset
   (struct elf_strtab_hash *, size_t);
 extern const char * _bfd_elf_strtab_str
-  (struct elf_strtab_hash *, size_t idx, size_t *offset);
+  (struct elf_strtab_hash *, size_t idx, bfd_size_type *offset);
 extern bfd_boolean _bfd_elf_strtab_emit
   (bfd *, struct elf_strtab_hash *);
 extern void _bfd_elf_strtab_finalize
@@ -2965,6 +2965,14 @@ extern asection _bfd_elf_large_com_section;
 	 || (H)->start_stop \
 	 || ((INFO)->dynamic && !(H)->dynamic)))
 
+/* Determine if a section contains CTF data, using its name.  */
+static inline bfd_boolean
+bfd_section_is_ctf (const asection *sec)
+{
+  const char *name = bfd_section_name (sec);
+  return strncmp (name, ".ctf", 4) == 0 && (name[4] == 0 || name[4] == '.');
+}
+
 #ifdef __cplusplus
 }
 #endif
diff --git a/bfd/elf.c b/bfd/elf.c
index bb994b5..cbec426 100644
--- a/bfd/elf.c
+++ b/bfd/elf.c
@@ -2671,6 +2671,7 @@ static const struct bfd_elf_special_section special_sections_b[] =
 static const struct bfd_elf_special_section special_sections_c[] =
 {
   { STRING_COMMA_LEN (".comment"), 0, SHT_PROGBITS, 0 },
+  { STRING_COMMA_LEN (".ctf"),	0, SHT_PROGBITS,    0 },
   { NULL,			0, 0, 0,	    0 }
 };
 
@@ -5893,7 +5894,8 @@ is_debuginfo_file (bfd *abfd)
   return TRUE;
 }
 
-/* Assign file positions for the other sections.  */
+/* Assign file positions for the other sections, except for compressed debugging
+   and other sections assigned in _bfd_elf_assign_file_positions_for_non_load().  */
 
 static bfd_boolean
 assign_file_positions_for_non_load_sections (bfd *abfd,
@@ -5952,9 +5954,12 @@ assign_file_positions_for_non_load_sections (bfd *abfd,
 	}
       else if (((hdr->sh_type == SHT_REL || hdr->sh_type == SHT_RELA)
 		&& hdr->bfd_section == NULL)
+	       /* We don't know the offset of these sections yet: their size has
+		  not been decided.  */
 	       || (hdr->bfd_section != NULL
-		   && (hdr->bfd_section->flags & SEC_ELF_COMPRESS))
-		   /* Compress DWARF debug sections.  */
+		   && (hdr->bfd_section->flags & SEC_ELF_COMPRESS
+		       || (bfd_section_is_ctf (hdr->bfd_section)
+			   && abfd->is_linker_output)))
 	       || hdr == i_shdrpp[elf_onesymtab (abfd)]
 	       || (elf_symtab_shndx_list (abfd) != NULL
 		   && hdr == i_shdrpp[elf_symtab_shndx_list (abfd)->ndx])
@@ -6222,11 +6227,12 @@ find_section_in_list (unsigned int i, elf_section_list * list)
    VMAs must be known before this is called.
 
    Reloc sections come in two flavours: Those processed specially as
-   "side-channel" data attached to a section to which they apply, and
-   those that bfd doesn't process as relocations.  The latter sort are
-   stored in a normal bfd section by bfd_section_from_shdr.   We don't
-   consider the former sort here, unless they form part of the loadable
-   image.  Reloc sections not assigned here will be handled later by
+   "side-channel" data attached to a section to which they apply, and those that
+   bfd doesn't process as relocations.  The latter sort are stored in a normal
+   bfd section by bfd_section_from_shdr.  We don't consider the former sort
+   here, unless they form part of the loadable image.  Reloc sections not
+   assigned here (and compressed debugging sections and CTF sections which
+   nothing else in the file can rely upon) will be handled later by
    assign_file_positions_for_relocs.
 
    We also don't set the positions of the .symtab and .strtab here.  */
@@ -6261,9 +6267,12 @@ assign_file_positions_except_relocs (bfd *abfd,
 	  hdr = *hdrpp;
 	  if (((hdr->sh_type == SHT_REL || hdr->sh_type == SHT_RELA)
 	       && hdr->bfd_section == NULL)
+	      /* Do not assign offsets for these sections yet: we don't know
+		 their sizes.  */
 	      || (hdr->bfd_section != NULL
-		  && (hdr->bfd_section->flags & SEC_ELF_COMPRESS))
-		  /* Compress DWARF debug sections.  */
+		  && (hdr->bfd_section->flags & SEC_ELF_COMPRESS
+		      || (bfd_section_is_ctf (hdr->bfd_section)
+			  && abfd->is_linker_output)))
 	      || i == elf_onesymtab (abfd)
 	      || (elf_symtab_shndx_list (abfd) != NULL
 		  && hdr == i_shdrpp[elf_symtab_shndx_list (abfd)->ndx])
@@ -6471,10 +6480,12 @@ _bfd_elf_assign_file_positions_for_non_load (bfd *abfd)
 	  asection *sec = shdrp->bfd_section;
 	  bfd_boolean is_rel = (shdrp->sh_type == SHT_REL
 				|| shdrp->sh_type == SHT_RELA);
+	  bfd_boolean is_ctf = sec && bfd_section_is_ctf (sec);
 	  if (is_rel
+	      || is_ctf
 	      || (sec != NULL && (sec->flags & SEC_ELF_COMPRESS)))
 	    {
-	      if (!is_rel)
+	      if (!is_rel && !is_ctf)
 		{
 		  const char *name = sec->name;
 		  struct bfd_elf_section_data *d;
@@ -6520,6 +6531,13 @@ _bfd_elf_assign_file_positions_for_non_load (bfd *abfd)
 		  shdrp->contents = sec->contents;
 		  shdrp->bfd_section->contents = NULL;
 		}
+	      else if (is_ctf)
+		{
+		  /* Update section size and contents.	*/
+		  shdrp->sh_size = sec->size;
+		  shdrp->contents = sec->contents;
+		}
+
 	      off = _bfd_elf_assign_file_position_for_section (shdrp,
 							       off,
 							       TRUE);
@@ -9099,6 +9117,11 @@ _bfd_elf_set_section_contents (bfd *abfd,
   hdr = &elf_section_data (section)->this_hdr;
   if (hdr->sh_offset == (file_ptr) -1)
     {
+      if (bfd_section_is_ctf (section))
+	/* Nothing to do with this section: the contents are generated
+	   later.  */
+	return TRUE;
+
       /* We must compress this section.  Write output to the buffer.  */
       unsigned char *contents = hdr->contents;
       if ((offset + count) > hdr->sh_size
diff --git a/bfd/elflink.c b/bfd/elflink.c
index 2169e2b..371c096 100644
--- a/bfd/elflink.c
+++ b/bfd/elflink.c
@@ -9539,6 +9539,14 @@ elf_link_swap_symbols_out (struct elf_final_link_info *flinfo)
 				+ elfsym->destshndx_index));
     }
 
+  /* Allow the linker to examine the strtab and symtab now they are
+     populated.  */
+
+  if (flinfo->info->callbacks->examine_strtab)
+    flinfo->info->callbacks->examine_strtab (hash_table->strtab,
+					     hash_table->strtabcount,
+					     flinfo->symstrtab);
+
   hdr = &elf_tdata (flinfo->output_bfd)->symtab_hdr;
   pos = hdr->sh_offset + hdr->sh_size;
   amt = hash_table->strtabcount * bed->s->sizeof_sym;
@@ -11810,7 +11818,7 @@ bfd_elf_final_link (bfd *abfd, struct bfd_link_info *info)
 
   /* The object attributes have been merged.  Remove the input
      sections from the link, and set the contents of the output
-     secton.  */
+     section.  */
   std_attrs_section = get_elf_backend_data (abfd)->obj_attrs_section;
   for (o = abfd->sections; o != NULL; o = o->next)
     {
@@ -12032,7 +12040,8 @@ bfd_elf_final_link (bfd *abfd, struct bfd_link_info *info)
       esdo->rel.count = 0;
       esdo->rela.count = 0;
 
-      if (esdo->this_hdr.sh_offset == (file_ptr) -1)
+      if ((esdo->this_hdr.sh_offset == (file_ptr) -1)
+	  && !bfd_section_is_ctf (o))
 	{
 	  /* Cache the section contents so that they can be compressed
 	     later.  Use bfd_malloc since it will be freed by
@@ -12048,10 +12057,10 @@ bfd_elf_final_link (bfd *abfd, struct bfd_link_info *info)
 	}
     }
 
-  /* We have now assigned file positions for all the sections except
-     .symtab, .strtab, and non-loaded reloc sections.  We start the
-     .symtab section at the current file position, and write directly
-     to it.  We build the .strtab section in memory.  */
+  /* We have now assigned file positions for all the sections except .symtab,
+     .strtab, and non-loaded reloc and compressed debugging sections.  We start
+     the .symtab section at the current file position, and write directly to it.
+     We build the .strtab section in memory.  */
   abfd->symcount = 0;
   symtab_hdr = &elf_tdata (abfd)->symtab_hdr;
   /* sh_name is set in prep_headers.  */
@@ -12837,6 +12846,9 @@ bfd_elf_final_link (bfd *abfd, struct bfd_link_info *info)
   if (! _bfd_elf_write_section_eh_frame_hdr (abfd, info))
     goto error_return;
 
+  if (info->callbacks->emit_ctf)
+      info->callbacks->emit_ctf ();
+
   elf_final_link_free (abfd, &flinfo);
 
   if (attr_section)