libctf: support getting strings from the ELF strtab

The CTF file format has always supported "external strtabs", which
internally are strtab offsets with their MSB on: such refs
get their strings from the strtab passed in at CTF file open time:
this is usually intended to be the ELF strtab, and that's what this
implementation is meant to support, though in theory the external
strtab could come from anywhere.

This commit adds support for these external strings in the ctf-string.c
strtab tracking layer.  It's quite easy: we just add a field csa_offset
to the atoms table that tracks all strings: this field tracks the offset
of the string in the ELF strtab (with its MSB already on, courtesy of a
new macro CTF_SET_STID), and adds a new function that sets the
csa_offset to the specified offset (plus MSB).  Then we just need to
avoid writing out strings to the internal strtab if they have csa_offset
set, and note that the internal strtab is shorter than it might
otherwise be.

(We could in theory save a little more time here by eschewing sorting
such strings, since we never actually write the strings out anywhere,
but that would mean storing them separately and it's just not worth the
complexity cost until profiling shows it's worth doing.)

We also have to go through a bit of extra effort at variable-sorting
time.  This was previously using direct references to the internal
strtab: it couldn't use ctf_strptr or ctf_strraw because the new strtab
is not yet ready to put in its usual field (in a ctf_file_t that hasn't
even been allocated yet at this stage): but now we're using the external
strtab, this will no longer do because it'll be looking things up in the
wrong strtab, with disastrous results.  Instead, pass the new internal
strtab in to a new ctf_strraw_explicit function which is just like
ctf_strraw except you can specify a ne winternal strtab to use.

But even now that it is using a new internal strtab, this is not quite
enough: it can't look up strings in the external strtab because ld
hasn't written it out yet, and when it does will write it straight to
disk.  Instead, when we write the internal strtab, note all the offset
-> string mappings that we have noted belong in the *external* strtab to
a new "synthetic external strtab" dynhash, ctf_syn_ext_strtab, and look
in there at ctf_strraw time if it is set.  This uses minimal extra
memory (because only strings in the external strtab that we actually use
are stored, and even those come straight out of the atoms table), but
let both variable sorting and name interning when ctf_bufopen is next
called work fine.  (This also means that we don't need to filter out
spurious ECTF_STRTAB warnings from ctf_bufopen but can pass them back to
the caller, once we wrap ctf_bufopen so that we have a new internal
variant of ctf_bufopen etc that we can pass the synthetic external
strtab to. That error has been filtered out since the days of Solaris
libctf, which didn't try to handle the problem of getting external
strtabs right at construction time at all.)

v3: add the synthetic strtab and all associated machinery.
v5: fix tabdamage.

include/
	* ctf.h (CTF_SET_STID): New.

libctf/
	* ctf-impl.h (ctf_str_atom_t) <csa_offset>: New field.
	(ctf_file_t) <ctf_syn_ext_strtab>: Likewise.
	(ctf_str_add_ref): Name the last arg.
	(ctf_str_add_external) New.
	(ctf_str_add_strraw_explicit): Likewise.
	(ctf_simple_open_internal): Likewise.
	(ctf_bufopen_internal): Likewise.

	* ctf-string.c (ctf_strraw_explicit): Split from...
	(ctf_strraw): ... here, with new support for ctf_syn_ext_strtab.
	(ctf_str_add_ref_internal): Return the atom, not the
	string.
	(ctf_str_add): Adjust accordingly.
	(ctf_str_add_ref): Likewise.  Move up in the file.
	(ctf_str_add_external): New: update the csa_offset.
	(ctf_str_count_strtab): Only account for strings with no csa_offset
	in the internal strtab length.
	(ctf_str_write_strtab): If the csa_offset is set, update the
	string's refs without writing the string out, and update the
	ctf_syn_ext_strtab.  Make OOM handling less ugly.
	* ctf-create.c (struct ctf_sort_var_arg_cb): New.
	(ctf_update): Handle failure to populate the strtab.  Pass in the
	new ctf_sort_var arg.  Adjust for ctf_syn_ext_strtab addition.
	Call ctf_simple_open_internal, not ctf_simple_open.
	(ctf_sort_var): Call ctf_strraw_explicit rather than looking up
	strings by hand.
	* ctf-hash.c (ctf_hash_insert_type): Likewise (but using
	ctf_strraw).  Adjust to diagnose ECTF_STRTAB nonetheless.
	* ctf-open.c (init_types): No longer filter out ECTF_STRTAB.
	(ctf_file_close): Destroy the ctf_syn_ext_strtab.
	(ctf_simple_open): Rename to, and reimplement as a wrapper around...
	(ctf_simple_open_internal): ... this new function, which calls
	ctf_bufopen_internal.
	(ctf_bufopen): Rename to, and reimplement as a wrapper around...
	(ctf_bufopen_internal): ... this new function, which sets
	ctf_syn_ext_strtab.

1 files changed, 129 insertions, 36 deletions

diff --git a/libctf/ctf-string.c b/libctf/ctf-string.c
index 27bd7c2..44cd447 100644
--- a/libctf/ctf-string.c
+++ b/libctf/ctf-string.c
@@ -20,13 +20,24 @@
 #include <ctf-impl.h>
 #include <string.h>
 
-/* Convert an encoded CTF string name into a pointer to a C string by looking
-  up the appropriate string table buffer and then adding the offset.  */
+/* Convert an encoded CTF string name into a pointer to a C string, using an
+  explicit internal strtab rather than the fp-based one.  */
 const char *
-ctf_strraw (ctf_file_t *fp, uint32_t name)
+ctf_strraw_explicit (ctf_file_t *fp, uint32_t name, ctf_strs_t *strtab)
 {
   ctf_strs_t *ctsp = &fp->ctf_str[CTF_NAME_STID (name)];
 
+  if ((CTF_NAME_STID (name) == CTF_STRTAB_0) && (strtab != NULL))
+    ctsp = strtab;
+
+  /* If this name is in the external strtab, and there is a synthetic strtab,
+     use it in preference.  */
+
+  if (CTF_NAME_STID (name) == CTF_STRTAB_1
+      && fp->ctf_syn_ext_strtab != NULL)
+    return ctf_dynhash_lookup (fp->ctf_syn_ext_strtab,
+			       (void *) (uintptr_t) name);
+
   if (ctsp->cts_strs != NULL && CTF_NAME_OFFSET (name) < ctsp->cts_len)
     return (ctsp->cts_strs + CTF_NAME_OFFSET (name));
 
@@ -34,6 +45,14 @@ ctf_strraw (ctf_file_t *fp, uint32_t name)
   return NULL;
 }
 
+/* Convert an encoded CTF string name into a pointer to a C string by looking
+  up the appropriate string table buffer and then adding the offset.  */
+const char *
+ctf_strraw (ctf_file_t *fp, uint32_t name)
+{
+  return ctf_strraw_explicit (fp, name, NULL);
+}
+
 /* Return a guaranteed-non-NULL pointer to the string with the given CTF
    name.  */
 const char *
@@ -88,11 +107,11 @@ ctf_str_free_atoms (ctf_file_t *fp)
   ctf_dynhash_destroy (fp->ctf_str_atoms);
 }
 
-/* Add a string to the atoms table and return it, or return an existing string
-   if present, copying the passed-in string.  Returns NULL only when out of
-   memory (and do not touch the passed-in string in that case).  Possibly
-   augment the ref list with the passed-in ref.  */
-static const char *
+/* Add a string to the atoms table, copying the passed-in string.  Return the
+   atom added. Return NULL only when out of memory (and do not touch the
+   passed-in string in that case).  Possibly augment the ref list with the
+   passed-in ref.  */
+static ctf_str_atom_t *
 ctf_str_add_ref_internal (ctf_file_t *fp, const char *str,
 			  int add_ref, uint32_t *ref)
 {
@@ -116,7 +135,7 @@ ctf_str_add_ref_internal (ctf_file_t *fp, const char *str,
 	  ctf_list_append (&atom->csa_refs, aref);
 	  fp->ctf_str_num_refs++;
 	}
-      return atom->csa_str;
+      return atom;
     }
 
   if ((atom = ctf_alloc (sizeof (struct ctf_str_atom))) == NULL)
@@ -136,7 +155,7 @@ ctf_str_add_ref_internal (ctf_file_t *fp, const char *str,
       ctf_list_append (&atom->csa_refs, aref);
       fp->ctf_str_num_refs++;
     }
-  return newstr;
+  return atom;
 
  oom:
   ctf_free (atom);
@@ -150,9 +169,48 @@ ctf_str_add_ref_internal (ctf_file_t *fp, const char *str,
 const char *
 ctf_str_add (ctf_file_t *fp, const char *str)
 {
-  if (str)
-    return ctf_str_add_ref_internal (fp, str, FALSE, 0);
-  return NULL;
+  ctf_str_atom_t *atom;
+  if (!str)
+    return NULL;
+
+  atom = ctf_str_add_ref_internal (fp, str, FALSE, 0);
+  if (!atom)
+    return NULL;
+
+  return atom->csa_str;
+}
+
+/* Like ctf_str_add(), but additionally augment the atom's refs list with the
+   passed-in ref, whether or not the string is already present.  There is no
+   attempt to deduplicate the refs list (but duplicates are harmless).  */
+const char *
+ctf_str_add_ref (ctf_file_t *fp, const char *str, uint32_t *ref)
+{
+  ctf_str_atom_t *atom;
+  if (!str)
+    return NULL;
+
+  atom = ctf_str_add_ref_internal (fp, str, TRUE, ref);
+  if (!atom)
+    return NULL;
+
+  return atom->csa_str;
+}
+
+/* Add an external strtab reference at OFFSET.  */
+const char *
+ctf_str_add_external (ctf_file_t *fp, const char *str, uint32_t offset)
+{
+  ctf_str_atom_t *atom;
+  if (!str)
+    return NULL;
+
+  atom = ctf_str_add_ref_internal (fp, str, FALSE, 0);
+  if (!atom)
+    return NULL;
+
+  atom->csa_offset = CTF_SET_STID (offset, CTF_STRTAB_1);
+  return atom->csa_str;
 }
 
 /* A ctf_dynhash_iter_remove() callback that removes atoms later than a given
@@ -173,17 +231,6 @@ ctf_str_rollback (ctf_file_t *fp, ctf_snapshot_id_t id)
   ctf_dynhash_iter_remove (fp->ctf_str_atoms, ctf_str_rollback_atom, &id);
 }
 
-/* Like ctf_str_add(), but additionally augment the atom's refs list with the
-   passed-in ref, whether or not the string is already present.  There is no
-   attempt to deduplicate the refs list (but duplicates are harmless).  */
-const char *
-ctf_str_add_ref (ctf_file_t *fp, const char *str, uint32_t *ref)
-{
-  if (str)
-    return ctf_str_add_ref_internal (fp, str, TRUE, ref);
-  return NULL;
-}
-
 /* An adaptor around ctf_purge_atom_refs.  */
 static void
 ctf_str_purge_one_atom_refs (void *key _libctf_unused_, void *value,
@@ -238,7 +285,11 @@ ctf_str_count_strtab (void *key _libctf_unused_, void *value,
   ctf_str_atom_t *atom = (ctf_str_atom_t *) value;
   ctf_strtab_write_state_t *s = (ctf_strtab_write_state_t *) arg;
 
-  s->strtab->cts_len += strlen (atom->csa_str) + 1;
+  /* We only factor in the length of items that have no offset:
+     other items are in the external strtab.  They still contribute to the
+     total count, though, because we still have to sort them.  */
+  if (!atom->csa_offset)
+    s->strtab->cts_len += strlen (atom->csa_str) + 1;
   s->strtab_count++;
 }
 
@@ -268,8 +319,10 @@ ctf_str_sort_strtab (const void *a, const void *b)
 }
 
 /* Write out and return a strtab containing all strings with recorded refs,
-   adjusting the refs to refer to the corresponding string.  The returned
-   strtab may be NULL on error.  */
+   adjusting the refs to refer to the corresponding string.  The returned strtab
+   may be NULL on error.  Also populate the synthetic strtab with mappings from
+   external strtab offsets to names, so we can look them up with ctf_strptr().
+   Only external strtab offsets with references are added.  */
 ctf_strs_writable_t
 ctf_str_write_strtab (ctf_file_t *fp)
 {
@@ -279,6 +332,7 @@ ctf_str_write_strtab (ctf_file_t *fp)
   ctf_strtab_write_state_t s;
   ctf_str_atom_t **sorttab;
   size_t i;
+  int any_external = 0;
 
   memset (&strtab, 0, sizeof (struct ctf_strs_writable));
   memset (&s, 0, sizeof (struct ctf_strtab_write_state));
@@ -300,7 +354,7 @@ ctf_str_write_strtab (ctf_file_t *fp)
   /* Sort the strtab.  Force the null string to be first.  */
   sorttab = calloc (s.strtab_count, sizeof (ctf_str_atom_t *));
   if (!sorttab)
-      return strtab;
+    goto oom;
 
   sorttab[0] = nullstr;
   s.i = 1;
@@ -312,19 +366,58 @@ ctf_str_write_strtab (ctf_file_t *fp)
 	 ctf_str_sort_strtab);
 
   if ((strtab.cts_strs = ctf_alloc (strtab.cts_len)) == NULL)
-    {
-      free (sorttab);
-      return strtab;
-    }
+    goto oom_sorttab;
+
+  if (!fp->ctf_syn_ext_strtab)
+    fp->ctf_syn_ext_strtab = ctf_dynhash_create (ctf_hash_integer,
+						 ctf_hash_eq_integer,
+						 NULL, NULL);
+  if (!fp->ctf_syn_ext_strtab)
+    goto oom_strtab;
 
-  /* Update the strtab, and all refs.  */
+  /* Update all refs: also update the strtab appropriately.  */
   for (i = 0; i < s.strtab_count; i++)
     {
-      strcpy (&strtab.cts_strs[cur_stroff], sorttab[i]->csa_str);
-      ctf_str_update_refs (sorttab[i], cur_stroff);
-      cur_stroff += strlen (sorttab[i]->csa_str) + 1;
+      if (sorttab[i]->csa_offset)
+	{
+	  /* External strtab entry: populate the synthetic external strtab.
+
+	     This is safe because you cannot ctf_rollback to before the point
+	     when a ctf_update is done, and the strtab is written at ctf_update
+	     time.  So any atoms we reference here are sure to stick around
+	     until ctf_file_close.  */
+
+	  any_external = 1;
+	  ctf_str_update_refs (sorttab[i], sorttab[i]->csa_offset);
+	  if (ctf_dynhash_insert (fp->ctf_syn_ext_strtab,
+				  (void *) (uintptr_t) sorttab[i]->csa_offset,
+				  (void *) sorttab[i]->csa_str) < 0)
+	    goto oom_strtab;
+	}
+      else
+	{
+	  /* Internal strtab entry: actually add to the string table.  */
+
+	  ctf_str_update_refs (sorttab[i], cur_stroff);
+	  strcpy (&strtab.cts_strs[cur_stroff], sorttab[i]->csa_str);
+	  cur_stroff += strlen (sorttab[i]->csa_str) + 1;
+	}
     }
   free (sorttab);
 
+  if (!any_external)
+    {
+      ctf_dynhash_destroy (fp->ctf_syn_ext_strtab);
+      fp->ctf_syn_ext_strtab = NULL;
+    }
+
+  return strtab;
+
+ oom_strtab:
+  free (strtab.cts_strs);
+  strtab.cts_strs = NULL;
+ oom_sorttab:
+  free (sorttab);
+ oom:
   return strtab;
 }