libctf: a couple of small error-handling fixes

Out-of-memory errors initializing the string atoms table were
disregarded (though they would have caused a segfault very shortly
afterwards).  Errors hashing types during deduplication were only
reported if they happened on the output dict, which is almost never the
case (most errors are going to be on the dict we're working over, which
is going to be one of the inputs).  (The error was detected in both
cases, but the errno was extracted from the wrong dict.)

libctf/ChangeLog
2021-03-18  Nick Alcock  <nick.alcock@oracle.com>

	* ctf-dedup.c (ctf_dedup_rhash_type): Report errors on the input
	dict properly.
	* ctf-open.c (ctf_bufopen_internal): Report errors initializing
	the atoms table.

1 files changed, 6 insertions, 1 deletions

diff --git a/libctf/ctf-open.c b/libctf/ctf-open.c
index c2d9a33..8d11134 100644
--- a/libctf/ctf-open.c
+++ b/libctf/ctf-open.c
@@ -1545,7 +1545,12 @@ ctf_bufopen_internal (const ctf_sect_t *ctfsect, const ctf_sect_t *symsect,
      ctf_set_base().  */
 
   ctf_set_version (fp, hp, hp->cth_version);
-  ctf_str_create_atoms (fp);
+  if (ctf_str_create_atoms (fp) < 0)
+    {
+      err = ENOMEM;
+      goto bad;
+    }
+
   fp->ctf_parmax = CTF_MAX_PTYPE;
   memcpy (&fp->ctf_data, ctfsect, sizeof (ctf_sect_t));