libctf: get the encoding of non-ints/fps in the dynamic space right

If you call ctf_type_encoding() on a slice, you are meant to get the
encoding of the slice with the format of the underlying type.  If
you call it on a non-int, non-fp, non-slice, you're meant to get the
error ECTF_INTNOTFP.

None of this was implemented for types in the dynamic space (which, now,
is *all* types in writable containers).  Instead, we were always
returning the encoding as if it were a float, which for all other types
consulted the wrong part of a discriminated union and returned garbage.
(Curiously, existing users were more disturbed by the lack of an error
in the non-int/fp/slice case than they were about getting garbage back.)

libctf/
	* ctf-types.c (ctf_type_encoding): Fix the dynamic case to
	work right for non-int/fps.

1 files changed, 21 insertions, 1 deletions

diff --git a/libctf/ctf-types.c b/libctf/ctf-types.c
index ec221d7..27cbfb9 100644
--- a/libctf/ctf-types.c
+++ b/libctf/ctf-types.c
@@ -739,7 +739,27 @@ ctf_type_encoding (ctf_file_t *fp, ctf_id_t type, ctf_encoding_t *ep)
 
   if ((dtd = ctf_dynamic_type (ofp, type)) != NULL)
     {
-      *ep = dtd->dtd_u.dtu_enc;
+      switch (LCTF_INFO_KIND (fp, tp->ctt_info))
+	{
+	case CTF_K_INTEGER:
+	case CTF_K_FLOAT:
+	  *ep = dtd->dtd_u.dtu_enc;
+	  break;
+	case CTF_K_SLICE:
+	  {
+	    const ctf_slice_t *slice;
+	    ctf_encoding_t underlying_en;
+	    slice = &dtd->dtd_u.dtu_slice;
+
+	    data = ctf_type_encoding (fp, slice->cts_type, &underlying_en);
+	    ep->cte_format = underlying_en.cte_format;
+	    ep->cte_offset = slice->cts_offset;
+	    ep->cte_bits = slice->cts_bits;
+	    break;
+	  }
+	default:
+	  return (ctf_set_errno (ofp, ECTF_NOTINTFP));
+	}
       return 0;
     }