| Age | Commit message (Collapse) | Author | Files | Lines |
|---|
|
GCC permits not only unnamed structs and unions, but cv-qualified ones.
Our earlier fix in 6c3a38777b38a2ad87e2b2bcec4567578d1c83ec supported
unnamed structs and unions, but only unqualified ones.
Resolving away cvr-quals of nameless fields (and, irrelevantly, typedefs)
is easy and fixes this problem.
Tests adjusted accordingly.
libctf/
PR libctf/32746
* ctf-types.c (ctf_member_next): Resolve away cv-quals.
(ctf_member_info): Likewise.
* testsuite/libctf-lookup/struct-iteration-ctf.c: Add a cv-qualified
type or two: make sure to keep a non-qualified one.
* testsuite/libctf-lookup/struct-iteration.c: Verify consistency
of ctf_member_next and ctf_member_info.
* testsuite/libctf-lookup/struct-iteration.lk: Adjust.
Tested-by: Stephen Brennan <stephen.s.brennan@oracle.com>
|
|
Slices had a bunch of horrible usability problems. In particular, while
towers of cv-quals are resolved away by functions that need to do it, towers
of cv-quals with slices in the middle are not resolved away by functions
like ctf_enum_value that can see through slices: resolving volatile -> slice
-> const -> enum will leave it with a 'const', which will error pointlessly,
annoying callers, who reasonably expect slices to be more invisible than
this. (The user-callable ctf_type_resolve still does not resolve away
slices, because this is the only way users can see that the slices are there
at all.)
This is induced by a fix for another wart: ctf_add_enumerator does not
resolve anything away at all, so you can't even add enumerators to const or
volatile enums -- and more problematically, you can't add enumerators to
enums with an explicit encoding without resolving away the types by hand,
since ctf_add_enum_encoded works by returning a slice! ctf_add_enumerator
now resolves away all of those, so any cvr-or-typedef-or-slice-qual
terminating in an enum can be added to, exactly as callers likely expect.
(New tests added.)
libctf/
* ctf-create.c (ctf_add_enumerator): Resolve away cvr-qualness.
* ctf-types.c (ctf_type_resolve_unsliced): Don't terminate at
the first slice.
* testsuite/libctf-writable/slice-of-slice.*: New test.
|
|
|
|
Previous code included the full $srcdir pathnames in the individual
subtest PASS/FAIL names, which makes it difficult to compute
comparisons or regressions between test runs on different machines.
This version switches to the basename only, which are common.
Signed-off-by: Frank Ch. Eigler <fche@redhat.com>
|
|
The recent change to detect duplicate enum values and return ECTF_DUPLICATE
when found turns out to perturb a great many callers. In particular, the
pahole-created kernel BTF has the same problem we historically did, and
gleefully emits duplicated enum constants in profusion. Handling the
resulting duplicate errors from BTF -> CTF converters reasonably is
unreasonably difficult (it amounts to forcing them to skip some types or
reimplement the deduplicator).
So let's step back a bit. What we care about mostly is that the
deduplicator treat enums with conflicting enumeration constants as
conflicting types: programs that want to look up enumeration constant ->
value mappings using the new APIs to do so might well want the same checks
to apply to any ctf_add_* operations they carry out (and since they're
*using* the new APIs, added at the same time as this restriction was
imposed, there is likely to be no negative consequence of this).
So we want some way to allow processes that know about duplicate detection
to opt into it, while allowing everyone else to stay clear of it: but we
want ctf_link to get this behaviour even if its caller has opted out.
So add a new concept to the API: dict-wide CTF flags, set via
ctf_dict_set_flag, obtained via ctf_dict_get_flag. They are not bitflags
but simple arbitrary integers and an on/off value, stored in an unspecified
manner (the one current flag, we translate into an LCTF_* flag value in the
internal ctf_dict ctf_flags word). If you pass in an invalid flag or value
you get a new ECTF_BADFLAG error, so the caller can easily tell whether
flags added in future are valid with a particular libctf or not.
We check this flag in ctf_add_enumerator, and set it around the link
(including on child per-CU dicts). The newish enumerator-iteration test is
souped up to check the semantics of the flag as well.
The fact that the flag can be set and unset at any time has curious
consequences. You can unset the flag, insert a pile of duplicates, then set
it and expect the new duplicates to be detected, not only by
ctf_add_enumerator but also by ctf_lookup_enumerator. This means we now
have to maintain the ctf_names and conflicting_enums enum-duplication
tracking as new enums are added, not purely as the dict is opened.
Move that code out of init_static_types_internal and into a new
ctf_track_enumerator function that addition can also call.
(None of this affects the file format or serialization machinery, which has
to be able to handle duplicate enumeration constants no matter what.)
include/
* ctf-api.h (CTF_ERRORS) [ECTF_BADFLAG]: New.
(ECTF_NERR): Update.
(CTF_STRICT_NO_DUP_ENUMERATORS): New flag.
(ctf_dict_set_flag): New function.
(ctf_dict_get_flag): Likewise.
libctf/
* ctf-impl.h (LCTF_STRICT_NO_DUP_ENUMERATORS): New flag.
(ctf_track_enumerator): Declare.
* ctf-dedup.c (ctf_dedup_emit_type): Set it.
* ctf-link.c (ctf_create_per_cu): Likewise.
(ctf_link_deduplicating_per_cu): Likewise.
(ctf_link): Likewise.
(ctf_link_write): Likewise.
* ctf-subr.c (ctf_dict_set_flag): New function.
(ctf_dict_get_flag): New function.
* ctf-open.c (init_static_types_internal): Move enum tracking to...
* ctf-create.c (ctf_track_enumerator): ... this new function.
(ctf_add_enumerator): Call it.
* libctf.ver: Add the new functions.
* testsuite/libctf-lookup/enumerator-iteration.c: Test them.
|
|
Commit 483546ce4f3 ("libctf: make ctf_serialize() actually serialize")
accidentally broke dict compression. There were two bugs:
- ctf_arc_write_one_ctf was still making its own decision about
whether to compress the dict via direct ctf_size comparison, which is
unfortunate because now that it no longer calls ctf_serialize itself,
ctf_size is always zero when it does this: it should let the writing
functions decide on the threshold, which they contain code to do which is
simply not used for lack of one trivial wrapper to write to an fd and
also provide a compression threshold
- ctf_write_mem, the function underlying all writing as of the commit
above, was calling zlib's compressBound and avoiding compression if this
returned a value larger than the input. Unfortunately compressBound does
not do a trial compression and determine whether the result is
compressible: it just adds zlib header sizes to the value passed in, so
our test would *always* have concluded that the value was incompressible!
Avoid by simply always compressing if the raw size is larger than the
threshold: zlib is quite clever enough to avoid actually compressing
if the data is incompressible.
Add a testcase for this.
libctf/
* ctf-impl.h (ctf_write_thresholded): New...
* ctf-serialize.c (ctf_write_thresholded): ... defined here,
a wrapper around...
(ctf_write_mem): ... this. Don't check compressibility.
(ctf_compress_write): Reimplement as a ctf_write_thresholded
wrapper.
(ctf_write): Likewise.
* ctf-archive.c (arc_write_one_ctf): Just call
ctf_write_thresholded rather than trying to work out whether
to compress.
* testsuite/libctf-writable/ctf-compressed.*: New test.
|
|
If you deduplicate non-root-visible types, the resulting type should still
be non-root-visible! We were promoting all such types to root-visible, and
re-demoting them only if their names collided (which might happen on
cu-mapped links if multiple compilation units with conflicting types are
fused into one child dict).
This "worked" before now, in that linking at least didn't fail (if you don't
mind having your non-root flag value destroyed if you're adding
non-root-visible types), but now that conflicting enumerators cause their
containing enums to become conflicted (enums which might have *different
names*), this caused the linker to crash when it hit two enumerators with
conflicting values.
Not testable in ld because cu-mapped links are not exposed to ld, but can be
tested via direct creation of libraries and calls to ctf_link directly.
(This also tests the ctf_dump non-root type printout, which before now
was untested.)
libctf/
* ctf-dedup.c (ctf_dedup_emit_type): Non-root-visible input types
should be emitted as non-root-visible output types.
* testsuite/libctf-writable/ctf-nonroot-linking.c: New test.
* testsuite/libctf-writable/ctf-nonroot-linking.lk: New test.
|
|
Most of these are harmless, but some of the type confusions and especially
a missing ctf_strerror() on an error path were actual bugs that could
have resulted in test failures crashing rather than printing an error
message.
libctf/
* testsuite/libctf-lookup/enumerator-iteration.c: Fix type
confusion, signedness confusion and a missing ctf_errmsg().
* testsuite/libctf-regression/libctf-repeat-cu-main.c: Return 0 from
the test function.
* testsuite/libctf-regression/open-error-free.c: Fix signedness
confusion.
* testsuite/libctf-regression/zrewrite.c: Remove unused label.
|
|
Three new functions for looking up the enum type containing a given
enumeration constant, and optionally that constant's value.
The simplest, ctf_lookup_enumerator, looks up a root-visible enumerator by
name in one dict: if the dict contains multiple such constants (which is
possible for dicts created by older versions of the libctf deduplicator),
ECTF_DUPLICATE is returned.
The next simplest, ctf_lookup_enumerator_next, is an iterator which returns
all enumerators with a given name in a given dict, whether root-visible or
not.
The most elaborate, ctf_arc_lookup_enumerator_next, finds all
enumerators with a given name across all dicts in an entire CTF archive,
whether root-visible or not, starting looking in the shared parent dict;
opened dicts are cached (as with all other ctf_arc_*lookup functions) so
that repeated use does not incur repeated opening costs.
All three of these return enumerator values as int64_t: unfortunately, API
compatibility concerns prevent us from doing the same with the other older
enum-related functions, which all return enumerator constant values as ints.
We may be forced to add symbol-versioning compatibility aliases that fix the
other functions in due course, bumping the soname for platforms that do not
support such things.
ctf_arc_lookup_enumerator_next is implemented as a nested ctf_archive_next
iterator, and inside that, a nested ctf_lookup_enumerator_next iterator
within each dict. To aid in this, add support to ctf_next_t iterators for
iterators that are implemented in terms of two simultaneous nested iterators
at once. (It has always been possible for callers to use as many nested or
semi-overlapping ctf_next_t iterators as they need, which is one of the
advantages of this style over the _iter style that calls a function for each
thing iterated over: the iterator change here permits *ctf_next_t iterators
themselves* to be implemented by iterating using multiple other iterators as
part of their internal operation, transparently to the caller.)
Also add a testcase that tests all these functions (which is fairly easy
because ctf_arc_lookup_enumerator_next is implemented in terms of
ctf_lookup_enumerator_next) in addition to enumeration addition in
ctf_open()ed dicts, ctf_add_enumerator duplicate enumerator addition, and
conflicting enumerator constant deduplication.
include/
* ctf-api.h (ctf_lookup_enumerator): New.
(ctf_lookup_enumerator_next): Likewise.
(ctf_arc_lookup_enumerator_next): Likewise.
libctf/
* libctf.ver: Add them.
* ctf-impl.h (ctf_next_t) <ctn_next_inner>: New.
* ctf-util.c (ctf_next_copy): Copy it.
(ctf_next_destroy): Destroy it.
* ctf-lookup.c (ctf_lookup_enumerator): New.
(ctf_lookup_enumerator_next): New.
* ctf-archive.c (ctf_arc_lookup_enumerator_next): New.
* testsuite/libctf-lookup/enumerator-iteration.*: New test.
* testsuite/libctf-lookup/enum-ctf-2.c: New test CTF, used by the
above.
|
|
The libctf-regression/open-error-free.c test works by interposing malloc
and counting mallocs and frees across libctf operations. This only
works under suitably-interposable mallocs on systems supporting
dlsym (RTLD_NEXT, ...), so its operation is restricted to glibc
systems for now, but also it interacts badly with valgrind, which
interposes malloc itself. Detect a running valgrind and skip the test.
Add new facilities allowing libctf lookup tests to declare themselves
unsupported, by printing "UNSUPPORTED: " and then some meaningful
message instead of their normal output.
libctf/
* configure.ac: Check for <valgrind/valgrind.h>.
* config.h.in: Regenerate.
* configure: Likewise.
* testsuite/lib/ctf-lib.exp (run_lookup_test): Add support for
UNSUPPORTED tests.
* testsuite/libctf-regression/open-error-free.c: When running
under valgrind, this test is unsupported.
|
|
|
|
* testsuite/libctf-regression/open-error-free.c (main): Correct
format length modifier.
|
|
Ever since commit 1fa7a0c24e78e7f ("libctf: sort out potential refcount
loops") ctf_dict_close has only freed anything if the refcount on entry
to the function is precisely 1. >1 obviously just decrements the
refcount, but the linker machinery can sometimes cause freeing to recurse
from a dict to another dict and then back to the first dict again, so
we interpret a refcount of 0 as an indication that this is a recursive call
and we should just return, because a caller is already freeing this dict.
Unfortunately there is one situation in which this is not true: the bad:
codepath in ctf_bufopen entered when opening fails. Because the refcount is
bumped only at the very end of ctf_bufopen, any failure causes
ctf_dict_close to be entered with a refcount of zero, and it frees nothing
and we leak the entire dict.
The solution is to bump the refcount to 1 right before freeing... but this
codepath is clearly delicate enough that we need to properly validate it,
so we add a test that uses malloc interposition to count allocations and
frees, creates a dict, writes it out, intentionally corrupts it (by setting
a bunch of bytes after the header to a value high enough that it is
definitely not a valid CTF type kind), then tries to open it again and
counts the malloc/free pairs to make sure they're matched. (Test run only
on *-linux-gnu, because malloc interposition is not a thing you can rely
upon working everywhere, and this test is not arch-dependent so if it
passes on one arch it can be assumed to pass on all of them.)
libctf/
* ctf-open.c (ctf_bufopen): Bump the refcount on failure.
* testsuite/libctf-regression/open-error-free.*: New test.
|
|
This .lk option lets you run the lookup program via a wrapper executable.
For example, to run under valgrind and check for leaks (albeit noisily
because of the libtool shell script wrapper):
libctf/
* testsuite/lib/ctf-lib.exp (run_lookup_test): Add wrapper.
|
|
This .lk option lets you execute particular tests only on specific host
architectures.
libctf/
* testsuite/lib/ctf-lib.exp (run_lookup_test): Add host.
|
|
This .lk option lets you link the lookup program with extra libraries
in addition to -lctf.
libctf/
* testsuite/lib/ctf-lib.exp (run_lookup_test): Add lookup_link.
|
|
* testsuite/libctf-regression/gzrewrite.c (main): Don't overflow
"a" buffer in "after adding types" check.
* testsuite/libctf-regression/zrewrite.c (main): Likewise.
|
|
Seen on 64-bit targets.
ERROR: compilation of lookup program .../libctf-regression/gzrewrite.c failed
* testsuite/libctf-regression/gzrewrite.c (main): Use %zu to
print size_t values.
* testsuite/libctf-regression/zrewrite.c (main): Likewise.
|
|
This purely serves to make it easier to interpret valgrind output.
No functional effect.
libctf/
* testsuite/libctf-lookup/conflicting-type-syms.c: Free everything.
|
|
Now there's a chance of it actually working, we can add more tests for
the long-broken dict read-and-rewrite cases. This is the first ever
test for the (rarely-used, unpleasant, and until recently completely
broken) ctf_gzwrite function.
libctf/
* testsuite/libctf-regression/gzrewrite*: New test.
* testsuite/libctf-regression/zrewrite*: Likewise.
|
|
ctf_serialize() evolved from the old ctf_update(), which mutated the
in-memory CTF dict to make all the dynamic in-memory types into static,
unchanging written-to-the-dict types (by deserializing and reserializing
it): back in the days when you could only do type lookups on static types,
this meant you could see all the types you added recently, at the small,
small cost of making it impossible to change those older types ever again
and inducing an amortized O(n^2) cost if you actually wanted to add
references to types you added at arbitrary times to later types.
It also reset things so that ctf_discard() would throw away only types you
added after the most recent ctf_update() call.
Some time ago this was all changed so that you could look up dynamic types
just as easily as static types: ctf_update() changed so that only its
visible side-effect of affecting ctf_discard() remained: the old
ctf_update() was renamed to ctf_serialize(), made internal to libctf, and
called from the various functions that wrote files out.
... but it was still working by serializing and deserializing the entire
dict, swapping out its guts with the newly-serialized copy in an invasive
and horrible fashion that coupled ctf_serialize() to almost every field in
the ctf_dict_t. This is totally useless, and fixing it is easy: just rip
all that code out and have ctf_serialize return a serialized representation,
and let everything use that directly. This simplifies most of its callers
significantly.
(It also points up another bug: ctf_gzwrite() failed to call ctf_serialize()
at all, so it would only ever work for a dict you just ctf_write_mem()ed
yourself, just for its invisible side-effect of serializing the dict!)
This lets us simplify away a bunch of internal-only open-side functionality
for overriding the syn_ext_strtab and some just-added functionality for
forcing in an existing atoms table, without loss of functionality, and lets
us lift the restriction on reserializing a dict that was ctf_open()ed rather
than being ctf_create()d: it's now perfectly OK to open a dict, modify it
(except for adding members to existing structs, unions, or enums, which
fails with -ECTF_RDONLY), and write it out again, just as one would expect.
libctf/
* ctf-serialize.c (ctf_symtypetab_sect_sizes): Fix typos.
(ctf_type_sect_size): Add static type sizes too.
(ctf_serialize): Return the new dict rather than updating the
existing dict. No longer fail for dicts with static types;
copy them onto the start of the new types table.
(ctf_gzwrite): Actually serialize before gzwriting.
(ctf_write_mem): Improve forced (test-mode) endian-flipping:
flip dicts even if they are too small to be compressed.
Improve confusing variable naming.
* ctf-archive.c (arc_write_one_ctf): Don't bother to call
ctf_serialize: both the functions we call do so.
* ctf-string.c (ctf_str_create_atoms): Drop serializing case
(atoms arg).
* ctf-open.c (ctf_simple_open): Call ctf_bufopen directly.
(ctf_simple_open_internal): Delete.
(ctf_bufopen_internal): Delete/rename to ctf_bufopen: no
longer bother with syn_ext_strtab or forced atoms table,
serialization no longer needs them.
* ctf-create.c (ctf_create): Call ctf_bufopen directly.
* ctf-impl.h (ctf_str_create_atoms): Drop atoms arg.
(ctf_simple_open_internal): Delete.
(ctf_bufopen_internal): Likewise.
(ctf_serialize): Adjust.
* testsuite/libctf-lookup/add-to-opened.c: Adjust now that
this is supposed to work.
|
|
libctf has long declared deserialized dictionaries (out of files or ELF
sections or memory buffers or whatever) to be read-only: back in the
furthest prehistory this was not the case, in that you could add a
few sorts of type to such dicts, but attempting to do so often caused
horrible memory corruption, so I banned the lot.
But it turns out real consumers want it (notably DTrace, which
synthesises pointers to types that don't have them and adds them to the
ctf_open()ed dicts if it needs them). Let's bring it back again, but
without the memory corruption and without the massive code duplication
required in days of yore to distinguish between static and dynamic
types: the representation of both types has been identical for a few
years, with the only difference being that types as a whole are stored in
a big buffer for types read in via ctf_open and per-type hashtables for
newly-added types.
So we discard the internally-visible concept of "readonly dictionaries"
in favour of declaring the *range of types* that were already present
when the dict was read in to be read-only: you can't modify them (say,
by adding members to them if they're structs, or calling ctf_set_array
on them), but you can add more types and point to them. (The API
remains the same, with calls sometimes returning ECTF_RDONLY, but now
they do so less often.)
This is a fairly invasive change, mostly because code written since the
ban was introduced didn't take the possibility of a static/dynamic split
into account. Some of these irregularities were hard to define as
anything but bugs.
Notably:
- The symbol handling was assuming that symbols only needed to be
looked for in dynamic hashtabs or static linker-laid-out indexed/
nonindexed layouts, but now we want to check both in case people
added more symbols to a dict they opened.
- The code that handles type additions wasn't checking to see if types
with the same name existed *at all* (so you could do
ctf_add_typedef (fp, "foo", bar) repeatedly without error). This
seems reasonable for types you just added, but we probably *do* want
to ban addition of types with names that override names we already
used in the ctf_open()ed portion, since that would probably corrupt
existing type relationships. (Doing things this way also avoids
causing new errors for any existing code that was doing this sort of
thing.)
- ctf_lookup_variable entirely failed to work for variables just added
by ctf_add_variable: you had to write the dict out and read it back
in again before they appeared.
- The symbol handling remembered what symbols you looked up but didn't
remember their types, so you could look up an object symbol and then
find it popping up when you asked for function symbols, which seems
less than ideal. Since we had to rejig things enough to be able to
distinguish function and object symbols internally anyway (in order
to give suitable errors if you try to add a symbol with a name that
already existed in the ctf_open()ed dict), this bug suddenly became
more visible and was easily fixed.
We do not (yet) support writing out dicts that have been previously read
in via ctf_open() or other deserializer (you can look things up in them,
but not write them out a second time). This never worked, so there is
no incompatibility; if it is needed at a later date, the serializer is a
little bit closer to having it work now (the only table we don't deal
with is the types table, and that's because the upcoming CTFv4 changes
are likely to make major changes to the way that table is represented
internally, so adding more code that depends on its current form seems
like a bad idea).
There is a new testcase that tests much of this, in particular that
modification of existing types is still banned and that you can add new
ones and chase them without error.
libctf/
* ctf-impl.h (struct ctf_dict.ctf_symhash): Split into...
(ctf_dict.ctf_symhash_func): ... this and...
(ctf_dict.ctf_symhash_objt): ... this.
(ctf_dict.ctf_stypes): New, counts static types.
(LCTF_INDEX_TO_TYPEPTR): Use it instead of CTF_RDWR.
(LCTF_RDWR): Deleted.
(LCTF_DIRTY): Renumbered.
(LCTF_LINKING): Likewise.
(ctf_lookup_variable_here): New.
(ctf_lookup_by_sym_or_name): Likewise.
(ctf_symbol_next_static): Likewise.
(ctf_add_variable_forced): Likewise.
(ctf_add_funcobjt_sym_forced): Likewise.
(ctf_simple_open_internal): Adjust.
(ctf_bufopen_internal): Likewise.
* ctf-create.c (ctf_grow_ptrtab): Adjust a lot to start with.
(ctf_create): Migrate a bunch of initializations into bufopen.
Force recreation of name tables. Do not forcibly override the
model, let ctf_bufopen do it.
(ctf_static_type): New.
(ctf_update): Drop LCTF_RDWR check.
(ctf_dynamic_type): Likewise.
(ctf_add_function): Likewise.
(ctf_add_type_internal): Likewise.
(ctf_rollback): Check ctf_stypes, not LCTF_RDWR.
(ctf_set_array): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_enumerator): Likewise (only on the target dict).
(ctf_add_member_offset): Likewise.
(ctf_add_generic): Drop LCTF_RDWR check. Ban addition of types
with colliding names.
(ctf_add_forward): Note safety under the new rules.
(ctf_add_variable): Split all but the existence check into...
(ctf_add_variable_forced): ... this new function.
(ctf_add_funcobjt_sym): Likewise...
(ctf_add_funcobjt_sym_forced): ... for this new function.
* ctf-link.c (ctf_link_add_linker_symbol): Ban calling on dicts
with any stypes.
(ctf_link_add_strtab): Likewise.
(ctf_link_shuffle_syms): Likewise.
(ctf_link_intern_extern_string): Note pre-existing prohibition.
* ctf-lookup.c (ctf_lookup_by_id): Drop LCTF_RDWR check.
(ctf_lookup_variable): Split out looking in a dict but not
its parent into...
(ctf_lookup_variable_here): ... this new function.
(ctf_lookup_symbol_idx): Track whether looking up a function or
object: cache them separately.
(ctf_symbol_next): Split out looking in non-dynamic symtypetab
entries to...
(ctf_symbol_next_static): ... this new function. Don't get confused
by the simultaneous presence of static and dynamic symtypetab entries.
(ctf_try_lookup_indexed): Don't waste time looking up symbols by
index before there can be any idea how symbols are numbered.
(ctf_lookup_by_sym_or_name): Distinguish between function and
data object lookups. Drop LCTF_RDWR.
(ctf_lookup_by_symbol): Adjust.
(ctf_lookup_by_symbol_name): Likewise.
* ctf-open.c (init_types): Rename to...
(init_static_types): ... this. Drop LCTF_RDWR. Populate ctf_stypes.
(ctf_simple_open): Drop writable arg.
(ctf_simple_open_internal): Likewise.
(ctf_bufopen): Likewise.
(ctf_bufopen_internal): Populate fields only used for writable dicts.
Drop LCTF_RDWR.
(ctf_dict_close): Cater for symhash cache split.
* ctf-serialize.c (ctf_serialize): Use ctf_stypes, not LCTF_RDWR.
* ctf-types.c (ctf_variable_next): Drop LCTF_RDWR.
* testsuite/libctf-lookup/add-to-opened*: New test.
|
|
The intent of the name lookup code was for lookups to yield non-bitfield
basic types except if none existed with a given name, and only then
return bitfield types with that name. Unfortunately, the code as
written only does this if the base type has a type ID higher than all
bitfield types, which is most unlikely (the opposite is almost always
the case).
Adjust it so that what ends up in the name table is the highest-width
zero-offset type with a given name, if any such exist, and failing that
the first type with that name we see, no matter its offset. (We don't
define *which* bitfield type you get, after all, so we might as well
just stuff in the first we find.)
Reported by Stephen Brennan <stephen.brennan@oracle.com>.
libctf/
* ctf-open.c (init_types): Modify to allow some lookups during open;
detect bitfield name reuse and prefer less bitfieldy types.
* testsuite/libctf-writable/libctf-bitfield-name-lookup.*: New test.
|
|
Just because a path is an error path doesn't mean the program terminates
there if you don't ask it to. And we don't want to -- but that means
we need to initialize the variables that are missed if an error happens to
*something*. Type ID 0 (unimplemented) will do: it'll induce further
ECTF_BADID errors, but that's no bad thing.
libctf/ChangeLog:
* testsuite/libctf-writable/libctf-errors.c: Initialize variables.
|
|
Adds two new external authors to etc/update-copyright.py to cover
bfd/ax_tls.m4, and adds gprofng to dirs handled automatically, then
updates copyright messages as follows:
1) Update cgen/utils.scm emitted copyrights.
2) Run "etc/update-copyright.py --this-year" with an extra external
author I haven't committed, 'Kalray SA.', to cover gas testsuite
files (which should have their copyright message removed).
3) Build with --enable-maintainer-mode --enable-cgen-maint=yes.
4) Check out */po/*.pot which we don't update frequently.
|
|
The fixes applied a few years ago to resolve confusions between parent and
child dicts at lookup time also apply in various forms to creation. In
general, if you have a type in a parent dict ctf_imported into a child and
you do something to it, and the parent dict is writable (created via
ctf_create, not opened via ctf_open*) it should work just the same to make
changes to that type via a child dict as it does to make the change
to the parent dict directly -- and nothing you're prohibited from doing
to the parent dict when done directly should be allowed just because
you're doing it via a child.
Specifically, the following don't work when doing things from the child, but
should:
- adding a member of a type in the parent to a struct or union in the
parent via ctf_add_member or ctf_add_member_offset: this yields
ECTF_BADID
- adding a member of a type in the parent to a struct or union in the
parent via ctf_add_member_encoded: this dumps core (!).
- adding an enumerand to an enumerator in the parent: this yields
ECTF_BADID
- setting the properties of an array in the parent via ctf_set_array;
this yields ECTF_BADID
Relatedly, some things work when doing things via a child that should fail,
yielding a CTF dictionary with invalid content (readable, but meaningless):
in particular, you can add a child type to a struct in the parent via
any of the ctf_add_member* family and nothing complains at all, even though
you should never be able to add references to children to parents (since any
given parent can be associated with many different children).
A family of tests is added to check each of these cases independently, since
some can result in coredumps and it would be nice to test the other cases
even if some dump core. They use a common library to do all the actual
work. The set of affected API calls was determined by code inspection
(auditing all calls to ctf_dtd_lookup): it's possible that I missed a few,
but I doubt it, since other cases use ctf_lookup* functions, which already
climb to the parent where appropriate.
libctf/ChangeLog:
PR libctf/30985
* ctf-create.c (ctf_dtd_lookup): Traverse to parents if necessary.
(ctf_set_array): Likewise. Report errors on the child; require
both parent and child to be writable.
(ctf_add_enumerator): Likewise.
(ctf_add_member_offset): Likewise. Prohibit addition of child types
to structs in the parent.
(ctf_add_member_encoded): Do not dereference a NULL dtd: report
ECTF_BADID instead.
* ctf-string.c (ctf_str_add_ref_internal): Report ENOMEM on the
dict if addition of a string ref fails.
* testsuite/libctf-writable/parent-child-dtd-crash-lib.c: New library.
* testsuite/libctf-writable/parent-child-dtd-enum.*: New test.
* testsuite/libctf-writable/parent-child-dtd-enumerator.*: New test.
* testsuite/libctf-writable/parent-child-dtd-member-encoded.*: New test.
* testsuite/libctf-writable/parent-child-dtd-member-offset.*: New test.
* testsuite/libctf-writable/parent-child-dtd-set-array.*: New test.
* testsuite/libctf-writable/parent-child-dtd-struct.*: New test.
* testsuite/libctf-writable/parent-child-dtd-union.*: New test.
|
|
We do this as a writable test because the only known-affected platforms
(with ssize_t longer than unsigned long) use PE, and we do not have support
for CTF linkage in the PE linker yet.
PR libctf/30836
* libctf/testsuite/libctf-writable/libctf-errors.*: New test.
|
|
CTF dicts have per-dict errno values: as with other errno values these
are set on error and left unchanged on success. This means that all
errors *must* set the CTF errno: if a call leaves it unchanged, the
caller is apt to find a previous, lingering error and misinterpret
it as the real error.
There are many places in libctf where we carry out operations on parent
dicts as a result of carrying out other user-requested operations on
child dicts (e.g. looking up information on a pointer to a type will
look up the type as well: the pointer might well be in a child and the
type it's a pointer to in the parent). Those operations on the parent
might fail; if they do, the error must be correctly reflected on the
child that the user-visible operation was carried out on. In many
places this was not happening.
So, audit and fix all those places. Add tests for as many of those
cases as possible so they don't regress.
libctf/
* ctf-create.c (ctf_add_slice): Use the original dict.
* ctf-lookup.c (ctf_lookup_variable): Propagate errors.
(ctf_lookup_symbol_idx): Likewise.
* ctf-types.c (ctf_member_next): Likewise.
(ctf_type_resolve_unsliced): Likewise.
(ctf_type_aname): Likewise.
(ctf_member_info): Likewise.
(ctf_type_rvisit): Likewise.
(ctf_func_type_info): Set the error on the right dict.
(ctf_type_encoding): Use the original dict.
* testsuite/libctf-writable/error-propagation.*: New test.
|
|
The newly-introduced libctf-lookup unnamed-field-info test checks
C compiler-observed field offsets against libctf-computed ones
by #including the testcase in the lookup runner as well as
generating CTF for it. This only works if the host, on which
the lookup runner is compiled and executed, is the same architecture as
the target, for which the CTF is generated: when crossing, the trick
may fail.
So pass down an indication of whether this is a cross into the
testsuite, and add a new no_cross flag to .lk files that is used to
suppress test execution when a cross-compiler is being tested.
libctf/
* Makefile.am (check_DEJAGNU): Pass down TEST_CROSS.
* Makefile.in: Regenerated.
* testsuite/lib/ctf-lib.exp (run_lookup_test): Use it to
implement the new no_cross option.
* testsuite/libctf-lookup/unnamed-field-info.lk: Mark as
no_cross.
|
|
We were failing to add the offsets of the containing struct/union
in this case, leading to all offsets being relative to the unnamed
struct/union itself.
libctf/
PR libctf/30264
* ctf-types.c (ctf_member_info): Add the offset of the unnamed
member of the current struct as necessary.
* testsuite/libctf-lookup/unnamed-field-info*: New test.
|
|
The libctf testsuite uses Tcl try/catch to trap run_output errors. This
is only supported in reasonably recent Tcls, so we detect the lack of
try/catch and suppress the testsuite via an Automake conditional in its
absence.
But this turns out not to work: Automake produces a check-DEJAGNU target
regardless of the value of this conditional and sticks it in an
unconditionally-executed part of the makefile, so the testsuite gets
executed anyway, and fails with a nasty-looking syntax error. We can't
disable it by taking "dejagnu" out of AUTOMAKE_OPTIONS, because if you
do that Automake stops you using RUNTEST, RUNTESTFLAGS and other
variables users would expect to work.
So move to disabling the testsuite from inside the testsuite itself,
importing the value of the former Automake conditional as a Tcl variable
and exiting very early in default.exp if it's false.
* configure.ac (TCL_TRY): No longer an Automake conditional.
Rename to...
(HAVE_TCL_TRY): ... this.
* Makefile.am: Drop TCL_TRY.
(development.exp): Set have_tcl_try.
* testsuite/config/default.exp: Exit if have_tcl_try is false.
* configure: Regenerated.
* Makefile.in: Likewise.
|
|
The newer update-copyright.py fixes file encoding too, removing cr/lf
on binutils/bfdtest2.c and ld/testsuite/ld-cygwin/exe-export.exp, and
embedded cr in binutils/testsuite/binutils-all/ar.exp string match.
|
|
We were failing to call prune_warnings appropriately, leading to
false-positive test failures on some platforms (observed on
sparclinux).
libctf/ChangeLog:
* testsuite/lib/ctf-lib.exp: Prune warnings from compiler and
linker output.
* testsuite/libctf-regression/libctf-repeat-cu.exp: Likewise,
and ar output too.
|
|
Right now, if you compile the same .c input repeatedly with CTF enabled
and different compilation flags, then arrange to link all of these
together, then things misbehave in various ways. libctf may conflate
either inputs (if the .o files have the same name, say if they are
stored in different .a archives), or per-CU outputs when conflicting
types are found: the latter can lead to entirely spurious errors when
it tries to produce multiple per-CU outputs with the same name
(discarding all but the last, but then looking for types in the earlier
ones which have just been thrown away).
Fixing this is multi-pronged. Both inputs and outputs need to be
differentiated in the hashtables libctf keeps them in: inputs with the
same cuname and filename need to be considered distinct as long as they
have different associated CTF dicts, and per-CU outputs need to be
considered distinct as long as they have different associated input
dicts. Right now there is nothing tying the two together other than the
CU name: fix this by introducing a new field in the ctf_dict_t named
ctf_link_in_out, which (for input dicts) points to the associated per-CU
output dict (if any), and for output dicts points to the associated
input dict. At creation time the name used is completely arbitrary:
it's only important that it be distinct if CTF dicts are distinct. So,
when a clash is found, adjust the CU name by sticking the number of
elements in the input on the end. At output time, the CU name will
appear in the linked object, so it matters a little more that it look
slightly less ugly: in conflicting cases, append an incrementing
integer, starting at 0.
This naming scheme is not very helpful, but it's hard to see what else
we can do. The input .o name may be the same. The input .a name is not
even visible to ctf_link, and even *that* might be the same, because
.a's can contain many members with the same name, all of which
participate in the link. All we really know is that the two have
distinct dictionaries with distinct types in them, and at least this way
they are all represented, any any symbols, variables etc referring to
those types are accurately stored.
(As a side-effect this also fixes a use-after-free and double-free when
errors are found during variable or symbol emission.)
Use the opportunity to prevent a couple of sources of problems, to wit
changing the active CU mappings when a link has already been done
(no effect on ld, which doesn't use CU mappings at all), and causing
multiple consecutive ctf_link's to have the same net effect as just
doing the last one (no effect on ld, which only ever does one
ctf_link) rather than having the links be a sort of half-incremental
not-really-intended mess.
libctf/ChangeLog:
PR libctf/29242
* ctf-impl.h (struct ctf_dict) [ctf_link_in_out]: New.
* ctf-dedup.c (ctf_dedup_emit_type): Set it.
* ctf-link.c (ctf_link_add_ctf_internal): Set the input
CU name uniquely when clashes are found.
(ctf_link_add): Document what repeated additions do.
(ctf_new_per_cu_name): New, come up with a consistent
name for a new per-CU dict.
(ctf_link_deduplicating): Use it.
(ctf_create_per_cu): Use it, and ctf_link_in_out, and set
ctf_link_in_out properly. Don't overwrite per-CU dicts with
per-CU dicts relating to different inputs.
(ctf_link_add_cu_mapping): Prevent per-CU mappings being set up
if we already have per-CU outputs.
(ctf_link_one_variable): Adjust ctf_link_per_cu call.
(ctf_link_deduplicating_one_symtypetab): Likewise.
(ctf_link_empty_outputs): New, delete all the ctf_link_outputs
and blank out ctf_link_in_out on the corresponding inputs.
(ctf_link): Clarify the effect of multiple ctf_link calls.
Empty ctf_link_outputs if it already exists rather than
having the old output leak into the new link. Fix a variable
name.
* testsuite/config/default.exp (AR): Add.
(OBJDUMP): Likewise.
* testsuite/libctf-regression/libctf-repeat-cu.exp: New test.
* testsuite/libctf-regression/libctf-repeat-cu*: Main program,
library, and expected results for the test.
|
|
The result of running etc/update-copyright.py --this-year, fixing all
the files whose mode is changed by the script, plus a build with
--enable-maintainer-mode --enable-cgen-maint=yes, then checking
out */po/*.pot which we don't update frequently.
The copy of cgen was with commit d1dd5fcc38ead reverted as that commit
breaks building of bfp opcodes files.
|
|
An off-by-one bug in the check for pptrtab lookup meant that we could
access the pptrtab past its bounds (*well* past its bounds),
particularly if we called ctf_lookup_by_name in a child dict with "*foo"
where "foo" is a type that exists in the parent but not the child and no
previous lookups by name have been carried out. (Note that "*foo" is
not even a valid thing to call ctf_lookup_by_name with: foo * is.
Nonetheless, users sometimes do call ctf_lookup_by_name with invalid
content, and it should return ECTF_NOTYPE, not crash.)
ctf_pptrtab_len, as its name suggests (and as other tests of it in
ctf-lookup.c confirm), is one higher than the maximum valid permissible
index, so the comparison is wrong.
(Test added, which should fail pretty reliably in the presence of this
bug on any machine with 4KiB pages.)
libctf/ChangeLog
2021-09-27 Nick Alcock <nick.alcock@oracle.com>
* ctf-lookup.c (ctf_lookup_by_name_internal): Fix pptrtab bounds.
* testsuite/libctf-writable/pptrtab-writable-page-deep-lookup.*:
New test.
|
|
These warnings are all off by default, but if they do fire you get
spurious ERRORs when running make check-libctf.
libctf/ChangeLog
2021-09-27 Nick Alcock <nick.alcock@oracle.com>
* testsuite/libctf-lookup/enum-symbol.c: Remove unused label.
* testsuite/libctf-lookup/conflicting-type-syms.c: Remove unused
variables.
* testsuite/libctf-regression/pptrtab.c: Likewise.
* testsuite/libctf-regression/type-add-unnamed-struct.c: Likewise.
* testsuite/libctf-writable/pptrtab.c: Likewise.
* testsuite/libctf-writable/reserialize-strtab-corruption.c:
Likewise.
* testsuite/libctf-regression/nonstatic-var-section-ld-r.c: Fix
format string.
* testsuite/libctf-regression/nonstatic-var-section-ld.c:
Likewise.
* testsuite/libctf-regression/nonstatic-var-section-ld.lk: Adjust.
* testsuite/libctf-writable/symtypetab-nonlinker-writeout.c: Fix
initializer.
|
|
The top level Makefile, the ld Makefile and others, define
CC_FOR_TARGET to be a compiler for the binutils target machine. This
is the compiler that should be used for almost all tests with C
source. There are _FOR_TARGET versions of CFLAGS, CXX, and CXXFLAGS
too. This was all supposed to work with the testsuite .exp files
using CC for the target compiler, and CC_FOR_HOST for the host
compiler, with the makefiles passing CC=$CC_FOR_TARGET and
CC_FOR_HOST=$CC to the runtest invocation.
One exception to the rule of using CC_FOR_TARGET is the native-only ld
bootstrap test, which uses the newly built ld to link a copy of
itself. Since the files being linked were created with the host
compiler, the boostrap test should use CC and CFLAGS, in case some
host compiler option provides needed libraries automatically.
However, bootstrap.exp used CC where it should have used CC_FOR_HOST.
I set about fixing that problem, then decided that playing games in
the makefiles with CC was a bad idea. Not only is it confusing, but
other dejagnu code knows about CC_FOR_TARGET. See dejagnu/target.exp.
So this patch gets rid of the makefile variable renaming and changes
all the .exp files to use the correct _FOR_TARGET variables.
CC_FOR_HOST and CFLAGS_FOR_HOST disappear. A followup patch will
correct bootstrap.exp to use CFLAGS, and a number of other things I
noticed.
binutils/
* testsuite/lib/binutils-common.exp (run_dump_test): Use
CC_FOR_TARGET and CFLAGS_FOR_TARGET rather than CC and CFLAGS.
ld/
* Makefile.am (check-DEJAGNU): Don't set CC to CC_FOR_TARGET
and similar. Pass variables with unchanged names. Don't set
CC_FOR_HOST or CFLAGS_FOR_HOST.
* Makefile.in: Regenerate.
* testsuite/config/default.exp: Update default CC and similar.
(compiler_supports, plug_opt): Use CC_FOR_TARGET.
* testsuite/ld-cdtest/cdtest.exp: Replace all uses of CC with
CC_FOR_TARGET, and similarly for CFLAGS, CXX and CXXFLAGS.
* testsuite/ld-auto-import/auto-import.exp: Likewise.
* testsuite/ld-cygwin/exe-export.exp: Likewise.
* testsuite/ld-elf/dwarf.exp: Likewise.
* testsuite/ld-elf/indirect.exp: Likewise.
* testsuite/ld-elf/shared.exp: Likewise.
* testsuite/ld-elfcomm/elfcomm.exp: Likewise.
* testsuite/ld-elfvers/vers.exp: Likewise.
* testsuite/ld-elfvsb/elfvsb.exp: Likewise.
* testsuite/ld-elfweak/elfweak.exp: Likewise.
* testsuite/ld-gc/gc.exp: Likewise.
* testsuite/ld-ifunc/ifunc.exp: Likewise.
* testsuite/ld-mn10300/mn10300.exp: Likewise.
* testsuite/ld-pe/pe-compile.exp: Likewise.
* testsuite/ld-pe/pe-run.exp: Likewise.
* testsuite/ld-pe/pe-run2.exp: Likewise.
* testsuite/ld-pie/pie.exp: Likewise.
* testsuite/ld-plugin/lto.exp: Likewise.
* testsuite/ld-plugin/plugin.exp: Likewise.
* testsuite/ld-scripts/crossref.exp: Likewise.
* testsuite/ld-selective/selective.exp: Likewise.
* testsuite/ld-sh/sh.exp: Likewise.
* testsuite/ld-shared/shared.exp: Likewise.
* testsuite/ld-srec/srec.exp: Likewise.
* testsuite/ld-undefined/undefined.exp: Likewise.
* testsuite/ld-unique/unique.exp: Likewise.
* testsuite/ld-x86-64/tls.exp: Likewise.
* testsuite/lib/ld-lib.exp: Likewise.
libctf/
* Makefile.am (check-DEJAGNU): Don't set CC to CC_FOR_TARGET.
Pass CC and CC_FOR_TARGET. Don't set CC_FOR_HOST.
* Makefile.in: Regenerate.
* testsuite/config/default.exp: Update default CC and similar.
* testsuite/lib/ctf-lib.exp (run_native_host_cmd): Use CC rather
than CC_FOR_HOST.
(run_lookup_test): Use CC_FOR_TARGET and CFLAGS_FOR_TARGET.
|
|
The tests currently in binutils are aimed at the original GCC-based
implementation of CTF, which emitted CTF directly from GCC's internal
representation. The approach now under review emits CTF from DWARF,
with an eye to eventually doing this for all non-DWARF debuginfo-like
formats GCC supports. It also uses a different flag to enable
CTF emission (-gctf rather than -gt).
Adjust the testsuite accordingly.
Given that the ld testsuite results are dependent on type ordering,
which we do not guarantee at all, it's amazing how little changes. We
see a few type ordering differences, slices change because the old GCC
was buggy (slices were emitted "backwards", from the wrong end of the
machine word) and its expected results were wrong, and GCC now emits the
underlying integral type for enumerated types, though CTF has no way to
record this yet (coming in v4).
GCC also now emits even hidden symbols into the symtab (and thus
symtypetab), so one symtypetab test changes its expected results
slightly to compensate.
Also add tests for the CTF_K_UNKNOWN nonrepresentable type: this
couldn't be done before now since the only GCC that emits CTF_K_UNKNOWN
for nonrepresentable types is the new one.
ld/ChangeLog
2021-05-06 Nick Alcock <nick.alcock@oracle.com>
* testsuite/ld-ctf/ctf.exp: Use -gctf, not -gt.
* testsuite/lib/ld-lib.exp: Likewise.
* testsuite/ld-ctf/nonrepresentable-1.c: New test for nonrepresentable types.
* testsuite/ld-ctf/nonrepresentable-2.c: Likewise.
* testsuite/ld-ctf/nonrepresentable.d: Likewise.
* testsuite/ld-ctf/array.d: Larger type section.
* testsuite/ld-ctf/data-func-conflicted.d: Likewise.
* testsuite/ld-ctf/enums.d: Likewise.
* testsuite/ld-ctf/conflicting-enums.d: Don't compare types.
* testsuite/ld-ctf/cross-tu-cyclic-conflicting.d: Changed type order.
* testsuite/ld-ctf/cross-tu-noncyclic.d: Likewise.
* testsuite/ld-ctf/slice.d: Adjust for improved slice emission.
libctf/ChangeLog
2021-05-06 Nick Alcock <nick.alcock@oracle.com>
* testsuite/lib/ctf-lib.exp: Use -gctf, not -gt.
* testsuite/libctf-regression/nonstatic-var-section-ld-r.lk:
Hidden symbols now get into the symtypetab anyway.
|
|
Harmless, but causes noise that makes it harder to spot other leaks.
libctf/ChangeLog
2021-03-25 Nick Alcock <nick.alcock@oracle.com>
* testsuite/libctf-writable/symtypetab-nonlinker-writeout.c: Don't
leak buf.
|
|
Eliminate the dynamic member storage for structs and unions as we have
for other dynamic types. This is much like the previous enum
elimination, except that structs and unions are the only types for which
a full-sized ctf_type_t might be needed. Up to now, this decision has
been made in the individual ctf_add_{struct,union}_sized functions and
duplicated in ctf_add_member_offset. The vlen machinery lets us
simplify this, always allocating a ctf_lmember_t and setting the
dtd_data's ctt_size to CTF_LSIZE_SENT: we figure out whether this is
really justified and (almost always) repack things down into a
ctf_stype_t at ctf_serialize time.
This allows us to eliminate the dynamic member paths from the iterators and
query functions in ctf-types.c in favour of always using the large-structure
vlen stuff for dynamic types (the diff is ugly but that's just because of the
volume of reindentation this calls for). This also means the large-structure
vlen stuff gets more heavily tested, which is nice because it was an almost
totally unused code path before now (it only kicked in for structures of size
>4GiB, and how often do you see those?)
The only extra complexity here is ctf_add_type. Back in the days of the
nondeduplicating linker this was called a ridiculous number of times for
countless identical copies of structures: eschewing the repeated lookups of the
dtd in ctf_add_member_offset and adding the members directly saved an amazing
amount of time. Now the nondeduplicating linker is gone, this is extreme
overoptimization: we can rip out the direct addition and use ctf_member_next and
ctf_add_member_offset, just like ctf_dedup_emit does.
We augment a ctf_add_type test to try adding a self-referential struct, the only
thing the ctf_add_type part of this change really perturbs.
This completes the elimination of dtd_u.
libctf/ChangeLog
2021-03-18 Nick Alcock <nick.alcock@oracle.com>
* ctf-impl.h (ctf_dtdef_t) <dtu_members>: Remove.
<dtd_u>: Likewise.
(ctf_dmdef_t): Remove.
(struct ctf_next) <u.ctn_dmd>: Remove.
* ctf-create.c (INITIAL_VLEN): New, more-or-less arbitrary initial
vlen size.
(ctf_add_enum): Use it.
(ctf_dtd_delete): Do not free the (removed) dmd; remove string
refs from the vlen on struct deletion.
(ctf_add_struct_sized): Populate the vlen: do it by hand if
promoting forwards. Always populate the full-size
lsizehi/lsizelo members.
(ctf_add_union_sized): Likewise.
(ctf_add_member_offset): Set up the vlen rather than the dmd.
Expand it as needed, repointing string refs via
ctf_str_move_pending. Add the member names as pending strings.
Always populate the full-size lsizehi/lsizelo members.
(membadd): Remove, folding back into...
(ctf_add_type_internal): ... here, adding via an ordinary
ctf_add_struct_sized and _next iteration rather than doing
everything by hand.
* ctf-serialize.c (ctf_copy_smembers): Remove this...
(ctf_copy_lmembers): ... and this...
(ctf_emit_type_sect): ... folding into here. Figure out if a
ctf_stype_t is needed here, not in ctf_add_*_sized.
(ctf_type_sect_size): Figure out the ctf_stype_t stuff the same
way here.
* ctf-types.c (ctf_member_next): Remove the dmd path and always
use the vlen. Force large-structure usage for dynamic types.
(ctf_type_align): Likewise.
(ctf_member_info): Likewise.
(ctf_type_rvisit): Likewise.
* testsuite/libctf-regression/type-add-unnamed-struct-ctf.c: Add a
self-referential type to this test.
* testsuite/libctf-regression/type-add-unnamed-struct.c: Adjusted
accordingly.
* testsuite/libctf-regression/type-add-unnamed-struct.lk: Likewise.
|
|
This is the first tricky one, the first complex multi-entry vlen
containing strings. To handle this in vlen form, we have to handle
pending refs moving around on realloc.
We grow vlen regions using a new ctf_grow_vlen function, and iterate
through the existing enums every time a grow happens, telling the string
machinery the distance between the old and new vlen region and letting
it adjust the pending refs accordingly. (This avoids traversing all
outstanding refs to find the refs that need adjusting, at the cost of
having to traverse one enum: an obvious major performance win.)
Addition of enums themselves (and also structs/unions later) is a bit
trickier than earlier forms, because the type might be being promoted
from a forward, and forwards have no vlen: so we have to spot that and
create it if needed.
Serialization of enums simplifies down to just telling the string
machinery about the string refs; all the enum type-lookup code loses all
its dynamic member lookup complexity entirely.
A new test is added that iterates over (and gets values of) an enum with
enough members to force a round of vlen growth.
libctf/ChangeLog
2021-03-18 Nick Alcock <nick.alcock@oracle.com>
* ctf-impl.h (ctf_dtdef_t) <dtd_vlen_alloc>: New.
(ctf_str_move_pending): Declare.
* ctf-string.c (ctf_str_add_ref_internal): Fix error return.
(ctf_str_move_pending): New.
* ctf-create.c (ctf_grow_vlen): New.
(ctf_dtd_delete): Zero out the vlen_alloc after free. Free the
vlen later: iterate over it and free enum name refs first.
(ctf_add_generic): Populate dtd_vlen_alloc from vlen.
(ctf_add_enum): populate the vlen; do it by hand if promoting
forwards.
(ctf_add_enumerator): Set up the vlen rather than the dmd. Expand
it as needed, repointing string refs via ctf_str_move_pending. Add
the enumerand names as pending strings.
* ctf-serialize.c (ctf_copy_emembers): Remove.
(ctf_emit_type_sect): Copy the vlen into place and ref the
strings.
* ctf-types.c (ctf_enum_next): The dynamic portion now uses
the same code as the non-dynamic.
(ctf_enum_name): Likewise.
(ctf_enum_value): Likewise.
* testsuite/libctf-lookup/enum-many-ctf.c: New test.
* testsuite/libctf-lookup/enum-many.lk: New test.
|
|
The preceding change revealed a new bug: the string table is sorted for
better compression, so repeated serialization with type (or member)
additions in the middle can move strings around. But every
serialization flushes the set of refs (the memory locations that are
automatically updated with a final string offset when the strtab is
updated), so if we are not to have string offsets go stale, we must do
all ref additions within the serialization code (which walks the
complete set of types and symbols anyway). Unfortunately, we were adding
one ref in another place: the type name in the dynamic type definitions,
which has a ref added to it by ctf_add_generic.
So adding a type, serializing (via, say, one of the ctf_write
functions), adding another type with a name that sorts earlier, and
serializing again will corrupt the name of the first type because it no
longer had a ref pointing to its dtd entry's name when its string offset
was shifted later in the strtab to mae way for the other type.
To ensure that we don't miss strings, we also maintain a set of *pending
refs* that will be added later (during serialization), and remove
entries from that set when the ref is finally added. We always use
ctf_str_add_pending outside ctf-serialize.c, ensure that ctf_serialize
adds all strtab offsets as refs (even those in the dtds) on every
serialization, and mandate that no refs are live on entry to
ctf_serialize and that all pending refs are gone before strtab
finalization. (Of necessity ctf_serialize has to traverse all strtab
offsets in the dtds in order to serialize them, so adding them as refs
at the same time is easy.)
(Note that we still can't erase unused atoms when we roll back, though
we can erase unused refs: members and enums are still not removed by
rollbacks and might reference strings added after the snapshot.)
libctf/ChangeLog
2021-03-18 Nick Alcock <nick.alcock@oracle.com>
* ctf-hash.c (ctf_dynset_elements): New.
* ctf-impl.h (ctf_dynset_elements): Declare it.
(ctf_str_add_pending): Likewise.
(ctf_dict_t) <ctf_str_pending_ref>: New, set of refs that must be
added during serialization.
* ctf-string.c (ctf_str_create_atoms): Initialize it.
(CTF_STR_ADD_REF): New flag.
(CTF_STR_MAKE_PROVISIONAL): Likewise.
(CTF_STR_PENDING_REF): Likewise.
(ctf_str_add_ref_internal): Take a flags word rather than int
params. Populate, and clear out, ctf_str_pending_ref.
(ctf_str_add): Adjust accordingly.
(ctf_str_add_external): Likewise.
(ctf_str_add_pending): New.
(ctf_str_remove_ref): Also remove the potential ref if it is a
pending ref.
* ctf-serialize.c (ctf_serialize): Prohibit addition of strings
with ctf_str_add_ref before serialization. Ensure that the
ctf_str_pending_ref set is empty before strtab finalization.
(ctf_emit_type_sect): Add a ref to the ctt_name.
* ctf-create.c (ctf_add_generic): Add the ctt_name as a pending
ref.
* testsuite/libctf-writable/reserialize-strtab-corruption.*: New test.
|
|
ctf-link.c is unnecessarily confusing because ctf_link_lazy_open is
positioned near functions that have nothing to do with opening files.
Move it around, and fix some tabdamage that's crept in lately.
libctf/ChangeLog
2021-03-18 Nick Alcock <nick.alcock@oracle.com>
* ctf-link.c (ctf_link_lazy_open): Move up in the file, to near
ctf_link_add_ctf.
* ctf-lookup.c (ctf_lookup_symbol_idx): Repair tabdamage.
(ctf_lookup_by_sym_or_name): Likewise.
* testsuite/libctf-lookup/struct-iteration.c: Likewise.
* testsuite/libctf-regression/type-add-unnamed-struct.c: Likewise.
|
|
The existing ctf_lookup_by_symbol and ctf_arc_lookup_symbol functions
suffice to look up the types of symbols if the caller already has a
symbol number. But the caller often doesn't have one of those and only
knows the name of the symbol: also, in object files, the caller might
not have a useful symbol number in any sense (and neither does libctf:
the 'symbol number' we use in that case literally starts at 0 for the
lexicographically first-sorted symbol in the symtypetab and counts those
symbols, so it corresponds to nothing useful).
This means that even though object files have a symtypetab (generated by
the compiler or by ld -r), the only way we can look up anything in it is
to iterate over all symbols in turn with ctf_symbol_next until we find
the one we want.
This is unhelpful and pointlessly inefficient.
So add a pair of functions to look up symbols by name in a dict and in a
whole archive: ctf_lookup_by_symbol_name and ctf_arc_lookup_symbol_name.
These are identical to the existing functions except that they take
symbol names rather than symbol numbers.
To avoid insane repetition, we do some refactoring in the process, so
that both ctf_lookup_by_symbol and ctf_arc_lookup_symbol turn into thin
wrappers around internal functions that do both lookup by symbol index
and lookup by name. This massively reduces code duplication because
even the existing lookup-by-index stuff wants to use a name sometimes
(when looking up in indexed sections), and the new lookup-by-name stuff
has to turn it into an index sometimes (when looking up in non-indexed
sections): doing it this way lets us share most of that.
The actual name->index lookup is done by ctf_lookup_symbol_idx. We do
not anticipate this lookup to be as heavily used as ld.so symbol lookup
by many orders of magnitude, so using the ELF symbol hashes would
probably take more time to read them than is saved by using the hashes,
and it adds a lot of complexity. Instead, do a linear search for the
symbol name, caching all the name -> index mappings as we go, so that
future searches are likely to hit in the cache. To avoid having to
repeat this search over and over in a CTF archive when
ctf_arc_lookup_symbol_name is used, have cached archive lookups (the
sort done by ctf_arc_lookup_symbol* and the ctf_archive_next iterator)
pick out the first dict they cache in a given archive and store it in a
new ctf_archive field, ctfi_crossdict_cache. This can be used to store
cross-dictionary cached state that depends on things like the ELF symbol
table rather than the contents of any one dict. ctf_lookup_symbol_idx
then caches its name->index mappings in the dictionary named in the
crossdict cache, if any, so that ctf_lookup_symbol_idx in other dicts
in the same archive benefit from the previous linear search, and the
symtab only needs to be scanned at most once.
(Note that if you call ctf_lookup_by_symbol_name in one specific dict,
and then follow it with a ctf_arc_lookup_symbol_name, the former will
not use the crossdict cache because it's only populated by the dict
opens in ctf_arc_lookup_symbol_name. This is harmless except for a small
one-off waste of memory and time: it's only a cache, after all. We can
fix this later by using the archive caching machinery more
aggressively.)
In ctf-archive, we do similar things, turning ctf_arc_lookup_symbol into
a wrapper around a new function that does both index -> ID and name ->
ID lookups across all dicts in an archive. We add a new
ctfi_symnamedicts cache that maps symbol names to the ctf_dict_t * that
it was found in (so that linear searches for symbols don't need to be
repeated): but we also *remove* a cache, the ctfi_syms cache that was
memoizing the actual ctf_id_t returned from every call to
ctf_arc_lookup_symbol. This is pointless: all it saves is one call to
ctf_lookup_by_symbol, and that's basically an array lookup and nothing
more so isn't worth caching. (Equally, given that symbol -> index
mappings are cached by ctf_lookup_by_symbol_name, those calls are nearly
free after the first call, so there's no point caching the ctf_id_t in
that case either.)
We fix up one test that was doing manual symbol lookup to use
ctf_arc_lookup_symbol instead, and enhance it to check that the caching
layer is not totally broken: we also add a new test to do lookups in a
.o file, and another to do lookups in an archive with conflicted types
and make sure that sort of multi-dict lookup is actually working.
include/ChangeLog
2021-02-17 Nick Alcock <nick.alcock@oracle.com>
* ctf-api.h (ctf_arc_lookup_symbol_name): New.
(ctf_lookup_by_symbol_name): Likewise.
libctf/ChangeLog
2021-02-17 Nick Alcock <nick.alcock@oracle.com>
* ctf-impl.h (ctf_dict_t) <ctf_symhash>: New.
<ctf_symhash_latest>: Likewise.
(struct ctf_archive_internal) <ctfi_crossdict_cache>: New.
<ctfi_symnamedicts>: New.
<ctfi_syms>: Remove.
(ctf_lookup_symbol_name): Remove.
* ctf-lookup.c (ctf_lookup_symbol_name): Propagate errors from
parent properly. Make static.
(ctf_lookup_symbol_idx): New, linear search for the symbol name,
cached in the crossdict cache's ctf_symhash (if available), or
this dict's (otherwise).
(ctf_try_lookup_indexed): Allow the symname to be passed in.
(ctf_lookup_by_symbol): Turn into a wrapper around...
(ctf_lookup_by_sym_or_name): ... this, supporting name lookup too,
using ctf_lookup_symbol_idx in non-writable dicts. Special-case
name lookup in dynamic dicts without reported symbols, which have
no symtab or dynsymidx but where name lookup should still work.
(ctf_lookup_by_symbol_name): New, another wrapper.
* ctf-archive.c (enosym): Note that this is present in
ctfi_symnamedicts too.
(ctf_arc_close): Adjust for removal of ctfi_syms. Free the
ctfi_symnamedicts.
(ctf_arc_flush_caches): Likewise.
(ctf_dict_open_cached): Memoize the first cached dict in the
crossdict cache.
(ctf_arc_lookup_symbol): Turn into a wrapper around...
(ctf_arc_lookup_sym_or_name): ... this. No longer cache
ctf_id_t lookups: just call ctf_lookup_by_symbol as needed (but
still cache the dicts those lookups succeed in). Add
lookup-by-name support, with dicts of successful lookups cached in
ctfi_symnamedicts. Refactor the caching code a bit.
(ctf_arc_lookup_symbol_name): New, another wrapper.
* ctf-open.c (ctf_dict_close): Free the ctf_symhash.
* libctf.ver (LIBCTF_1.2): New version. Add
ctf_lookup_by_symbol_name, ctf_arc_lookup_symbol_name.
* testsuite/libctf-lookup/enum-symbol.c (main): Use
ctf_arc_lookup_symbol rather than looking up the name ourselves.
Fish it out repeatedly, to make sure that symbol caching isn't
broken.
(symidx_64): Remove.
(symidx_32): Remove.
* testsuite/libctf-lookup/enum-symbol-obj.lk: Test symbol lookup
in an unlinked object file (indexed symtypetab sections only).
* testsuite/libctf-writable/symtypetab-nonlinker-writeout.c
(try_maybe_reporting): Check symbol types via
ctf_lookup_by_symbol_name as well as ctf_symbol_next.
* testsuite/libctf-lookup/conflicting-type-syms.*: New test of
lookups in a multi-dict archive.
|
|
* testsuite/config/default.exp (ld_L_opt): Define.
* testsuite/lib/ctf-lib.exp (load_common_lib): Delete. Instead load
ld-lib.exp.
(run_host_cmd, run_host_cmd_yesno, check_compiler_available): Delete.
(compile_one_cc, check_ctf_available): Delete.
|
|
The variable section in a CTF dict is meant to contain the types of
variables that do not appear in the symbol table (mostly file-scope
static declarations). We implement this by having the compiler emit
all potential data symbols into both sections, then delete those
symbols from the variable section that correspond to data symbols the
linker has reported.
Unfortunately, the check for this in ctf_serialize is wrong: rather than
checking the set of linker-reported symbols, we check the set of names
in the data object symtypetab section: if the linker has reported no
symbols at all (usually if ld -r has been run, or if a non-linker
program that does not use symbol tables is calling ctf_link) this will
include every single symbol, emptying the variable section completely.
Worse, when ld -r is in use, we want to force writeout of every
symtypetab entry on the inputs, in an indexed section, whether or not
the linker has reported them, since this isn't a final link yet and the
symbol table is not finalized (and may grow more symbols than the linker
has yet reported). But the check for this is flawed too: we were
relying on ctf_link_shuffle_syms not having been called if no symbols
exist, but that function is *always* called by ld even when ld -r is in
use: ctf_link_add_linker_symbol is the one that's not called when there
are no symbols.
We clearly need to rethink this. Using the emptiness of the set of
reported symbols as a test for ld -r is just ugly: the linker already
knows if ld -r is underway and can just tell us. So add a new linker
flag CTF_LINK_NO_FILTER_REPORTED_SYMS that is set to stop the linker
filtering the symbols in the symtypetab sections using the set that the
linker has reported: use the presence or absence of this flag to
determine whether to emit unindexed symtabs: we only remove entries from
the variable section when filtering symbols, and we only remove them if
they are in the reported symbol set, fixing the case where no symbols
are reported by the linker at all.
(The negative sense of the new CTF_LINK flag is intentional: the common
case, both for ld and for simple tools that want to do a ctf_link with
no ELF symbol table in sight, is probably to filter out symbols that no
linker has reported: i.e., for the simple tools, all of them.)
There's another wrinkle, though. It is quite possible for a non-linker
to add symbols to a dict via ctf_add_*_sym and then write it out via the
ctf_write APIs: perhaps it's preparing a dict for a later linker
invocation. Right now this would not lead to anything terribly
meaningful happening: ctf_serialize just assumes it was called via
ctf_link if symbols are present. So add an (internal-to-libctf) flag
that indicates that a writeout is happening via ctf_link_write, and set
it there (propagating it to child dicts as needed). ctf_serialize can
then spot when it is not being called by a linker, and arrange to always
write out an indexed, sorted symtypetab for fastest possible future
symbol lookup by name in that case. (The writeouts done by ld -r are
unsorted, because the only thing likely to use those symtabs is the
linker, which doesn't benefit from symtypetab sorting.)
Tests added for all three linking cases (ld -r, ld -shared, ld), with a
bit of testsuite framework enhancement to stop it unconditionally
linking the CTF to be checked by the lookup program with -shared, so
tests can now examine CTF linked with -r or indeed with no flags at all,
though the output filename is still foo.so even in this case.
Another test added for the non-linker case that endeavours to determine
whether the symtypetab is sorted by examining the order of entries
returned from ctf_symbol_next: nobody outside libctf should rely on
this ordering, but this test is not outside libctf :)
include/ChangeLog
2021-01-26 Nick Alcock <nick.alcock@oracle.com>
* ctf-api.h (CTF_LINK_NO_FILTER_REPORTED_SYMS): New.
ld/ChangeLog
2021-01-26 Nick Alcock <nick.alcock@oracle.com>
* ldlang.c (lang_merge_ctf): Set CTF_LINK_NO_FILTER_REPORTED_SYMS
when appropriate.
libctf/ChangeLog
2021-01-27 Nick Alcock <nick.alcock@oracle.com>
* ctf-impl.c (_libctf_nonnull_): Add parameters.
(LCTF_LINKING): New flag.
(ctf_dict_t) <ctf_link_flags>: Mention it.
* ctf-link.c (ctf_link): Keep LCTF_LINKING set across call.
(ctf_write): Likewise, including in child dictionaries.
(ctf_link_shuffle_syms): Make sure ctf_dynsyms is NULL if there
are no reported symbols.
* ctf-create.c (symtypetab_delete_nonstatic_vars): Make sure
the variable has been reported as a symbol by the linker.
(symtypetab_skippable): Mention relationship between SYMFP and the
flags.
(symtypetab_density): Adjust nonnullity. Exit early if no symbols
were reported and force-indexing is off (i.e., we are doing a
final link).
(ctf_serialize): Handle the !LCTF_LINKING case by writing out an
indexed, sorted symtypetab (and allow SYMFP to be NULL in this
case). Turn sorting off if this is a non-final link. Only delete
nonstatic vars if we are filtering symbols and the linker has
reported some.
* testsuite/libctf-regression/nonstatic-var-section-ld-r*:
New test of variable and symtypetab section population when
ld -r is used.
* testsuite/libctf-regression/nonstatic-var-section-ld-executable.lk:
Likewise, when ld of an executable is used.
* testsuite/libctf-regression/nonstatic-var-section-ld.lk:
Likewise, when ld -shared alone is used.
* testsuite/libctf-regression/nonstatic-var-section-ld*.c:
Lookup programs for the above.
* testsuite/libctf-writable/symtypetab-nonlinker-writeout.*: New
test, testing survival of symbols across ctf_write paths.
* testsuite/lib/ctf-lib.exp (run_lookup_test): New option,
nonshared, suppressing linking of the SOURCE with -shared.
|
|
Our recent commit to support unnamed structure members better ditched
the old ctf_member_iter iterator body in favour of ctf_member_next.
However, these functions treat unnamed structure members differently:
ctf_member_iter just returned whatever the internal representation
contained, while ctf_member_next took care to always return "" rather
than sometimes returning "" and sometimes NULL depending on whether the
dict was dynamic (a product of ctf_create) or not (a product of
ctf_open). After this commit, ctf_member_iter did the same.
It was always a bug for external callers not to treat a "" return from
these functions as if it were NULL, so only buggy callers could be
affected -- but one of those buggy callers was ctf_add_type, which
assumed that it could just take whatever name was returned from
ctf_member_iter and slam it directly into the internal representation of
a dynamic dict -- which expects NULL for unnamed members, not "". The
net effect of all of this is that taking a struct containing unnamed
members and ctf_add_type'ing it into a dynamic dict produced a dict
whose unnamed members were inaccessible to ctf_member_info (though if
you wrote that dict out and then ctf_open'ed it, they would magically
reappear again).
Compensate for this by suitably transforming a "" name into NULL in the
internal representation, as should have been done all along.
libctf/ChangeLog
2021-01-19 Nick Alcock <nick.alcock@oracle.com>
* ctf-create.c (membadd): Transform ""-named members into
NULL-named ones.
* testsuite/libctf-regression/type-add-unnamed-struct*: New test.
|
|
The recent work allowing lookups of pointers in child dicts when the
pointed-to type is in the parent dict broke the case where a pointer
type that does not exist at all is looked up: we mistakenly return the
pointed-to type, which is likely not a pointer at all. This causes
considerable confusion.
Fixed, with a new testcase.
libctf/ChangeLog
2021-01-19 Nick Alcock <nick.alcock@oracle.com>
* ctf-lookup.c (ctf_lookup_by_name_internal): Do not return the
base type if looking up a nonexistent pointer type.
* testsuite/libctf-regression/pptrtab*: Test it.
|
|
This returns an int, not a long int or an ssize_t (as one test was
inconsistently assuming).
libctf/ChangeLog
2021-01-05 Nick Alcock <nick.alcock@oracle.com>
* testsuite/libctf-lookup/struct-iteration.c (main):
ctf_member_count returns an int.
|
