| Age | Commit message (Collapse) | Author | Files | Lines |
|---|
|
libctf/
* testsuite/libctf-writable/ctf-nonroot-addition.*: New test.
|
|
The previous commits dropped preservation of the non-root flag in ctf_link
and arranged to use it somewhat differently to track conflicting types in
cu-mapped CUs when doing cu-mapped links. This was necessary to prevent
entirely spuriously hidden types from appearing on the output of such links.
Bring it (and the test for it) back. The problem with the previous design
was that it implicitly assumed that the non-root flag it saw on the input
was always meant to be preserved (when in the final phase of cu-mapped links
it merely means that conflicting types were found in intermediate links),
and also that it could figure out what the non-root flag on the input was by
sucking in the non-root flag of the input type corresponding to an output in
the output mapping (which maps type hashes to a corresponding type on some
input).
This method of getting properties of the input type *does* work *if* that
property was one of those hashed by the ctf_dedup_hash_type process. In
that case, every type with a given hash will have the same value for all
hashed-in properties, so it doesn't matter which one is consulted (the
output mapping points at an arbitrary one of those input types). But the
non-root flag is explicitly *not* hashed in: as a comment in
ctf_dedup_rhash_type notes, being non-root is not a property of a type, and
two types (one non-root, one not) can perfectly well be the same type even
though one is visible and one isn't. So just copying the non-root flag from
the output mapping's idea of the input type will copy in a value that is not
stabilized by the hash, so is more-or-less random!
So we cannot do that. We have to do something else, which means we have to
decide what to do if two identical types with different nonroot flag values
pop up. The most sensible thing to do is probably to say that if all
instances of a type are non-root-visible, the linked output should also be
non-root-visible: any root-visible types in that set, and the output type is
root-visible again.
We implement this with a new cd_nonroot_consistency dynhash, which maps type
hashes to the value 0 ("all instances root-visible"), 1 ("all instances
non-root-visible") or 2 ("inconsistent"). After hashing is over, we save a
bit of memory by deleting everything from this hashtab that doesn't have a
value of 1 ("non-root-visible"), then use this to decide whether to emit any
given type as non-root-visible or not.
However... that's not quite enough. In cu-mapped links, we want to
disregard this whole thing because we just hide everything -- but in phase
2, when we take the smushed-together CUs resulting from phase 1 and
deduplicate them against each other, we want to do what the previous commits
implemented and ignore the non-root flag entirely, instead falling back to
preventing clashes by hiding anything that would be considered conflicting.
We extend the existing cu_mapped parameter to various bits of ctf_dedup so
that it is now tristate: 0 means a normal link, 1 means the smush-it-
together phase of cu-mapped links, and 2 means the final phase of cu-mapped
links. We do the hide-conflicting stuff only in phase 2, meaning that
normal links by GNU ld can always respect the value of the nonroot flag put
on types in the input.
(One extra thing added as part of this: you can now efficiently delete the
last value returned by ctf_dynhash_next() by calling
ctf_dynhash_next_remove.)
We bring back the ctf-nonroot-linking test with one tweak: linking now works
on mingw as long as you're using the ucrt libc, so re-enable it for better
test coverage on that platform.
libctf/
PR libctf/33047
* ctf-hash.c (ctf_dynhash_next_remove): New.
* ctf-impl.h (struct ctf_dedup) [cd_nonroot_consistency]: New.
* ctf-link.c (ctf_link_deduplicating): Differentiate between
cu-mapped and non-cu-mapped links, even in the final phase.
* ctf-dedup.c (ctf_dedup_hash_type): Callback prototype addition.
Get the non-root flag and pass it down.
(ctf_dedup_rhash_type): Callback prototype addition. Document
restrictions on use of the nonroot flag.
(ctf_dedup_populate_mappings): Populate cd_nonroot_consistency.
(ctf_dedup_hash_type_fini): New function: delete now-unnecessary
values from cd_nonroot_consistency.
(ctf_dedup_init): Initialize it.
(ctf_dedup_fini): Destroy it.
(ctf_dedup): cu_mapping is now cu_mapping_phase. Call
ctf_dedup_hash_type_fini.
(ctf_dedup_emit_type): Use cu_mapping_phase and
cd_nonroot_consistency to propagate the non-root flag into outputs
for normal links, and to do name-based conflict checking only for
phase 2 of cu-mapped links.
(ctf_dedup_emit): cu_mapping is now cu_mapping_phase. Adjust
assertion accordingly.
* testsuite/libctf-writable/ctf-nonroot-linking.c: Bring back.
* testsuite/libctf-writable/ctf-nonroot-linking.lk: Likewise.
|
|
This reverts commit 87b2f673102884d7c69144c85a26ed5dbaa4f86a.
It is based on a misconception, that hidden types in the deduplicator
input should always be hidden in the output. For cu-mapped links,
and final links following cu-mapped links, this is not true: we want
to hide inputs if they were conflicting on the output and no more.
We will reintroduce the testcase once a better fix is found.
libctf/
PR libctf/33047
* ctf-dedup.c (ctf_dedup_emit_type): Don't respect the nonroot flag.
* testsuite/libctf-writable/ctf-nonroot-linking.c: Removed.
* testsuite/libctf-writable/ctf-nonroot-linking.lk: Removed.
|
|
ctf_arc_import_parent, called by the cached-opening machinery used by
ctf_archive_next and archive-wide lookup functions like
ctf_arc_lookup_symbol, has an err-pointer parameter like all other opening
functions. Unfortunately it unconditionally initializes it whenever
provided, even if there was no error, which can lead to its being
initialized to an uninitialized value. This is not technically an
API-contract violation, since we don't define what happens to the error
value except when an error happens, but it is still unpleasant.
Initialize it only when there is an actual error, so we never initialize it
to an uninitialized value.
While we're at it, improve all the opening pathways: on success, set errp to
0, rather than leaving it what it was, reducing the likelihood of
uninitialized error param returns in callers too. (This is inconsistent
with the treatment of ctf_errno(), but the err value being a parameter
passed in from outside makes the divergence acceptable: in open functions,
you're never going to be overwriting some old error value someone might want
to keep around across multiple calls, some of which are successful and some
of which are not.)
Soup up existing tests to verify all this.
Thanks to Bruce McCulloch for the original patch, and Stephen Brennan for
the report.
libctf/
PR libctf/32903
* ctf-archive.c (ctf_arc_open_internal): Zero errp on success.
(ctf_dict_open_sections): Zero errp at the start.
(ctf_arc_import_parent): Intialize err.
* ctf-open.c (ctf_bufopen): Zero errp at the start.
* testsuite/libctf-lookup/add-to-opened.c: Make sure one-element
archive opens update errp.
* testsuite/libctf-writable/ctf-compressed.c: Make sure real archive
opens update errp.
|
|
Slices had a bunch of horrible usability problems. In particular, while
towers of cv-quals are resolved away by functions that need to do it, towers
of cv-quals with slices in the middle are not resolved away by functions
like ctf_enum_value that can see through slices: resolving volatile -> slice
-> const -> enum will leave it with a 'const', which will error pointlessly,
annoying callers, who reasonably expect slices to be more invisible than
this. (The user-callable ctf_type_resolve still does not resolve away
slices, because this is the only way users can see that the slices are there
at all.)
This is induced by a fix for another wart: ctf_add_enumerator does not
resolve anything away at all, so you can't even add enumerators to const or
volatile enums -- and more problematically, you can't add enumerators to
enums with an explicit encoding without resolving away the types by hand,
since ctf_add_enum_encoded works by returning a slice! ctf_add_enumerator
now resolves away all of those, so any cvr-or-typedef-or-slice-qual
terminating in an enum can be added to, exactly as callers likely expect.
(New tests added.)
libctf/
* ctf-create.c (ctf_add_enumerator): Resolve away cvr-qualness.
* ctf-types.c (ctf_type_resolve_unsliced): Don't terminate at
the first slice.
* testsuite/libctf-writable/slice-of-slice.*: New test.
|
|
|
|
Commit 483546ce4f3 ("libctf: make ctf_serialize() actually serialize")
accidentally broke dict compression. There were two bugs:
- ctf_arc_write_one_ctf was still making its own decision about
whether to compress the dict via direct ctf_size comparison, which is
unfortunate because now that it no longer calls ctf_serialize itself,
ctf_size is always zero when it does this: it should let the writing
functions decide on the threshold, which they contain code to do which is
simply not used for lack of one trivial wrapper to write to an fd and
also provide a compression threshold
- ctf_write_mem, the function underlying all writing as of the commit
above, was calling zlib's compressBound and avoiding compression if this
returned a value larger than the input. Unfortunately compressBound does
not do a trial compression and determine whether the result is
compressible: it just adds zlib header sizes to the value passed in, so
our test would *always* have concluded that the value was incompressible!
Avoid by simply always compressing if the raw size is larger than the
threshold: zlib is quite clever enough to avoid actually compressing
if the data is incompressible.
Add a testcase for this.
libctf/
* ctf-impl.h (ctf_write_thresholded): New...
* ctf-serialize.c (ctf_write_thresholded): ... defined here,
a wrapper around...
(ctf_write_mem): ... this. Don't check compressibility.
(ctf_compress_write): Reimplement as a ctf_write_thresholded
wrapper.
(ctf_write): Likewise.
* ctf-archive.c (arc_write_one_ctf): Just call
ctf_write_thresholded rather than trying to work out whether
to compress.
* testsuite/libctf-writable/ctf-compressed.*: New test.
|
|
If you deduplicate non-root-visible types, the resulting type should still
be non-root-visible! We were promoting all such types to root-visible, and
re-demoting them only if their names collided (which might happen on
cu-mapped links if multiple compilation units with conflicting types are
fused into one child dict).
This "worked" before now, in that linking at least didn't fail (if you don't
mind having your non-root flag value destroyed if you're adding
non-root-visible types), but now that conflicting enumerators cause their
containing enums to become conflicted (enums which might have *different
names*), this caused the linker to crash when it hit two enumerators with
conflicting values.
Not testable in ld because cu-mapped links are not exposed to ld, but can be
tested via direct creation of libraries and calls to ctf_link directly.
(This also tests the ctf_dump non-root type printout, which before now
was untested.)
libctf/
* ctf-dedup.c (ctf_dedup_emit_type): Non-root-visible input types
should be emitted as non-root-visible output types.
* testsuite/libctf-writable/ctf-nonroot-linking.c: New test.
* testsuite/libctf-writable/ctf-nonroot-linking.lk: New test.
|
|
The intent of the name lookup code was for lookups to yield non-bitfield
basic types except if none existed with a given name, and only then
return bitfield types with that name. Unfortunately, the code as
written only does this if the base type has a type ID higher than all
bitfield types, which is most unlikely (the opposite is almost always
the case).
Adjust it so that what ends up in the name table is the highest-width
zero-offset type with a given name, if any such exist, and failing that
the first type with that name we see, no matter its offset. (We don't
define *which* bitfield type you get, after all, so we might as well
just stuff in the first we find.)
Reported by Stephen Brennan <stephen.brennan@oracle.com>.
libctf/
* ctf-open.c (init_types): Modify to allow some lookups during open;
detect bitfield name reuse and prefer less bitfieldy types.
* testsuite/libctf-writable/libctf-bitfield-name-lookup.*: New test.
|
|
Just because a path is an error path doesn't mean the program terminates
there if you don't ask it to. And we don't want to -- but that means
we need to initialize the variables that are missed if an error happens to
*something*. Type ID 0 (unimplemented) will do: it'll induce further
ECTF_BADID errors, but that's no bad thing.
libctf/ChangeLog:
* testsuite/libctf-writable/libctf-errors.c: Initialize variables.
|
|
Adds two new external authors to etc/update-copyright.py to cover
bfd/ax_tls.m4, and adds gprofng to dirs handled automatically, then
updates copyright messages as follows:
1) Update cgen/utils.scm emitted copyrights.
2) Run "etc/update-copyright.py --this-year" with an extra external
author I haven't committed, 'Kalray SA.', to cover gas testsuite
files (which should have their copyright message removed).
3) Build with --enable-maintainer-mode --enable-cgen-maint=yes.
4) Check out */po/*.pot which we don't update frequently.
|
|
The fixes applied a few years ago to resolve confusions between parent and
child dicts at lookup time also apply in various forms to creation. In
general, if you have a type in a parent dict ctf_imported into a child and
you do something to it, and the parent dict is writable (created via
ctf_create, not opened via ctf_open*) it should work just the same to make
changes to that type via a child dict as it does to make the change
to the parent dict directly -- and nothing you're prohibited from doing
to the parent dict when done directly should be allowed just because
you're doing it via a child.
Specifically, the following don't work when doing things from the child, but
should:
- adding a member of a type in the parent to a struct or union in the
parent via ctf_add_member or ctf_add_member_offset: this yields
ECTF_BADID
- adding a member of a type in the parent to a struct or union in the
parent via ctf_add_member_encoded: this dumps core (!).
- adding an enumerand to an enumerator in the parent: this yields
ECTF_BADID
- setting the properties of an array in the parent via ctf_set_array;
this yields ECTF_BADID
Relatedly, some things work when doing things via a child that should fail,
yielding a CTF dictionary with invalid content (readable, but meaningless):
in particular, you can add a child type to a struct in the parent via
any of the ctf_add_member* family and nothing complains at all, even though
you should never be able to add references to children to parents (since any
given parent can be associated with many different children).
A family of tests is added to check each of these cases independently, since
some can result in coredumps and it would be nice to test the other cases
even if some dump core. They use a common library to do all the actual
work. The set of affected API calls was determined by code inspection
(auditing all calls to ctf_dtd_lookup): it's possible that I missed a few,
but I doubt it, since other cases use ctf_lookup* functions, which already
climb to the parent where appropriate.
libctf/ChangeLog:
PR libctf/30985
* ctf-create.c (ctf_dtd_lookup): Traverse to parents if necessary.
(ctf_set_array): Likewise. Report errors on the child; require
both parent and child to be writable.
(ctf_add_enumerator): Likewise.
(ctf_add_member_offset): Likewise. Prohibit addition of child types
to structs in the parent.
(ctf_add_member_encoded): Do not dereference a NULL dtd: report
ECTF_BADID instead.
* ctf-string.c (ctf_str_add_ref_internal): Report ENOMEM on the
dict if addition of a string ref fails.
* testsuite/libctf-writable/parent-child-dtd-crash-lib.c: New library.
* testsuite/libctf-writable/parent-child-dtd-enum.*: New test.
* testsuite/libctf-writable/parent-child-dtd-enumerator.*: New test.
* testsuite/libctf-writable/parent-child-dtd-member-encoded.*: New test.
* testsuite/libctf-writable/parent-child-dtd-member-offset.*: New test.
* testsuite/libctf-writable/parent-child-dtd-set-array.*: New test.
* testsuite/libctf-writable/parent-child-dtd-struct.*: New test.
* testsuite/libctf-writable/parent-child-dtd-union.*: New test.
|
|
We do this as a writable test because the only known-affected platforms
(with ssize_t longer than unsigned long) use PE, and we do not have support
for CTF linkage in the PE linker yet.
PR libctf/30836
* libctf/testsuite/libctf-writable/libctf-errors.*: New test.
|
|
CTF dicts have per-dict errno values: as with other errno values these
are set on error and left unchanged on success. This means that all
errors *must* set the CTF errno: if a call leaves it unchanged, the
caller is apt to find a previous, lingering error and misinterpret
it as the real error.
There are many places in libctf where we carry out operations on parent
dicts as a result of carrying out other user-requested operations on
child dicts (e.g. looking up information on a pointer to a type will
look up the type as well: the pointer might well be in a child and the
type it's a pointer to in the parent). Those operations on the parent
might fail; if they do, the error must be correctly reflected on the
child that the user-visible operation was carried out on. In many
places this was not happening.
So, audit and fix all those places. Add tests for as many of those
cases as possible so they don't regress.
libctf/
* ctf-create.c (ctf_add_slice): Use the original dict.
* ctf-lookup.c (ctf_lookup_variable): Propagate errors.
(ctf_lookup_symbol_idx): Likewise.
* ctf-types.c (ctf_member_next): Likewise.
(ctf_type_resolve_unsliced): Likewise.
(ctf_type_aname): Likewise.
(ctf_member_info): Likewise.
(ctf_type_rvisit): Likewise.
(ctf_func_type_info): Set the error on the right dict.
(ctf_type_encoding): Use the original dict.
* testsuite/libctf-writable/error-propagation.*: New test.
|
|
The newer update-copyright.py fixes file encoding too, removing cr/lf
on binutils/bfdtest2.c and ld/testsuite/ld-cygwin/exe-export.exp, and
embedded cr in binutils/testsuite/binutils-all/ar.exp string match.
|
|
The result of running etc/update-copyright.py --this-year, fixing all
the files whose mode is changed by the script, plus a build with
--enable-maintainer-mode --enable-cgen-maint=yes, then checking
out */po/*.pot which we don't update frequently.
The copy of cgen was with commit d1dd5fcc38ead reverted as that commit
breaks building of bfp opcodes files.
|
|
An off-by-one bug in the check for pptrtab lookup meant that we could
access the pptrtab past its bounds (*well* past its bounds),
particularly if we called ctf_lookup_by_name in a child dict with "*foo"
where "foo" is a type that exists in the parent but not the child and no
previous lookups by name have been carried out. (Note that "*foo" is
not even a valid thing to call ctf_lookup_by_name with: foo * is.
Nonetheless, users sometimes do call ctf_lookup_by_name with invalid
content, and it should return ECTF_NOTYPE, not crash.)
ctf_pptrtab_len, as its name suggests (and as other tests of it in
ctf-lookup.c confirm), is one higher than the maximum valid permissible
index, so the comparison is wrong.
(Test added, which should fail pretty reliably in the presence of this
bug on any machine with 4KiB pages.)
libctf/ChangeLog
2021-09-27 Nick Alcock <nick.alcock@oracle.com>
* ctf-lookup.c (ctf_lookup_by_name_internal): Fix pptrtab bounds.
* testsuite/libctf-writable/pptrtab-writable-page-deep-lookup.*:
New test.
|
|
These warnings are all off by default, but if they do fire you get
spurious ERRORs when running make check-libctf.
libctf/ChangeLog
2021-09-27 Nick Alcock <nick.alcock@oracle.com>
* testsuite/libctf-lookup/enum-symbol.c: Remove unused label.
* testsuite/libctf-lookup/conflicting-type-syms.c: Remove unused
variables.
* testsuite/libctf-regression/pptrtab.c: Likewise.
* testsuite/libctf-regression/type-add-unnamed-struct.c: Likewise.
* testsuite/libctf-writable/pptrtab.c: Likewise.
* testsuite/libctf-writable/reserialize-strtab-corruption.c:
Likewise.
* testsuite/libctf-regression/nonstatic-var-section-ld-r.c: Fix
format string.
* testsuite/libctf-regression/nonstatic-var-section-ld.c:
Likewise.
* testsuite/libctf-regression/nonstatic-var-section-ld.lk: Adjust.
* testsuite/libctf-writable/symtypetab-nonlinker-writeout.c: Fix
initializer.
|
|
Harmless, but causes noise that makes it harder to spot other leaks.
libctf/ChangeLog
2021-03-25 Nick Alcock <nick.alcock@oracle.com>
* testsuite/libctf-writable/symtypetab-nonlinker-writeout.c: Don't
leak buf.
|
|
The preceding change revealed a new bug: the string table is sorted for
better compression, so repeated serialization with type (or member)
additions in the middle can move strings around. But every
serialization flushes the set of refs (the memory locations that are
automatically updated with a final string offset when the strtab is
updated), so if we are not to have string offsets go stale, we must do
all ref additions within the serialization code (which walks the
complete set of types and symbols anyway). Unfortunately, we were adding
one ref in another place: the type name in the dynamic type definitions,
which has a ref added to it by ctf_add_generic.
So adding a type, serializing (via, say, one of the ctf_write
functions), adding another type with a name that sorts earlier, and
serializing again will corrupt the name of the first type because it no
longer had a ref pointing to its dtd entry's name when its string offset
was shifted later in the strtab to mae way for the other type.
To ensure that we don't miss strings, we also maintain a set of *pending
refs* that will be added later (during serialization), and remove
entries from that set when the ref is finally added. We always use
ctf_str_add_pending outside ctf-serialize.c, ensure that ctf_serialize
adds all strtab offsets as refs (even those in the dtds) on every
serialization, and mandate that no refs are live on entry to
ctf_serialize and that all pending refs are gone before strtab
finalization. (Of necessity ctf_serialize has to traverse all strtab
offsets in the dtds in order to serialize them, so adding them as refs
at the same time is easy.)
(Note that we still can't erase unused atoms when we roll back, though
we can erase unused refs: members and enums are still not removed by
rollbacks and might reference strings added after the snapshot.)
libctf/ChangeLog
2021-03-18 Nick Alcock <nick.alcock@oracle.com>
* ctf-hash.c (ctf_dynset_elements): New.
* ctf-impl.h (ctf_dynset_elements): Declare it.
(ctf_str_add_pending): Likewise.
(ctf_dict_t) <ctf_str_pending_ref>: New, set of refs that must be
added during serialization.
* ctf-string.c (ctf_str_create_atoms): Initialize it.
(CTF_STR_ADD_REF): New flag.
(CTF_STR_MAKE_PROVISIONAL): Likewise.
(CTF_STR_PENDING_REF): Likewise.
(ctf_str_add_ref_internal): Take a flags word rather than int
params. Populate, and clear out, ctf_str_pending_ref.
(ctf_str_add): Adjust accordingly.
(ctf_str_add_external): Likewise.
(ctf_str_add_pending): New.
(ctf_str_remove_ref): Also remove the potential ref if it is a
pending ref.
* ctf-serialize.c (ctf_serialize): Prohibit addition of strings
with ctf_str_add_ref before serialization. Ensure that the
ctf_str_pending_ref set is empty before strtab finalization.
(ctf_emit_type_sect): Add a ref to the ctt_name.
* ctf-create.c (ctf_add_generic): Add the ctt_name as a pending
ref.
* testsuite/libctf-writable/reserialize-strtab-corruption.*: New test.
|
|
The existing ctf_lookup_by_symbol and ctf_arc_lookup_symbol functions
suffice to look up the types of symbols if the caller already has a
symbol number. But the caller often doesn't have one of those and only
knows the name of the symbol: also, in object files, the caller might
not have a useful symbol number in any sense (and neither does libctf:
the 'symbol number' we use in that case literally starts at 0 for the
lexicographically first-sorted symbol in the symtypetab and counts those
symbols, so it corresponds to nothing useful).
This means that even though object files have a symtypetab (generated by
the compiler or by ld -r), the only way we can look up anything in it is
to iterate over all symbols in turn with ctf_symbol_next until we find
the one we want.
This is unhelpful and pointlessly inefficient.
So add a pair of functions to look up symbols by name in a dict and in a
whole archive: ctf_lookup_by_symbol_name and ctf_arc_lookup_symbol_name.
These are identical to the existing functions except that they take
symbol names rather than symbol numbers.
To avoid insane repetition, we do some refactoring in the process, so
that both ctf_lookup_by_symbol and ctf_arc_lookup_symbol turn into thin
wrappers around internal functions that do both lookup by symbol index
and lookup by name. This massively reduces code duplication because
even the existing lookup-by-index stuff wants to use a name sometimes
(when looking up in indexed sections), and the new lookup-by-name stuff
has to turn it into an index sometimes (when looking up in non-indexed
sections): doing it this way lets us share most of that.
The actual name->index lookup is done by ctf_lookup_symbol_idx. We do
not anticipate this lookup to be as heavily used as ld.so symbol lookup
by many orders of magnitude, so using the ELF symbol hashes would
probably take more time to read them than is saved by using the hashes,
and it adds a lot of complexity. Instead, do a linear search for the
symbol name, caching all the name -> index mappings as we go, so that
future searches are likely to hit in the cache. To avoid having to
repeat this search over and over in a CTF archive when
ctf_arc_lookup_symbol_name is used, have cached archive lookups (the
sort done by ctf_arc_lookup_symbol* and the ctf_archive_next iterator)
pick out the first dict they cache in a given archive and store it in a
new ctf_archive field, ctfi_crossdict_cache. This can be used to store
cross-dictionary cached state that depends on things like the ELF symbol
table rather than the contents of any one dict. ctf_lookup_symbol_idx
then caches its name->index mappings in the dictionary named in the
crossdict cache, if any, so that ctf_lookup_symbol_idx in other dicts
in the same archive benefit from the previous linear search, and the
symtab only needs to be scanned at most once.
(Note that if you call ctf_lookup_by_symbol_name in one specific dict,
and then follow it with a ctf_arc_lookup_symbol_name, the former will
not use the crossdict cache because it's only populated by the dict
opens in ctf_arc_lookup_symbol_name. This is harmless except for a small
one-off waste of memory and time: it's only a cache, after all. We can
fix this later by using the archive caching machinery more
aggressively.)
In ctf-archive, we do similar things, turning ctf_arc_lookup_symbol into
a wrapper around a new function that does both index -> ID and name ->
ID lookups across all dicts in an archive. We add a new
ctfi_symnamedicts cache that maps symbol names to the ctf_dict_t * that
it was found in (so that linear searches for symbols don't need to be
repeated): but we also *remove* a cache, the ctfi_syms cache that was
memoizing the actual ctf_id_t returned from every call to
ctf_arc_lookup_symbol. This is pointless: all it saves is one call to
ctf_lookup_by_symbol, and that's basically an array lookup and nothing
more so isn't worth caching. (Equally, given that symbol -> index
mappings are cached by ctf_lookup_by_symbol_name, those calls are nearly
free after the first call, so there's no point caching the ctf_id_t in
that case either.)
We fix up one test that was doing manual symbol lookup to use
ctf_arc_lookup_symbol instead, and enhance it to check that the caching
layer is not totally broken: we also add a new test to do lookups in a
.o file, and another to do lookups in an archive with conflicted types
and make sure that sort of multi-dict lookup is actually working.
include/ChangeLog
2021-02-17 Nick Alcock <nick.alcock@oracle.com>
* ctf-api.h (ctf_arc_lookup_symbol_name): New.
(ctf_lookup_by_symbol_name): Likewise.
libctf/ChangeLog
2021-02-17 Nick Alcock <nick.alcock@oracle.com>
* ctf-impl.h (ctf_dict_t) <ctf_symhash>: New.
<ctf_symhash_latest>: Likewise.
(struct ctf_archive_internal) <ctfi_crossdict_cache>: New.
<ctfi_symnamedicts>: New.
<ctfi_syms>: Remove.
(ctf_lookup_symbol_name): Remove.
* ctf-lookup.c (ctf_lookup_symbol_name): Propagate errors from
parent properly. Make static.
(ctf_lookup_symbol_idx): New, linear search for the symbol name,
cached in the crossdict cache's ctf_symhash (if available), or
this dict's (otherwise).
(ctf_try_lookup_indexed): Allow the symname to be passed in.
(ctf_lookup_by_symbol): Turn into a wrapper around...
(ctf_lookup_by_sym_or_name): ... this, supporting name lookup too,
using ctf_lookup_symbol_idx in non-writable dicts. Special-case
name lookup in dynamic dicts without reported symbols, which have
no symtab or dynsymidx but where name lookup should still work.
(ctf_lookup_by_symbol_name): New, another wrapper.
* ctf-archive.c (enosym): Note that this is present in
ctfi_symnamedicts too.
(ctf_arc_close): Adjust for removal of ctfi_syms. Free the
ctfi_symnamedicts.
(ctf_arc_flush_caches): Likewise.
(ctf_dict_open_cached): Memoize the first cached dict in the
crossdict cache.
(ctf_arc_lookup_symbol): Turn into a wrapper around...
(ctf_arc_lookup_sym_or_name): ... this. No longer cache
ctf_id_t lookups: just call ctf_lookup_by_symbol as needed (but
still cache the dicts those lookups succeed in). Add
lookup-by-name support, with dicts of successful lookups cached in
ctfi_symnamedicts. Refactor the caching code a bit.
(ctf_arc_lookup_symbol_name): New, another wrapper.
* ctf-open.c (ctf_dict_close): Free the ctf_symhash.
* libctf.ver (LIBCTF_1.2): New version. Add
ctf_lookup_by_symbol_name, ctf_arc_lookup_symbol_name.
* testsuite/libctf-lookup/enum-symbol.c (main): Use
ctf_arc_lookup_symbol rather than looking up the name ourselves.
Fish it out repeatedly, to make sure that symbol caching isn't
broken.
(symidx_64): Remove.
(symidx_32): Remove.
* testsuite/libctf-lookup/enum-symbol-obj.lk: Test symbol lookup
in an unlinked object file (indexed symtypetab sections only).
* testsuite/libctf-writable/symtypetab-nonlinker-writeout.c
(try_maybe_reporting): Check symbol types via
ctf_lookup_by_symbol_name as well as ctf_symbol_next.
* testsuite/libctf-lookup/conflicting-type-syms.*: New test of
lookups in a multi-dict archive.
|
|
The variable section in a CTF dict is meant to contain the types of
variables that do not appear in the symbol table (mostly file-scope
static declarations). We implement this by having the compiler emit
all potential data symbols into both sections, then delete those
symbols from the variable section that correspond to data symbols the
linker has reported.
Unfortunately, the check for this in ctf_serialize is wrong: rather than
checking the set of linker-reported symbols, we check the set of names
in the data object symtypetab section: if the linker has reported no
symbols at all (usually if ld -r has been run, or if a non-linker
program that does not use symbol tables is calling ctf_link) this will
include every single symbol, emptying the variable section completely.
Worse, when ld -r is in use, we want to force writeout of every
symtypetab entry on the inputs, in an indexed section, whether or not
the linker has reported them, since this isn't a final link yet and the
symbol table is not finalized (and may grow more symbols than the linker
has yet reported). But the check for this is flawed too: we were
relying on ctf_link_shuffle_syms not having been called if no symbols
exist, but that function is *always* called by ld even when ld -r is in
use: ctf_link_add_linker_symbol is the one that's not called when there
are no symbols.
We clearly need to rethink this. Using the emptiness of the set of
reported symbols as a test for ld -r is just ugly: the linker already
knows if ld -r is underway and can just tell us. So add a new linker
flag CTF_LINK_NO_FILTER_REPORTED_SYMS that is set to stop the linker
filtering the symbols in the symtypetab sections using the set that the
linker has reported: use the presence or absence of this flag to
determine whether to emit unindexed symtabs: we only remove entries from
the variable section when filtering symbols, and we only remove them if
they are in the reported symbol set, fixing the case where no symbols
are reported by the linker at all.
(The negative sense of the new CTF_LINK flag is intentional: the common
case, both for ld and for simple tools that want to do a ctf_link with
no ELF symbol table in sight, is probably to filter out symbols that no
linker has reported: i.e., for the simple tools, all of them.)
There's another wrinkle, though. It is quite possible for a non-linker
to add symbols to a dict via ctf_add_*_sym and then write it out via the
ctf_write APIs: perhaps it's preparing a dict for a later linker
invocation. Right now this would not lead to anything terribly
meaningful happening: ctf_serialize just assumes it was called via
ctf_link if symbols are present. So add an (internal-to-libctf) flag
that indicates that a writeout is happening via ctf_link_write, and set
it there (propagating it to child dicts as needed). ctf_serialize can
then spot when it is not being called by a linker, and arrange to always
write out an indexed, sorted symtypetab for fastest possible future
symbol lookup by name in that case. (The writeouts done by ld -r are
unsorted, because the only thing likely to use those symtabs is the
linker, which doesn't benefit from symtypetab sorting.)
Tests added for all three linking cases (ld -r, ld -shared, ld), with a
bit of testsuite framework enhancement to stop it unconditionally
linking the CTF to be checked by the lookup program with -shared, so
tests can now examine CTF linked with -r or indeed with no flags at all,
though the output filename is still foo.so even in this case.
Another test added for the non-linker case that endeavours to determine
whether the symtypetab is sorted by examining the order of entries
returned from ctf_symbol_next: nobody outside libctf should rely on
this ordering, but this test is not outside libctf :)
include/ChangeLog
2021-01-26 Nick Alcock <nick.alcock@oracle.com>
* ctf-api.h (CTF_LINK_NO_FILTER_REPORTED_SYMS): New.
ld/ChangeLog
2021-01-26 Nick Alcock <nick.alcock@oracle.com>
* ldlang.c (lang_merge_ctf): Set CTF_LINK_NO_FILTER_REPORTED_SYMS
when appropriate.
libctf/ChangeLog
2021-01-27 Nick Alcock <nick.alcock@oracle.com>
* ctf-impl.c (_libctf_nonnull_): Add parameters.
(LCTF_LINKING): New flag.
(ctf_dict_t) <ctf_link_flags>: Mention it.
* ctf-link.c (ctf_link): Keep LCTF_LINKING set across call.
(ctf_write): Likewise, including in child dictionaries.
(ctf_link_shuffle_syms): Make sure ctf_dynsyms is NULL if there
are no reported symbols.
* ctf-create.c (symtypetab_delete_nonstatic_vars): Make sure
the variable has been reported as a symbol by the linker.
(symtypetab_skippable): Mention relationship between SYMFP and the
flags.
(symtypetab_density): Adjust nonnullity. Exit early if no symbols
were reported and force-indexing is off (i.e., we are doing a
final link).
(ctf_serialize): Handle the !LCTF_LINKING case by writing out an
indexed, sorted symtypetab (and allow SYMFP to be NULL in this
case). Turn sorting off if this is a non-final link. Only delete
nonstatic vars if we are filtering symbols and the linker has
reported some.
* testsuite/libctf-regression/nonstatic-var-section-ld-r*:
New test of variable and symtypetab section population when
ld -r is used.
* testsuite/libctf-regression/nonstatic-var-section-ld-executable.lk:
Likewise, when ld of an executable is used.
* testsuite/libctf-regression/nonstatic-var-section-ld.lk:
Likewise, when ld -shared alone is used.
* testsuite/libctf-regression/nonstatic-var-section-ld*.c:
Lookup programs for the above.
* testsuite/libctf-writable/symtypetab-nonlinker-writeout.*: New
test, testing survival of symbols across ctf_write paths.
* testsuite/lib/ctf-lib.exp (run_lookup_test): New option,
nonshared, suppressing linking of the SOURCE with -shared.
|
|
When you look up a type by name using ctf_lookup_by_name, in most cases
libctf can just strip off any qualifiers and look for the name, but for
pointer types this doesn't work, since the caller will want the pointer
type itself. But pointer types are nameless, and while they cite the
types they point to, looking up a type by name requires a link going the
*other way*, from the type pointed to to the pointer type that points to
it.
libctf has always built this up at open time: ctf_ptrtab is an array of
type indexes pointing from the index of every type to the index of the
type that points to it. But because it is built up at open time (and
because it uses type indexes and not type IDs) it is restricted to
working within a single dict and ignoring parent/child
relationships. This is normally invisible, unless you manage to get a
dict with a type in the parent but the only pointer to it in a child.
The ctf_ptrtab will not track this relationship, so lookups of this
pointer type by name will fail. Since which type is in the parent and
which in the child is largely opaque to the user (which goes where is up
to the deduplicator, and it can and does reshuffle things to save
space), this leads to a very bad user experience, with an
obviously-visible pointer type which ctf_lookup_by_name claims doesn't
exist.
The fix is to have another array, ctf_pptrtab, which is populated in
child dicts: like the parent's ctf_ptrtab, it has one element per type
in the parent, but is all zeroes except for those types which are
pointed to by types in the child: so it maps parent dict indices to
child dict indices. The array is grown, and new child types scanned,
whenever a lookup happens and new types have been added to the child
since the last time a lookup happened that might need the pptrtab.
(So for non-writable dicts, this only happens once, since new types
cannot be added to non-writable dicts at all.)
Since this introduces new complexity (involving updating only part of
the ctf_pptrtab) which is only seen when a writable dict is in use, we
introduce a new libctf-writable testsuite that contains lookup tests
with no corresponding CTF-containing .c files (which can thus be run
even on platforms with no .ctf-section support in the linker yet), and
add a test to check that creation of pointers in children to types in
parents and a following lookup by name works as expected. The non-
writable case is tested in a new libctf-regression testsuite which is
used to track now-fixed outright bugs in libctf.
libctf/ChangeLog
2021-01-05 Nick Alcock <nick.alcock@oracle.com>
* ctf-impl.h (ctf_dict_t) <ctf_pptrtab>: New.
<ctf_pptrtab_len>: New.
<ctf_pptrtab_typemax>: New.
* ctf-create.c (ctf_serialize): Update accordingly.
(ctf_add_reftype): Note that we don't need to update pptrtab here,
despite updating ptrtab.
* ctf-open.c (ctf_dict_close): Destroy the pptrtab.
(ctf_import): Likewise.
(ctf_import_unref): Likewise.
* ctf-lookup.c (grow_pptrtab): New.
(refresh_pptrtab): New, update a pptrtab.
(ctf_lookup_by_name): Turn into a wrapper around (and rename to)...
(ctf_lookup_by_name_internal): ... this: construct the pptrtab, and
use it in addition to the parent's ptrtab when parent dicts are
searched.
* testsuite/libctf-regression/regression.exp: New testsuite for
regression tests.
* testsuite/libctf-regression/pptrtab*: New test.
* testsuite/libctf-writable/writable.exp: New testsuite for tests of
writable CTF dicts.
* testsuite/libctf-writable/pptrtab*: New test.
|
