aboutsummaryrefslogtreecommitdiff
path: root/llvm/lib/LTO/LTO.cpp
AgeCommit message (Collapse)AuthorFilesLines
2019-05-17[ELF] Implement Dependent Libraries FeatureBen Dunbobbin1-0/+1
This patch implements a limited form of autolinking primarily designed to allow either the --dependent-library compiler option, or "comment lib" pragmas ( https://docs.microsoft.com/en-us/cpp/preprocessor/comment-c-cpp?view=vs-2017) in C/C++ e.g. #pragma comment(lib, "foo"), to cause an ELF linker to automatically add the specified library to the link when processing the input file generated by the compiler. Currently this extension is unique to LLVM and LLD. However, care has been taken to design this feature so that it could be supported by other ELF linkers. The design goals were to provide: - A simple linking model for developers to reason about. - The ability to to override autolinking from the linker command line. - Source code compatibility, where possible, with "comment lib" pragmas in other environments (MSVC in particular). Dependent library support is implemented differently for ELF platforms than on the other platforms. Primarily this difference is that on ELF we pass the dependent library specifiers directly to the linker without manipulating them. This is in contrast to other platforms where they are mapped to a specific linker option by the compiler. This difference is a result of the greater variety of ELF linkers and the fact that ELF linkers tend to handle libraries in a more complicated fashion than on other platforms. This forces us to defer handling the specifiers to the linker. In order to achieve a level of source code compatibility with other platforms we have restricted this feature to work with libraries that meet the following "reasonable" requirements: 1. There are no competing defined symbols in a given set of libraries, or if they exist, the program owner doesn't care which is linked to their program. 2. There may be circular dependencies between libraries. The binary representation is a mergeable string section (SHF_MERGE, SHF_STRINGS), called .deplibs, with custom type SHT_LLVM_DEPENDENT_LIBRARIES (0x6fff4c04). The compiler forms this section by concatenating the arguments of the "comment lib" pragmas and --dependent-library options in the order they are encountered. Partial (-r, -Ur) links are handled by concatenating .deplibs sections with the normal mergeable string section rules. As an example, #pragma comment(lib, "foo") would result in: .section ".deplibs","MS",@llvm_dependent_libraries,1 .asciz "foo" For LTO, equivalent information to the contents of a the .deplibs section can be retrieved by the LLD for bitcode input files. LLD processes the dependent library specifiers in the following way: 1. Dependent libraries which are found from the specifiers in .deplibs sections of relocatable object files are added when the linker decides to include that file (which could itself be in a library) in the link. Dependent libraries behave as if they were appended to the command line after all other options. As a consequence the set of dependent libraries are searched last to resolve symbols. 2. It is an error if a file cannot be found for a given specifier. 3. Any command line options in effect at the end of the command line parsing apply to the dependent libraries, e.g. --whole-archive. 4. The linker tries to add a library or relocatable object file from each of the strings in a .deplibs section by; first, handling the string as if it was specified on the command line; second, by looking for the string in each of the library search paths in turn; third, by looking for a lib<string>.a or lib<string>.so (depending on the current mode of the linker) in each of the library search paths. 5. A new command line option --no-dependent-libraries tells LLD to ignore the dependent libraries. Rationale for the above points: 1. Adding the dependent libraries last makes the process simple to understand from a developers perspective. All linkers are able to implement this scheme. 2. Error-ing for libraries that are not found seems like better behavior than failing the link during symbol resolution. 3. It seems useful for the user to be able to apply command line options which will affect all of the dependent libraries. There is a potential problem of surprise for developers, who might not realize that these options would apply to these "invisible" input files; however, despite the potential for surprise, this is easy for developers to reason about and gives developers the control that they may require. 4. This algorithm takes into account all of the different ways that ELF linkers find input files. The different search methods are tried by the linker in most obvious to least obvious order. 5. I considered adding finer grained control over which dependent libraries were ignored (e.g. MSVC has /nodefaultlib:<library>); however, I concluded that this is not necessary: if finer control is required developers can fall back to using the command line directly. RFC thread: http://lists.llvm.org/pipermail/llvm-dev/2019-March/131004.html. Differential Revision: https://reviews.llvm.org/D60274 llvm-svn: 360984
2019-05-13[ThinLTO] Don't internalize weak writeable variablesEugene Leviant1-1/+14
Variables with linkonce_odr and weak_odr linkage shouldn't be internalized if they're not readonly. Otherwise we may end up with multiple copies of such variable, so reads and writes will become inconsistent Differential revision: https://reviews.llvm.org/D61255 llvm-svn: 360577
2019-05-10[ThinLTO] Auto-hide prevailing linkonce_odr only when all copies eligibleTeresa Johnson1-14/+29
Summary: We hit undefined references building with ThinLTO when one source file contained explicit instantiations of a template method (weak_odr) but there were also implicit instantiations in another file (linkonce_odr), and the latter was the prevailing copy. In this case the symbol was marked hidden when the prevailing linkonce_odr copy was promoted to weak_odr. It led to unsats when the resulting shared library was linked with other code that contained a reference (expecting to be resolved due to the explicit instantiation). Add a CanAutoHide flag to the GV summary to allow the thin link to identify when all copies are eligible for auto-hiding (because they were all originally linkonce_odr global unnamed addr), and only do the auto-hide in that case. Most of the changes here are due to plumbing the new flag through the bitcode and llvm assembly, and resulting test changes. I augmented the existing auto-hide test to check for this situation. Reviewers: pcc Subscribers: mehdi_amini, inglorion, eraman, dexonsmith, arphaman, dang, llvm-commits, steven_wu, wmi Tags: #llvm Differential Revision: https://reviews.llvm.org/D59709 llvm-svn: 360466
2019-04-19[LTO] Add plumbing to save stats during LTO on Darwin.Florian Hahn1-10/+21
Gold and ld on Linux already support saving stats, but the infrastructure is missing on Darwin. Unfortunately it seems like the configuration from lib/LTO/LTO.cpp is not used. This patch adds a new LTOStatsFile option and adds plumbing in Clang to use it on Darwin, similar to the way remarks are handled. Currnetly the handling of LTO flags seems quite spread out, with a bunch of duplication. But I am not sure if there is an easy way to improve that? Reviewers: anemet, tejohnson, thegameg, steven_wu Reviewed By: steven_wu Differential Revision: https://reviews.llvm.org/D60516 llvm-svn: 358753
2019-04-17[ThinLTO] Fix ThinLTOCodegenerator to export llvm.used symbolsSteven Wu1-0/+5
Summary: Reapply r357931 with fixes to ThinLTO testcases and llvm-lto tool. ThinLTOCodeGenerator currently does not preserve llvm.used symbols and it can internalize them. In order to pass the necessary information to the legacy ThinLTOCodeGenerator, the input to the code generator is rewritten to be based on lto::InputFile. Now ThinLTO using the legacy LTO API will requires data layout in Module. "internalize" thinlto action in llvm-lto is updated to run both "promote" and "internalize" with the same configuration as ThinLTOCodeGenerator. The old "promote" + "internalize" option does not produce the same output as ThinLTOCodeGenerator. This fixes: PR41236 rdar://problem/49293439 Reviewers: tejohnson, pcc, kromanova, dexonsmith Reviewed By: tejohnson Subscribers: ormris, bd1976llvm, mehdi_amini, inglorion, eraman, hiraditya, jkorous, dexonsmith, arphaman, dang, llvm-commits Tags: #llvm Differential Revision: https://reviews.llvm.org/D60421 llvm-svn: 358601
2019-04-08Revert [ThinLTO] Fix ThinLTOCodegenerator to export llvm.used symbolsSteven Wu1-5/+0
This reverts r357931 (git commit 8b70a5c11e08116955a875b9085433f14737bcaf) llvm-svn: 357932
2019-04-08[ThinLTO] Fix ThinLTOCodegenerator to export llvm.used symbolsSteven Wu1-0/+5
Summary: ThinLTOCodeGenerator currently does not preserve llvm.used symbols and it can internalize them. In order to pass the necessary information to the legacy ThinLTOCodeGenerator, the input to the code generator is rewritten to be based on lto::InputFile. This fixes: PR41236 rdar://problem/49293439 Reviewers: tejohnson, pcc, dexonsmith Reviewed By: tejohnson Subscribers: mehdi_amini, inglorion, eraman, hiraditya, jkorous, dang, llvm-commits Tags: #llvm Differential Revision: https://reviews.llvm.org/D60226 llvm-svn: 357931
2019-03-12Reland "[Remarks] Add -foptimization-record-passes to filter remark emission"Francis Visoiu Mistrih1-0/+6
Currently we have -Rpass for filtering the remarks that are displayed as diagnostics, but when using -fsave-optimization-record, there is no way to filter the remarks while generating them. This adds support for filtering remarks by passes using a regex. Ex: `clang -fsave-optimization-record -foptimization-record-passes=inline` will only emit the remarks coming from the pass `inline`. This adds: * `-fsave-optimization-record` to the driver * `-opt-record-passes` to cc1 * `-lto-pass-remarks-filter` to the LTOCodeGenerator * `--opt-remarks-passes` to lld * `-pass-remarks-filter` to llc, opt, llvm-lto, llvm-lto2 * `-opt-remarks-passes` to gold-plugin Differential Revision: https://reviews.llvm.org/D59268 Original llvm-svn: 355964 llvm-svn: 355984
2019-03-12Revert "[Remarks] Add -foptimization-record-passes to filter remark emission"Francis Visoiu Mistrih1-6/+0
This reverts commit 20fff32b7d1f1a1bd417b22aa9f26ededd97a3e5. llvm-svn: 355976
2019-03-12[Remarks] Add -foptimization-record-passes to filter remark emissionFrancis Visoiu Mistrih1-0/+6
Currently we have -Rpass for filtering the remarks that are displayed as diagnostics, but when using -fsave-optimization-record, there is no way to filter the remarks while generating them. This adds support for filtering remarks by passes using a regex. Ex: `clang -fsave-optimization-record -foptimization-record-passes=inline` will only emit the remarks coming from the pass `inline`. This adds: * `-fsave-optimization-record` to the driver * `-opt-record-passes` to cc1 * `-lto-pass-remarks-filter` to the LTOCodeGenerator * `--opt-remarks-passes` to lld * `-pass-remarks-filter` to llc, opt, llvm-lto, llvm-lto2 * `-opt-remarks-passes` to gold-plugin Differential Revision: https://reviews.llvm.org/D59268 llvm-svn: 355964
2019-03-06Reland "[Remarks] Refactor remark diagnostic emission in a RemarkStreamer"Francis Visoiu Mistrih1-2/+3
This allows us to store more info about where we're emitting the remarks without cluttering LLVMContext. This is needed for future support for the remark section. Differential Revision: https://reviews.llvm.org/D58996 Original llvm-svn: 355507 llvm-svn: 355514
2019-03-06Revert "[Remarks] Refactor remark diagnostic emission in a RemarkStreamer"Francis Visoiu Mistrih1-3/+2
This reverts commit 2e8c4997a2089f8228c843fd81b148d903472e02. Breaks bots. llvm-svn: 355511
2019-03-06[Remarks] Refactor remark diagnostic emission in a RemarkStreamerFrancis Visoiu Mistrih1-2/+3
This allows us to store more info about where we're emitting the remarks without cluttering LLVMContext. This is needed for future support for the remark section. Differential Revision: https://reviews.llvm.org/D58996 llvm-svn: 355507
2019-02-14[ThinLTO] Detect partially split modules during the thin linkTeresa Johnson1-5/+51
Summary: The changes to disable LTO unit splitting by default (r350949) and detect inconsistently split LTO units (r350948) are causing some crashes when the inconsistency is detected in multiple threads simultaneously. Fix that by having the code always look for the inconsistently split LTO units during the thin link, by checking for the presence of type tests recorded in the summaries. Modify test added in r350948 to remove single threading required to fix a bot failure due to this issue (and some debugging options added in the process of diagnosing it). Reviewers: pcc Subscribers: mehdi_amini, inglorion, eraman, steven_wu, dexonsmith, llvm-commits Tags: #llvm Differential Revision: https://reviews.llvm.org/D57561 llvm-svn: 354062
2019-01-19Update the file headers across all of the LLVM projects in the monorepoChandler Carruth1-4/+3
to reflect the new license. We understand that people may be surprised that we're moving the header entirely to discuss the new license. We checked this carefully with the Foundation's lawyer and we believe this is the correct approach. Essentially, all code in the project is now made available by the LLVM project under our new license, so you will see that the license headers include that license only. Some of our contributors have contributed code under our old license, and accordingly, we have retained a copy of our old license notice in the top-level files in each project and repository. llvm-svn: 351636
2019-01-11[LTO] Record whether LTOUnit splitting is enabled in indexTeresa Johnson1-0/+9
Summary: Records in the module summary index whether the bitcode was compiled with the option necessary to enable splitting the LTO unit (e.g. -fsanitize=cfi, -fwhole-program-vtables, or -fsplit-lto-unit). The information is passed down to the ModuleSummaryIndex builder via a new module flag "EnableSplitLTOUnit", which is propagated onto a flag on the summary index. This is then used during the LTO link to check whether all linked summaries were built with the same value of this flag. If not, an error is issued when we detect a situation requiring whole program visibility of the class hierarchy. This is the case when both of the following conditions are met: 1) We are performing LowerTypeTests or Whole Program Devirtualization. 2) There are type tests or type checked loads in the code. Note I have also changed the ThinLTOBitcodeWriter to also gate the module splitting on the value of this flag. Reviewers: pcc Subscribers: ormris, mehdi_amini, Prazek, inglorion, eraman, steven_wu, dexonsmith, arphaman, dang, llvm-commits Differential Revision: https://reviews.llvm.org/D53890 llvm-svn: 350948
2018-12-19[ThinLTO] Remove dllimport attribute from locally defined symbolsMatthew Voss1-1/+5
Summary: The LTO/ThinLTO driver currently creates invalid bitcode by setting symbols marked dllimport as dso_local. The compiler often has access to the definition (often dllexport) and the declaration (often dllimport) of an object at link-time, leading to a conflicting declaration. This patch resolves the inconsistency by removing the dllimport attribute. Reviewers: tejohnson, pcc, rnk, echristo Reviewed By: rnk Subscribers: dmikulin, wristow, mehdi_amini, inglorion, eraman, steven_wu, dexonsmith, dang, llvm-commits Differential Revision: https://reviews.llvm.org/D55627 llvm-svn: 349667
2018-12-13[ThinLTO] Compute synthetic function entry countEaswaran Raman1-0/+5
Summary: This patch computes the synthetic function entry count on the whole program callgraph (based on module summary) and writes the entry counts to the summary. After function importing, this count gets attached to the IR as metadata. Since it adds a new field to the summary, this bumps up the version. Reviewers: tejohnson Subscribers: mehdi_amini, inglorion, llvm-commits Differential Revision: https://reviews.llvm.org/D43521 llvm-svn: 349076
2018-12-05LTO: Don't internalize available_externally globals.Peter Collingbourne1-1/+4
This breaks C and C++ semantics because it can cause the address of the global inside the module to differ from the address outside of the module. Differential Revision: https://reviews.llvm.org/D55237 llvm-svn: 348321
2018-12-04[ThinLTO] Look through aliases when computing hash keysGeorge Burgess IV1-2/+8
Without this, we don't consider types used by aliasees in our cache key. This caused issues when using the same cache for thin-linking the same TU with different sets of virtual call candidates for a virtual call inside of a constructor. That's sort of a mouthful. :) Differential Revision: https://reviews.llvm.org/D55060 llvm-svn: 348216
2018-11-26[ThinLTO] Consolidate cache key computation between new/old LTO APIsTeresa Johnson1-5/+6
Summary: The old legacy LTO API had a separate cache key computation, which was a subset of the cache key computation in the new LTO API (from what I can tell this is largely just because certain features such as CFI, dsoLocal, etc are only utilized via the new LTO API). However, having separate computations is unnecessary (much of the code is duplicated), and can lead to bugs when adding new optimizations if both cache computation algorithms aren't updated properly - it's much easier to maintain if we have a single facility. This patch refactors the old LTO API code to use the cache key computation from the new LTO API. To do this, we set up an lto::Config object and fill in the fields that the old LTO was hashing (the others will just use the defaults). There are two notable changes: - I added a Freestanding flag to the LTO Config. Currently this is only used by the legacy LTO API. In the patch that added it (D30791) I had asked about adding it to the new LTO API, but it looks like that was not addressed. This should probably be discussed as a follow up to this change, as it is orthogonal. - The legacy LTO API had some code that was hashing the GUID of all preserved symbols defined in the module. I looked back at the history of this (which was added with the original hashing in the legacy LTO API in D18494), and there is a comment in the review thread that it was added in preparation for future internalization. We now do the internalization of course, and that is handled in the new LTO API cache key computation by hashing the recorded linkage type of all defined globals. Therefore I didn't try to move over and keep the preserved symbols handling. Reviewers: steven_wu, pcc Subscribers: mehdi_amini, inglorion, eraman, dexonsmith, dang, llvm-commits Differential Revision: https://reviews.llvm.org/D54635 llvm-svn: 347592
2018-11-16[ThinLTO] Internalize readonly globalsEugene Leviant1-1/+4
An attempt to recommit r346584 after failure on OSX build bot. Fixed cache key computation in ThinLTOCodeGenerator and added test case llvm-svn: 347033
2018-11-13Revert "[ThinLTO] Internalize readonly globals"Steven Wu1-4/+1
This reverts commit 10c84a8f35cae4a9fc421648d9608fccda3925f2. llvm-svn: 346768
2018-11-10[ThinLTO] Internalize readonly globalsEugene Leviant1-1/+4
This patch allows internalising globals if all accesses to them (from live functions) are from non-volatile load instructions Differential revision: https://reviews.llvm.org/D49362 llvm-svn: 346584
2018-11-08[LTO] Drop non-prevailing definitions only if linkage is not local or appendingPirama Arumuga Nainar1-10/+16
Summary: This fixes PR 37422 In ELF, non-weak symbols can also be non-prevailing. In this particular PR, the __llvm_profile_* symbols are non-prevailing but weren't getting dropped - causing multiply-defined errors with lld. Also add a test, strong_non_prevailing.ll, to ensure that multiple copies of a strong symbol are dropped. To fix the test regressions exposed by this fix, - do not mark prevailing copies for symbols with 'appending' linkage. There's no one prevailing copy for such symbols. - fix the prevailing version in dead-strip-fulllto.ll - explicitly pass exported symbols to llvm-lto in fumcimport.ll and funcimport_var.ll Reviewers: tejohnson, pcc Subscribers: mehdi_amini, inglorion, eraman, steven_wu, dexonsmith, dang, srhines, llvm-commits Differential Revision: https://reviews.llvm.org/D54125 llvm-svn: 346436
2018-11-05[ThinLTO] Add an option to disable (thin)lto internalization.Xin Tong1-2/+8
Summary: LTO and ThinLTO optimizes the IR differently. One source of differences is the amount of internalizations that can happen. Add an option to enable/disable internalization so that other differences can be studied in isolation. e.g. inlining. There are other things lto and thinlto do differently, I will add flags to enable/disable them as needed. Reviewers: tejohnson, pcc, steven_wu Subscribers: mehdi_amini, inglorion, steven_wu, dexonsmith, dang, llvm-commits Differential Revision: https://reviews.llvm.org/D53294 llvm-svn: 346140
2018-10-10Add a flag to remap manglings when reading profile data information.Richard Smith1-1/+8
This can be used to preserve profiling information across codebase changes that have widespread impact on mangled names, but across which most profiling data should still be usable. For example, when switching from libstdc++ to libc++, or from the old libstdc++ ABI to the new ABI, or even from a 32-bit to a 64-bit build. The user can provide a remapping file specifying parts of mangled names that should be treated as equivalent (eg, std::__1 should be treated as equivalent to std::__cxx11), and profile data will be treated as applying to a particular function if its name is equivalent to the name of a function in the profile data under the provided equivalences. See the documentation change for a description of how this is configured. Remapping is supported for both sample-based profiling and instruction profiling. We do not support remapping indirect branch target information, but all other profile data should be remapped appropriately. Support is only added for the new pass manager. If someone wants to also add support for this for the old pass manager, doing so should be straightforward. This is the LLVM side of Clang r344199. Reviewers: davidxl, tejohnson, dlj, erik.pilkington Subscribers: mehdi_amini, steven_wu, dexonsmith, llvm-commits Differential Revision: https://reviews.llvm.org/D51249 llvm-svn: 344200
2018-09-25[ThinLTO] Efficiency fix for writing type id records in per-module indexesTeresa Johnson1-27/+9
Summary: In D49565/r337503, the type id record writing was fixed so that only referenced type ids were emitted into each per-module index for ThinLTO distributed builds. However, this still left an efficiency issue: each per-module index checked all type ids for membership in the referenced set, yielding O(M*N) performance (M indexes and N type ids). Change the TypeIdMap in the summary to be indexed by GUID, to facilitate correlating with type identifier GUIDs referenced in the function summary TypeIdInfo structures. This allowed simplifying other places where a map from type id GUID to type id map entry was previously being used to aid this correlation. Also fix AsmWriter code to handle the rare case of type id GUID collision. For a large internal application, this reduced the thin link time by almost 15%. Reviewers: pcc, vitalybuka Subscribers: mehdi_amini, inglorion, steven_wu, dexonsmith, llvm-commits Differential Revision: https://reviews.llvm.org/D51330 llvm-svn: 343021
2018-07-27[LTO] Don't internalize declarationsBob Haarman1-1/+2
Summary: Some links were failing with "Global is external, but doesn't have external or weak linkage!" in ThinLTO builds with debug information. This happened when we elide the body of a global that is referenced by debug info. This results in a declaration, which we would then internalize - but declarations cannot be internal. This change avoids the problem by not internalizing these declarations. Fixes PR38046. Reviewers: pcc, tejohnson Subscribers: mehdi_amini, aprantl, hiraditya, JDevlieghere, steven_wu, dexonsmith, llvm-commits Differential Revision: https://reviews.llvm.org/D49777 llvm-svn: 338100
2018-07-23[LTO] Handle __imp_ (dllimport) symbols consistently with lldTeresa Johnson1-1/+8
Summary: Similar to what lld already does for dllimport symbols which are prefaced with __imp_ (see lld patch r240620), strip off the __imp_ prefix in LTO. Otherwise we can get 2 separate GlobalResolution for a single symbol, the dllimport declaration, and the definition, which leads to incorrect LTO handling. Fixes PR38105. Reviewers: pcc Subscribers: mehdi_amini, inglorion, steven_wu, dexonsmith, llvm-commits Differential Revision: https://reviews.llvm.org/D49138 llvm-svn: 337762
2018-07-16Restore "[ThinLTO] Ensure we always select the same function copy to import"Teresa Johnson1-2/+2
This reverts commit r337081, therefore restoring r337050 (and fix in r337059), with test fix for bot failure described after the original description below. In order to always import the same copy of a linkonce function, even when encountering it with different thresholds (a higher one then a lower one), keep track of the summary we decided to import. This ensures that the backend only gets a single definition to import for each GUID, so that it doesn't need to choose one. Move the largest threshold the GUID was considered for import into the current module out of the ImportMap (which is part of a larger map maintained across the whole index), and into a new map just maintained for the current module we are computing imports for. This saves some memory since we no longer have the thresholds maintained across the whole index (and throughout the in-process backends when doing a normal non-distributed ThinLTO build), at the cost of some additional information being maintained for each invocation of ComputeImportForModule (the selected summary pointer for each import). There is an additional map lookup for each callee being considered for importing, however, this was able to subsume a map lookup in the Worklist iteration that invokes computeImportForFunction. We also are able to avoid calling selectCallee if we already failed to import at the same or higher threshold. I compared the run time and peak memory for the SPEC2006 471.omnetpp benchmark (running in-process ThinLTO backends), as well as for a large internal benchmark with a distributed ThinLTO build (so just looking at the thin link time/memory). Across a number of runs with and without this change there was no significant change in the time and memory. (I tried a few other variations of the change but they also didn't improve time or peak memory). The new commit removes a test that no longer makes sense (Transforms/FunctionImport/hotness_based_import2.ll), as exposed by the reverse-iteration bot. The test depends on the order of processing the summary call edges, and actually depended on the old problematic behavior of selecting more than one summary for a given GUID when encountered with different thresholds. There was no guarantee even before that we would eventually pick the linkonce copy with the hottest call edges, it just happened to work with the test and the old code, and there was no guarantee that we would end up importing the selected version of the copy that had the hottest call edges (since the backend would effectively import only one of the selected copies). Reviewers: davidxl Subscribers: mehdi_amini, inglorion, llvm-commits Differential Revision: https://reviews.llvm.org/D48670 llvm-svn: 337184
2018-07-14Revert "[ThinLTO] Ensure we always select the same function copy to import"Teresa Johnson1-2/+2
This reverts commits r337050 and r337059. Caused failure in reverse-iteration bot that needs more investigation. llvm-svn: 337081
2018-07-13[ThinLTO] Ensure we always select the same function copy to importTeresa Johnson1-2/+2
In order to always import the same copy of a linkonce function, even when encountering it with different thresholds (a higher one then a lower one), keep track of the summary we decided to import. This ensures that the backend only gets a single definition to import for each GUID, so that it doesn't need to choose one. Move the largest threshold the GUID was considered for import into the current module out of the ImportMap (which is part of a larger map maintained across the whole index), and into a new map just maintained for the current module we are computing imports for. This saves some memory since we no longer have the thresholds maintained across the whole index (and throughout the in-process backends when doing a normal non-distributed ThinLTO build), at the cost of some additional information being maintained for each invocation of ComputeImportForModule (the selected summary pointer for each import). There is an additional map lookup for each callee being considered for importing, however, this was able to subsume a map lookup in the Worklist iteration that invokes computeImportForFunction. We also are able to avoid calling selectCallee if we already failed to import at the same or higher threshold. I compared the run time and peak memory for the SPEC2006 471.omnetpp benchmark (running in-process ThinLTO backends), as well as for a large internal benchmark with a distributed ThinLTO build (so just looking at the thin link time/memory). Across a number of runs with and without this change there was no significant change in the time and memory. (I tried a few other variations of the change but they also didn't improve time or peak memory). Reviewers: davidxl Subscribers: mehdi_amini, inglorion, llvm-commits Differential Revision: https://reviews.llvm.org/D48670 llvm-svn: 337050
2018-07-10[ThinLTO] Use std::map to get determistic imports filesTeresa Johnson1-1/+2
Summary: I noticed that the .imports files emitted for distributed ThinLTO backends do not have consistent ordering. This is because StringMap iteration order is not guaranteed to be deterministic. Since we already have a std::map with this information, used when emitting the individual index files (ModuleToSummariesForIndex), use it for the imports files as well. This issue is likely causing some unnecessary rebuilds of the ThinLTO backends in our distributed build system as the imports files are inputs to those backends. Reviewers: pcc, steven_wu, mehdi_amini Subscribers: mehdi_amini, inglorion, eraman, steven_wu, dexonsmith, llvm-commits Differential Revision: https://reviews.llvm.org/D48783 llvm-svn: 336721
2018-06-06[ThinLTO] Rename index IsAnalysis flag to HaveGVs (NFC)Teresa Johnson1-1/+1
With the upcoming patch to add summary parsing support, IsAnalysis would be true in contexts where we are not performing module summary analysis. Rename to the more specific and approprate HaveGVs, which is essentially what this flag is indicating. llvm-svn: 334140
2018-05-04[LTO] Allow pass remarks with hotness to be set when emitting to stderrTeresa Johnson1-2/+2
Summary: Set setDiagnosticsHotnessRequested before the early exit check for a diagnostic output file, so that pass remarks with hotness works when emitting pass remarks to stderr (e.g. via -pass-remarks=.). Also fix the llvm-lto2 diagnistic handler so that it only calls exit(1) when the diagnistic is an error type. Otherwise the new test invocation of llvm-lto2 with -pass-remarks causes it to fail. The new code is consistent with the diagnostic handler elsewhere (e.g. on the LLVMContext). Reviewers: pcc, davide Subscribers: fhahn, mehdi_amini, llvm-commits, inglorion Differential Revision: https://reviews.llvm.org/D46387 llvm-svn: 331569
2018-04-30IWYU for llvm-config.h in llvm, additions.Nico Weber1-0/+1
See r331124 for how I made a list of files missing the include. I then ran this Python script: for f in open('filelist.txt'): f = f.strip() fl = open(f).readlines() found = False for i in xrange(len(fl)): p = '#include "llvm/' if not fl[i].startswith(p): continue if fl[i][len(p):] > 'Config': fl.insert(i, '#include "llvm/Config/llvm-config.h"\n') found = True break if not found: print 'not found', f else: open(f, 'w').write(''.join(fl)) and then looked through everything with `svn diff | diffstat -l | xargs -n 1000 gvim -p` and tried to fix include ordering and whatnot. No intended behavior change. llvm-svn: 331184
2018-04-20[LTO] Add stats-file option to LTO/Config.h.Florian Hahn1-3/+21
This patch adds a StatsFile option to LTO/Config.h and updates both LLVMGold and llvm-lto2 to set it. Reviewers: MatzeB, tejohnson, espindola Reviewed By: tejohnson Differential Revision: https://reviews.llvm.org/D45531 llvm-svn: 330411
2018-04-13Enable debug fission for thinLTO linked via gold-pluginYunlian Jiang1-0/+1
Summary: This enables debug fission on implicit ThinLTO when linked with gold. It will put the .dwo files in a directory specified by user. Reviewers: tejohnson, pcc, dblaikie Reviewed By: pcc Subscribers: JDevlieghere, mehdi_amini, inglorion Differential Revision: https://reviews.llvm.org/D44792 llvm-svn: 329988
2018-03-08Revert "[LTO] Support filtering by hotness threshold"Bob Haarman1-6/+4
This reverts commit 1f3bd185c53beb6aa68446974b7e80837abd6ef0 (r326107) because it fails ThinLTO/X86/diagnostic-handler-remarks-with-hotness.ll. llvm-svn: 326975
2018-02-26[LTO] Support filtering by hotness thresholdAdam Nemet1-4/+6
This wires up -pass-remarks-hotness-threshold to LTO and ThinLTO. Next is to change the clang driver to pass this with -fdiagnostics-hotness-threshold. Differential Revision: https://reviews.llvm.org/D41465 llvm-svn: 326107
2018-02-22[ThinLTO] Always create linked objects file for --thinlto-index-only=Vitaly Buka1-20/+8
Summary: ThinLTO indexing may decide to skip all objects. If we don't write something to the list build system may consider this as failure or linker can reuse a file from the previews build. Reviewers: pcc, tejohnson Subscribers: mehdi_amini, inglorion, eraman, hiraditya, llvm-commits Differential Revision: https://reviews.llvm.org/D43415 llvm-svn: 325819
2018-02-19[ThinLTO] Add GraphTraits for FunctionSummariesCharles Saternos1-0/+7
Add GraphTraits definitions to the FunctionSummary and ModuleSummaryIndex classes. These GraphTraits will be used to construct find SCC's in ThinLTO analysis passes. Third attempt - moved function from lambda to static function due to build failures. llvm-svn: 325506
2018-02-18Revert: [llvm] r325448 - [ThinLTO] Add GraphTraits for FunctionSummaries Simon Pilgrim1-7/+0
Add GraphTraits definitions to the FunctionSummary and ModuleSummaryIndex classes. These GraphTraits will be used to construct find SCC's in ThinLTO analysis passes. Second attempt, since last patch caused stage2 build to fail (now using function_ref rather than std::function). Reverted due to buildbot failures llvm-svn: 325454
2018-02-17[ThinLTO] Add GraphTraits for FunctionSummariesCharles Saternos1-0/+7
Add GraphTraits definitions to the FunctionSummary and ModuleSummaryIndex classes. These GraphTraits will be used to construct find SCC's in ThinLTO analysis passes. Second attempt, since last patch caused stage2 build to fail (now using function_ref rather than std::function). llvm-svn: 325448
2018-02-12Revert "[ThinLTO] Add GraphTraits for FunctionSummaries"Volodymyr Sapsai1-7/+0
It caused assertion failure Assertion failed: (!DD.IsLambda && !MergeDD.IsLambda && "faked up lambda definition?"), function MergeDefinitionData, file /Users/buildslave/jenkins/workspace/clang-stage1-configure-RA/llvm/tools/clang/lib/Serialization/ASTReaderDecl.cpp, line 1675. on the second stage build bots. llvm-svn: 324932
2018-02-11[ThinLTO] Add GraphTraits for FunctionSummariesCharles Saternos1-0/+7
Add GraphTraits definitions to the FunctionSummary and ModuleSummaryIndex classes. These GraphTraits will be used to construct find SCC's in ThinLTO analysis passes. llvm-svn: 324854
2018-02-09LTO: Include live bit in ThinLTO cache key.Peter Collingbourne1-0/+1
As of r323633, this bit started controlling whether symbol definitions appear in object files, and it also became sensitive to the prevailing bit, so it needs to be included in the key. Differential Revision: https://reviews.llvm.org/D43109 llvm-svn: 324711
2018-02-05LTO: Also include dso-local bit for calls in ThinLTO cache key.Peter Collingbourne1-1/+3
Differential Revision: https://reviews.llvm.org/D42934 llvm-svn: 324291
2018-02-05LTO: Include dso-local bit in ThinLTO cache key.Peter Collingbourne1-1/+3
Differential Revision: https://reviews.llvm.org/D42713 llvm-svn: 324253