| Age | Commit message (Collapse) | Author | Files | Lines |
|---|
|
These are identified by misc-include-cleaner. I've filtered out those
that break builds. Also, I'm staying away from llvm-config.h,
config.h, and Compiler.h, which likely cause platform- or
compiler-specific build failures.
|
|
This reverts commit 7a242387c950c7060143da6da0e6fb91f36bb458. Even after 175f8a44, the Modules/fmodules-validate-once-per-build-session.c test is not fixed on the clang-armv8-quick build bot. (Failure occurs on line 114.)
|
|
This reverts commit 18b885f66babff3a10451bc811ffc077d61ed8ee, effectively reapplying #139987. This commit fixes unit tests (for example ASTUnitTest.SaveLoadPreservesLangOptionsInPrintingPolicy) where the `ASTUnit::ModCache` pointer dereferenced within `ASTUnit::serialize()` was null. This commit makes sure each factory function does initialize `ASTUnit::ModCache`.
|
|
Reverts llvm/llvm-project#138983
|
|
Timestamps are an implementation detail of the cross-process module
cache implementation. This PR hides it from the `ModuleCache` API, which
simplifies the in-process implementation.
|
|
(#139637)
|
|
imports are (#138920)
The dependency scanner mixes implicitly- and explicitly-built modules.
When an implicitly-built module imports an explicitly-built one, we
never run the modification time validation checks, resulting in an
out-of-date module cache. This PR fixes that by only skipping the
modification time validation checks when both the imported module and
its importer are built explicitly.
rdar://150230022
|
|
In the past, timestamps used for
`-fmodules-validate-once-per-build-session` were found to be a source of
contention in the dependency scanner
([D149802](https://reviews.llvm.org/D149802),
https://github.com/llvm/llvm-project/pull/112452). This PR is yet
another attempt to optimize these. We now make use of the new
`ModuleCache` interface to implement the in-process version in terms of
atomic `std::time_t` variables rather the mtime attribute on
`.timestamp` files.
|
|
|
|
This PR adds new `ModuleCache` interface to Clang's implicitly-built
modules machinery. The main motivation for this change is to create a
second implementation that uses a more efficient kind of
`llvm::AdvisoryLock` during dependency scanning.
In addition to the lock abstraction, the `ModuleCache` interface also
manages the existing `InMemoryModuleCache` instance. I found that
compared to keeping these separate/independent, the code is a bit
simpler now, since these are two tightly coupled concepts. I can
envision a more efficient implementation of the `InMemoryModuleCache`
for the single-process case too, which will be much easier to implement
with the current setup.
This is not intended to be a functional change.
|
|
When a pcm file has a different size or modification time than it had
when it was written to another module's IMPORT table Clang emits:
`<pcm> is out of date and needs to be rebuilt: module file out of date`
This is difficult to understand what's happening because there are a lot
of reasons that a module file can be out of date. This changes the
latter part of that message to:
`module file has a different size or mtime than expected`
Which makes it clearer what the issue is. For future work it would be
nice if a more detailed explanation of the issue could be emitted as a
note instead.
|
|
Clang uses timestamp files to track the last time an implicitly-built
PCM file was verified to be up-to-date with regard to its inputs. With
`-fbuild-session-{file,timestamp}=` and
`-fmodules-validate-once-per-build-session` this reduces the number of
times a PCM file is checked per "build session".
The behavior I'm seeing with the current scheme is that when lots of
Clang instances wait for the same PCM to be built, they race to validate
it as soon as the file lock gets released, causing lots of concurrent
IO.
This patch makes it so that the timestamp is written by the same Clang
instance responsible for building the PCM while still holding the lock.
This makes it so that whenever a PCM file gets compiled, it's never
re-validated in the same build session.
I believe this is as sound as the current scheme. One thing to be aware
of is that there might be a time interval between accessing input file N
and writing the timestamp file, where changes to input files 0..<N would
not result in a rebuild. Since this is the case current scheme too, I'm
not too concerned about that.
I've seen this speed up `clang-scan-deps` by ~27%.
|
|
Some `FileManager` APIs still return `{File,Directory}Entry` instead of
the preferred `{File,Directory}EntryRef`. These are documented to be
deprecated, but don't have the attribute that warns on their usage. This
PR marks them as such with `LLVM_DEPRECATED()` and replaces their usage
with the recommended counterparts. NFCI.
|
|
|
|
AFAICT, `ModuleFile::File` can be `std::nullopt` only for PCM files
loaded from the standard input. This patch starts setting that variable
to `FileManager::getSTDIN()` in that case, which makes it possible to
remove the optionality, and also simplifies code that actually reads the
file.
This is part of an effort to get rid of
`Optional{File,Directory}EntryRefDegradesTo{File,Directory}EntryPtr`.
|
|
|
|
|
|
|
|
The needed tweaks are mostly trivial, the one nasty bit is Clang's usage
of OptionalStorage. To keep this working old Optional stays around as
clang::CustomizableOptional, with the default Storage removed.
Optional<File/DirectoryEntryRef> is replaced with a typedef.
I tested this with GCC 7.5, the oldest supported GCC I had around.
Differential Revision: https://reviews.llvm.org/D140332
|
|
std::optional"
This reverts commit 8f0df9f3bbc6d7f3d5cbfd955c5ee4404c53a75d.
The Optional*RefDegradesTo*EntryPtr types want to keep the same size as
the underlying type, which std::optional doesn't guarantee. For use with
llvm::Optional, they define their own storage class, and there is no way
to do that in std::optional.
On top of that, that commit broke builds with older GCCs, where
std::optional was not trivially copyable (static_assert in the clang
sources was failing).
|
|
|
|
This patch mechanically replaces None with std::nullopt where the
compiler would warn if None were deprecated. The intent is to reduce
the amount of manual work required in migrating from Optional to
std::optional.
This is part of an effort to migrate from llvm::Optional to
std::optional:
https://discourse.llvm.org/t/deprecating-llvm-optional-x-hasvalue-getvalue-getvalueor/63716
|
|
refers to unknown module...'".
When a framework can be found at a new location, all references to it in
the module cache become outdated. When we try to load such outdated .pcm
file, we shouldn't change any already loaded and processed modules.
If `Module` has `ASTFile`, it means we've read its AST block already and
it is too late to undo that. If `ASTFile` is `None`, there is no value
in setting it to `None` again. So we don't reset `ASTFile` in
`ModuleManager::removeModules` at all.
rdar://97216258
Differential Revision: https://reviews.llvm.org/D134249
|
|
|
|
|
|
|
|
Change `Module::ASTFile` and `ModuleFile::File` to use
`Optional<FileEntryRef>` instead of `const FileEntry *`. One of many
steps toward removing `FileEntry::getName`.
Differential Revision: https://reviews.llvm.org/D89836
|
|
once, NFC
Differential Revision: https://reviews.llvm.org/D89835
|
|
cache for implicit modules.
The ModuleManager's use of FileEntry nodes as the keys for its map of
loaded modules is less than ideal. Uniqueness for FileEntry nodes is
maintained by FileManager, which in turn uses inode numbers on hosts
that support that. When coupled with the module cache's proclivity for
turning over and deleting stale PCMs, this means entries for different
module files can wind up reusing the same underlying inode. When this
happens, subsequent accesses to the Modules map will disagree on the
ModuleFile associated with a given file.
In general, it is not sufficient to resolve this conundrum with a type
like FileEntryRef that stores the name of the FileEntry node on first
access because of path canonicalization issues. However, the paths
constructed for implicit module builds are fully under Clang's
control. We *can*, therefore, rely on their structure being consistent
across operating systems and across subsequent accesses to the Modules
map.
To mitigate the effects of inode reuse, perform an extra name check when
implicit modules are returned from the cache. This has the effect of
forcing reused FileEntry nodes to stomp over existing-but-stale entries
in the cache, which simulates a miss - exactly the desired behavior.
rdar://48443680
Patch by Robert Widmann!
Differential Revision: https://reviews.llvm.org/D86823
|
|
This change got missed while upstreaming
https://reviews.llvm.org/D77772. This is the part of that change that
actually passes the correct arguments when opening a PCM.
The test didn't catch this because it starts at the
`MemoryBuffer::getOpenFile` level. It's not really possible to test
`ModuleManager::addModule` itself to verify how the file was opened.
|
|
to load."
As per comment on https://reviews.llvm.org/D72860, it is suggested to
revert this change in the meantime, since it has introduced regression.
This reverts commit 83f4c3af021cd5322ea10fd1c4e839874c1dae49.
|
|
sed -Ei 's/[[:space:]]+$//' include/**/*.{def,h,td} lib/**/*.{cpp,h,td}
|
|
Partially reverts 0a2be46cfdb698fefcc860a56b47dde0884d5335 as it turned
out to cause redundant module rebuilds in multi-process incremental builds.
When a module was getting out of date, all compilation processes started at the
same time were marking it as `ToBuild`. So each process was building the same
module instead of checking if it was built by someone else and using that
result. In addition to the work duplication, contention on the same .pcm file
wasn't making builds faster.
Note that for a single-process build this change would cause redundant module
reads and validations. But reading a module is faster than building it and
multi-process builds are more common than single-process. So I'm willing to
make such a trade-off.
rdar://problem/54395127
Reviewed By: dexonsmith
Differential Revision: https://reviews.llvm.org/D72860
|
|
Remove some cognitive load by renaming clang/Serialization/Module.h to
clang/Serialization/ModuleFile.h, since it declares the ModuleFile
class. This also makes editing a bit easier, since the basename of the
file no long conflicts with clang/Basic/Module.h, which declares the
Module class. Also move lib/Serialization/Module.cpp to
lib/Serialization/ModuleFile.cpp.
|
|
The other paremeters appear to be sufficient to determine which modules
have just been loaded and need to be removed, so stop collecting and
sending in that set explicitly.
|
|
Remove this dead code. We always close it.
llvm-svn: 370488
|
|
Marking a module for a rebuild when its signature differs from the
expected one causes redundant module rebuilds for incremental builds.
When a module is updated, its signature changes. But its consumers still
have the old signature and loading them will result in signature
mismatches. It will correctly cause the rebuilds for the consumers but
we don't need to rebuild the common module for each of them as it is
already up to date.
In practice this bug causes longer build times. We are doing more work
than required and only a single process can build a module, so parallel
builds degrade to a single-process mode where extra processes are just
waiting on a file lock.
Fix by not marking a module dependency for a rebuild on signature
mismatch. We'll check if it is up to date when we load it.
rdar://problem/50212358
Reviewers: dexonsmith, bruno, rsmith
Reviewed By: dexonsmith, bruno
Subscribers: jkorous, ributzka, cfe-commits, aprantl
Differential Revision: https://reviews.llvm.org/D66907
llvm-svn: 370274
|
|
Now that we've moved to C++14, we no longer need the llvm::make_unique
implementation from STLExtras.h. This patch is a mechanical replacement
of (hopefully) all the llvm::make_unique instances across the monorepo.
Differential revision: https://reviews.llvm.org/D66259
llvm-svn: 368942
|
|
Update the callers of FileManager::getFile and FileManager::getDirectory to handle the new llvm::ErrorOr-returning methods.
Signed-off-by: Harlan Haskins <harlan@apple.com>
llvm-svn: 367616
|
|
This patch applies clang-tidy's bugprone-argument-comment tool
to LLVM, clang and lld source trees. Here is how I created this
patch:
$ git clone https://github.com/llvm/llvm-project.git
$ cd llvm-project
$ mkdir build
$ cd build
$ cmake -GNinja -DCMAKE_BUILD_TYPE=Debug \
-DLLVM_ENABLE_PROJECTS='clang;lld;clang-tools-extra' \
-DCMAKE_EXPORT_COMPILE_COMMANDS=On -DLLVM_ENABLE_LLD=On \
-DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++ ../llvm
$ ninja
$ parallel clang-tidy -checks='-*,bugprone-argument-comment' \
-config='{CheckOptions: [{key: StrictMode, value: 1}]}' -fix \
::: ../llvm/lib/**/*.{cpp,h} ../clang/lib/**/*.{cpp,h} ../lld/**/*.{cpp,h}
llvm-svn: 366177
|
|
llvm-svn: 357359
|
|
Leverage the InMemoryModuleCache to invalidate a module the first time
it fails to import (and to lock a module as soon as it's built or
imported successfully). For implicit module builds, this optimizes
importing deep graphs where the leaf module is out-of-date; see example
near the end of the commit message.
Previously the cache finalized ("locked in") all modules imported so far
when starting a new module build. This was sufficient to prevent
loading two versions of the same module, but was somewhat arbitrary and
hard to reason about.
Now the cache explicitly tracks module state, where each module must be
one of:
- Unknown: module not in the cache (yet).
- Tentative: module in the cache, but not yet fully imported.
- ToBuild: module found on disk could not be imported; need to build.
- Final: module in the cache has been successfully built or imported.
Preventing repeated failed imports avoids variation in builds based on
shifting filesystem state. Now it's guaranteed that a module is loaded
from disk exactly once. It now seems safe to remove
FileManager::invalidateCache, but I'm leaving that for a later commit.
The new, precise logic uncovered a pre-existing problem in the cache:
the map key is the module filename, and different contexts use different
filenames for the same PCM file. (In particular, the test
Modules/relative-import-path.c does not build without this commit.
r223577 started using a relative path to describe a module's base
directory when importing it within another module. As a result, the
module cache sees an absolute path when (a) building the module or
importing it at the top-level, and a relative path when (b) importing
the module underneath another one.)
The "obvious" fix is to resolve paths using FileManager::getVirtualFile
and change the map key for the cache to a FileEntry, but some contexts
(particularly related to ASTUnit) have a shorter lifetime for their
FileManager than the InMemoryModuleCache. This is worth pursuing
further in a later commit; perhaps by tying together the FileManager and
InMemoryModuleCache lifetime, or moving the in-memory PCM storage into a
VFS layer.
For now, use the PCM's base directory as-written for constructing the
filename to check the ModuleCache.
Example
=======
To understand the build optimization, first consider the build of a
module graph TU -> A -> B -> C -> D with an empty cache:
TU builds A'
A' builds B'
B' builds C'
C' builds D'
imports D'
B' imports C'
imports D'
A' imports B'
imports C'
imports D'
TU imports A'
imports B'
imports C'
imports D'
If we build TU again, where A, B, C, and D are in the cache and D is
out-of-date, we would previously get this build:
TU imports A
imports B
imports C
imports D (out-of-date)
TU builds A'
A' imports B
imports C
imports D (out-of-date)
builds B'
B' imports C
imports D (out-of-date)
builds C'
C' imports D (out-of-date)
builds D'
imports D'
B' imports C'
imports D'
A' imports B'
imports C'
imports D'
TU imports A'
imports B'
imports C'
imports D'
After this commit, we'll immediateley invalidate A, B, C, and D when we
first observe that D is out-of-date, giving this build:
TU imports A
imports B
imports C
imports D (out-of-date)
TU builds A' // The same graph as an empty cache.
A' builds B'
B' builds C'
C' builds D'
imports D'
B' imports C'
imports D'
A' imports B'
imports C'
imports D'
TU imports A'
imports B'
imports C'
imports D'
The new build matches what we'd naively expect, pretty closely matching
the original build with the empty cache.
rdar://problem/48545366
llvm-svn: 355778
|
|
Change MemoryBufferCache to InMemoryModuleCache, moving it from Basic to
Serialization. Another patch will start using it to manage module build
more explicitly, but this is split out because it's mostly mechanical.
Because of the move to Serialization we can no longer abuse the
Preprocessor to forward it to the ASTReader. Besides the rename and
file move, that means Preprocessor::Preprocessor has one fewer parameter
and ASTReader::ASTReader has one more.
llvm-svn: 355777
|
|
to reflect the new license.
We understand that people may be surprised that we're moving the header
entirely to discuss the new license. We checked this carefully with the
Foundation's lawyer and we believe this is the correct approach.
Essentially, all code in the project is now made available by the LLVM
project under our new license, so you will see that the license headers
include that license only. Some of our contributors have contributed
code under our old license, and accordingly, we have retained a copy of
our old license notice in the top-level files in each project and
repository.
llvm-svn: 351636
|
|
Fix a layering violation. Frontend depends on Serialization, so anything used
by both should be in Serialization.
llvm-svn: 348907
|
|
This patch moves the virtual file system form clang to llvm so it can be
used by more projects.
Concretely the patch:
- Moves VirtualFileSystem.{h|cpp} from clang/Basic to llvm/Support.
- Moves the corresponding unit test from clang to llvm.
- Moves the vfs namespace from clang::vfs to llvm::vfs.
- Formats the lines affected by this change, mostly this is the result of
the added llvm namespace.
RFC on the mailing list:
http://lists.llvm.org/pipermail/llvm-dev/2018-October/126657.html
Differential revision: https://reviews.llvm.org/D52783
llvm-svn: 344140
|
|
While investigating why LLDB (which can build hundreds of clang
modules during one debug session) was getting "too many open files"
errors, I found that most of them are .pcm files that are kept open by
ModuleManager. Pretty much all of the open file dscriptors are
FileEntries that are refering to `.pcm` files for which a buffer
already exists in a CompilerInstance's PCMCache.
Before PCMCache was added it was necessary to hold on to open file
descriptors to ensure that all ModuleManagers using the same
FileManager read the a consistent version of a given `.pcm` file on
disk, even when a concurrent clang process overwrites the file halfway
through. The PCMCache makes this practice unnecessary, since it caches
the entire contents of a `.pcm` file, while the FileManager caches all
the stat() information.
This patch adds a call to FileEntry::closeFile() to the path where a
Buffer has already been created. This is necessary because even for a
freshly written `.pcm` file the file is stat()ed once immediately
after writing to generate a FileEntry in the FileManager. Because a
freshly-generated file's contents is stored in the PCMCache, it is
fine to close the file immediately thereafter. The second change this
patch makes is to set the `ShouldClose` flag to true when reading a
`.pcm` file into the PCMCache for the first time.
[For reference, in 1 Clang instance there is
- 1 FileManager and
- n ModuleManagers with
- n PCMCaches.]
rdar://problem/40906753
Differential Revision: https://reviews.llvm.org/D50870
llvm-svn: 340188
|
|
sed -Ei 's/[[:space:]]+$//' include/**/*.{def,h,td} lib/**/*.{cpp,h}
llvm-svn: 338291
|
|
warnings; other minor fixes (NFC).
llvm-svn: 317953
|
|
Extend the -fmodule-file option to support the [<name>=]<file> value format.
If the name is omitted, then the old semantics is preserved (the module file
is loaded whether needed or not). If the name is specified, then the mapping
is treated as just another prebuilt module search mechanism, similar to
-fprebuilt-module-path, and the module file is only loaded if actually used
(e.g., via import). With one exception: this mapping also overrides module
file references embedded in other modules (which can be useful if module files
are moved/renamed as often happens during remote compilation).
This override semantics requires some extra work: we now store the module name
in addition to the file name in the serialized AST representation.
Reviewed By: rsmith
Differential Revision: https://reviews.llvm.org/D35020
llvm-svn: 312220
|
