| Age | Commit message (Collapse) | Author | Files | Lines |
|---|
|
Clients of the dependency scanning service may need to add dependencies
based on the visibility of importing modules, for example, when
determining whether a Swift overlay dependency should be brought in
based on whether there's a corresponding **visible** clang module for
it.
This patch introduces a new field `VisibleModules` that contains all the
visible top-level modules in a given TU.
Because visibility is determined by which headers or (sub)modules were
imported, and not top-level module dependencies, the scanner now
performs a separate DFS starting from what was directly imported for
this computation.
In my local performance testing, there was no observable performance
impact.
resolves: rdar://151416358
---------
Co-authored-by: Jan Svoboda <jan@svoboda.ai>
|
|
(#145221)
|
|
lookupModuleOutput callbacks, NFCI (#131688)
This allows clients to reference more read-only attributes, like IsInStableDirectories.
|
|
Update Dependency scanner so it can scan the dependency of a TU with
a provided buffer rather than relying on the on disk file system to
provide the input file.
|
|
This information is already exposed via `TranslationUnitDeps::ClangModuleDeps` on the `DependencyScanningTool` level, and this patch also adds it on the `DependencyScanningWorker` level via `DependencyConsumer::handleDirectModuleDependency()`.
Besides being redundant, this bit of information is misleading for clients that share single `ModuleDeps` instance between multiple TUs (by using the `AlreadySeen` set). The module can be imported directly in some TUs but transitively in others.
Reviewed By: benlangmuir
Differential Revision: https://reviews.llvm.org/D156563
|
|
Scanner's C++ API accepts a set of modular dependencies the client has already seen and for which it doesn't need the full details. This is currently a set of strings, which somewhat implies that it should contain the set of module names. However, scanner internally expects the values to be in the format "{hash}{name}". Besides not being documented, this is very unintuitive. This patch makes this expectation explicit by changing the type to set of `ModuleID`.
Reviewed By: benlangmuir
Differential Revision: https://reviews.llvm.org/D156492
|
|
The idea is to split the callbacks that are used to consume dependency
information (DependencyConsumer) from callbacks that modify the scan
behaviour itself in any way (DependencyActionController). Currently this
is just lookupModuleOutput, but we have additional callbacks related to
CAS support that we intend to upstream in the future.
Differential Revision: https://reviews.llvm.org/D144058
|
|
clang modules
Add a test to ensure that the clang-scan-deps won't crash due to the
unexpected use of clang modules.
|
|
dependency file within P1689 format (4/4)
Required in https://reviews.llvm.org/D137534.
The build systems needs the information to know that "header X changed,
scanning may have changed, so please rerun scanning". Although it is
possible to get the information by running clang-scan-deps for the
second time with make format, it is not user friendly clearly.
Reviewed By: jansvoboda11
Differential Revision: https://reviews.llvm.org/D139168
|
|
C++20 Named Modules in P1689 format (2/4)
Close https://github.com/llvm/llvm-project/issues/51792
Close https://github.com/llvm/llvm-project/issues/56770
This patch adds ClangScanDeps support for C++20 Named Modules in P1689
format. We can find the P1689 format at:
https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2022/p1689r5.html.
After we land the patch, we're able to compile C++20 Named
Modules with CMake! And although P1689 is written by kitware people,
other build systems should be able to use the format to compile C++20
Named Modules too.
TODO: Support header units in P1689 Format.
TODO2: Support C++20 Modules in the full dependency format of
ClangScanDeps. We also want to support C++20 Modules and clang modules
together according to
https://discourse.llvm.org/t/how-should-we-support-dependency-scanner-for-c-20-modules/66027.
But P1689 format cares about C++20 Modules only for now. So let's focus
on C++ Modules and P1689 format. And look at the full dependency format
later.
I'll add the ReleaseNotes and Documentations after the patch get landed.
Reviewed By: jansvoboda11
Differential Revision: https://reviews.llvm.org/D137527
|
|
C++20 Named Modules in P1689 format (2/4)"
This reverts commit de17c665e3f995c7f5a0e453461ce3a1b8aec196.
See also D137527
|
|
dependency file within P1689 format (4/4)"
This reverts commit e1354763b6e641e45cc2371270883fcd26edf705.
There is a build failure in m68k-linux testing bot
(https://lab.llvm.org/buildbot/#/builders/192/builds/267), which is
weird. Revert this for now and look at the reasons.
|
|
I introduced an unused parameter due to an oversight during rebasing.
Now the patch removes the parameter.
|
|
within P1689 format (4/4)
Required in https://reviews.llvm.org/D137534.
The build systems needs the information to know that "header X changed,
scanning may have changed, so please rerun scanning". Although it is
possible to get the information by running clang-scan-deps for the
second time with make format, it is not user friendly clearly.
Reviewed By: jansvoboda11
Differential Revision: https://reviews.llvm.org/D139168
|
|
Modules in P1689 format (2/4)
Close https://github.com/llvm/llvm-project/issues/51792
Close https://github.com/llvm/llvm-project/issues/56770
This patch adds ClangScanDeps support for C++20 Named Modules in P1689
format. We can find the P1689 format at:
https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2022/p1689r5.html.
After we land the patch, we're able to compile C++20 Named
Modules with CMake! And although P1689 is written by kitware people,
other build systems should be able to use the format to compile C++20
Named Modules too.
TODO: Support header units in P1689 Format.
TODO2: Support C++20 Modules in the full dependency format of
ClangScanDeps. We also want to support C++20 Modules and clang modules
together according to
https://discourse.llvm.org/t/how-should-we-support-dependency-scanner-for-c-20-modules/66027.
But P1689 format cares about C++20 Modules only for now. So let's focus
on C++ Modules and P1689 format. And look at the full dependency format
later.
I'll add the ReleaseNotes and Documentations after the patch get landed.
Reviewed By: jansvoboda11
Differential Revision: https://reviews.llvm.org/D137527
|
|
For users of the C++ API, the return type of `getFullDependencies` doesn't make sense when asking for dependencies of a module. In the returned `FullDependenciesResult` instance, only `DiscoveredModules` is useful (the graph of modular dependecies). The `FullDeps` member is trying to describe a translation unit it was never given. Its command line also refers to a file in the in-memory VFS we create in the scanner, leaking the implementation detail.
This patch splits the API and improves layering and naming of the return types.
Depends on D140175.
Reviewed By: artemcm
Differential Revision: https://reviews.llvm.org/D140176
|
|
This API is no longer necessary, so let's remove it to simplify the internal APIs.
Reviewed By: benlangmuir, artemcm
Differential Revision: https://reviews.llvm.org/D140175
|
|
This patch replaces (llvm::|)Optional< with std::optional<. I'll post
a separate patch to remove #include "llvm/ADT/Optional.h".
This is part of an effort to migrate from llvm::Optional to
std::optional:
https://discourse.llvm.org/t/deprecating-llvm-optional-x-hasvalue-getvalue-getvalueor/63716
|
|
This patch adds #include <optional> to those files containing
llvm::Optional<...> or Optional<...>.
I'll post a separate patch to actually replace llvm::Optional with
std::optional.
This is part of an effort to migrate from llvm::Optional to
std::optional:
https://discourse.llvm.org/t/deprecating-llvm-optional-x-hasvalue-getvalue-getvalueor/63716
|
|
commands"
Attempt to fix the test failures observed in CI:
* Add Option dependency, which caused BUILD_SHARED_LIBS builds to fail
* Adapt tests that accidentally depended on the host platform: platforms
that don't use an integrated assembler (e.g. AIX) get a different set
of commands from the driver. Most dependency scanner tests can use
-fsyntax-only or -E instead of -c to avoid this, and in the rare case
we want to check -c specifically, set an explicit target so the
behaviour is independent of the host.
Original commit message follows.
---
Instead of trying to "fix" the original driver invocation by appending
arguments to it, split it into multiple commands, and for each -cc1
command use a CompilerInvocation to give precise control over the
invocation.
This change should make it easier to (in the future) canonicalize the
command-line (e.g. to improve hits in something like ccache), apply
optimizations, or start supporting multi-arch builds, which would
require different modules for each arch.
In the long run it may make sense to treat the TU commands as a
dependency graph, each with their own dependencies on modules or earlier
TU commands, but for now they are simply a list that is executed in
order, and the dependencies are simply duplicated. Since we currently
only support single-arch builds, there is no parallelism available in
the execution.
Differential Revision: https://reviews.llvm.org/D132405
|
|
commands"
Failing on some bots, reverting until I can fix it.
This reverts commit f80a0ea760728e70f70debf744277bc3aa59bc17.
|
|
Instead of trying to "fix" the original driver invocation by appending
arguments to it, split it into multiple commands, and for each -cc1
command use a CompilerInvocation to give precise control over the
invocation.
This change should make it easier to (in the future) canonicalize the
command-line (e.g. to improve hits in something like ccache), apply
optimizations, or start supporting multi-arch builds, which would
require different modules for each arch.
In the long run it may make sense to treat the TU commands as a
dependency graph, each with their own dependencies on modules or earlier
TU commands, but for now they are simply a list that is executed in
order, and the dependencies are simply duplicated. Since we currently
only support single-arch builds, there is no parallelism available in
the execution.
Differential Revision: https://reviews.llvm.org/D132405
|
|
This patch introduces new option `-eager-load-pcm` to `clang-scan-deps`, which controls whether the resulting command-lines will load PCM files eagerly (at the start of compilation) or lazily (when handling import directive). This patch also switches the default from eager to lazy.
To reduce the potential for churn in LIT tests in the future, this patch also removes redundant checks of command-line arguments and introduces new test `modules-dep-args.c` as a substitute.
Reviewed By: benlangmuir
Differential Revision: https://reviews.llvm.org/D132066
|
|
Instead of delaying the generation of command-lines to after all
dependencies are reported, compute them immediately. This is partly in
preparation for splitting the TU driver command into its constituent cc1
and other jobs, but it also just simplifies working with the compiler
invocation for modules if they are not "without paths".
Also change the computation of the default output path in
clang-scan-deps to scrape the implicit module cache from the
command-line rather than get it from the dependency, since that is now
unavailable at the time we make the callback.
Differential Revision: https://reviews.llvm.org/D131934
|
|
|
|
Since D129389 (and downstream PR https://github.com/apple/llvm-project/pull/4965), the dependency scanner is responsible for generating full command-lines, including the modules paths. This patch removes the flag that was making this an opt-in behavior in clang-scan-deps.
Reviewed By: benlangmuir
Differential Revision: https://reviews.llvm.org/D131420
|
|
`DependencyScanningTool`
Also include a unit test to validate that the `vfs::FileSystem` object is properly used.
Differential Revision: https://reviews.llvm.org/D129912
|
|
When building modules, override secondary outputs (dependency file,
dependency targets, serialized diagnostic file) in addition to the pcm
file path. This avoids inheriting per-TU command-line options that
cause non-determinism in the results (non-deterministic command-line for
the module build, non-determinism in which TU's .diag and .d files will
contain the module outputs). In clang-scan-deps we infer whether to
generate dependency or serialized diagnostic files based on an original
command-line. In a real build system this should be modeled explicitly.
Differential Revision: https://reviews.llvm.org/D129389
|
|
`.cpp` file contents in namespaces, NFC
This makes the file consistent with the coding style of the rest of LLVM.
|
|
Disable or canonicalize compiler options that are not relevant in
explicit module builds, similar to what we already did for the modules
cache path. This reduces uninteresting differences between
command-lines, which is particularly useful if there is a tool that can
cache the compilations.
Differential Revision: https://reviews.llvm.org/D127883
|
|
This fixes the underlying module dependencies, which had a
non-deterministic order, which was also visible in the order of calls to
DependencyConsumer methods. This was not directly observable in
the clang-scan-deps utility, because it was previously seeing a sorted
order from std::map in DependencyScanningTool. However, the underlying
API previously created a likely issue for any other clients. Note: if
you only apply the change from DependencyScanningTool, you can see the
issue in clang-scan-deps, and existing tests will fail
non-deterministicaly.
Differential Revision: https://reviews.llvm.org/D127243
|
|
|
|
With explicit modules build, the '-fmodules-cache-path=' argument is unused.
This patch removes the argument to avoid warnings or errors (with '-Werror') stemming from that.
Depends on D118915.
Reviewed By: dexonsmith
Differential Revision: https://reviews.llvm.org/D120474
|
|
The `clang-scan-deps` tool currently generates `-fmodule-file=` command-line arguments for the whole transitive closure of modular dependencies. This is not necessary, we only need to provide the direct dependencies on the command line. Information about transitive dependencies is stored within the `.pcm` files of direct dependencies. This makes the command lines shorter, but should be a NFC otherwise (unless there are bugs in the loading mechanism for explicit modules).
Depends on D120465.
Reviewed By: Bigcheese
Differential Revision: https://reviews.llvm.org/D118915
|
|
Since D113473, we don't report any module map files via `-fmodule-map-file=` in explicit builds. The ultimate goal here is to make sure Clang doesn't open/read/parse/evaluate unnecessary module maps.
However, implicit module maps still end up reading all reachable module maps. This patch disables implicit module maps in explicit builds.
Unfortunately, we still need to report some module map files that aren't encoded in PCM files of dependencies: module maps that are necessary to correctly evaluate includes in modules marked as `[no_undeclared_includes]`.
Depends on D120464.
Reviewed By: dexonsmith
Differential Revision: https://reviews.llvm.org/D120465
|
|
The dependency scanner already generates canonical -cc1 command lines that can be used to compile discovered modular dependencies.
For translation unit command lines, the scanner only generates additional driver arguments the build system is expected to append to the original command line.
While this works most of the time, there are situations where that's not the case. For example with `-Wunused-command-line-argument`, Clang will complain about the `-fmodules-cache-path=` argument that's not being used in explicit modular builds. Combine that with `-Werror` and the build outright fails.
To prevent such failures, this patch changes the dependency scanner to return the full driver command line to compile the original translation unit. This gives us more opportunities to massage the arguments into something reasonable.
Reviewed By: Bigcheese
Differential Revision: https://reviews.llvm.org/D118986
|
|
In D113473, the dependency scanner stopped emitting "-fmodule-map-file=" arguments. Potential build systems are expected to not add any such arguments on their own. This commit removes mentions of such arguments to avoid confusion.
|
|
During explicit modules build, when all modules are provided via `-fmodule-file=<path>` and implicit modules and implicit module maps are disabled (`-fno-implicit-modules`, `-fno-implicit-module-maps`), we don't need to load the original module map files at all. This patch stops emitting the `-fmodule-map-file=` arguments we don't need, saving some compilation time due to avoiding parsing such module maps and making the command line shorter.
Reviewed By: bnbarham
Differential Revision: https://reviews.llvm.org/D113473
|
|
During explicit modular build, PCM files are typically specified via the `-fmodule-file=<path>` command-line option. Early during the compilation, Clang uses the `ASTReader` to read their contents and caches the result so that the module isn't loaded implicitly later on. A listener is attached to the `ASTReader` to collect names of the modules read from the PCM files. However, if the PCM has already been loaded previously via PCH:
1. the `ASTReader` doesn't do anything for the second time,
2. the listener is not invoked at all,
3. the module load result is not cached,
4. the compilation fails when attempting to load the module implicitly later on.
This patch solves this problem by attaching the listener to the `ASTReader` for PCH reading as well.
Reviewed By: dexonsmith
Differential Revision: https://reviews.llvm.org/D111560
|
|
This patch simplifies the dependency scanner API. Depends on D108980.
Reviewed By: dexonsmith
Differential Revision: https://reviews.llvm.org/D108981
|
|
This patch simplifies the dependency scanner API. Depends on D108979.
Reviewed By: dexonsmith
Differential Revision: https://reviews.llvm.org/D108980
|
|
module lookup by name alone
This removes the need to create a fake source file that imports a
module.
rdar://64538073
Differential Revision: https://reviews.llvm.org/D109485
|
|
Translation units with multiple direct modular dependencies trigger a non-deterministic ordering in `clang-scan-deps`. This boils down to usage of `std::unordered_map`, which gets replaced by `std::map` in this patch.
Depends on D103526.
Reviewed By: dexonsmith
Differential Revision: https://reviews.llvm.org/D103807
|
|
There's no need to pass `DependencyOutputOptions` to each call of `handleFileDependency`, since the options don't ever change.
This patch adds new `handleDependencyOutputOpts` method to the `DependencyConsumer` interface and the dependency scanner uses it to report the options only once.
Reviewed By: dexonsmith
Differential Revision: https://reviews.llvm.org/D104104
|
|
When a translation unit uses a PCH and imports the same modules as the PCH, we'd prefer to resolve to those modules instead of inventing new modules and reporting them as modular dependencies. Since the PCH modules have already been built nudge the compiler to reuse them when deciding whether to build a new module and don't report them as regular modular dependencies.
Depends on D103524 & D103802.
Reviewed By: dexonsmith
Differential Revision: https://reviews.llvm.org/D103526
|
|
The context hash of modular dependencies can be different from the context hash of the original translation unit if we modify their `CompilerInvocation`s.
Stop assuming the TU's context hash everywhere.
No functionality change here, since we're still currently using the unmodified TU CompilerInvocation to compute the context hash.
Reviewed By: dexonsmith
Differential Revision: https://reviews.llvm.org/D102473
|
|
Add option to `clang-scan-deps` to enable/disable generation of command-line arguments with absolute paths. This is essentially a revert of D100533, but with improved naming and added test.
Reviewed By: dexonsmith
Differential Revision: https://reviews.llvm.org/D101051
|
|
This patch removes the `FullDependencies::AdditionalNonPathCommandLine` member, as it's value-initialized and never mutated.
Reviewed By: dexonsmith
Differential Revision: https://reviews.llvm.org/D100536
|
|
This patch simplifies (and renames) the `appendCommonModuleArguments` function.
It no longer tries to construct the command line for explicitly building modules. Instead, it only performs the DFS traversal of modular dependencies and queries the callbacks to collect paths to `.pcm` and `.modulemap` files.
This makes it more flexible and usable in two contexts:
* Generating additional command line arguments for the main TU in modular build. The `std::vector<std::string>` output parameters can be used to manually generate appropriate command line flags.
* Generate full command line for a module. The output parameters can be the corresponding parts of `CompilerInvocation`. (In a follow-up patch.)
Reviewed By: dexonsmith
Differential Revision: https://reviews.llvm.org/D100531
|
|
This patch extracts the `ModuleName` and `ContextHash` members of `ClangModuleDep`, `FullDependencies` and `ModuleDeps` into a single struct `ModuleID`. This makes it easier to understand how the full dependency graph works.
Reviewed By: Bigcheese, dexonsmith
Differential Revision: https://reviews.llvm.org/D98943
|
