| Age | Commit message (Collapse) | Author | Files | Lines |
|---|
|
This pull request makes a minor fix to the
`llvm/utils/lit/lit/llvm/config.py` file.
The change corrects a missing comma in the list of environment
variables, ensuring that `"UBSAN_SYMBOLIZER_PATH"` and `"ASAN_OPTIONS"`
are treated as separate entries instead of a single concatenated string.
* Fixed a missing comma between `"UBSAN_SYMBOLIZER_PATH"` and
`"ASAN_OPTIONS"` in the environment variable list in
`llvm/utils/lit/lit/llvm/config.py`, preventing potential configuration
issues.
|
|
This patch removes support for %T from llvm-lit. For now we mark the
test unresolved and add an error message noting the substitution is
deprecated. This is exactly the same as the error handling for other
substitution failures. We intend to remove support for the nice error
message once 22 branches as users should have moved over by the they are
upgrading to v23.
Reviewers: petrhosek, jh7370, ilovepi, pogo59, cmtice
Reviewed By: cmtice, jh7370, ilovepi
Pull Request: https://github.com/llvm/llvm-project/pull/160028
|
|
Before this change, rm would assume that a symlink to a directory was
actually a directory and require the recursive flag to be passed,
differing from other shells. Given the change in lit is about the same
length as the test change would be (minus tests), I think it makes sense
to just support this in the internal shell.
Reviewers: cmtice, petrhosek, ilovepi
Reviewed By: petrhosek, cmtice, ilovepi
Pull Request: https://github.com/llvm/llvm-project/pull/158464
|
|
cat with no files passed to it is supposed to read from STDIN according
to POSIX. The builtin cat lacking this behavior led to the clang test in
dev-fd-fs.c to fail because it expected this behavior. This is a simple
modification and I do not think it is possible to rewrite the test
without this easily while preserving the semantics around named pipes.
Reviewers: petrhosek, arichardson, ilovepi, cmtice, jh7370
Reviewed By: jh7370, arichardson, ilovepi, cmtice
Pull Request: https://github.com/llvm/llvm-project/pull/158447
|
|
This patch adds support for the new lit %{readfile:<filename>}
substitution to the external shell. The implementation currently just
appends some test commands to ensure the file exists and uses a subshell
with cat. This is intended to enable running tests using the
substitution in the external shell before we fully switch over to the
internal shell.
This code is designed to be temporary with us deleting it once
everything has migrated over to the internal shell and we are able to
remove the external shell code paths.
Reviewers: petrhosek, cmtice, pogo59, ilovepi, arichardson
Reviewed By: cmtice
Pull Request: https://github.com/llvm/llvm-project/pull/159431
|
|
This patch adds a new %{readfile:<file name>} substitution to lit. This
is needed for porting a couple of tests to lit's internal shell. These
tests are all using subshells to pass some option to a command are not
feasible to run within the internal shell without this functionality.
Reviewers: petrhosek, jh7370, ilovepi, cmtice
Reviewed By: jh7370, cmtice
Pull Request: https://github.com/llvm/llvm-project/pull/158441
|
|
This reverts commit 330068a74bfb6333f9016e3c4053eeaf4989d601.
This was causing some test failures on MacOS that are now fixed in the reland.
These failures were related to calling ulimit -v despite XNU not having support
for that option. This patch simply disables the test on non-Linux platforms for
now until we can have a Linux specific test for ulimit -v.
|
|
This reverts commit 615d07ea55ea57afab0205aa739239070448a038.
This was causing some MacOS buildbolt failures.
|
|
This patch implements ulimit inside the lit internal shell.
Implementation wise, this functions similar to umask. But instead of
setting the limits within the lit test worker process, we set
environment variables and add a wrapper around the command to be
executed. The wrapper then sets the limits. This is because we cannot
increase the limits after lowering them, so we would otherwise end up
with a lit test worker stuck with a lower limit.
There are several tests where the use of ulimit is essential to the
semantics of the test (two in clang, ~7 in compiler-rt), so we need to
implement this in order to switch on the internal shell by default
without losing test coverage.
Reviewers: cmtice, petrhosek, ilovepi
Reviewed By: cmtice, ilovepi
Pull Request: https://github.com/llvm/llvm-project/pull/157958
|
|
|
|
There are some code paths within lit that still check what string types
are supported with the aim of being compatible with Python 2 and 3.
Given LLVM's minimum Python version is 3.8 and we do not have any
upstream testing for Python 2, I think we can safely drop this.
|
|
Most of these tests do not actually have a shell requirement. The shell
requirement ended up in the test either from cargo culting (from what I
can tell) or because the test authors actually meant to mark Windows as
unsupported. This prevents enablement of lit's internal shell within
clang.
Towards #102699.
Reviewers: rnk, efriedma-quic, Sirraide, petrhosek, ilovepi
Reviewed By: ilovepi
Pull Request: https://github.com/llvm/llvm-project/pull/156905
|
|
Previously we compared paths by string manipulation, however Windows
paths can use both '\' and '/' as path separators, which made this
fragile. This uses the pathlib.Path.samefile API instead.
|
|
|
|
StringIO was moved to io.StringIO in Python 3. These try/catch
statements were intended to provide cross compatibility between the two.
Now that LLVM's minimum required Python version is 3.8 and Python 2 has
been deprecated for a while, we can probably drop support. Especially
since there is no upstream testing for this configuration.
|
|
Lit's builtin diff command had some Python 2.7 code paths lying around
that we can probably get rid of at this point. LLVM at this point
requires Python 3.8 at minimum. Keeping lit working at a lower version
is a reasonable goal, but I think we can probably drop python 2 support
at this point given how long it has been deprecated and how long LLVM
has supported Python 3.
Reviewers: jdenny-ornl, ilovepi, petrhosek
Reviewed By: ilovepi
Pull Request: https://github.com/llvm/llvm-project/pull/157558
|
|
This adds an updater to lit's --update-tests flag with support for
`diff`. If a RUN line containing the `diff` command fails, this function
will use heuristics to try to deduce which file is the "reference" file,
and copy the contents of the other file to the reference. If it cannot
deduce which file is the reference file, it does nothing.
The heuristics are currently:
- does one of the files end in .expected while the other does not? Then
the .expected file is the reference.
- does one of the file paths contain the substring ".tmp" while the
other does not? Then the file not containing ".tmp" is the reference.
This matches cases where one file path is constructed using the `%t`
substitution.
|
|
env -i is needed for some lit tests. The feature requires a minimal
amount of work to support and there is no easy way to rewrite the tests
that require it.
At least two tests that need this:
1. clang/test/Driver/env.c
2. lldb/test/Shell/Host/TestCustomShell.test
|
|
double-quoted strings. (#156742)
PR 156125 removed the escape (backslash) in front of '$' for all quoted
strings. It has since been pointed out this should only happen for
double-quoted strings. This PR fixes that.
|
|
LLDB allows creation of 'persistent' variables, with names that start
with '$'. The lit internal shell was escaping the '$', making it '\\$',
in some CHECK lines, which causes an LLDB test,
TestExprWithSideEffectOnConvenienceVar, to fail when using the lit
internal shell.
Further explanation of the failing LLDB test: LLDB convenience variables
start with '$'. The test passes several quoted commands that use and
update convenience variables to lldb as arguments to be run in batch
mode. The tool that packages up the complete string and passes it to the
lit internal shell lexer for lexing inserts a backslash in front of the
'$' before passing the string in for lexing. The lexer was passing this
change along, causing the tests to fail.
This PR fixes the issue by having the lexer remove the newly added
escape on the '$'.
|
|
This reverts commit faa4e35c622c13c7a565b979a6676d6cf3040cd4.
This was originally reverted because it was using a Python 3.9 feature
(umask in subprocess.Popen) when LLVM only requires Python 3.8. This
patch uses os.umask instead, which has been around for longer.
|
|
The early exit we relied on to only invoke test updaters for failing
tests requires that there was no output to stdout or stderr, and that
timeouts weren't enabled. When these conditions weren't fulfilled, test
updaters would be invoked even on passing or XFAILed tests.
|
|
(#154317)
This patch adds a -r|--relative-paths option to llvm-lit, which when
enabled will print test case names using paths relative to the current
working directory. The legacy default without that option is that test
cases are identified using a path relative to the test suite.
Only the summary report is impacted. That normally include failed tests,
unless unless options such as --show-pass.
Background to this patch was the discussion here
https://discourse.llvm.org/t/test-case-paths-in-llvm-lit-output-are-lacking-the-location-of-the-test-dir-itself/87973
with a goal to making it easier to copy-n-paste the path to the failing
test cases.
Examples showing difference in "Passed Tests" and "Failed Tests":
> llvm-lit --show-pass test/Transforms/Foo
PASS: LLVM :: Transforms/Foo/test1.txt (1 of 2)
FAIL: LLVM :: Transforms/Foo/test2.txt (2 of 2)
Passed Tests (1):
LLVM :: Transforms/Foo/test1.txt
Failed Tests (1):
LLVM :: Transforms/Foo/test2.txt
> llvm-lit --show-pass --relative-paths test/Transforms/Foo
PASS: LLVM :: Transforms/Foo/test1.txt (1 of 2)
FAIL: LLVM :: Transforms/Foo/test2.txt (2 of 2)
Passed Tests (1):
test/Transforms/Foo/test1.txt
Failed Tests (1):
test/Transforms/Foo/test2.txt
|
|
This reverts commit
https://github.com/llvm/llvm-project/commit/e495231238b86ae2a3c7bb5f94634c19ca2af19a
to reland
the --update-tests feature, originally landed in
https://github.com/llvm/llvm-project/pull/108425.
|
|
There are a few tests in clang that gate on system-windows but also
don't work on Cygwin, so add a system-cygwin feature for them to use.
|
|
See the related issue. We want to set up a build bot where `opt` runs with `-enable-profcheck`, which inserts `MD_prof` before running the rest of the pipeline requested from `opt`, and then validates resulting profile information (more info in the RFC linked by the aforementioned issue)
In that setup, we will also ignore `FileCheck`: while the profile info inserted is, currently, equivalent to the profile info a pass would observe via `BranchProbabilityInfo`/`BlockFrequencyInfo`, (1) we may want to change that, and (2) some tests are quite sensitive to the output IR, and break if, for instance, extra metadata is present (which it would be due to `-enable-profcheck`). Since we're just interested in profile consistency on the upcoming bot, ignoring `FileCheck` is simpler and sufficient. However, this has the effect of passing XFAIL tests. Rather than listing them all, the alternative is to just exclude XFAIL tests.
This PR adds support for that by introducing a `--exclude-xfail` option to `llvm-lit`.
Issue #147390
|
|
Flang and Flang-RT have two flavours of unittests:
1. GTest unittests, using lit's `lit.formats.GoogleTest` format ending
with `Tests${CMAKE_EXECUTABLE_SUFFIX}`
2. "non-GTest" or "OldUnit" unittests, a plain executable ending with
`.test${CMAKE_EXECUTABLE_SUFFIX}`
Both executables are emitted into the same unittests/ subdirectory. When
running ...
1. `tests/Unit/lit.cfg.py`, only considers executable ending with
`Tests` (or `Tests.exe` on Windows), hence skips the non-GTest tests.
2. `tests/NonGtestUnit/lit.cfg.py` considers all tests ending with
`.test` or `.exe`. On Windows, The GTest unitests also end with `.exe`.
In Flang-RT, `.exe` is considered an extension for non-GTest unitests
which causes tests such as Flang's `RuntimeTests.exe` to be executed for
both on Windows. This particular test includes a file write test, using
a hard-coded filename `ucsfile`. If the two instances are executed
concurrently, they might interfere with each other reading/writing
`ucsfile` which results in a flaky test.
This patch avoids the redundant execution by requiring the suffix
`.test.exe` on Windows. lit has to be modified because it uses
`os.path.splitext` the extract the extension, which would only recognize
the last component. It was changed from the orginal `endswith` in
c865abe747aa72192f02ebfdcabe730f2553e42f
for unknown reasons.
In Flang, `.exe` is not considered a suffix for non-GTest unittests and
hence they are not run at all. Fixing by also added `.test.exe` as valid
suffix, like with Flang-RT.
Unfortunately, the ` Evaluate/real.test.exe` test was failing on
Windows:
```
FAIL: flang-OldUnit :: Evaluate/real.test.exe (3592 of 3592)
******************** TEST 'flang-OldUnit :: Evaluate/real.test.exe' FAILED ********************
..\_src\flang\unittests\Evaluate\real.cpp:511: FAIL: FlagsToBits(prod.flags) == 0x18, not 0x10
0 0x800001 * 0xbf7ffffe
..\_src\flang\unittests\Evaluate\real.cpp:511: FAIL: FlagsToBits(prod.flags) == 0x18, not 0x10
0 0x800001 * 0x3f7ffffe
..\_src\flang\unittests\Evaluate\real.cpp:511: FAIL: FlagsToBits(prod.flags) == 0x18, not 0x10
0 0x80800001 * 0xbf7ffffe
..\_src\flang\unittests\Evaluate\real.cpp:511: FAIL: FlagsToBits(prod.flags) == 0x18, not 0x10
0 0x80800001 * 0x3f7ffffe
...
```
This is due to the `__x86_64__` macro not being set by Microsoft's
cl.exe and hence floating point status flags not being read out. The
equivalent macro for Microsofts compiler is `_M_X64` (or `_M_X64`).
|
|
|
|
This patch refactors use_clang into two functions. This is intended for
use within the clang-tools-extra test suite to avoid a race condition
where the clang binary exists but is not yet ready for execution which
results in a lit configuration failure.
Details are in #145703.
Reviewers: Keenuts, pogo59, AaronBallman, DavidSpickett
Reviewed By: pogo59
Pull Request: https://github.com/llvm/llvm-project/pull/147436
|
|
(#145731)
Before this commit the LIT test framework of the static analyzer had a
file called `analyzer_test.py` which implemented a tricky system for
selecting the constraint manager:
- (A) Test files without `REQUIRES: z3` were executed with the default
range-based constraint manager.
- (B) If clang was built with Z3 support _and_ `USE_Z3_SOLVER=1` was
passed to the test run, the test was executed with Z3 as the
constraint manager.
(There was support for executing the same RUN line twice if both
conditions were satisfied.)
Unfortunately, using Z3 as the constraint manager does not work in
practice (very slow and causes many crashes), so the (B) pathway became
unused (or was never truly used?) and became broken due to bit rot. (In
the CI bots the analyzer is built without Z3 support, so only the
pathway (A) is used.)
This commit removes `analyzer_test.py` (+ related logic in other build
files + the test `z3/enabled.c` which just tested that
`analyzer_test.py` is active), because it tries to implement a feature
that we don't need (only one constraint manager is functional) and its
code is so complicated and buggy that it isn't useful as a starting
point for future development.
The fact that this logic was broken implied that tests with `REQUIRES:
z3` were not executed during normal testing, so they were also affected
by bit rot. Unfortunately this also affected the tests of the
`z3-crosscheck` mode (aka Z3 refutation) which also depends on Z3 but
uses Z3 in a different way which is actually stable and functional.
In this commit I'm fixing most of the `REQUIRES: z3` tests that were
broken by straightforward issues. Two test files, `PR37855.c` and
`z3-crosscheck.c` were affected by more complex issues, so I marked them
as `XFAIL` for now. We're planning to fix them with follow-up commits in
the foreseeable future.
For additional background information see also the discourse thread
https://discourse.llvm.org/t/taking-ownership-of-clang-test-analysis/84689
|
|
Reverts llvm/llvm-project#143930 as it causes build failures:
https://github.com/llvm/llvm-project/pull/143930#issuecomment-2969115461
|
|
Remove imports that are not used in some lit test files.
|
|
If a test took more than one attempt to complete, show the number of attempts and the maximum allowed attempts as `2 of 4 attempts` inside the `<progress info>` (see [TEST RUN OUTPUT FORMAT](https://llvm.org/docs/CommandGuide/lit.html#test-run-output-format)).
NOTE: Additionally this is a fixup for #141851 where the tests were not quite right. `max-retries-per-test/allow-retries-test_retry_attempts/test.py` was added but never used there. Now we're calling it. To correlate better between the test output and the test script I've used higher numbers of max allowed retries.
|
|
When packaging LLVM we've seen arbitrary tests fail.
It happened sporadically and most of the times the test
do work if they are run a second time on the next day.
The tests themselves were always different and we didn't
know ahead of time which ones we wanted to re-run.
That's we filter-out a lot of `libomp` and `libarcher` tests [1].
This change allows us to set
`LIT_OPTS="--max-retries-per-test=12"`
when running any "check-XXX" build target. Then any lit test
will at most be re-run 12 times, unless there's an `ALLOW_RETRIES:`
in one of the test scripts that's specifying a different value
than `12`. `12` is just an example here, any positive integer
will work.
Please note, that this only adds the possibility to re-run
lit tests. It does not actually do it until the caller specifies
`--max-retries-per-test=<POSITIVE_INT>` either on a call to `lit` or
in `LIT_OPTS`.
Also note, that one can still use `ALLOW_RETRIES:` in test scripts
and it will always rule over `--max-retries-per-test`. When
`--max-retries-per-test` is set too low, but the
`config.test_retry_attempts`
is high enough, it works as well.
Any option in the list below overrules its predecessor:
* `--max-retries-per-test`
* `config.test_retry_attempts`
* `ALLOW_RETRIES` keyword
From the above options to re-run tests, `--max-retries-per-test` is the
only one that doesn't require a change in the test scripts or the test
config.
[1]:
https://src.fedoraproject.org/rpms/llvm/blob/rawhide/f/llvm.spec#_2326
Downstream PR to make use of the `--max-retries-per-test` option:
https://src.fedoraproject.org/rpms/llvm/pull-request/434
Downstream ticket: https://issues.redhat.com/browse/LLVM-145
|
|
Incldue `HOME` among the variables that are passed through to tests when
environment is cleaned. This is necessary for Gentoo build environments,
where individual package builds are given temporary home directories
that are exposed via `HOME` variable. By stripping the variable, `lit`
made these tests attempt to access user's home directory, resulting in
permission errors.
|
|
Fixes #133914
When using the internal shell with a timeout set lit will hang on the
following call if an exception is thrown and not immediately caught
https://github.com/llvm/llvm-project/blob/19970535f92c0f2dcda01b7fc60f95945166e424/llvm/utils/lit/lit/run.py#L93
This can occur when using the internal lit shell and trying to run a
program that does not exist. In this case `_executeShCmd` will throw an
internal shell error, which will not be caught by the function directly
calling it, `executeShCmd`, rather it is caught one function higher in
the call stack in `executeScriptInternal`. Because that exception is
percolated up the call stack instead of being immediately caught lit
will hang until the test timeout expires. This patch changes the
location where we catch this exception to `executeShCmd` instead to
avoid this.
For more background on what causes this hang see:
https://stackoverflow.com/questions/15314189/python-multiprocessing-pool-hangs-at-join
|
|
|
|
This reverts commit 57e89c97c2c1b4e41f07a90c2f4d36649696e619.
Updated lit tests.
|
|
This reverts commit 8d3dc1ed5656a3e69e4195d58684a7f4bf0ff5cc.
Test needs to be updated.
|
|
When a developer copy/pastes a failing command line into their
shell to rerun it, they have to manually delete the "RUN: at line
N:" prefix. To make life easier for such developers, let's make it
possible to copy/paste a command without needing to modify it while
still showing the line number in the output by moving the line number
to a comment at the end of the command line.
Reviewers: jroelofs, MaskRay
Reviewed By: jroelofs, MaskRay
Pull Request: https://github.com/llvm/llvm-project/pull/132485
|
|
generic target (#122629)
The current parsing logic for the target string assumes it follows the
format `<kind>-<triple>-<target id>:<feature>`, such as
`hipv4-amdgcn-amd-amdhsa-gfx1030:+xnack`.
Specifically, it assumes that `<target id>` does not contain any `-`,
relying on `rsplit` for parsing.
However, this assumption breaks for AMDGPU's generic targets, which may
contain one or more `-`, such as `gfx10-3-generic` or `gfx12-generic`.
As a result, the existing approach using `rstrip` is no longer reliable.
This patch reworks the parsing logic to handle target strings more
robustly, including support for generic targets.
The bundler now strictly requires a 4-field target triple.
Additionally, a new Python helper function has been added to `config.py`
to normalize the target triple into the 4-field format when it is not,
ensuring tests pass reliably.
|
|
This patch fixes an issue introduced with commit:
https://github.com/llvm/llvm-project/commit/8507dbaec3f644b8a0c6291f097800d82a4f4b16
|
|
Fixes: https://github.com/llvm/llvm-project/issues/123903
Reviewed By: DavidSpickett, SixWeining
Pull Request: https://github.com/llvm/llvm-project/pull/124059
|
|
(#124475)
Extending the conditionals in `AugmentRegisterInfo` to support
alternative names for lldb.
Fixes #124023
There is an exception with register `X8` which is not covered here but
more details can be found in the issue
https://github.com/llvm/llvm-project/issues/127900.
|
|
Also clear the release notes.
|
|
- Add option (--report-failures-only) to generate a reduced report for
lit tests that only includes failing tests
- This is a continuation of proposed patches by @gregbedwell here:
- https://reviews.llvm.org/D143516
- https://reviews.llvm.org/D143519
---------
Co-authored-by: Greg Bedwell <greg.bedwell@sony.com>
Co-authored-by: James Henderson <James.Henderson@sony.com>
|
|
I was too hasty landing an option whose only known use at this time is
LLVM's own CI.
We may be able to remove it before the next branch that would be the
next llvm-lit release outside of llvm, but the timing may not work out.
So I am hiding the option in case that were to happen.
|
|
This is to address the latest lit regressions
https://lab.llvm.org/buildbot/#/builders/64/builds/1285 caused by using
the internal lit shell. This change will limit using the internal lit
shell to TableGen on AIX so we do not hit these regressions.
|
|
Diff on AIX doesn't have all the required features used in tests (see
<https://github.com/llvm/llvm-project/pull/108871> and
<https://github.com/llvm/llvm-project/pull/112997#issuecomment-2429656192>),
so always use the internal shell.
|
|
Fixes 22e21bc1e796406c89e4a24fd81a1623ab2d7d85.
|
