| Age | Commit message (Collapse) | Author | Files | Lines |
|---|
|
This patch removes support for %T from llvm-lit. For now we mark the
test unresolved and add an error message noting the substitution is
deprecated. This is exactly the same as the error handling for other
substitution failures. We intend to remove support for the nice error
message once 22 branches as users should have moved over by the they are
upgrading to v23.
Reviewers: petrhosek, jh7370, ilovepi, pogo59, cmtice
Reviewed By: cmtice, jh7370, ilovepi
Pull Request: https://github.com/llvm/llvm-project/pull/160028
|
|
This patch removes %T from tests in preparation for removing the
substitution completely. Splitting this off from the actual removal to
make it easier to revert the removal of %T in case I missed something.
Reviewers: RKSimon, jh7370, cmtice, petrhosek, ilovepi, pogo59
Reviewed By: cmtice, RKSimon, jh7370
Pull Request: https://github.com/llvm/llvm-project/pull/160027
|
|
This patch allows for two retry attempts for the readfile tests. This is
intended as a stop-gap until I have time to do proper investigation into
why exactly the tests are failing.
|
|
Python 3.13 considers "/file/does/not/exist" to not be an absolute path
on Windows, so the test runner does os.path.join(cwd, filePath), which
can end up with an output path such as "D:/file/does/not/exist".
Accept a potential prefix before the missing path here.
This fixes running the lit tests on Windows with Python 3.13.
|
|
This PR fixes a bunch of failures on AIX that occurred due to the switch
to lit internal shell by default. The failures deal with the following:
1. unset not being supported by lit internal shell
2. A bug with echo -n on AIX when there are multiple pipes in the RUN
command
3. ulimit test case not supported on AIX due to the -v option
4. platform specific error message for missing file
|
|
Before this change, rm would assume that a symlink to a directory was
actually a directory and require the recursive flag to be passed,
differing from other shells. Given the change in lit is about the same
length as the test change would be (minus tests), I think it makes sense
to just support this in the internal shell.
Reviewers: cmtice, petrhosek, ilovepi
Reviewed By: petrhosek, cmtice, ilovepi
Pull Request: https://github.com/llvm/llvm-project/pull/158464
|
|
cat with no files passed to it is supposed to read from STDIN according
to POSIX. The builtin cat lacking this behavior led to the clang test in
dev-fd-fs.c to fail because it expected this behavior. This is a simple
modification and I do not think it is possible to rewrite the test
without this easily while preserving the semantics around named pipes.
Reviewers: petrhosek, arichardson, ilovepi, cmtice, jh7370
Reviewed By: jh7370, arichardson, ilovepi, cmtice
Pull Request: https://github.com/llvm/llvm-project/pull/158447
|
|
This patch adds support for the new lit %{readfile:<filename>}
substitution to the external shell. The implementation currently just
appends some test commands to ensure the file exists and uses a subshell
with cat. This is intended to enable running tests using the
substitution in the external shell before we fully switch over to the
internal shell.
This code is designed to be temporary with us deleting it once
everything has migrated over to the internal shell and we are able to
remove the external shell code paths.
Reviewers: petrhosek, cmtice, pogo59, ilovepi, arichardson
Reviewed By: cmtice
Pull Request: https://github.com/llvm/llvm-project/pull/159431
|
|
This patch adds a new %{readfile:<file name>} substitution to lit. This
is needed for porting a couple of tests to lit's internal shell. These
tests are all using subshells to pass some option to a command are not
feasible to run within the internal shell without this functionality.
Reviewers: petrhosek, jh7370, ilovepi, cmtice
Reviewed By: jh7370, cmtice
Pull Request: https://github.com/llvm/llvm-project/pull/158441
|
|
There were some build failures because apparently running ulimit -n 50
actually means ulimit -n 48 there. Losing the test coverage on Solaris
shouldn't be a big deal.
|
|
Some of the ulimit limits do not work on some POSIX platforms. THe
motivating example here is ulimit -v on MacOS as the relevant system
calls are not implemented in XNU. Splitting the tests lets us keep test
coverage on POSIX (non-Linux) platforms.
|
|
This reverts commit 330068a74bfb6333f9016e3c4053eeaf4989d601.
This was causing some test failures on MacOS that are now fixed in the reland.
These failures were related to calling ulimit -v despite XNU not having support
for that option. This patch simply disables the test on non-Linux platforms for
now until we can have a Linux specific test for ulimit -v.
|
|
There was a recent patch that added in some tests to the lit test suite
that use split-file. An explicit dependency in CMake was not added,
which led to check-lit not working if being run without doing a full
build first. This patch explicitly adds the dependency inside the CMake
file to fix this configuration.
|
|
This reverts commit 615d07ea55ea57afab0205aa739239070448a038.
This was causing some MacOS buildbolt failures.
|
|
This patch implements ulimit inside the lit internal shell.
Implementation wise, this functions similar to umask. But instead of
setting the limits within the lit test worker process, we set
environment variables and add a wrapper around the command to be
executed. The wrapper then sets the limits. This is because we cannot
increase the limits after lowering them, so we would otherwise end up
with a lit test worker stuck with a lower limit.
There are several tests where the use of ulimit is essential to the
semantics of the test (two in clang, ~7 in compiler-rt), so we need to
implement this in order to switch on the internal shell by default
without losing test coverage.
Reviewers: cmtice, petrhosek, ilovepi
Reviewed By: cmtice, ilovepi
Pull Request: https://github.com/llvm/llvm-project/pull/157958
|
|
(#158297)
Should fix bot: https://lab.llvm.org/buildbot/#/builders/46/builds/23206
|
|
|
|
There are some code paths within lit that still check what string types
are supported with the aim of being compatible with Python 2 and 3.
Given LLVM's minimum Python version is 3.8 and we do not have any
upstream testing for Python 2, I think we can safely drop this.
|
|
Most of these tests do not actually have a shell requirement. The shell
requirement ended up in the test either from cargo culting (from what I
can tell) or because the test authors actually meant to mark Windows as
unsupported. This prevents enablement of lit's internal shell within
clang.
Towards #102699.
Reviewers: rnk, efriedma-quic, Sirraide, petrhosek, ilovepi
Reviewed By: ilovepi
Pull Request: https://github.com/llvm/llvm-project/pull/156905
|
|
Previously we compared paths by string manipulation, however Windows
paths can use both '\' and '/' as path separators, which made this
fragile. This uses the pathlib.Path.samefile API instead.
|
|
|
|
StringIO was moved to io.StringIO in Python 3. These try/catch
statements were intended to provide cross compatibility between the two.
Now that LLVM's minimum required Python version is 3.8 and Python 2 has
been deprecated for a while, we can probably drop support. Especially
since there is no upstream testing for this configuration.
|
|
Lit's builtin diff command had some Python 2.7 code paths lying around
that we can probably get rid of at this point. LLVM at this point
requires Python 3.8 at minimum. Keeping lit working at a lower version
is a reasonable goal, but I think we can probably drop python 2 support
at this point given how long it has been deprecated and how long LLVM
has supported Python 3.
Reviewers: jdenny-ornl, ilovepi, petrhosek
Reviewed By: ilovepi
Pull Request: https://github.com/llvm/llvm-project/pull/157558
|
|
Previously this test case would `touch %S/empty.txt` to create and empty
file. The test case then copies contents to that file, so if run a
second time the `touch` command would not create an empty file.
`llvm/utils/lit/tests/diff-test-update.py` also no longer matches against file
paths other than the final bit of the file name. This prevents test failures
on Windows due to different path separators.
|
|
This adds an updater to lit's --update-tests flag with support for
`diff`. If a RUN line containing the `diff` command fails, this function
will use heuristics to try to deduce which file is the "reference" file,
and copy the contents of the other file to the reference. If it cannot
deduce which file is the reference file, it does nothing.
The heuristics are currently:
- does one of the files end in .expected while the other does not? Then
the .expected file is the reference.
- does one of the file paths contain the substring ".tmp" while the
other does not? Then the file not containing ".tmp" is the reference.
This matches cases where one file path is constructed using the `%t`
substitution.
|
|
These removes another test that otherwise needs a shell. This does
remove test coverage for pipefail in the external shell, but the
external shell should be disabled by default pretty soon. This also adds
test coverage for the internal shell which did not exist before.
Reviewers: jh7370, ilovepi, petrhosek
Reviewed By: ilovepi, petrhosek
Pull Request: https://github.com/llvm/llvm-project/pull/157223
|
|
125c74bc56926d42251d17c661e6ecad21dd45d7 added these commands eight
years ago to aid in debugging that is presumably done at this point.
Removing them to make the test a bit cleaner.
|
|
env -i is needed for some lit tests. The feature requires a minimal
amount of work to support and there is no easy way to rewrite the tests
that require it.
At least two tests that need this:
1. clang/test/Driver/env.c
2. lldb/test/Shell/Host/TestCustomShell.test
|
|
double-quoted strings. (#156742)
PR 156125 removed the escape (backslash) in front of '$' for all quoted
strings. It has since been pointed out this should only happen for
double-quoted strings. This PR fixes that.
|
|
LLDB allows creation of 'persistent' variables, with names that start
with '$'. The lit internal shell was escaping the '$', making it '\\$',
in some CHECK lines, which causes an LLDB test,
TestExprWithSideEffectOnConvenienceVar, to fail when using the lit
internal shell.
Further explanation of the failing LLDB test: LLDB convenience variables
start with '$'. The test passes several quoted commands that use and
update convenience variables to lldb as arguments to be run in batch
mode. The tool that packages up the complete string and passes it to the
lit internal shell lexer for lexing inserts a backslash in front of the
'$' before passing the string in for lexing. The lexer was passing this
change along, causing the tests to fail.
This PR fixes the issue by having the lexer remove the newly added
escape on the '$'.
|
|
This is causing bot failures that need later investigation, but there
are now dependent patches on top of this that would otherwise be
annoying to pull out.
|
|
This reverts commit faa4e35c622c13c7a565b979a6676d6cf3040cd4.
This was originally reverted because it was using a Python 3.9 feature
(umask in subprocess.Popen) when LLVM only requires Python 3.8. This
patch uses os.umask instead, which has been around for longer.
|
|
Should fix Windows build bot failures such as
https://lab.llvm.org/buildbot/#/builders/46/builds/22281.
The test (and the followup fix in #155303) did not properly account for
Windows style path separators.
|
|
This test case matches against python traceback output, which seems to
vary slightly between versions. This relaxes the constraints a bit to
make the test pass on buildbots.
|
|
The early exit we relied on to only invoke test updaters for failing
tests requires that there was no output to stdout or stderr, and that
timeouts weren't enabled. When these conditions weren't fulfilled, test
updaters would be invoked even on passing or XFAILed tests.
|
|
This is a fixup for #154317
|
|
Make sure we support windows style paths.
This is a fixup for #154317
|
|
(#154317)
This patch adds a -r|--relative-paths option to llvm-lit, which when
enabled will print test case names using paths relative to the current
working directory. The legacy default without that option is that test
cases are identified using a path relative to the test suite.
Only the summary report is impacted. That normally include failed tests,
unless unless options such as --show-pass.
Background to this patch was the discussion here
https://discourse.llvm.org/t/test-case-paths-in-llvm-lit-output-are-lacking-the-location-of-the-test-dir-itself/87973
with a goal to making it easier to copy-n-paste the path to the failing
test cases.
Examples showing difference in "Passed Tests" and "Failed Tests":
> llvm-lit --show-pass test/Transforms/Foo
PASS: LLVM :: Transforms/Foo/test1.txt (1 of 2)
FAIL: LLVM :: Transforms/Foo/test2.txt (2 of 2)
Passed Tests (1):
LLVM :: Transforms/Foo/test1.txt
Failed Tests (1):
LLVM :: Transforms/Foo/test2.txt
> llvm-lit --show-pass --relative-paths test/Transforms/Foo
PASS: LLVM :: Transforms/Foo/test1.txt (1 of 2)
FAIL: LLVM :: Transforms/Foo/test2.txt (2 of 2)
Passed Tests (1):
test/Transforms/Foo/test1.txt
Failed Tests (1):
test/Transforms/Foo/test2.txt
|
|
This reverts commit
https://github.com/llvm/llvm-project/commit/e495231238b86ae2a3c7bb5f94634c19ca2af19a
to reland
the --update-tests feature, originally landed in
https://github.com/llvm/llvm-project/pull/108425.
|
|
There are a few tests in clang that gate on system-windows but also
don't work on Cygwin, so add a system-cygwin feature for them to use.
|
|
See the related issue. We want to set up a build bot where `opt` runs with `-enable-profcheck`, which inserts `MD_prof` before running the rest of the pipeline requested from `opt`, and then validates resulting profile information (more info in the RFC linked by the aforementioned issue)
In that setup, we will also ignore `FileCheck`: while the profile info inserted is, currently, equivalent to the profile info a pass would observe via `BranchProbabilityInfo`/`BlockFrequencyInfo`, (1) we may want to change that, and (2) some tests are quite sensitive to the output IR, and break if, for instance, extra metadata is present (which it would be due to `-enable-profcheck`). Since we're just interested in profile consistency on the upcoming bot, ignoring `FileCheck` is simpler and sufficient. However, this has the effect of passing XFAIL tests. Rather than listing them all, the alternative is to just exclude XFAIL tests.
This PR adds support for that by introducing a `--exclude-xfail` option to `llvm-lit`.
Issue #147390
|
|
Flang and Flang-RT have two flavours of unittests:
1. GTest unittests, using lit's `lit.formats.GoogleTest` format ending
with `Tests${CMAKE_EXECUTABLE_SUFFIX}`
2. "non-GTest" or "OldUnit" unittests, a plain executable ending with
`.test${CMAKE_EXECUTABLE_SUFFIX}`
Both executables are emitted into the same unittests/ subdirectory. When
running ...
1. `tests/Unit/lit.cfg.py`, only considers executable ending with
`Tests` (or `Tests.exe` on Windows), hence skips the non-GTest tests.
2. `tests/NonGtestUnit/lit.cfg.py` considers all tests ending with
`.test` or `.exe`. On Windows, The GTest unitests also end with `.exe`.
In Flang-RT, `.exe` is considered an extension for non-GTest unitests
which causes tests such as Flang's `RuntimeTests.exe` to be executed for
both on Windows. This particular test includes a file write test, using
a hard-coded filename `ucsfile`. If the two instances are executed
concurrently, they might interfere with each other reading/writing
`ucsfile` which results in a flaky test.
This patch avoids the redundant execution by requiring the suffix
`.test.exe` on Windows. lit has to be modified because it uses
`os.path.splitext` the extract the extension, which would only recognize
the last component. It was changed from the orginal `endswith` in
c865abe747aa72192f02ebfdcabe730f2553e42f
for unknown reasons.
In Flang, `.exe` is not considered a suffix for non-GTest unittests and
hence they are not run at all. Fixing by also added `.test.exe` as valid
suffix, like with Flang-RT.
Unfortunately, the ` Evaluate/real.test.exe` test was failing on
Windows:
```
FAIL: flang-OldUnit :: Evaluate/real.test.exe (3592 of 3592)
******************** TEST 'flang-OldUnit :: Evaluate/real.test.exe' FAILED ********************
..\_src\flang\unittests\Evaluate\real.cpp:511: FAIL: FlagsToBits(prod.flags) == 0x18, not 0x10
0 0x800001 * 0xbf7ffffe
..\_src\flang\unittests\Evaluate\real.cpp:511: FAIL: FlagsToBits(prod.flags) == 0x18, not 0x10
0 0x800001 * 0x3f7ffffe
..\_src\flang\unittests\Evaluate\real.cpp:511: FAIL: FlagsToBits(prod.flags) == 0x18, not 0x10
0 0x80800001 * 0xbf7ffffe
..\_src\flang\unittests\Evaluate\real.cpp:511: FAIL: FlagsToBits(prod.flags) == 0x18, not 0x10
0 0x80800001 * 0x3f7ffffe
...
```
This is due to the `__x86_64__` macro not being set by Microsoft's
cl.exe and hence floating point status flags not being read out. The
equivalent macro for Microsofts compiler is `_M_X64` (or `_M_X64`).
|
|
|
|
This patch refactors use_clang into two functions. This is intended for
use within the clang-tools-extra test suite to avoid a race condition
where the clang binary exists but is not yet ready for execution which
results in a lit configuration failure.
Details are in #145703.
Reviewers: Keenuts, pogo59, AaronBallman, DavidSpickett
Reviewed By: pogo59
Pull Request: https://github.com/llvm/llvm-project/pull/147436
|
|
(#145731)
Before this commit the LIT test framework of the static analyzer had a
file called `analyzer_test.py` which implemented a tricky system for
selecting the constraint manager:
- (A) Test files without `REQUIRES: z3` were executed with the default
range-based constraint manager.
- (B) If clang was built with Z3 support _and_ `USE_Z3_SOLVER=1` was
passed to the test run, the test was executed with Z3 as the
constraint manager.
(There was support for executing the same RUN line twice if both
conditions were satisfied.)
Unfortunately, using Z3 as the constraint manager does not work in
practice (very slow and causes many crashes), so the (B) pathway became
unused (or was never truly used?) and became broken due to bit rot. (In
the CI bots the analyzer is built without Z3 support, so only the
pathway (A) is used.)
This commit removes `analyzer_test.py` (+ related logic in other build
files + the test `z3/enabled.c` which just tested that
`analyzer_test.py` is active), because it tries to implement a feature
that we don't need (only one constraint manager is functional) and its
code is so complicated and buggy that it isn't useful as a starting
point for future development.
The fact that this logic was broken implied that tests with `REQUIRES:
z3` were not executed during normal testing, so they were also affected
by bit rot. Unfortunately this also affected the tests of the
`z3-crosscheck` mode (aka Z3 refutation) which also depends on Z3 but
uses Z3 in a different way which is actually stable and functional.
In this commit I'm fixing most of the `REQUIRES: z3` tests that were
broken by straightforward issues. Two test files, `PR37855.c` and
`z3-crosscheck.c` were affected by more complex issues, so I marked them
as `XFAIL` for now. We're planning to fix them with follow-up commits in
the foreseeable future.
For additional background information see also the discourse thread
https://discourse.llvm.org/t/taking-ownership-of-clang-test-analysis/84689
|
|
Reverts llvm/llvm-project#143930 as it causes build failures:
https://github.com/llvm/llvm-project/pull/143930#issuecomment-2969115461
|
|
Remove imports that are not used in some lit test files.
|
|
If a test took more than one attempt to complete, show the number of attempts and the maximum allowed attempts as `2 of 4 attempts` inside the `<progress info>` (see [TEST RUN OUTPUT FORMAT](https://llvm.org/docs/CommandGuide/lit.html#test-run-output-format)).
NOTE: Additionally this is a fixup for #141851 where the tests were not quite right. `max-retries-per-test/allow-retries-test_retry_attempts/test.py` was added but never used there. Now we're calling it. To correlate better between the test output and the test script I've used higher numbers of max allowed retries.
|
|
The subject test sporadically fails on the AIX builder:
https://lab.llvm.org/buildbot/#/builders/64/builds/3921/steps/6/logs/FAIL__lit___timeout-hang_py
This appears to be an environment issue potentially connected to high
load because the problem is not observed on other AIX machines.
This patch separates the "hard" timeout value from the value used to
signal a hang. This allows for a more generous "hard" timeout value,
which allows observation of cases that take longer to finish despite not
hanging.
|
|
When packaging LLVM we've seen arbitrary tests fail.
It happened sporadically and most of the times the test
do work if they are run a second time on the next day.
The tests themselves were always different and we didn't
know ahead of time which ones we wanted to re-run.
That's we filter-out a lot of `libomp` and `libarcher` tests [1].
This change allows us to set
`LIT_OPTS="--max-retries-per-test=12"`
when running any "check-XXX" build target. Then any lit test
will at most be re-run 12 times, unless there's an `ALLOW_RETRIES:`
in one of the test scripts that's specifying a different value
than `12`. `12` is just an example here, any positive integer
will work.
Please note, that this only adds the possibility to re-run
lit tests. It does not actually do it until the caller specifies
`--max-retries-per-test=<POSITIVE_INT>` either on a call to `lit` or
in `LIT_OPTS`.
Also note, that one can still use `ALLOW_RETRIES:` in test scripts
and it will always rule over `--max-retries-per-test`. When
`--max-retries-per-test` is set too low, but the
`config.test_retry_attempts`
is high enough, it works as well.
Any option in the list below overrules its predecessor:
* `--max-retries-per-test`
* `config.test_retry_attempts`
* `ALLOW_RETRIES` keyword
From the above options to re-run tests, `--max-retries-per-test` is the
only one that doesn't require a change in the test scripts or the test
config.
[1]:
https://src.fedoraproject.org/rpms/llvm/blob/rawhide/f/llvm.spec#_2326
Downstream PR to make use of the `--max-retries-per-test` option:
https://src.fedoraproject.org/rpms/llvm/pull-request/434
Downstream ticket: https://issues.redhat.com/browse/LLVM-145
|
