| Age | Commit message (Collapse) | Author | Files | Lines |
|---|
|
This patch removes support for %T from llvm-lit. For now we mark the
test unresolved and add an error message noting the substitution is
deprecated. This is exactly the same as the error handling for other
substitution failures. We intend to remove support for the nice error
message once 22 branches as users should have moved over by the they are
upgrading to v23.
Reviewers: petrhosek, jh7370, ilovepi, pogo59, cmtice
Reviewed By: cmtice, jh7370, ilovepi
Pull Request: https://github.com/llvm/llvm-project/pull/160028
|
|
This patch removes %T from tests in preparation for removing the
substitution completely. Splitting this off from the actual removal to
make it easier to revert the removal of %T in case I missed something.
Reviewers: RKSimon, jh7370, cmtice, petrhosek, ilovepi, pogo59
Reviewed By: cmtice, RKSimon, jh7370
Pull Request: https://github.com/llvm/llvm-project/pull/160027
|
|
This patch allows for two retry attempts for the readfile tests. This is
intended as a stop-gap until I have time to do proper investigation into
why exactly the tests are failing.
|
|
Python 3.13 considers "/file/does/not/exist" to not be an absolute path
on Windows, so the test runner does os.path.join(cwd, filePath), which
can end up with an output path such as "D:/file/does/not/exist".
Accept a potential prefix before the missing path here.
This fixes running the lit tests on Windows with Python 3.13.
|
|
This PR fixes a bunch of failures on AIX that occurred due to the switch
to lit internal shell by default. The failures deal with the following:
1. unset not being supported by lit internal shell
2. A bug with echo -n on AIX when there are multiple pipes in the RUN
command
3. ulimit test case not supported on AIX due to the -v option
4. platform specific error message for missing file
|
|
Before this change, rm would assume that a symlink to a directory was
actually a directory and require the recursive flag to be passed,
differing from other shells. Given the change in lit is about the same
length as the test change would be (minus tests), I think it makes sense
to just support this in the internal shell.
Reviewers: cmtice, petrhosek, ilovepi
Reviewed By: petrhosek, cmtice, ilovepi
Pull Request: https://github.com/llvm/llvm-project/pull/158464
|
|
cat with no files passed to it is supposed to read from STDIN according
to POSIX. The builtin cat lacking this behavior led to the clang test in
dev-fd-fs.c to fail because it expected this behavior. This is a simple
modification and I do not think it is possible to rewrite the test
without this easily while preserving the semantics around named pipes.
Reviewers: petrhosek, arichardson, ilovepi, cmtice, jh7370
Reviewed By: jh7370, arichardson, ilovepi, cmtice
Pull Request: https://github.com/llvm/llvm-project/pull/158447
|
|
This patch adds support for the new lit %{readfile:<filename>}
substitution to the external shell. The implementation currently just
appends some test commands to ensure the file exists and uses a subshell
with cat. This is intended to enable running tests using the
substitution in the external shell before we fully switch over to the
internal shell.
This code is designed to be temporary with us deleting it once
everything has migrated over to the internal shell and we are able to
remove the external shell code paths.
Reviewers: petrhosek, cmtice, pogo59, ilovepi, arichardson
Reviewed By: cmtice
Pull Request: https://github.com/llvm/llvm-project/pull/159431
|
|
This patch adds a new %{readfile:<file name>} substitution to lit. This
is needed for porting a couple of tests to lit's internal shell. These
tests are all using subshells to pass some option to a command are not
feasible to run within the internal shell without this functionality.
Reviewers: petrhosek, jh7370, ilovepi, cmtice
Reviewed By: jh7370, cmtice
Pull Request: https://github.com/llvm/llvm-project/pull/158441
|
|
There were some build failures because apparently running ulimit -n 50
actually means ulimit -n 48 there. Losing the test coverage on Solaris
shouldn't be a big deal.
|
|
Some of the ulimit limits do not work on some POSIX platforms. THe
motivating example here is ulimit -v on MacOS as the relevant system
calls are not implemented in XNU. Splitting the tests lets us keep test
coverage on POSIX (non-Linux) platforms.
|
|
This reverts commit 330068a74bfb6333f9016e3c4053eeaf4989d601.
This was causing some test failures on MacOS that are now fixed in the reland.
These failures were related to calling ulimit -v despite XNU not having support
for that option. This patch simply disables the test on non-Linux platforms for
now until we can have a Linux specific test for ulimit -v.
|
|
This reverts commit 615d07ea55ea57afab0205aa739239070448a038.
This was causing some MacOS buildbolt failures.
|
|
This patch implements ulimit inside the lit internal shell.
Implementation wise, this functions similar to umask. But instead of
setting the limits within the lit test worker process, we set
environment variables and add a wrapper around the command to be
executed. The wrapper then sets the limits. This is because we cannot
increase the limits after lowering them, so we would otherwise end up
with a lit test worker stuck with a lower limit.
There are several tests where the use of ulimit is essential to the
semantics of the test (two in clang, ~7 in compiler-rt), so we need to
implement this in order to switch on the internal shell by default
without losing test coverage.
Reviewers: cmtice, petrhosek, ilovepi
Reviewed By: cmtice, ilovepi
Pull Request: https://github.com/llvm/llvm-project/pull/157958
|
|
(#158297)
Should fix bot: https://lab.llvm.org/buildbot/#/builders/46/builds/23206
|
|
|
|
Previously this test case would `touch %S/empty.txt` to create and empty
file. The test case then copies contents to that file, so if run a
second time the `touch` command would not create an empty file.
`llvm/utils/lit/tests/diff-test-update.py` also no longer matches against file
paths other than the final bit of the file name. This prevents test failures
on Windows due to different path separators.
|
|
This adds an updater to lit's --update-tests flag with support for
`diff`. If a RUN line containing the `diff` command fails, this function
will use heuristics to try to deduce which file is the "reference" file,
and copy the contents of the other file to the reference. If it cannot
deduce which file is the reference file, it does nothing.
The heuristics are currently:
- does one of the files end in .expected while the other does not? Then
the .expected file is the reference.
- does one of the file paths contain the substring ".tmp" while the
other does not? Then the file not containing ".tmp" is the reference.
This matches cases where one file path is constructed using the `%t`
substitution.
|
|
These removes another test that otherwise needs a shell. This does
remove test coverage for pipefail in the external shell, but the
external shell should be disabled by default pretty soon. This also adds
test coverage for the internal shell which did not exist before.
Reviewers: jh7370, ilovepi, petrhosek
Reviewed By: ilovepi, petrhosek
Pull Request: https://github.com/llvm/llvm-project/pull/157223
|
|
125c74bc56926d42251d17c661e6ecad21dd45d7 added these commands eight
years ago to aid in debugging that is presumably done at this point.
Removing them to make the test a bit cleaner.
|
|
env -i is needed for some lit tests. The feature requires a minimal
amount of work to support and there is no easy way to rewrite the tests
that require it.
At least two tests that need this:
1. clang/test/Driver/env.c
2. lldb/test/Shell/Host/TestCustomShell.test
|
|
double-quoted strings. (#156742)
PR 156125 removed the escape (backslash) in front of '$' for all quoted
strings. It has since been pointed out this should only happen for
double-quoted strings. This PR fixes that.
|
|
LLDB allows creation of 'persistent' variables, with names that start
with '$'. The lit internal shell was escaping the '$', making it '\\$',
in some CHECK lines, which causes an LLDB test,
TestExprWithSideEffectOnConvenienceVar, to fail when using the lit
internal shell.
Further explanation of the failing LLDB test: LLDB convenience variables
start with '$'. The test passes several quoted commands that use and
update convenience variables to lldb as arguments to be run in batch
mode. The tool that packages up the complete string and passes it to the
lit internal shell lexer for lexing inserts a backslash in front of the
'$' before passing the string in for lexing. The lexer was passing this
change along, causing the tests to fail.
This PR fixes the issue by having the lexer remove the newly added
escape on the '$'.
|
|
This is causing bot failures that need later investigation, but there
are now dependent patches on top of this that would otherwise be
annoying to pull out.
|
|
This reverts commit faa4e35c622c13c7a565b979a6676d6cf3040cd4.
This was originally reverted because it was using a Python 3.9 feature
(umask in subprocess.Popen) when LLVM only requires Python 3.8. This
patch uses os.umask instead, which has been around for longer.
|
|
Should fix Windows build bot failures such as
https://lab.llvm.org/buildbot/#/builders/46/builds/22281.
The test (and the followup fix in #155303) did not properly account for
Windows style path separators.
|
|
This test case matches against python traceback output, which seems to
vary slightly between versions. This relaxes the constraints a bit to
make the test pass on buildbots.
|
|
The early exit we relied on to only invoke test updaters for failing
tests requires that there was no output to stdout or stderr, and that
timeouts weren't enabled. When these conditions weren't fulfilled, test
updaters would be invoked even on passing or XFAILed tests.
|
|
This is a fixup for #154317
|
|
Make sure we support windows style paths.
This is a fixup for #154317
|
|
(#154317)
This patch adds a -r|--relative-paths option to llvm-lit, which when
enabled will print test case names using paths relative to the current
working directory. The legacy default without that option is that test
cases are identified using a path relative to the test suite.
Only the summary report is impacted. That normally include failed tests,
unless unless options such as --show-pass.
Background to this patch was the discussion here
https://discourse.llvm.org/t/test-case-paths-in-llvm-lit-output-are-lacking-the-location-of-the-test-dir-itself/87973
with a goal to making it easier to copy-n-paste the path to the failing
test cases.
Examples showing difference in "Passed Tests" and "Failed Tests":
> llvm-lit --show-pass test/Transforms/Foo
PASS: LLVM :: Transforms/Foo/test1.txt (1 of 2)
FAIL: LLVM :: Transforms/Foo/test2.txt (2 of 2)
Passed Tests (1):
LLVM :: Transforms/Foo/test1.txt
Failed Tests (1):
LLVM :: Transforms/Foo/test2.txt
> llvm-lit --show-pass --relative-paths test/Transforms/Foo
PASS: LLVM :: Transforms/Foo/test1.txt (1 of 2)
FAIL: LLVM :: Transforms/Foo/test2.txt (2 of 2)
Passed Tests (1):
test/Transforms/Foo/test1.txt
Failed Tests (1):
test/Transforms/Foo/test2.txt
|
|
See the related issue. We want to set up a build bot where `opt` runs with `-enable-profcheck`, which inserts `MD_prof` before running the rest of the pipeline requested from `opt`, and then validates resulting profile information (more info in the RFC linked by the aforementioned issue)
In that setup, we will also ignore `FileCheck`: while the profile info inserted is, currently, equivalent to the profile info a pass would observe via `BranchProbabilityInfo`/`BlockFrequencyInfo`, (1) we may want to change that, and (2) some tests are quite sensitive to the output IR, and break if, for instance, extra metadata is present (which it would be due to `-enable-profcheck`). Since we're just interested in profile consistency on the upcoming bot, ignoring `FileCheck` is simpler and sufficient. However, this has the effect of passing XFAIL tests. Rather than listing them all, the alternative is to just exclude XFAIL tests.
This PR adds support for that by introducing a `--exclude-xfail` option to `llvm-lit`.
Issue #147390
|
|
If a test took more than one attempt to complete, show the number of attempts and the maximum allowed attempts as `2 of 4 attempts` inside the `<progress info>` (see [TEST RUN OUTPUT FORMAT](https://llvm.org/docs/CommandGuide/lit.html#test-run-output-format)).
NOTE: Additionally this is a fixup for #141851 where the tests were not quite right. `max-retries-per-test/allow-retries-test_retry_attempts/test.py` was added but never used there. Now we're calling it. To correlate better between the test output and the test script I've used higher numbers of max allowed retries.
|
|
The subject test sporadically fails on the AIX builder:
https://lab.llvm.org/buildbot/#/builders/64/builds/3921/steps/6/logs/FAIL__lit___timeout-hang_py
This appears to be an environment issue potentially connected to high
load because the problem is not observed on other AIX machines.
This patch separates the "hard" timeout value from the value used to
signal a hang. This allows for a more generous "hard" timeout value,
which allows observation of cases that take longer to finish despite not
hanging.
|
|
When packaging LLVM we've seen arbitrary tests fail.
It happened sporadically and most of the times the test
do work if they are run a second time on the next day.
The tests themselves were always different and we didn't
know ahead of time which ones we wanted to re-run.
That's we filter-out a lot of `libomp` and `libarcher` tests [1].
This change allows us to set
`LIT_OPTS="--max-retries-per-test=12"`
when running any "check-XXX" build target. Then any lit test
will at most be re-run 12 times, unless there's an `ALLOW_RETRIES:`
in one of the test scripts that's specifying a different value
than `12`. `12` is just an example here, any positive integer
will work.
Please note, that this only adds the possibility to re-run
lit tests. It does not actually do it until the caller specifies
`--max-retries-per-test=<POSITIVE_INT>` either on a call to `lit` or
in `LIT_OPTS`.
Also note, that one can still use `ALLOW_RETRIES:` in test scripts
and it will always rule over `--max-retries-per-test`. When
`--max-retries-per-test` is set too low, but the
`config.test_retry_attempts`
is high enough, it works as well.
Any option in the list below overrules its predecessor:
* `--max-retries-per-test`
* `config.test_retry_attempts`
* `ALLOW_RETRIES` keyword
From the above options to re-run tests, `--max-retries-per-test` is the
only one that doesn't require a change in the test scripts or the test
config.
[1]:
https://src.fedoraproject.org/rpms/llvm/blob/rawhide/f/llvm.spec#_2326
Downstream PR to make use of the `--max-retries-per-test` option:
https://src.fedoraproject.org/rpms/llvm/pull-request/434
Downstream ticket: https://issues.redhat.com/browse/LLVM-145
|
|
Fixes #133914
When using the internal shell with a timeout set lit will hang on the
following call if an exception is thrown and not immediately caught
https://github.com/llvm/llvm-project/blob/19970535f92c0f2dcda01b7fc60f95945166e424/llvm/utils/lit/lit/run.py#L93
This can occur when using the internal lit shell and trying to run a
program that does not exist. In this case `_executeShCmd` will throw an
internal shell error, which will not be caught by the function directly
calling it, `executeShCmd`, rather it is caught one function higher in
the call stack in `executeScriptInternal`. Because that exception is
percolated up the call stack instead of being immediately caught lit
will hang until the test timeout expires. This patch changes the
location where we catch this exception to `executeShCmd` instead to
avoid this.
For more background on what causes this hang see:
https://stackoverflow.com/questions/15314189/python-multiprocessing-pool-hangs-at-join
|
|
|
|
|
|
This reverts commit 57e89c97c2c1b4e41f07a90c2f4d36649696e619.
Updated lit tests.
|
|
I got a report that downstream this test failed and the cause was that
it took longer than the 1 second we expected to run one of the test
cases.
This test doesn't need to be that specific, so I am updating all the
time regexes to be the same one that allows 0-9 any number of digits,
requires a decimal point, then 0-9 any number of digits for the final
part.
|
|
The changes from https://github.com/llvm/llvm-project/pull/121376 has
broken the ppc64 aix bot:
https://lab.llvm.org/buildbot/#/builders/64/builds/1835. Adjusted the
testcase to account for `cat` behaviour on AIX prior to the changes
|
|
BusyBox `cat` has yet another slight variation of error formatting:
```console
$ cat --help 2>&1 | head -1
BusyBox v1.37.0 (2024-09-30 10:39:57 UTC) multi-call binary.
$ cat does-not-exist
cat: can't open 'does-not-exist': No such file or directory
```
Rather than extend the test result regex with a third case,
recognise that we only really care about the filename and errno string.
Weaken the regex to ignore all "noise" around the filename.
Note this also corrects what looks like a bug with the previous regex.
Previously, the `cannot open does-not-exist` alternate did not assert
the following errno message. This was introduced in
https://reviews.llvm.org/D60553 (apparently) due to differences in the
`cat` command on AIX. That bug doesn't include the specific
AIX output, so it's unclear if this omission was intended.
|
|
- Add option (--report-failures-only) to generate a reduced report for
lit tests that only includes failing tests
- This is a continuation of proposed patches by @gregbedwell here:
- https://reviews.llvm.org/D143516
- https://reviews.llvm.org/D143519
---------
Co-authored-by: Greg Bedwell <greg.bedwell@sony.com>
Co-authored-by: James Henderson <James.Henderson@sony.com>
|
|
already exist (#112729)
When running a build like:
```
ninja check-clang check-llvm
```
Prior to my changes you ended up with one results file, in this specific case Junit XML:
```
results.xml
```
This would only include the last set of tests lit ran, which were for
llvm. To get around this, many CI systems will run one check target,
move the file away, then run another, somehow propgating the return code
as well.
```
rectode=0
for target in targets:
ninja target
retcode=$?
mv results.xml results-${target}.xml
<report the overall return code>
```
I want to use something like this Buildkite reporting plugin in CI, which needs to have all the results available:
https://buildkite.com/docs/agent/v3/cli-annotate#using-annotations-to-report-test-results
Modifying CI's build scripts for Windows and Linux is a lot of work. So
my changes instead make lit detect an existing result file and modify
the file name to find a new file to write to. Now you will get:
```
results.xml results.<tempfile generated value>.xml
```
This will work for all result file types since I'm doing it in the base
Report class. Now you've got separate files, it's easy to collect them
with `<path>/*.xml`.
Note that the `<tempfile generated value>` is not ordered.
|
|
This reverts commit 9d98acb196a40fee5229afeb08f95fd36d41c10a.
|
|
This reverts commit dccebddb3b802c4c1fe287222e454b63f850f012.
|
|
Fix up the testcase failures (#107974)
Encountered several testcase failures when running `ninja check-all`. It
was due to the full path name were shown in the error message instead of
the binary name, and therefore causing the check string mismatch.
The machine was running CentOS 9 with binfmt_misc setup that uses
qemu-aarch64 (8.1.2). Built and ran the unittest as aarch64 host
(through qemu user).
Co-authored-by: h2h <h2h@meta.com>
|
|
Line ending policies were changed in the parent, dccebddb3b80. To make
it easier to resolve downstream merge conflicts after line-ending
policies are adjusted this is a separate whitespace-only commit. If you
have merge conflicts as a result, you can simply `git add --renormalize
-u && git merge --continue` or `git add --renormalize -u && git rebase
--continue` - depending on your workflow.
|
|
Historically, we've not automatically enforced how git tracks line
endings, but there are many, many commits that "undo" unintended CRLFs
getting into history.
`git log --pretty=oneline --grep=CRLF` shows nearly 100 commits
involving reverts of CRLF making its way into the index and then
history. As far as I can tell, there are none the other way round except
for specific cases like `.bat` files or tests for parsers that need to
accept such sequences.
Of note, one of the earliest of those listed in that output is:
```
commit 9795860250734e5c2a879546c534e35d9edd5944
Author: NAKAMURA Takumi <geek4civic@gmail.com>
Date: Thu Feb 3 11:41:27 2011 +0000
cmake/*: Add svn:eol-style=native and fix CRLF.
llvm-svn: 124793
```
...which introduced such a defacto policy for subversion.
With old versions of git, it's been a bit of a crap-shoot whether
enforcing storing line endings in the history will upset checkouts on
machines where such line endings are the norm. Indeed many users have
enforced that git checks out the working copy according to a global or
per-user config via core crlf, or core autocrlf.
For ~8 years now[1], however, git has supported the ability to "do as
the Romans do" on checkout, but internally store subsets of text files
with line-endings specified via a system of patterns in the
`.gitattributes` file. Since we now have this ability, and we've been
specifying attributes for various binary files, I think it makes sense
to rid us of all that work converting things "back", and just let git
handle the local checkout. Thus the new toplevel policy here is
* text=auto
In simple terms this means "unless otherwise specified, convert all
files considered "text" files to LF in the project history, but check
them out as expected on the local machine. What is "expected on the
local machine" is dependent on configuration and default.
For those files in the repository that *do* need CRLF endings, I've
adopted a policy of `eol=crlf` which means that git will store them in
history with LF, but regardless of user config, they'll be checked out
in tree with CRLF.
Finally, existing files have been "corrected" in history via `git add
--renormalize .`
End users should *not* need to adjust their local git config or
workflow.
[1]: git 2.10 was released with fixed support for fine-grained
line-ending tracking that respects user-config *and* repo policy. This
can be considered the point at which git will respect both the user's
local working tree preference *and* the history as specified by the
maintainers. See
https://github.com/git/git/blob/master/Documentation/RelNotes/2.10.0.txt#L248
for the release note.
|
|
Currently we write out a time taken to run all test suites:
```
<testsuites time="8.28">
```
And one for each test:
```
<testcase classname="lldb-shell.Breakpoint" name="breakpoint-command.test" time="2.38"/>
```
However, the schema says there should be one for each suite and test,
but none for testsuites:
https://github.com/windyroad/JUnit-Schema/blob/cfa434d4b8e102a8f55b8727b552a0063ee9044e/JUnit.xsd#L216
I'm leaving the `testsuites` time in though because no one has
complained so far, and someone out there probably has a script relying
on it by now. Most XML tools handle unknown attributes quite well
anyway.
I'm adding a per testsuite time to comply with the schema and maybe be
more compatible with other JUnit tools.
```
<testsuite name="lldb-shell" ... time="12.34">
```
The test suite time is the sum of the time taken for all tests in the
suite. This will ignore some overhead in setting up the suite, and means
that the sum of the times for all individual suites may not equal the
`testsuites` time.
As we're usually focusing on the execution time of particular tests, not
lit's book keeping, I think this is a reasonable choice.
|
