aboutsummaryrefslogtreecommitdiff
path: root/llvm/docs/CommandGuide/lit.rst
AgeCommit message (Collapse)AuthorFilesLines
3 days[lit] Optionally exclude xfail tests (#151191)Mircea Trofin1-0/+5
See the related issue. We want to set up a build bot where `opt` runs with `-enable-profcheck`, which inserts `MD_prof` before running the rest of the pipeline requested from `opt`, and then validates resulting profile information (more info in the RFC linked by the aforementioned issue) In that setup, we will also ignore `FileCheck`: while the profile info inserted is, currently, equivalent to the profile info a pass would observe via `BranchProbabilityInfo`/`BlockFrequencyInfo`, (1) we may want to change that, and (2) some tests are quite sensitive to the output IR, and break if, for instance, extra metadata is present (which it would be due to `-enable-profcheck`). Since we're just interested in profile consistency on the upcoming bot, ignoring `FileCheck` is simpler and sufficient. However, this has the effect of passing XFAIL tests. Rather than listing them all, the alternative is to just exclude XFAIL tests. This PR adds support for that by introducing a `--exclude-xfail` option to `llvm-lit`. Issue #147390
2025-06-06[lit] show retry attempts (#142413)Konrad Kleine1-0/+8
If a test took more than one attempt to complete, show the number of attempts and the maximum allowed attempts as `2 of 4 attempts` inside the `<progress info>` (see [TEST RUN OUTPUT FORMAT](https://llvm.org/docs/CommandGuide/lit.html#test-run-output-format)). NOTE: Additionally this is a fixup for #141851 where the tests were not quite right. `max-retries-per-test/allow-retries-test_retry_attempts/test.py` was added but never used there. Now we're calling it. To correlate better between the test output and the test script I've used higher numbers of max allowed retries.
2025-06-04[docs] don't use "=" in lit options with arguments (#142340)Konrad Kleine1-14/+14
This is a fixup for #141851 and removes `=` from all options with additional arguments. Before 14 out of 22 options with arguments used "=" and 7 didn't.
2025-05-31[lit] add --max-retries-per-test execution option (#141851)Konrad Kleine1-0/+13
When packaging LLVM we've seen arbitrary tests fail. It happened sporadically and most of the times the test do work if they are run a second time on the next day. The tests themselves were always different and we didn't know ahead of time which ones we wanted to re-run. That's we filter-out a lot of `libomp` and `libarcher` tests [1]. This change allows us to set `LIT_OPTS="--max-retries-per-test=12"` when running any "check-XXX" build target. Then any lit test will at most be re-run 12 times, unless there's an `ALLOW_RETRIES:` in one of the test scripts that's specifying a different value than `12`. `12` is just an example here, any positive integer will work. Please note, that this only adds the possibility to re-run lit tests. It does not actually do it until the caller specifies `--max-retries-per-test=<POSITIVE_INT>` either on a call to `lit` or in `LIT_OPTS`. Also note, that one can still use `ALLOW_RETRIES:` in test scripts and it will always rule over `--max-retries-per-test`. When `--max-retries-per-test` is set too low, but the `config.test_retry_attempts` is high enough, it works as well. Any option in the list below overrules its predecessor: * `--max-retries-per-test` * `config.test_retry_attempts` * `ALLOW_RETRIES` keyword From the above options to re-run tests, `--max-retries-per-test` is the only one that doesn't require a change in the test scripts or the test config. [1]: https://src.fedoraproject.org/rpms/llvm/blob/rawhide/f/llvm.spec#_2326 Downstream PR to make use of the `--max-retries-per-test` option: https://src.fedoraproject.org/rpms/llvm/pull-request/434 Downstream ticket: https://issues.redhat.com/browse/LLVM-145
2025-04-28Clarify `lit`'s definition of failure and conditions when it exits with exit ↵i-ky1-5/+8
code 1 (#136190) Documentation of when `lit` exits with code 1 is out of date. It is no longer just "FAIL or XPASS", there are more failure types: https://github.com/llvm/llvm-project/blob/b30100b87f24847afd6407b4939a184ebcf16ef9/llvm/utils/lit/lit/Test.py#L51-L55 Exit code can also be affected by `--ignore-fail` option: https://github.com/llvm/llvm-project/blob/b30100b87f24847afd6407b4939a184ebcf16ef9/llvm/utils/lit/lit/main.py#L154-L162 This PR extracts a clear definition of "failure" from the description of `--report-failures-only` option: https://github.com/llvm/llvm-project/blob/b30100b87f24847afd6407b4939a184ebcf16ef9/llvm/docs/CommandGuide/lit.rst?plain=1#L194-L196 ...puts it into "Test Status Results" section and references it when describing exit codes and `--ignore-fail` option.
2025-03-24Reapply "lit: Move RUN at line comment after the command."Peter Collingbourne1-1/+1
This reverts commit 57e89c97c2c1b4e41f07a90c2f4d36649696e619. Updated lit tests.
2025-03-24Revert "lit: Move RUN at line comment after the command."Peter Collingbourne1-1/+1
This reverts commit 8d3dc1ed5656a3e69e4195d58684a7f4bf0ff5cc. Test needs to be updated.
2025-03-24lit: Move RUN at line comment after the command.Peter Collingbourne1-1/+1
When a developer copy/pastes a failing command line into their shell to rerun it, they have to manually delete the "RUN: at line N:" prefix. To make life easier for such developers, let's make it possible to copy/paste a command without needing to modify it while still showing the line number in the output by moving the line number to a comment at the end of the command line. Reviewers: jroelofs, MaskRay Reviewed By: jroelofs, MaskRay Pull Request: https://github.com/llvm/llvm-project/pull/132485
2024-11-13[lit] Add --report-failures-only option for lit test reports (#115439)Rakshit Patel1-0/+4
- Add option (--report-failures-only) to generate a reduced report for lit tests that only includes failing tests - This is a continuation of proposed patches by @gregbedwell here: - https://reviews.llvm.org/D143516 - https://reviews.llvm.org/D143519 --------- Co-authored-by: Greg Bedwell <greg.bedwell@sony.com> Co-authored-by: James Henderson <James.Henderson@sony.com>
2024-10-29[llvm][docs] Update list of llvm-lit optionsDavid Spickett1-49/+113
Fixes #62899 In this commit I have updated the list of options to include any missing options and re-rordered some of them to match the order in lit's --help. Where there was a larger description in this document I've used that instead of the --help description. This *does not* include --use-unique-output-file-name as this was only added recently and we are still debating whether it will be kept.
2024-10-03[LIT] Rename substitution `%basename_s` to `%{s:basename}` (#111062)Rahul Joshi1-2/+3
Also added `%{t:stem}` as an alias for `%basename_t` and modified unit test to test these new substitutions.
2024-10-03[LIT] Add support for `%basename_s` to get base name of source file (#110993)Rahul Joshi1-0/+1
Add support for `%basename_s` pattern in the RUN commands to get the base name of the source file, and adopt it in a TableGen LIT test.
2024-10-01Revert "[Utils] Add new --update-tests flag to llvm-lit" (#110772)Henrik G. Olsson1-5/+0
Reverts llvm/llvm-project#108425
2024-10-01[Utils] Add new --update-tests flag to llvm-lit (#108425)Henrik G. Olsson1-0/+5
This adds a flag to lit for detecting and updating failing tests when possible to do so automatically. The flag uses a plugin architecture where config files can add additional auto-updaters for the types of tests in the test suite. When a test fails with `--update-tests` enabled lit passes the test RUN invocation and output to each registered test updater until one of them signals that it updated the test (or all test updaters have been run). As such it is the responsibility of the test updater to only update tests where it is reasonably certain that it will actually fix the test, or come close to doing so. Initially adds support for UpdateVerifyTests and UpdateTestChecks. The flag is currently only implemented for lit's internal shell, so `--update-tests` implies `LIT_USE_INTERNAL_SHELL=1`. Builds on work in #97369 Fixes #81320
2024-07-20[lit] Add a flag to disable lit time tests (#98270)Vincent Lee1-0/+4
LLVM lit assumes control of the test parallelism when running a test suite. This style of testing doesn't play nicely with build systems like Buck or Bazel since it prefers finer grained actions on a per-test level. In order for external build systems to control the test parallelism, add an option to disable `.lit_test_times.txt` under the `--skip-test-time-recording` flag, thus allowing other build systems to determine the parallelism and avoid race conditions when writing to that file. I went for `--skip-test-time-recording` instead of `--time-tests` in order to preserve the original functionality of writing to `.lit_test_times.txt` as the default behavior and only opt-in for those who do _not_ want `.lit_test_times.txt` file.
2024-06-11[llvm][lit] Add tsan feature when enabled (#94573)Keith Smiley1-0/+17
2023-09-19[lit] Improve test output from lit's internal shellJoel E. Denny1-4/+3
This patch and D154984 were discussed in <https://discourse.llvm.org/t/rfc-improving-lits-debug-output/72839>. Motivation ---------- D154984 removes the "Script:" section that lit prints along with a test's output, and it makes -v and -a imply -vv. For example, after D154984, the "Script:" section below is never shown, but -v is enough to produce the execution trace following it: ``` Script: -- : 'RUN: at line 1'; echo hello | FileCheck bogus.txt && echo success -- Exit Code: 2 Command Output (stdout): -- $ ":" "RUN: at line 1" $ "echo" "hello" # command output: hello $ "FileCheck" "bogus.txt" # command stderr: Could not open check file 'bogus.txt': No such file or directory error: command failed with exit status: 2 -- ``` In the D154984 review, some reviewers point out that they have been using the "Script:" section for copying and pasting a test's shell commands to a terminal window. The shell commands as printed in the execution trace can be harder to copy and paste for the following reasons: - They drop redirections and break apart RUN lines at `&&`, `|`, etc. - They add `$` at the start of every command, which makes it hard to copy and paste multiple commands in bulk. - Command stdout, stderr, etc. are interleaved with the commands and are not clearly delineated. - They don't always use proper shell quoting. Instead, they blindly enclose all command-line arguments in double quotes. Changes ------- D154984 plus this patch converts the above example into: ``` Exit Code: 2 Command Output (stdout): -- # RUN: at line 1 echo hello | FileCheck bogus-file.txt && echo success # executed command: echo hello # .---command stdout------------ # | hello # `----------------------------- # executed command: FileCheck bogus-file.txt # .---command stderr------------ # | Could not open check file 'bogus-file.txt': No such file or directory # `----------------------------- # error: command failed with exit status: 2 -- ``` Thus, this patch addresses the above issues as follows: - The entire execution trace can be copied and pasted in bulk to a terminal for correct execution of the RUN lines, which are printed intact as they appeared in the original RUN lines except lit substitutions are expanded. Everything else in the execution trace appears in shell comments so it has no effect in a terminal. - Each of the RUN line's commands is repeated (in shell comments) as it executes to show (1) that the command actually executed (e.g., `echo success` above didn't) and (2) what stdout, stderr, non-zero exit status, and output files are associated with the command, if any. Shell quoting in the command is now correct and minimal but is not necessarily the original shell quoting from the RUN line. - The start and end of the contents of stdout, stderr, or an output file is now delineated clearly in the trace. To help produce some of the above output, this patch extends lit's internal shell with a built-in `@echo` command. It's like `echo` except lit suppresses the normal execution trace for `@echo` and just prints its stdout directly. For now, `@echo` isn't documented for use in lit tests. Without this patch, libcxx's custom lit test format tries to parse the stdout from `lit.TestRunner.executeScriptInternal` (which runs lit's internal shell) to extract the stdout and stderr produced by shell commands, and that parse no longer works after the above changes. This patch makes a small adjustment to `lit.TestRunner.executeScriptInternal` so libcxx can just request stdout and stderr without an execution trace. (As a minor drive-by fix that came up in testing: lit's internal `not` command now always produces a numeric exit status and never `True`.) Caveat ------ This patch only makes the above changes for lit's internal shell. In most cases, we do not know how to force external shells (e.g., bash, sh, window's `cmd`) to produce execution traces in the manner we want. To configure a test suite to use lit's internal shell (which is usually better for test portability than external shells anyway), add this to the test suite's `lit.cfg` or other configuration file: ``` config.test_format = lit.formats.ShTest(execute_external=False) ``` Reviewed By: MaskRay, awarzynski Differential Revision: https://reviews.llvm.org/D156954
2023-09-19[lit] Drop "Script:", make -v and -a imply -vvJoel E. Denny1-10/+9
This patch and D156954 were discussed in <https://discourse.llvm.org/t/rfc-improving-lits-debug-output/72839>. **Motivation**: -a shows output from all tests, and -v shows output from just failed tests. Without this patch, that output from each test includes a section called "Script:", which includes all shell commands that lit has computed from RUN directives and will attempt to run for that test. The effect of -vv (which also implies -v if neither -a or -v is specified) is to extend that output with shell commands as they are executing so you can easily see which one failed. For example, when using lit's internal shell and -vv: ``` Script: -- : 'RUN: at line 1'; echo hello world : 'RUN: at line 2'; 3c40 hello world : 'RUN: at line 3'; echo hello world -- Exit Code: 127 Command Output (stdout): -- $ ":" "RUN: at line 1" $ "echo" "hello" "world" hello world $ ":" "RUN: at line 2" $ "3c40" "hello" "world" '3c40': command not found error: command failed with exit status: 127 -- ``` Notice that all shell commands that actually execute appear in the output twice, once for "Script:" and once for -vv. Especially for tests with many RUN directives, the result is noisy. When searching through the output for a particular shell command, it is easy to get lost and mistake shell commands under "Script:" for shell commands that actually executed. **Change**: With this patch, a test's output changes in two ways. First, the "Script:" section is never shown. Second, omitting -vv no longer disables printing of shell commands as they execute. That is, -a and -v imply -vv, and so -vv is deprecated as it is just an alias for -v. **Secondary motivation**: We are also working to introduce a PYTHON directive, which can appear between RUN directives. How should PYTHON directives be represented in the "Script:" section, which has previously been just a shell script? We could probably think of something, but adding info about PYTHON directive execution in the -vv trace seems more straight-forward and more useful. (This patch also removes a confusing point in the -vv documentation: at least when using bash as an external shell, -vv echoes commands to the shell's stderr not stdout.) Reviewed By: awarzynski, Endill, ldionne, MaskRay Differential Revision: https://reviews.llvm.org/D154984
2023-09-07Revert "[lit] Drop "Script:", make -v and -a imply -vv"Joel E. Denny1-9/+10
This reverts commit 09b6e457d91ce84088e6e21783913e5f1e5bd227. The reason for the revert is discussed at: https://discourse.llvm.org/t/rfc-improving-lits-debug-output/72839/52
2023-08-29[lit] Drop "Script:", make -v and -a imply -vvJoel E. Denny1-10/+9
This patch and D156954 were discussed in <https://discourse.llvm.org/t/rfc-improving-lits-debug-output/72839>. **Motivation**: -a shows output from all tests, and -v shows output from just failed tests. Without this patch, that output from each test includes a section called "Script:", which includes all shell commands that lit has computed from RUN directives and will attempt to run for that test. The effect of -vv (which also implies -v if neither -a or -v is specified) is to extend that output with shell commands as they are executing so you can easily see which one failed. For example, when using lit's internal shell and -vv: ``` Script: -- : 'RUN: at line 1'; echo hello world : 'RUN: at line 2'; 3c40 hello world : 'RUN: at line 3'; echo hello world -- Exit Code: 127 Command Output (stdout): -- $ ":" "RUN: at line 1" $ "echo" "hello" "world" hello world $ ":" "RUN: at line 2" $ "3c40" "hello" "world" '3c40': command not found error: command failed with exit status: 127 -- ``` Notice that all shell commands that actually execute appear in the output twice, once for "Script:" and once for -vv. Especially for tests with many RUN directives, the result is noisy. When searching through the output for a particular shell command, it is easy to get lost and mistake shell commands under "Script:" for shell commands that actually executed. **Change**: With this patch, a test's output changes in two ways. First, the "Script:" section is never shown. Second, omitting -vv no longer disables printing of shell commands as they execute. That is, -a and -v imply -vv, and so -vv is deprecated as it is just an alias for -v. **Secondary motivation**: We are also working to introduce a PYTHON directive, which can appear between RUN directives. How should PYTHON directives be represented in the "Script:" section, which has previously been just a shell script? We could probably think of something, but adding info about PYTHON directive execution in the -vv trace seems more straight-forward and more useful. (This patch also removes a confusing point in the -vv documentation: at least when using bash as an external shell, -vv echoes commands to the shell's stderr not stdout.) Reviewed By: awarzynski, Endill, ldionne, MaskRay Differential Revision: https://reviews.llvm.org/D154984
2023-08-01[lit][clang] Avoid realpath on Windows due to MAX_PATH limitationsSaleem Abdulrasool1-0/+10
Running lit tests on Windows can fail because its use of `os.path.realpath` expands substitute drives, which are used to keep paths short and avoid hitting MAX_PATH limitations. Changes lit logic to: Use `os.path.abspath` on Windows, where `MAX_PATH` is a concern that we can work around using substitute drives, which `os.path.realpath` would resolve. Use `os.path.realpath` on Unix, where the current directory always has symlinks resolved, so it is impossible to preserve symlinks in the presence of relative paths, and so we must make sure that all code paths use real paths. Also updates clang's `FileManager::getCanonicalName` and `ExtractAPI` code to avoid resolving substitute drives (i.e. resolving to a path under a different root). How tested: built with `-DLLVM_ENABLE_PROJECTS=clang` and built `check-all` on both Windows Differential Revision: https://reviews.llvm.org/D154130 Reviewed By: @benlangmuir Patch by Tristan Labelle <tristan@thebrowser.company>!
2023-07-26[LIT] Added an option to llvm-lit to emit the necessary test coverage data, ↵Shivam Gupta1-0/+6
divided per test case This patch is the first part of https://llvm.org/OpenProjects.html#llvm_patch_coverage. We have first define a new variable LLVM_TEST_COVERAGE which when set, pass --per-test-coverage option to llvm-lit which will help in setting a unique value to LLVM_PROFILE_FILE for each RUN. So for example coverage data for test case llvm/test/Analysis/AliasSet/memtransfer.ll will be emitted as build/test/Analysis/AliasSet/memtransfer0.profraw Reviewed By: hnrklssn Differential Revision: https://reviews.llvm.org/D154280
2023-07-21Revert "[LIT] Added an option to llvm-lit to emit the necessary test ↵Shivam Gupta1-6/+0
coverage data, divided per test case" This reverts commit d8e26bccb3016d255298b7db78fe8bf05dd880b2. Test case are meant to run only when LLVM_INDIVIDUAL_TEST_COVERAGE is set.
2023-07-21[LIT] Added an option to llvm-lit to emit the necessary test coverage data, ↵Shivam Gupta1-0/+6
divided per test case This patch is the first part of https://llvm.org/OpenProjects.html#llvm_patch_coverage. We have first define a new variable LLVM_TEST_COVERAGE which when set, pass --emit-coverage option to llvm-lit which will help in setting a unique value to LLVM_PROFILE_FILE for each RUN. So for example coverage data for test case llvm/test/Analysis/AliasSet/memtransfer.ll will be emitted as build/test/Analysis/AliasSet/memtransfer.profraw Reviewed By: hnrklssn Differential Revision: https://reviews.llvm.org/D154280
2023-07-17[lit] Remove the --no-indirectly-run-check optionLouis Dionne1-8/+2
This option had originally been added in D83069 to allow disabling the check that something is going to get run at all when a specific test name is used on the command-line. Since we now use getTestsForPath() (from D151664) to get the tests to run for a specific path, we don't need a specific check for this anymore -- Lit will produce the same complaint it would produce if you provided a directory with no tests. If one needs to run a specific test on the command-line and the Lit configuration would normally not include that test, the configuration should be set up as a "standalone" configuration or it should be fixed to allow for that test to be found (i.e. probably fix the allowed test suffixes). Differential Revision: https://reviews.llvm.org/D153967
2023-02-09[Docs] Clarify behavior of llvm-lit -vvJay Foad1-1/+1
Differential Revision: https://reviews.llvm.org/D143586
2022-09-21[lit] Implement DEFINE and REDEFINE directivesJoel E. Denny1-8/+0
These directives define per-test lit substitutions. The concept was discussed at <https://discourse.llvm.org/t/iterating-lit-run-lines/62596/10>. For example, the following directives can be inserted into a test file to define `%{cflags}` and `%{fcflags}` substitutions with empty initial values, which serve as the parameters of another newly defined `%{check}` substitution: ``` // DEFINE: %{cflags} = // DEFINE: %{fcflags} = // DEFINE: %{check} = %clang_cc1 %{cflags} -emit-llvm -o - %s | \ // DEFINE: FileCheck %{fcflags} %s ``` The following directives then redefine the parameters before each use of `%{check}`: ``` // REDEFINE: %{cflags} = -foo // REDEFINE: %{fcflags} = -check-prefix=FOO // RUN: %{check} // REDEFINE: %{cflags} = -bar // REDEFINE: %{fcflags} = -check-prefix=BAR // RUN: %{check} ``` Of course, `%{check}` would typically be more elaborate, increasing the benefit of the reuse. One issue is that the strings `DEFINE:` and `REDEFINE:` already appear in 5 tests. This patch adjusts those tests not to use those strings. Our prediction is that, in the vast majority of cases, if a test author mistakenly uses one of those strings for another purpose, the text appearing after the string will not happen to have the syntax required for these directives. Thus, the test author will discover the mistake immediately when lit reports the syntax error. This patch also expands the documentation on existing lit substitution behavior. Reviewed By: jhenderson, MaskRay, awarzynski Differential Revision: https://reviews.llvm.org/D132513
2022-08-29[docs] Expand lit documentation on response filesPetr Hosek1-1/+4
This was requested in https://reviews.llvm.org/D132437.
2022-08-24[lit] Support reading arguments from a filePetr Hosek1-0/+3
This allows reading arguments from file using the response file syntax. We would like to use this in the LLVM build to pass test suites from subbuilds. Differential Revision: https://reviews.llvm.org/D132437
2022-03-14[test] Add lit helper for windows pathsKeith Smiley1-0/+3
This adds 2 new lit helpers `%{fs-src-root}` and `%{fs-sep}`, these allow writing tests that correctly handle slashes on Windows. In the case of tests like clang/test/CodeGen/debug-prefix-map.c, these are unable to correctly test behavior on both platforms, unless they fork and add OS requirements, because the relevant logic hits host specific codepaths like checking if paths are absolute. Differential Revision: https://reviews.llvm.org/D111457
2021-08-27[llvm] [lit] Support forcing lexical test orderMichał Górny1-2/+17
Add a new --order option to choose between available test orders: the default "smart" order, predictable "lexical" order or "random" order. Default to using lexical order and one job in the lit test suite. Differential Revision: https://reviews.llvm.org/D107695
2021-07-16[lit] Add --xfail-not/LIT_XFAIL_NOTJoel E. Denny1-0/+9
For example, I need this lately in my CI config: LIT_XFAIL_NOT='libomptarget :: nvptx64-nvidia-cuda :: unified_shared_memory/api.c' That test specifies an XFAIL directive, but I get an XPASS result. Reviewed By: jhenderson Differential Revision: https://reviews.llvm.org/D106022
2021-07-01[lit] Extend --xfail/LIT_XFAIL to take full test nameJoel E. Denny1-0/+25
The new documentation entry gives an example use case from libomptarget. Reviewed By: yln, jhenderson, davezarzycki Differential Revision: https://reviews.llvm.org/D105208
2021-03-16[lit] Sort test start times based on prior test timing dataDavid Zarzycki1-14/+13
Lit as it exists today has three hacks that allow users to run tests earlier: 1) An entire test suite can set the `is_early` boolean. 2) A very recently introduced "early_tests" feature. 3) The `--incremental` flag forces failing tests to run first. All of these approaches have problems. 1) The `is_early` feature was until very recently undocumented. Nevertheless it still lacks testing and is a imprecise way of optimizing test starting times. 2) The `early_tests` feature requires manual updates and doesn't scale. 3) `--incremental` is undocumented, untested, and it requires modifying the *source* file system by "touching" the file. This "touch" based approach is arguably a hack because it confuses editors (because it looks like the test was modified behind the back of the editor) and "touching" the test source file doesn't work if the test suite is read only from the perspective of `lit` (via advanced filesystem/build tricks). This patch attempts to simplify and address all of the above problems. This patch formalizes, documents, tests, and defaults lit to recording the execution time of tests and then reordering all tests during the next execution. By reordering the tests, high core count machines run faster, sometimes significantly so. This patch also always runs failing tests first, which is a positive user experience win for those that didn't know about the hidden `--incremental` flag. Finally, if users want, they can _optionally_ commit the test timing data (or a subset thereof) back to the repository to accelerate bots and first-time runs of the test suite. Reviewed By: jhenderson, yln Differential Revision: https://reviews.llvm.org/D98179
2021-02-24[lit] Add --ignore-failJoel E. Denny1-0/+4
For some build configurations, `check-all` calls lit multiple times to run multiple lit test suites. Most recently, I've found this to be true when configuring openmp as part of `LLVM_ENABLE_RUNTIMES`, but this is not the first time. If one test suite fails, none of the remaining test suites run, so you cannot determine if your patch has broken them. It can then be frustrating to try to determine which `check-` targets will run the remaining tests without getting stuck on the failing tests. When such cases arise, it is probably best to adjust the cmake configuration for `check-all` to run all test suites as part of one lit invocation. Because that fix will likely not be implemented and land immediately, this patch introduces `--ignore-fail` to serve as a workaround for developers trying to see test results until it does land: ``` $ LIT_OPTS=--ignore-fail ninja check-all ``` One problem with `--ignore-fail` is that it makes it challenging to detect test failures in a script, perhaps in CI. This problem should serve as motivation to actually fix the cmake configuration instead of continuing to use `--ignore-fail` indefinitely. Reviewed By: jhenderson, thopre Differential Revision: https://reviews.llvm.org/D96371
2021-02-20[lit] Add --xfail and --filter-out (inverse of --filter)David Zarzycki1-0/+15
In semi-automated environments, XFAILing or filtering out known regressions without actually committing changes or temporarily modifying the test suite can be quite useful. Reviewed By: yln Differential Revision: https://reviews.llvm.org/D96662
2021-02-17[lit] Add "early_tests" config optionDavid Zarzycki1-0/+7
With enough cores, the slowest tests can significantly change the total testing time if they happen to run late. With this change, a test suite can improve performance (for high-end systems) by listing just a few of the slowest tests up front. Reviewed By: jdenny, jhenderson Differential Revision: https://reviews.llvm.org/D96594
2021-02-17Add lit config for dir with standalone testsThomas Preud'homme1-0/+10
Some test systems do not use lit for test discovery but only for its substitution and test selection because they use another way of managing test collections, e.g. CTest. This forces those tests to be invoked with lit --no-indirectly-run-check. When a mix of lit version is in use, it requires to detect the availability of that option. This commit provides a new config option standalone_tests to signal a directory made of tests meant to run as standalone. When this option is set, lit skips test discovery and the indirectly run check. It also adds the missing documentation for --no-indirectly-run-check. Reviewed By: jdenny Differential Revision: https://reviews.llvm.org/D94766
2020-08-06fix doc typo to cycle botsNico Weber1-1/+1
2020-07-28[docs] [lit] Add a more helpful description for lit.py's -s flag.Varun Gandhi1-0/+1
Reviewed By: yln Differential Revision: https://reviews.llvm.org/D82808
2020-06-27[LIT] Correcting max-failures option in lit documentation.Mike Edwards1-4/+3
2020-06-06[LIT] NFC adding max-failures option to lit documentation.Mike Edwards1-0/+7
Differential Revision: https://reviews.llvm.org/D81337
2020-04-25[docs] Fix :option: linksJon Roelofs1-2/+2
2020-04-22[docs] Fix :option: linksJon Roelofs1-2/+2
2020-04-22[docs] Document lit's --timeout=N flagJon Roelofs1-4/+12
2020-04-06[lit] Move the recursiveExpansionLimit setting to TestingConfigLouis Dionne1-4/+4
The LitConfig is shared across the whole test suite. However, since enabling recursive expansion can be a breaking change for some test suites, it's important to confine the setting to test suites that enable it explicitly. Note that other issues were raised with the way recursiveExpansionLimit operates. However, this commit simply moves the setting to the right place -- the mechanism by which it works can be improved independently. Differential Revision: https://reviews.llvm.org/D77415
2020-03-27[lit] Recursively expand substitutionsLouis Dionne1-4/+13
This allows defining substitutions in terms of other substitutions. For example, a %build substitution could be defined in terms of a %cxx substitution as '%cxx %s -o %t.exe' and the script would be properly expanded. Differential Revision: https://reviews.llvm.org/D76178
2020-03-24[lit] NFC: Document missing result codesLouis Dionne1-1/+11
These result codes already exist, but they were not documented. I assume this is an oversight when adding these result codes.
2020-01-22[llvm] NFC: fix trivial typos in documentsKazuaki Ishizaki1-1/+1
Reviewers: hans, Jim Reviewed By: Jim Subscribers: jvesely, nhaehnle, mgorny, arphaman, bmahjour, kerbowa, llvm-commits Tags: #llvm Differential Revision: https://reviews.llvm.org/D73017
2019-12-04[lit] Document the undocumented pre-defined substitutionsDaniel Sanders1-11/+32