aboutsummaryrefslogtreecommitdiff
path: root/llvm/utils/lit
AgeCommit message (Collapse)AuthorFilesLines
2016-09-05[lit] Downgrade error to warning on gtest crashes during discovery.Ahmed Bougacha1-2/+2
Lots of unittests started failing under asan after r280455. It seems they've been failing for a long time, but lit silently ignored them. Downgrade the error so we can figure out what is going on. Filed http://llvm.org/PR30285. llvm-svn: 280674
2016-09-05lit/util.py: Another fix for py3.NAKAMURA Takumi1-0/+2
'str' object has no attribute 'decode'. llvm-svn: 280641
2016-09-03Make lit/util.py py3-compatible.NAKAMURA Takumi1-1/+1
llvm-svn: 280579
2016-09-02lit: print process output, if getting the list of google-tests failed.Ivan Krasin2-6/+11
Summary: This is a follow up to r280455, where a check for the process exit code was introduced. Some ASAN bots throw this error now, but it's impossible to understand what's wrong with them, and the issue is not reproducible. Reviewers: vitalybuka Differential Revision: https://reviews.llvm.org/D24210 llvm-svn: 280550
2016-09-02Quick fix to make LIT_PRESERVES_TMP work againReid Kleckner1-1/+2
llvm-svn: 280502
2016-09-02[lit] Clean up temporary files created by testsReid Kleckner2-11/+28
Do this by creating a temp directory in the normal system temp directory, and cleaning it up on exit. It is still possible for this temp directory to leak if Python exits abnormally, but this is probably good enough for now. Fixes PR18335 llvm-svn: 280501
2016-09-02[lit] Fail testing if a googletest executable crashes during test discoveryGreg Parker2-2/+6
googletest formatted tests are discovered by running the test executable. Previously testing would silently succeed if the test executable crashed during the discovery process. Now testing fails with "error: unable to discover google-tests ..." if the test executable exits with a non-zero status. llvm-svn: 280455
2016-09-01[lit] Use multiprocessing by default on WindowsReid Kleckner1-6/+3
Apparently nobody evaluated multiprocessing on Windows since Daniel enabled multiprocessing on Unix in r193279. It works so far as I can tell. Today this is worth about an 8x speedup (631.29s to 73.25s) on my 24 core Windows machine. Hopefully this will improve Windows buildbot cycle time, where currently it takes more time to run check-all than it does to self-host with assertions enabled: http://lab.llvm.org:8011/builders/clang-x86-windows-msvc2015/builds/20 build stage 2 ninja all ( 28 mins, 22 secs ) ninja check 2 stage 2 ( 37 mins, 38 secs ) llvm-svn: 280382
2016-07-21[lit] Use full config path in diagnostics.Daniel Dunbar1-1/+1
- This allows tools like emacs to automatically find the config file path when you step through errors. - Patch by Dave Abrahams. llvm-svn: 276357
2016-07-21[lit] Bump version number.Daniel Dunbar1-2/+2
llvm-svn: 276353
2016-06-26[lit] Add SANITIZER_IGNORE_CVE_2016_2143 to pass_vars.Marcin Koscielnicki1-1/+2
This variable is used by ASan (and other sanitizers in the future) on s390x-linux to override a check for CVE-2016-2143 in the running kernel (see revision 267747 on compiler-rt). Since the check simply checks if the kernel version is in a whitelist of known-good versions, it may miss distribution kernels, or manually-patched kernels - hence the need for this variable. To enable running the ASan testsuite on such kernels, this variable should be passed from the environment down to the testcases. Differential Revision: http://reviews.llvm.org/D19888 llvm-svn: 273825
2016-06-10[lit] Only gather redirected files for command failures.Daniel Dunbar1-10/+11
- The intended use of this was just in diagnostics, so we shouldn't pay the cost of reading these all the time. - This will avoid including the full output of each command in tests which fail, but the most important use case for this was to gather the output of the specific command which failed. llvm-svn: 272365
2016-06-09Revert "[lit] Use os.devnull instead of named temp files"Vedant Kumar1-5/+19
This reverts commit r272290. It breaks a test that depends on being able to seek the /dev/null equivalent on Windows: http://bb.pgr.jp/builders/ninja-clang-x64-mingw64-RA/builds/11360 llvm-svn: 272293
2016-06-09[lit] Use os.devnull instead of named temp filesVedant Kumar1-19/+5
Use os.devnull instead of tempfiles when substituting '/dev/null' on Windows machines. This should make the bots just a bit speedier. Thanks to Yunzhong Gao for testing this patch on Windows! Differential Revision: http://reviews.llvm.org/D20549 llvm-svn: 272290
2016-06-08[lit] Ensure we get bytes when reading redirected output files.Daniel Dunbar1-1/+1
llvm-svn: 272147
2016-06-07[lit] Ignore errors when decoding redirected output.Daniel Dunbar1-1/+1
llvm-svn: 272066
2016-06-07[lit] Fix an uninitialized var on Windows.Daniel Dunbar1-1/+2
llvm-svn: 272052
2016-06-07[utils/lit] Show available_features with --show-suites.Daniel Dunbar1-0/+3
llvm-svn: 272022
2016-06-07[lit] Improve logging with file redirection.Daniel Dunbar4-9/+39
- This will cause lit to automatically include the first 1K of data in redirected output files when a command fails (previously if the command failed, but the main point of the test was, say, a `FileCheck` later on, then the log wasn't helpful in showing why the command failed). llvm-svn: 272021
2016-06-02[lit] Improve readability of failing scripts.Daniel Dunbar6-14/+69
- This only applies to scripts executed by the _internal_ shell script interpreter. - This patch reworks the log to look more like a shell transcript, and be less verbose (but in the interest of calling attention to the important parts). Here is an example of the new format, for commands with/without failures and with/without output: ``` $ true $ echo hi hi $ false note: command had no output on stdout or stderr error: command failed with exit status 1 ``` llvm-svn: 271610
2016-06-02[lit] Factor out a helper for shell command results.Daniel Dunbar1-8/+22
llvm-svn: 271608
2016-06-02Add "REQUIRES-ANY" feature testEric Fiselier4-3/+19
Summary: This patch adds a "REQUIRES-ANY" feature test that is disjunctive. This marks a test as `UNSUPPORTED` if none of the specified features are available. Libc++ has the need to write feature test such as `// REQUIRES-ANY: c++98, c++03` when testing of behavior that is specific to older dialects but has since changed. Reviewers: rnk, ddunbar Subscribers: ddunbar, probinson, llvm-commits, cfe-commits Differential Revision: http://reviews.llvm.org/D20757 llvm-svn: 271468
2016-04-30[lit] Add %:[STpst] to represent paths without colons on Windows.Rui Ueyama1-0/+18
Summary: We need these variables to concatenate two absolute paths to construct a valid path. Currently, %t\%t is, for example, expanded to C:\foo\C:\foo, which is not a valid path because ":" is not a valid path character on Windows. With this patch, %t will be expanded to C\foo. Differential Revision: http://reviews.llvm.org/D19757 llvm-svn: 268168
2016-03-31lit: python3 compatibility fixMatthias Braun1-1/+8
llvm-svn: 265070
2016-03-31[lit][googletest] Handle upstream gtest outputDaniel Dunbar4-0/+67
Summary: Upstream googletest prints "Running main() from gtest_main.cc" to stdout prior to running tests. LLVM removed that print statement in r61540. If a user were to use lit to run tests that use upstream googletest, however, lit reports "Running main()" as an invalid test name. To avoid such a failure, add an extra conditional to `formats/googletest.py`. Also add tests to demonstrate the modified behavior. Reviewers: abdulras, ddunbar Subscribers: ddunbar, llvm-commits, kastiglione Differential Revision: http://reviews.llvm.org/D18606 llvm-svn: 265034
2016-03-17[lit] Enqueue tests on a separate thread to not hit limits on parallel queuesFilipe Cabecinhas1-2/+14
Summary: The multiprocessing.Queue.put() call can hang if we try queueing all the tests before starting to take them out of the queue. The current implementation hangs if tests exceed 2^^15, on Mac OS X. This might happen with a ninja check-all if one has a bunch of llvm projects. Reviewers: delcypher, bkramer Subscribers: llvm-commits Differential Revision: http://reviews.llvm.org/D17609 llvm-svn: 263731
2016-03-12[lit] Hack lit to allow a test suite to request that it is run "early".Chandler Carruth3-2/+14
This lets us for example start running the unit test suite early. For 'check-llvm' on my machine, this drops the tim e from 44s to 32s!!!!! It's pretty ugly. I barely know how to write Python, so feel free to just tell me how I should write it instead. =D Thanks to Filipe and others for help. Differential Revision: http://reviews.llvm.org/D18089 llvm-svn: 263329
2016-02-04The canonical way to XFAIL a test for all targets is XFAIL: *, not XFAIL:Benjamin Kramer1-1/+1
Fix the lit bug that enabled this "feature" (empty triple is substring of all possible target triples) and change the two outliers to use the documented * syntax. llvm-svn: 259799
2016-01-27Add "/dev/tty" as a special file name for lit tests.Yunzhong Gao1-0/+4
If a lit test has a RUN line that includes a redirection to "/dev/tty", the redirection goes to the special device file corresponding to the console. It is /dev/tty on UNIX-like systems and "CON" on Windows. This patch is needed to implement a test like PR25717 (caused by the size limit of the Windows system call WriteConsole() prior to Windows 8) where the test only breaks when outputing to the console and won't fail if using a pipe. llvm-svn: 258898
2016-01-13[lit] Fix handling of per test timeout when the installed psutil versionDan Liew1-1/+8
is < ``2.0``. Older versions of psutil (e.g. ``1.2.1`` which is the version shipped with Ubuntu 14.04) use a different API for retrieving the child processes. To handle this try the new API first and if that fails try the old API. llvm-svn: 257616
2016-01-09Revert "Teach the CMake build system to run lit's test suite. These can be run"Dan Liew3-50/+2
This reverts r257221. This caused several build bot failures * It looks like some of the tests don't work correctly under Windows * It looks like the lit per test timeout tests fail So I'm reverting for now. Once the above failures are fixed running lit's tests can be enabled again. llvm-svn: 257268
2016-01-08Teach the CMake build system to run lit's test suite. These can be runDan Liew3-2/+50
directy with ``make check-lit`` and are run as part of ``make check-all``. In principle we should run lit's testsuite before testing LLVM using lit so that any problems with lit get discovered before testing LLVM so we can bail out early. However this implementation (``check-all`` runs all tests together) seemed simpler and will still report failing lit tests. Note that the tests and the configured ``lit.site.cfg`` have to be copied into the build directory to avoid polluting the source tree. llvm-svn: 257221
2015-12-27[lit] Implement support of per test timeout in lit.Dan Liew17-32/+585
This should work with ShTest (executed externally or internally) and GTest test formats. To set the timeout a new option ``--timeout=`` has been added which specifies the maximum run time of an individual test in seconds. By default this 0 which causes no timeout to be enforced. The timeout can also be set from a lit configuration file by modifying the ``lit_config.maxIndividualTestTime`` property. To implement a timeout we now require the psutil Python module if a timeout is requested. This dependency is confined to the newly added ``lit.util.killProcessAndChildren()``. A note has been added into the TODO document describing how we can remove the dependency on the ``pustil`` module in the future. It would be nice to remove this immediately but that is a lot more work and Daniel Dunbar believes it is better that we get a working implementation first and then improve it. To avoid breaking the existing behaviour the psutil module will not be imported if no timeout is requested. The included testcases are derived from test cases provided by Jonathan Roelofs which were in an previous attempt to add a per test timeout to lit (http://reviews.llvm.org/D6584). Thanks Jonathan! Reviewers: ddunbar, jroelofs, cmatthews, MatzeB Subscribers: cmatthews, llvm-commits Differential Revision: http://reviews.llvm.org/D14706 llvm-svn: 256471
2015-12-22lit: Limit number of processes on Windows to 32.Nico Weber1-1/+3
llvm-svn: 256291
2015-11-19[lit] Fix bug when using Python3 where a failing test would not showDan Liew1-1/+3
the script when running a ShTest with an external or internal shell. This bug is caused by use of the ``map`` function in Python 3 which returns an iterable (rather than a list in Python 2). After the iterable is exhausted it won't return any more output and consequently when ``_runShTest()`` tries to access the ``script`` which has already been iterated over it is empty. Converting to a list immediatley after calling ``map()`` fixes this. This fixes the ``tests/shtest-format.py`` test when running under Python3 which was previously failing. llvm-svn: 253556
2015-11-13[lit] Improve error message when lit fails to executable a command byDan Liew1-1/+1
showing the executable it tried to use. llvm-svn: 253032
2015-11-13[lit] Fix bug where ``lit.util.which()`` would return a directoryDan Liew1-1/+1
instead of executable if the argument was found inside a directory contained in PATH. An example where this could cause a problem is if there was a RUN line that ran the ``test`` command and if the user had a directory in their PATH that contained a directory called ``test/`` (that occured before ``/usr/bin/``). Lit would try to use the directory as the executable which would fail with the rather cryptic message. ``` Could not create process due to [Errno 13] Permission denied ``` llvm-svn: 253031
2015-11-11lit: Show all output with --show-all, even in combination with --succinctMatthias Braun1-0/+1
I missed an earlier exit for the --succinct case when I introduced the -a option. llvm-svn: 252698
2015-11-02lit: Add '-a' option to display commands+output of all testsMatthias Braun1-4/+9
The existing -v option only displays commands and outputs for failed tests, the newly introduced -a displays it for all executed tests. llvm-svn: 251806
2015-10-28lit/TestRunner.py: Factor variable subsitution into an own function; NFCIMatthias Braun1-25/+26
This is a clearer separation of concerns and makes it easier to reuse the function. llvm-svn: 251481
2015-10-28lit/TestRunner.py: Factor out Substitution construction; NFCMatthias Braun1-24/+27
This is a clearer separation of concerns and makes it easier to reuse the functions. llvm-svn: 251480
2015-10-28lit/TestRunner.py: Get execdir from test.getExecPath() instead of passing it ↵Matthias Braun1-5/+5
around; NFC llvm-svn: 251479
2015-10-28lit/TestRunner.py: Make parseIntegratedTestScriptCommands() keyword list a ↵Matthias Braun1-3/+3
parameter; NFC This allows the function to be easily reused and also simplifies the code as the keyword list is next to the keyword handling now. llvm-svn: 251478
2015-10-15Lit: Rework r249161; Move RLIMIT_NPROC to main.py.NAKAMURA Takumi2-22/+22
The message "raised the process limit..." prevented the progress bar. llvm-svn: 250420
2015-10-02[lit] Raise the default soft process limit when possibleHal Finkel1-0/+22
It is common to have a default soft process limit, at least on some families of Linux distributions, of 1024. This is normally more than enough, but if you have many cores, and you're running tests that create many threads, this can become a problem. My POWER7 development machine has 48 cores, and when running the lld regression tests, which often want to create up to 48 threads, I run into problems. lit, by default, will want to run 48 tests in parallel, and 48*48 < 1024, and so many tests fail like this: terminate called after throwing an instance of 'std::system_error' what(): Resource temporarily unavailable or lit fails like this when launching a test: OSError: [Errno 11] Resource temporarily unavailable lit can easily detect this situation and attempt to repair it before launching tests (by raising the soft process limit to something that will allow ncpus^2 threads to be created), and should do so to prevent spurious test failures. This is the follow-up to this thread: http://lists.llvm.org/pipermail/llvm-dev/2015-October/090942.html llvm-svn: 249161
2015-09-04Fix passed env var name in lit for Android tests.Evgeniy Stepanov1-1/+1
The variable is actually called ANDROID_SERIAL. This was not exercised on the bots until today. Should fix the sanitizer-x86_64-linux failures. llvm-svn: 246898
2015-09-02[lit] Add basic flaky test retry functionalityReid Kleckner3-2/+16
The plan is to use this for the sanitizer test suite on Windows. See PR24554 for more details on why we need this. Tested manually by injecting rand() into a sanitizer test and watching what it does. llvm-svn: 246704
2015-08-31Revert "[lit] Speculatively fix PR24554 by manually closing the process handle"Reid Kleckner1-3/+0
This reverts commit r245946. It didn't help the problem: http://lab.llvm.org:8011/builders/sanitizer-windows/builds/9179/steps/run%20tests/logs/stdio LINK : fatal error LNK1104: cannot open file llvm-svn: 246491
2015-08-28lit: Match UNSUPPORTED against target triple as we do for XFAIL.Peter Collingbourne1-0/+7
llvm-svn: 246343
2015-08-25[lit] Speculatively fix PR24554 by manually closing the process handleReid Kleckner1-0/+3
My theory is that somehow Python's refcounting and GC strategy isn't closing the subprocess handle in a timely fashion. This accesses the private '_handle' field of the Popen object, but I see no other way to do this. If this doesn't address the problem on the sanitizer-windows buildbot, we can revert this change. If it does, then let's keep the hack. llvm-svn: 245946
generated by cgit v1.1 at 2025-10-19 00:01:43 +0000