aboutsummaryrefslogtreecommitdiff
path: root/llvm/utils/lit
AgeCommit message (Collapse)AuthorFilesLines
2016-04-30[lit] Add %:[STpst] to represent paths without colons on Windows.Rui Ueyama1-0/+18
Summary: We need these variables to concatenate two absolute paths to construct a valid path. Currently, %t\%t is, for example, expanded to C:\foo\C:\foo, which is not a valid path because ":" is not a valid path character on Windows. With this patch, %t will be expanded to C\foo. Differential Revision: http://reviews.llvm.org/D19757 llvm-svn: 268168
2016-03-31lit: python3 compatibility fixMatthias Braun1-1/+8
llvm-svn: 265070
2016-03-31[lit][googletest] Handle upstream gtest outputDaniel Dunbar4-0/+67
Summary: Upstream googletest prints "Running main() from gtest_main.cc" to stdout prior to running tests. LLVM removed that print statement in r61540. If a user were to use lit to run tests that use upstream googletest, however, lit reports "Running main()" as an invalid test name. To avoid such a failure, add an extra conditional to `formats/googletest.py`. Also add tests to demonstrate the modified behavior. Reviewers: abdulras, ddunbar Subscribers: ddunbar, llvm-commits, kastiglione Differential Revision: http://reviews.llvm.org/D18606 llvm-svn: 265034
2016-03-17[lit] Enqueue tests on a separate thread to not hit limits on parallel queuesFilipe Cabecinhas1-2/+14
Summary: The multiprocessing.Queue.put() call can hang if we try queueing all the tests before starting to take them out of the queue. The current implementation hangs if tests exceed 2^^15, on Mac OS X. This might happen with a ninja check-all if one has a bunch of llvm projects. Reviewers: delcypher, bkramer Subscribers: llvm-commits Differential Revision: http://reviews.llvm.org/D17609 llvm-svn: 263731
2016-03-12[lit] Hack lit to allow a test suite to request that it is run "early".Chandler Carruth3-2/+14
This lets us for example start running the unit test suite early. For 'check-llvm' on my machine, this drops the tim e from 44s to 32s!!!!! It's pretty ugly. I barely know how to write Python, so feel free to just tell me how I should write it instead. =D Thanks to Filipe and others for help. Differential Revision: http://reviews.llvm.org/D18089 llvm-svn: 263329
2016-02-04The canonical way to XFAIL a test for all targets is XFAIL: *, not XFAIL:Benjamin Kramer1-1/+1
Fix the lit bug that enabled this "feature" (empty triple is substring of all possible target triples) and change the two outliers to use the documented * syntax. llvm-svn: 259799
2016-01-27Add "/dev/tty" as a special file name for lit tests.Yunzhong Gao1-0/+4
If a lit test has a RUN line that includes a redirection to "/dev/tty", the redirection goes to the special device file corresponding to the console. It is /dev/tty on UNIX-like systems and "CON" on Windows. This patch is needed to implement a test like PR25717 (caused by the size limit of the Windows system call WriteConsole() prior to Windows 8) where the test only breaks when outputing to the console and won't fail if using a pipe. llvm-svn: 258898
2016-01-13[lit] Fix handling of per test timeout when the installed psutil versionDan Liew1-1/+8
is < ``2.0``. Older versions of psutil (e.g. ``1.2.1`` which is the version shipped with Ubuntu 14.04) use a different API for retrieving the child processes. To handle this try the new API first and if that fails try the old API. llvm-svn: 257616
2016-01-09Revert "Teach the CMake build system to run lit's test suite. These can be run"Dan Liew3-50/+2
This reverts r257221. This caused several build bot failures * It looks like some of the tests don't work correctly under Windows * It looks like the lit per test timeout tests fail So I'm reverting for now. Once the above failures are fixed running lit's tests can be enabled again. llvm-svn: 257268
2016-01-08Teach the CMake build system to run lit's test suite. These can be runDan Liew3-2/+50
directy with ``make check-lit`` and are run as part of ``make check-all``. In principle we should run lit's testsuite before testing LLVM using lit so that any problems with lit get discovered before testing LLVM so we can bail out early. However this implementation (``check-all`` runs all tests together) seemed simpler and will still report failing lit tests. Note that the tests and the configured ``lit.site.cfg`` have to be copied into the build directory to avoid polluting the source tree. llvm-svn: 257221
2015-12-27[lit] Implement support of per test timeout in lit.Dan Liew17-32/+585
This should work with ShTest (executed externally or internally) and GTest test formats. To set the timeout a new option ``--timeout=`` has been added which specifies the maximum run time of an individual test in seconds. By default this 0 which causes no timeout to be enforced. The timeout can also be set from a lit configuration file by modifying the ``lit_config.maxIndividualTestTime`` property. To implement a timeout we now require the psutil Python module if a timeout is requested. This dependency is confined to the newly added ``lit.util.killProcessAndChildren()``. A note has been added into the TODO document describing how we can remove the dependency on the ``pustil`` module in the future. It would be nice to remove this immediately but that is a lot more work and Daniel Dunbar believes it is better that we get a working implementation first and then improve it. To avoid breaking the existing behaviour the psutil module will not be imported if no timeout is requested. The included testcases are derived from test cases provided by Jonathan Roelofs which were in an previous attempt to add a per test timeout to lit (http://reviews.llvm.org/D6584). Thanks Jonathan! Reviewers: ddunbar, jroelofs, cmatthews, MatzeB Subscribers: cmatthews, llvm-commits Differential Revision: http://reviews.llvm.org/D14706 llvm-svn: 256471
2015-12-22lit: Limit number of processes on Windows to 32.Nico Weber1-1/+3
llvm-svn: 256291
2015-11-19[lit] Fix bug when using Python3 where a failing test would not showDan Liew1-1/+3
the script when running a ShTest with an external or internal shell. This bug is caused by use of the ``map`` function in Python 3 which returns an iterable (rather than a list in Python 2). After the iterable is exhausted it won't return any more output and consequently when ``_runShTest()`` tries to access the ``script`` which has already been iterated over it is empty. Converting to a list immediatley after calling ``map()`` fixes this. This fixes the ``tests/shtest-format.py`` test when running under Python3 which was previously failing. llvm-svn: 253556
2015-11-13[lit] Improve error message when lit fails to executable a command byDan Liew1-1/+1
showing the executable it tried to use. llvm-svn: 253032
2015-11-13[lit] Fix bug where ``lit.util.which()`` would return a directoryDan Liew1-1/+1
instead of executable if the argument was found inside a directory contained in PATH. An example where this could cause a problem is if there was a RUN line that ran the ``test`` command and if the user had a directory in their PATH that contained a directory called ``test/`` (that occured before ``/usr/bin/``). Lit would try to use the directory as the executable which would fail with the rather cryptic message. ``` Could not create process due to [Errno 13] Permission denied ``` llvm-svn: 253031
2015-11-11lit: Show all output with --show-all, even in combination with --succinctMatthias Braun1-0/+1
I missed an earlier exit for the --succinct case when I introduced the -a option. llvm-svn: 252698
2015-11-02lit: Add '-a' option to display commands+output of all testsMatthias Braun1-4/+9
The existing -v option only displays commands and outputs for failed tests, the newly introduced -a displays it for all executed tests. llvm-svn: 251806
2015-10-28lit/TestRunner.py: Factor variable subsitution into an own function; NFCIMatthias Braun1-25/+26
This is a clearer separation of concerns and makes it easier to reuse the function. llvm-svn: 251481
2015-10-28lit/TestRunner.py: Factor out Substitution construction; NFCMatthias Braun1-24/+27
This is a clearer separation of concerns and makes it easier to reuse the functions. llvm-svn: 251480
2015-10-28lit/TestRunner.py: Get execdir from test.getExecPath() instead of passing it ↵Matthias Braun1-5/+5
around; NFC llvm-svn: 251479
2015-10-28lit/TestRunner.py: Make parseIntegratedTestScriptCommands() keyword list a ↵Matthias Braun1-3/+3
parameter; NFC This allows the function to be easily reused and also simplifies the code as the keyword list is next to the keyword handling now. llvm-svn: 251478
2015-10-15Lit: Rework r249161; Move RLIMIT_NPROC to main.py.NAKAMURA Takumi2-22/+22
The message "raised the process limit..." prevented the progress bar. llvm-svn: 250420
2015-10-02[lit] Raise the default soft process limit when possibleHal Finkel1-0/+22
It is common to have a default soft process limit, at least on some families of Linux distributions, of 1024. This is normally more than enough, but if you have many cores, and you're running tests that create many threads, this can become a problem. My POWER7 development machine has 48 cores, and when running the lld regression tests, which often want to create up to 48 threads, I run into problems. lit, by default, will want to run 48 tests in parallel, and 48*48 < 1024, and so many tests fail like this: terminate called after throwing an instance of 'std::system_error' what(): Resource temporarily unavailable or lit fails like this when launching a test: OSError: [Errno 11] Resource temporarily unavailable lit can easily detect this situation and attempt to repair it before launching tests (by raising the soft process limit to something that will allow ncpus^2 threads to be created), and should do so to prevent spurious test failures. This is the follow-up to this thread: http://lists.llvm.org/pipermail/llvm-dev/2015-October/090942.html llvm-svn: 249161
2015-09-04Fix passed env var name in lit for Android tests.Evgeniy Stepanov1-1/+1
The variable is actually called ANDROID_SERIAL. This was not exercised on the bots until today. Should fix the sanitizer-x86_64-linux failures. llvm-svn: 246898
2015-09-02[lit] Add basic flaky test retry functionalityReid Kleckner3-2/+16
The plan is to use this for the sanitizer test suite on Windows. See PR24554 for more details on why we need this. Tested manually by injecting rand() into a sanitizer test and watching what it does. llvm-svn: 246704
2015-08-31Revert "[lit] Speculatively fix PR24554 by manually closing the process handle"Reid Kleckner1-3/+0
This reverts commit r245946. It didn't help the problem: http://lab.llvm.org:8011/builders/sanitizer-windows/builds/9179/steps/run%20tests/logs/stdio LINK : fatal error LNK1104: cannot open file llvm-svn: 246491
2015-08-28lit: Match UNSUPPORTED against target triple as we do for XFAIL.Peter Collingbourne1-0/+7
llvm-svn: 246343
2015-08-25[lit] Speculatively fix PR24554 by manually closing the process handleReid Kleckner1-0/+3
My theory is that somehow Python's refcounting and GC strategy isn't closing the subprocess handle in a timely fashion. This accesses the private '_handle' field of the Popen object, but I see no other way to do this. If this doesn't address the problem on the sanitizer-windows buildbot, we can revert this change. If it does, then let's keep the hack. llvm-svn: 245946
2015-07-22[lit] Fix launching executables relative to the cwd after 'cd'Reid Kleckner1-1/+8
This was affecting test/asan/TestCases/Windows/coverage-basic.cc in compiler-rt. It does something like: cd %T/mydir %clang %s -o t.exe ./t.exe Previously, we'd end up looking for t.exe relative to the cwd of the lit process, not the cwd of the test. llvm-svn: 242941
2015-07-21[LIT] Fix discovery.py test once againEric Fiselier1-4/+4
llvm-svn: 242758
2015-07-20[lit] Implement 'env' in the internal shellReid Kleckner1-7/+28
The MSys 2 version of 'env' cannot be used to set 'TZ' in the environment due to some portability hacks in the process spawning compatibility layer[1]. This affects test/Object/archive-toc.test, which tries to set TZ in the environment. Other than that, this saves a subprocess invocation of a small unix utility, which is makes the tests faster. The internal shell does not support shell variable expansion, so this idiom in the ASan tests isn't supported yet: RUN: env ASAN_OPTIONS=$ASAN_OPTIONS:opt=1 ... [1] https://github.com/Alexpux/MSYS2-packages/issues/294 Differential Revision: http://reviews.llvm.org/D11350 llvm-svn: 242696
2015-07-19[LIT] Allow for executeCommand to take the stdin input.Eric Fiselier1-3/+3
Summary: This patch allows executeCommand to pass a string to the processes stdin. Reviewers: ddunbar, jroelofs Subscribers: llvm-commits Differential Revision: http://reviews.llvm.org/D11332 llvm-svn: 242631
2015-06-23Pass 2 more variables to lit tests.Evgeniy Stepanov1-1/+1
Pass ADB and ADB_SERIAL environment variables to lit tests. This would allow running Android tests in compiler-rt when there is more than one device attached to the host. llvm-svn: 240459
2015-06-13[LIT] Fix failing LIT testsEric Fiselier4-6/+8
Summary: I spend some time trying to get the LIT test suite passing. Here are the changes that I needed to make on my machine. I made the following changes for the following reasons. 1. google-test.py: The Google test format now checks for "[ PASSED ] 1 test." to check if a test passes. 2. discovery.py: The output appears in a different order on my machine than it did in the test. 3. unittest-adaptor.py: The output appears in a different order on my machine than it did in the test. 4. The classname is now formed differently in `getJUnitXML(...)`. I'm not sure what is causing the output order to differ in discovery.py and unittest-adaptor.py. Does anybody have any thoughts? Reviewers: ddunbar, danalbert, jroelofs Reviewed By: jroelofs Subscribers: llvm-commits Differential Revision: http://reviews.llvm.org/D9864 llvm-svn: 239663
2015-06-01lit: Allow configurations to restrict the set of tests to runArnold Schwaighofer2-1/+14
By setting limit_to_features to a non empty list of features a configuration can restrict the set of tests to run to only include tests that require a feature in this list. rdar://21082253 llvm-svn: 238766
2015-05-04Lit: Allow overriding llvm tool paths+arguments, make -D an alias for --paramMatthias Braun1-1/+1
These changes allow usages where you want to pass an additional commandline option to all invocations of a specific llvm tool. Example: > llvm-lit -Dllc=llc -enable-misched -verify-machineinstrs Differential Revision: http://reviews.llvm.org/D9487 llvm-svn: 236461
2015-04-07[lit] Allow disabling an entire gtest suite, as is done in tsanReid Kleckner1-1/+2
llvm-svn: 234336
2015-04-06[lit] Skip gtest names starting with DISABLED_Reid Kleckner1-0/+4
The sanitizer test suite uses this idiom to disable a test. Now that we actually check if a test ran after invoking it, we see that zero tests ran, and complain. Instead, ignore tests starting with DISABLED_ completely. Fixes the sanitizer test suite failures on Windows. llvm-svn: 234247
2015-04-06[lit] Fix running gtest type-parameterized tests on WindowsReid Kleckner1-4/+11
The '/' character in the test name of a type-parameterized test is not a path separator, and should not be '\' on Windows. We were passing a test name to --gtest_filter which found no tests, so the exit code was zero, indicating a passed test. This bug has been here since r84387 in 2009, when Jeff Yasskin added the original lit support for type-paratermized tests. Somewhere along the line some of the ValueMapTests started failing, but we can fix those separately. llvm-svn: 234242
2015-03-02lit: Add 'cd' support to the internal shell and port some testsReid Kleckner1-13/+39
The internal shell was already threading around a 'cwd' parameter. We just have to make it mutable so that we can update it as the test script executes. If the shell ever grows support for environment variable substitution, we could also implement support for export. llvm-svn: 231017
2015-02-04Don't warn or note if bash is missingReid Kleckner1-2/+0
We haven't needed bash on Windows to run the test suite for a long time now. Patch by Michael Edwards! llvm-svn: 228221
2015-02-01Fix some bashims. More information on https://wiki.ubuntu.com/DashAsBinSh. ↵Sylvestre Ledru1-1/+1
Reported initially on https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=772302 & https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=772301 llvm-svn: 227744
2015-01-21[lit] Format JSONMetricValue strings better.Eric Fiselier1-1/+2
llvm-svn: 226672
2015-01-14Don't set LD_PRELOAD to ''. It doesn't work on OpenBSD.Rafael Espindola1-1/+5
Patch by Brad Smith. llvm-svn: 225890
2015-01-12[LIT] Remove string decoding in gtest discovery code. lit.util.capture now ↵Eric Fiselier1-1/+0
does decoding. llvm-svn: 225693
2015-01-12[LIT] Decode string result in lit.util.captureEric Fiselier1-9/+9
Summary: I think this is probably a bug, but I'm putting this up for review just to be sure. I think that `lit.util.capture` should decode the resulting string in the same way `lit.util.executeCommand` does. Reviewers: ddunbar, EricWF Reviewed By: EricWF Subscribers: llvm-commits Differential Revision: http://reviews.llvm.org/D6769 llvm-svn: 225681
2014-12-23Pass LSAN_OPTIONS down so that it is possible to add suppressions.Rafael Espindola1-1/+2
llvm-svn: 224777
2014-12-20Split executeShTest into two parts so that it can be better leveraged by libc++Eric Fiselier1-13/+18
llvm-svn: 224672
2014-12-19[LIT] Add JSONMetricValue type to wrap types supported by the json encoder.Eric Fiselier1-0/+36
Summary: The following types can be encoded and decoded by the json library: `dict`, `list`, `tuple`, `str`, `unicode`, `int`, `long`, `float`, `bool`, `NoneType`. `JSONMetricValue` can be constructed with any of these types, and used as part of Test.Result. This patch also adds a toMetricValue function that converts a value into a MetricValue. Reviewers: ddunbar, EricWF Reviewed By: EricWF Subscribers: cfe-commits, llvm-commits Differential Revision: http://reviews.llvm.org/D6576 llvm-svn: 224628
2014-12-12Now that we require a newer python, use the new exception syntax.Rafael Espindola1-1/+1
llvm-svn: 224130
generated by cgit v1.1 at 2025-10-30 06:02:08 +0000