aboutsummaryrefslogtreecommitdiff
path: root/contrib/testsuite-management/validate_failures.py
AgeCommit message (Collapse)AuthorFilesLines
2023-06-17[contrib] validate_failures.py: Don't consider summary line in wrong placeThiago Jung Bauermann1-4/+14
When parsing a summary or manifest file, if we're not either after a tool line (e.g. "=== gdb tests ===") or before a summary line (e.g., "=== gdb Summary ===") then the current line can't be a valid result line so ignore it. This addresses a problem we're seeing when running the GDB testsuite in our CI environment where it produces a valid summary file, but then after the "=== gdb Summary ===" section it outputs a series of Tcl errors that match _VALID_TEST_RESULTS_REX and thus confuse the parsing logic: 05: 14:32 .sum file seems to be broken: tool="None", exp="None", summary_line="ERROR: -------------------------------------------" 05: 14:32 Traceback (most recent call last): 05: 14:32 File "/path/to/gcc/contrib/testsuite-management/validate_failures.py", line 706, in <module> 05: 14:32 retval = Main(sys.argv) 05: 14:32 File "/path/to/gcc/contrib/testsuite-management/validate_failures.py", line 697, in Main 05: 14:32 retval = CheckExpectedResults() 05: 14:32 File "/path/to/gcc/contrib/testsuite-management/validate_failures.py", line 572, in CheckExpectedResults 05: 14:32 actual = GetResults(sum_files) 05: 14:32 File "/path/to/gcc/contrib/testsuite-management/validate_failures.py", line 447, in GetResults 05: 14:32 build_results.update(ParseSummary(sum_fname)) 05: 14:32 File "/path/to/gcc/contrib/testsuite-management/validate_failures.py", line 389, in ParseSummary 05: 14:32 result = result_set.MakeTestResult(line, ordinal) 05: 14:32 File "/path/to/gcc/contrib/testsuite-management/validate_failures.py", line 236, in MakeTestResult 05: 14:32 return TestResult(summary_line, ordinal, 05: 14:32 File "/path/to/gcc/contrib/testsuite-management/validate_failures.py", line 148, in __init__ 05: 14:32 raise contrib/ChangeLog: * testsuite-management/validate_failures.py (IsInterestingResult): Add result_set argument and use it. Adjust callers.
2023-06-14[contrib] validate_failures.py: Ignore stray filesystem paths in resultsMaxim Kuvyrkov1-0/+15
This patch simplifies comparison of results that have filesystem paths. E.g., (assuming different values of <N>): <cut> Running /home/user/gcc-N/gcc/testsuite/gcc.target/aarch64/sve/acle/aarch64-sve-acle-asm.exp ... ERROR: tcl error sourcing /home/user/gcc-N/gcc/testsuite/gcc.target/aarch64/sve/acle/aarch64-sve-acle-asm.exp. </cut> We add "--srcpath <regex>", option, and set it by default to "[^ ]+/testsuite/", which works well for all components of the GNU Toolchain. We then remove substrings matching <regex> from paths of .exp files and from occasional "ERROR:" results. contrib/ChangeLog: * testsuite-management/validate_failures.py (TestResult,) (ParseManifestWorker, ParseSummary, Main): Handle new option "--srcpath <regex>".
2023-06-14[contrib] validate_failures.py: Add "--expiry_date YYYYMMDD" optionMaxim Kuvyrkov1-2/+22
This option sets "today" date to compare expiration entries against. Setting expiration date into the future allows re-detection of flaky tests and creating fresh entries for them before the current flaky entries expire. contrib/ChangeLog: * testsuite-management/validate_failures.py (TestResult): Update. (Main): Handle new option "--expiry_date YYYYMMDD".
2023-06-14[contrib] validate_failures.py: Add new option --invert_matchMaxim Kuvyrkov1-2/+32
This option is used to detect flaky tests that FAILed in the clean build (or manifest), but PASSed in the current build (or manifest). The option inverts output logic similar to what "-v/--invert-match" does for grep. contrib/ChangeLog: * testsuite-management/validate_failures.py (ResultSet.update,) (ResultSet.HasTestsuite): New methods. (GetResults): Update. (ParseSummary, CompareResults, PerformComparison, Main): Handle new option --invert_match.
2023-06-14[contrib] validate_failures.py: Improve error outputThiago Bauermann1-1/+4
- Print message in case of broken sum file error. - Print error messages to stderr. The script's stdout is, usually, redirected to a file, and error messages shouldn't go there. contrib/ChangeLog: * testsuite-management/validate_failures.py (TestResult): Improve error output.
2023-06-14[contrib] validate_failures.py: Support "$tool:" prefix in exp namesChristophe Lyon1-3/+5
This makes it easier to extract the $tool:$exp pair when iterating over failures/flaky tests, which, in turn, simplifies re-running testsuite parts that have unexpected failures or passes. contrib/ChangeLog: * testsuite-management/validate_failures.py (_EXP_LINE_FORMAT,) (_EXP_LINE_REX, ResultSet): Support "$tool:" prefix in exp names.
2023-06-14[contrib] validate_failures.py: Use exit code "2" to indicate regressionMaxim Kuvyrkov1-1/+1
... in the results. Python exits with code "1" on exceptions and internal errors, which we use to detect failure to parse results. contrib/ChangeLog: * testsuite-management/validate_failures.py (Main): Use exit code "2" to indicate regression.
2023-06-14[contrib] validate_failures.py: Be more stringent in parsing result linesMaxim Kuvyrkov1-4/+4
Before this patch we would identify malformed line "UNRESOLVEDTest run by tcwg-buildslave on Mon Aug 23 10:17:50 2021" as an interesting result, only to fail in TestResult:__init__ due to missing ":" after UNRESOLVED. This patch makes all places that parse result lines use a single compiled regex. contrib/ChangeLog: * testsuite-management/validate_failures.py (_VALID_TEST_RESULTS_REX): Update. (TestResult): Use _VALID_TEST_RESULTS_REX.
2023-06-14[contrib] validate_failures.py: Add more verbosity levelsMaxim Kuvyrkov1-36/+46
... to control validate_failures.py output contrib/ChangeLog: * testsuite-management/validate_failures.py: Add more verbosity levels.
2023-06-14[contrib] validate_failures.py: Simplify GetManifestPath()Maxim Kuvyrkov1-8/+5
... and don't require a valid build directory when no data from it is necessary. contrib/ChangeLog: * testsuite-management/validate_failures.py: Simplify GetManifestPath().
2023-06-14[contrib] validate_failures.py: Read in manifest when comparing build dirsMaxim Kuvyrkov1-3/+11
This allows comparison of two build directories with a manifest listing known flaky tests on the side. contrib/ChangeLog: * testsuite-management/validate_failures.py (GetResults): Update. (CompareBuilds): Read in manifest.
2023-06-14[contrib] validate_failures.py: Support expiry attributes in manifestsMaxim Kuvyrkov1-1/+9
contrib/ChangeLog: * testsuite-management/validate_failures.py (ParseManifestWorker): Support expiry attributes in manifests. (ParseSummary): Add a comment.
2023-06-14[contrib] validate_failures.py: Avoid testsuite aliasingMaxim Kuvyrkov1-22/+115
This patch adds tracking of current testsuite "tool" and "exp" to the processing of .sum files. This avoids aliasing between tests from different testsuites with same name+description. E.g., this is necessary for testsuite/c-c++-common, which is ran for both gcc and g++ "tools". This patch changes manifest format from ... <cut> FAIL: gcc_test FAIL: g++_test </cut> ... to ... <cut> === gcc tests === Running gcc/foo.exp ... FAIL: gcc_test === gcc Summary == === g++ tests === Running g++/bar.exp ... FAIL: g++_test === g++ Summary == </cut>. The new format uses same formatting as DejaGnu's .sum files to specify which "tool" and "exp" the test belongs to. contrib/ChangeLog: * testsuite-management/validate_failures.py: Avoid testsuite aliasing.
2023-01-16Update copyright years.Jakub Jelinek1-1/+1
2021-11-05contrib: testsuite-management: Update to be python3 compatibleBernhard Reutner-Fischer1-24/+20
contrib/ChangeLog: * testsuite-management/validate_failures.py: 2to3
2013-06-06validate_failures.py: Fix handling of summary lines with "|" characters or ↵Brooks Moses1-12/+17
empty description fields. * testsuite-management/validate_failures.py: Fix handling of summary lines with "|" characters or empty description fields. From-SVN: r199765
2013-04-18* validate_failures.py: Fix typo.Diego Novillo1-1/+1
From-SVN: r198074
2013-04-18Loosen check for build directory.Diego Novillo1-8/+11
* validate_failures.py: Loosen check for build directory. State what failed if we couldn't find the source tree or the target triplet. From-SVN: r198064
2013-02-06* testsuite-management/validate_failures.py: Fix typo.Diego Novillo1-1/+1
From-SVN: r195819
2013-02-06Fix validate_failures.py in standalone testing.Diego Novillo1-7/+21
When using validate_failures.py with --manifest and --results, we don't need a GCC build directory. This is useful when using the validator outside of the build tree. We were insisting on finding a valid build tree regardless of those options. Tested on x86_64. Committed to trunk. * testsuite-management/validate_failures.py: Update Copyright years. Request contributions not to use Python features newer than 2.4. (GetBuildData): If this is not a build directory, emit an error only if --results or --manifest are missing. From-SVN: r195817
2013-02-06validate_failures.py: Fix performance regressionBernhard Reutner-Fischer1-1/+2
2013-02-06 Bernhard Reutner-Fischer <aldot@gcc.gnu.org> * testsuite-management/validate_failures.py (IsInterestingResult): Fix performance regression From-SVN: r195811
2012-12-05validate_failures.py: also ignore .gitBernhard Reutner-Fischer1-7/+5
2012-12-01 Bernhard Reutner-Fischer <aldot@gcc.gnu.org> * testsuite-management/validate_failures.py (IsInterestingResult): Only strip line a second time if we did split. Rephrase return statement while at it. (CollectSumFiles): Also ignore .git directory. From-SVN: r194182
2012-12-03Fix stale use of old 'options' argument.Diego Novillo1-1/+1
An earlier patch had made the command line options a global variable _OPTIONS, but it had not renamed all the uses of the old options argument. * testsuite-management/validate_failures.py: Fix stale use of 'options'. From-SVN: r194092
2012-11-30validate_failures.py: Add support for @include, @remove directives in ↵Doug Evans1-5/+67
manifest files. * testsuite-management/validate_failures.py: Add support for @include, @remove directives in manifest files. From-SVN: r194008
2012-11-30validate_failures.py: Add function GetManifestPath.Doug Evans1-14/+25
* testsuite-management/validate_failures.py: Add function GetManifestPath. New global _MANIFEST_SUBDIR. From-SVN: r194007
2012-11-29validate_failures.py: Remove pass/fail indicator from result of GetBuildData.Doug Evans1-11/+4
* testsuite-management/validate_failures.py: Remove pass/fail indicator from result of GetBuildData. From-SVN: r193968
2012-11-29validate_failures.py: Store options in global variable _OPTIONS.Doug Evans1-28/+32
* testsuite-management/validate_failures.py: Store options in global variable _OPTIONS. From-SVN: r193967
2012-11-29validate_failures.py: Rename variable manifest_name to manifest_path everywhere.Doug Evans1-13/+13
* testsuite-management/validate_failures.py: Rename variable manifest_name to manifest_path everywhere. From-SVN: r193965
2012-11-29validate_failures.py (CompareBuilds): Pass options.results to GetSumFiles ↵Doug Evans1-1/+1
for clean build. * testsuite-management/validate_failures.py (CompareBuilds): Pass options.results to GetSumFiles for clean build. From-SVN: r193952
2012-11-28validate_failures.py: Use <target_alias> instead of <target>.Doug Evans1-3/+3
* testsuite-management/validate_failures.py: Use <target_alias> instead of <target>. Minor whitespace changes. From-SVN: r193904
2012-11-28validate_failures.py: Record ordinal with TestResult.Doug Evans1-3/+12
* testsuite-management/validate_failures.py: Record ordinal with TestResult. From-SVN: r193903
2012-11-02Add a new option --clean_build to validate_failures.pyDiego Novillo1-19/+47
This is useful when you have two builds of the same compiler. One with your changes. The other one, a clean build at the same revision. Instead of using a manifest file, --clean_build will compare the results it gather from the patched build against those it gathers from the clean build. Usage $ cd /top/of/patched/gcc/bld $ validate_failures.py --clean_build=clean/bld-gcc Source directory: /usr/local/google/home/dnovillo/gcc/trunk Build target: x86_64-unknown-linux-gnu Getting actual results from build directory . ./x86_64-unknown-linux-gnu/libstdc++-v3/testsuite/libstdc++.sum ./x86_64-unknown-linux-gnu/libffi/testsuite/libffi.sum ./x86_64-unknown-linux-gnu/libgomp/testsuite/libgomp.sum ./x86_64-unknown-linux-gnu/libgo/libgo.sum ./x86_64-unknown-linux-gnu/boehm-gc/testsuite/boehm-gc.sum ./x86_64-unknown-linux-gnu/libatomic/testsuite/libatomic.sum ./x86_64-unknown-linux-gnu/libmudflap/testsuite/libmudflap.sum ./x86_64-unknown-linux-gnu/libitm/testsuite/libitm.sum ./x86_64-unknown-linux-gnu/libjava/testsuite/libjava.sum ./gcc/testsuite/g++/g++.sum ./gcc/testsuite/gnat/gnat.sum ./gcc/testsuite/ada/acats/acats.sum ./gcc/testsuite/gcc/gcc.sum ./gcc/testsuite/gfortran/gfortran.sum ./gcc/testsuite/obj-c++/obj-c++.sum ./gcc/testsuite/go/go.sum ./gcc/testsuite/objc/objc.sum Getting actual results from build directory clean/bld-gcc clean/bld-gcc/x86_64-unknown-linux-gnu/libstdc++-v3/testsuite/libstdc++.sum clean/bld-gcc/x86_64-unknown-linux-gnu/libffi/testsuite/libffi.sum clean/bld-gcc/x86_64-unknown-linux-gnu/libgomp/testsuite/libgomp.sum clean/bld-gcc/x86_64-unknown-linux-gnu/libgo/libgo.sum clean/bld-gcc/x86_64-unknown-linux-gnu/boehm-gc/testsuite/boehm-gc.sum clean/bld-gcc/x86_64-unknown-linux-gnu/libatomic/testsuite/libatomic.sum clean/bld-gcc/x86_64-unknown-linux-gnu/libmudflap/testsuite/libmudflap.sum clean/bld-gcc/x86_64-unknown-linux-gnu/libitm/testsuite/libitm.sum clean/bld-gcc/x86_64-unknown-linux-gnu/libjava/testsuite/libjava.sum clean/bld-gcc/gcc/testsuite/g++/g++.sum clean/bld-gcc/gcc/testsuite/gnat/gnat.sum clean/bld-gcc/gcc/testsuite/ada/acats/acats.sum clean/bld-gcc/gcc/testsuite/gcc/gcc.sum clean/bld-gcc/gcc/testsuite/gfortran/gfortran.sum clean/bld-gcc/gcc/testsuite/obj-c++/obj-c++.sum clean/bld-gcc/gcc/testsuite/go/go.sum clean/bld-gcc/gcc/testsuite/objc/objc.sum SUCCESS: No unexpected failures. 2012-11-02 Diego Novillo <dnovillo@google.com> * testsuite-management/validate_failures.py: Add option --clean_build to compare test results against another build. From-SVN: r193105
2012-10-31validate_failures.py: Fix parsing of summary lines.Diego Novillo1-1/+1
* testsuite-management/validate_failures.py: Fix parsing of summary lines. From-SVN: r193039
2012-08-13validate_failures.py (ExpirationDate): Tidy comment.Diego Novillo1-1/+1
* testsuite-management/validate_failures.py (ExpirationDate): Tidy comment. From-SVN: r190354
2012-08-13validate_failures.py (ParseSummary): Fix comment.Diego Novillo1-2/+3
* testsuite-management/validate_failures.py (ParseSummary): Fix comment. From-SVN: r190352
2012-08-13Implement support for expiring expected results in validate_failures.py.Diego Novillo1-4/+35
I noticed recently that while the validator was accepting the 'expire=YYYYMMDD' attribute, it was not actually doing anything with it. This patch fixes the oversight. 2012-08-13 Diego Novillo <dnovillo@google.com> * testsuite-management/validate_failures.py: Import datetime. (TestResult.ExpirationDate): New. (TestResult.HasExpired): New. (ParseSummary): Call it. If it returns True, warn that the expected failure has expired and do not add it to the set of expected results. (GetResults): Clarify documentation. From-SVN: r190351
2012-07-26Do not use 'with ... as ...' in validate_failures.pyDiego Novillo1-14/+18
Some of the hosts were we run this script are still using Python 2.4. This patch replaces the use of 'with ... as ...' to avoid syntax errors. 2012-07-26 Diego Novillo <dnovillo@google.com> * testsuite-management/validate_failures.py: Do not use 'with ... as ...' constructs. From-SVN: r189893
2012-07-19Fix --produce_manifest flag in validate_failures.py.Diego Novillo1-9/+14
When I added the functionality to use other summary files for reporting, I broke the generation of manifests. When sum files are passed on, we need to use the list. Otherwise, we need to find summary files in the build directory. This patch factors out that logic into a new function and calls it from both the reporting and generation routines. Committed to trunk. * testsuite-management/validate_failures.py (CollectSumFiles): Rename from GetSumFiles. (GetSumFiles): Factor out of CheckExpectedResults. (CheckExpectedResults): Call it. (ProduceManifest): Call it. From-SVN: r189662
2012-07-18validate_failures.py: Fix /usr/bin/python invocation.Diego Novillo1-1/+1
* testsuite-management/validate_failures.py: Fix /usr/bin/python invocation. From-SVN: r189631
2012-06-04Extend validate_failures.py to run outside the build directory.Diego Novillo1-18/+48
This patch adds three new arguments to validate_failures.py so it can be used outside the build directory: --ignore_missing_failures When a failure is expected in the manifest but it is not found in the actual results, the script produces a note alerting to this fact. This means that the expected failure has been fixed, or it did not run, or it may simply be flaky. With this option, one can ask the script not to show the missing failures. --manifest Name of the manifest file to use. By default, the script will look for the manifest file in the source directory associated with this build. With this option, one can point to any arbitrary manifest file. I renamed the old --manifest flag to --produce_manifest. --results Space-separated list of .sum files with the testing results to check. The only content needed from these files are the lines starting with FAIL, XPASS or UNRESOLVED. From-SVN: r188217
2012-04-16Use target_alias in validate_failures.pyRainer Orth1-2/+2
* testsuite-management/validate_failures.py (GetBuildData): Use target_alias. From-SVN: r186497
2012-03-02This fixes an edge case in parsing summary lines.Diego Novillo1-4/+8
This fixes an edge case in parsing summary lines. Some times, the description field is missing (e.g., 'FAIL: libstdc++/abi_check'), so the space that the pattern was looking for does not exist. I've changed it to match any whitespace, which includes '\n'. I also made it print the line that it fails to parse, in case there are other problems like this in the future. 2012-03-02 Diego Novillo <dnovillo@google.com> * testsuite-management/validate_failures.py (class TestResult): Fix match pattern for the summary line. If there is a parsing failure, show the line we failed to parse. From-SVN: r184822
2012-02-172012-02-17 Doug Kwan <dougkwan@google.com>Doug Kwan1-1/+2
* contrib/testsuite-management/validate_failures.py (GetMakefileValue): Check for cross compilers. From-SVN: r184357
2011-09-13testsuite-management: New.Diego Novillo1-0/+337
* testsuite-management: New. * testsuite-management/validate_failures.py: New. From-SVN: r178833
generated by cgit v1.1 at 2025-06-01 00:52:24 +0000