Adds candidate formatter for replacing legacy summary results.

Also cleans up some usages of strings where symbolic names
were safer and made more sense.

Try a test run with something like this to check out the new
basic results formatter (not used by default):

time test/dotest.py --executable `pwd`/build/Debug/lldb --results-formatter lldbsuite.test.basic_results_formatter.BasicResultsFormatter --results-file stdout

This will yield something like:

Testing: 1 test suites, 8 threads
1 out of 1 test suites processed - TestHelp.py
Test Results
Total Test Methods Run (excluding reruns): 13
Test Method rerun count: 0

===================
Test Result Summary
===================
Success:              13
Expected Failure:      0
Failure:               0
Error:                 0
Unexpected Success:    0
Skip:                  0

Whereas something with a bit of error will look more like this:

42 out of 42 test suites processed - TestSymbolTable.py
Test Results
Total Test Methods Run (excluding reruns): 166
Test Method rerun count: 0

===================
Test Result Summary
===================
Success:              93
Expected Failure:     10
Failure:               2
Error:                 2
Unexpected Success:    0
Skip:                 59

Details:
FAIL:
TestModulesInlineFunctions.ModulesInlineFunctionsTestCase.test_expr_dsym
(/Users/tfiala/work/lldb-tot/git-svn/lldb/packages/Python/lldbsuite/test/lang/objc/modules-inline-functions/TestModulesInlineFunctions.py)
FAIL:
TestModulesInlineFunctions.ModulesInlineFunctionsTestCase.test_expr_dwarf
(/Users/tfiala/work/lldb-tot/git-svn/lldb/packages/Python/lldbsuite/test/lang/objc/modules-inline-functions/TestModulesInlineFunctions.py)
ERROR: TestObjCCheckers.ObjCCheckerTestCase.test_objc_checker_dsym
(/Users/tfiala/work/lldb-tot/git-svn/lldb/packages/Python/lldbsuite/test/lang/objc/objc-checker/TestObjCCheckers.py)
ERROR: TestObjCCheckers.ObjCCheckerTestCase.test_objc_checker_dwarf
(/Users/tfiala/work/lldb-tot/git-svn/lldb/packages/Python/lldbsuite/test/lang/objc/objc-checker/TestObjCCheckers.py)

The Details header only prints if there are any issues to report.  The
Details section has tags that should get picked up using the normal
issue text scrapers (e.g. buildbot).

Test numbers reported are strictly test method runs.

The rerun bit at the top is in support of the multi-pass test
runner code (to run the low-load, single worker test pass for
tests that failed the first run), which I'll be able to put up
for review after this.

ResultsFormatters now have the ability to indicate they replace
the legacy summary, as this one does.

Once we come to agreement on the exact format, I will switch
us over to using this by default.

llvm-svn: 254530

1 files changed, 63 insertions, 29 deletions

diff --git a/lldb/packages/Python/lldbsuite/test/dosep.py b/lldb/packages/Python/lldbsuite/test/dosep.py
index b4d2208..c98136f 100644
--- a/lldb/packages/Python/lldbsuite/test/dosep.py
+++ b/lldb/packages/Python/lldbsuite/test/dosep.py
@@ -1,5 +1,3 @@
-#!/usr/bin/env python
-
 """
 Run the test suite using a separate process for each test file.
 
@@ -55,6 +53,7 @@ import lldbsuite.support.seven as seven
 
 from . import dotest_channels
 from . import dotest_args
+from . import test_results
 
 # Todo: Convert this folder layout to be relative-import friendly and don't hack up
 # sys.path like this
@@ -1406,33 +1405,68 @@ def main(print_details_on_success, num_threads, test_subdir,
             test_name = os.path.splitext(xtime)[0]
             touch(os.path.join(session_dir, "{}-{}".format(result, test_name)))
 
-    print()
-    sys.stdout.write("Ran %d test suites" % num_test_files)
-    if num_test_files > 0:
-        sys.stdout.write(" (%d failed) (%f%%)" % (
-            len(failed), 100.0 * len(failed) / num_test_files))
-    print()
-    sys.stdout.write("Ran %d test cases" % num_test_cases)
-    if num_test_cases > 0:
-        sys.stdout.write(" (%d failed) (%f%%)" % (
-            fail_count, 100.0 * fail_count / num_test_cases))
-    print()
-    exit_code = 0
-
-    if len(failed) > 0:
-        failed.sort()
-        print("Failing Tests (%d)" % len(failed))
-        for f in failed:
-            print("%s: LLDB (suite) :: %s (%s)" % (
-                "TIMEOUT" if f in timed_out else "FAIL", f, system_info
-            ))
-        exit_code = 1
-
-    if len(unexpected_successes) > 0:
-        unexpected_successes.sort()
-        print("\nUnexpected Successes (%d)" % len(unexpected_successes))
-        for u in unexpected_successes:
-            print("UNEXPECTED SUCCESS: LLDB (suite) :: %s (%s)" % (u, system_info))
+    # Only run the old summary logic if we don't have a results formatter
+    # that already prints the summary.
+    if results_formatter is None or not results_formatter.replaces_summary():
+        print_legacy_summary = True
+    else:
+        print_legacy_summary = False
+
+    if not print_legacy_summary:
+        # Remove this timeout handling once
+        # https://llvm.org/bugs/show_bug.cgi?id=25703
+        # is addressed.
+        #
+        # Use non-event-based structures to count timeouts.
+        timeout_count = len(timed_out)
+        if timeout_count > 0:
+            failed.sort()
+            print("Timed out test files: {}".format(len(timed_out)))
+            for f in failed:
+                if f in timed_out:
+                    print("TIMEOUT: %s (%s)" % (f, system_info))
+
+        # Figure out exit code by count of test result types.
+        issue_count = (
+            results_formatter.counts_by_test_result_status(
+                test_results.EventBuilder.STATUS_ERROR) +
+            results_formatter.counts_by_test_result_status(
+                test_results.EventBuilder.STATUS_FAILURE) +
+            timeout_count)
+        # Return with appropriate result code
+        if issue_count > 0:
+            sys.exit(1)
+        else:
+            sys.exit(0)
+    else:
+        # Print the legacy test results summary.
+        print()
+        sys.stdout.write("Ran %d test suites" % num_test_files)
+        if num_test_files > 0:
+            sys.stdout.write(" (%d failed) (%f%%)" % (
+                len(failed), 100.0 * len(failed) / num_test_files))
+        print()
+        sys.stdout.write("Ran %d test cases" % num_test_cases)
+        if num_test_cases > 0:
+            sys.stdout.write(" (%d failed) (%f%%)" % (
+                fail_count, 100.0 * fail_count / num_test_cases))
+        print()
+        exit_code = 0
+
+        if len(failed) > 0:
+            failed.sort()
+            print("Failing Tests (%d)" % len(failed))
+            for f in failed:
+                print("%s: LLDB (suite) :: %s (%s)" % (
+                    "TIMEOUT" if f in timed_out else "FAIL", f, system_info
+                ))
+            exit_code = 1
+
+        if len(unexpected_successes) > 0:
+            unexpected_successes.sort()
+            print("\nUnexpected Successes (%d)" % len(unexpected_successes))
+            for u in unexpected_successes:
+                print("UNEXPECTED SUCCESS: LLDB (suite) :: %s (%s)" % (u, system_info))
 
     sys.exit(exit_code)