diff options
| -rw-r--r-- | ChangeLog | 137 | ||||
| -rw-r--r-- | Makefile.am | 7 | ||||
| -rw-r--r-- | Makefile.in | 2 | ||||
| -rw-r--r-- | NEWS | 7 | ||||
| -rw-r--r-- | doc/dejagnu.texi | 5 | ||||
| -rw-r--r-- | doc/runtest.1 | 3 | ||||
| -rw-r--r-- | lib/framework.exp | 60 | ||||
| -rw-r--r-- | runtest.exp | 25 | ||||
| -rw-r--r-- | testsuite/runtest.main/error.exp | 81 | ||||
| -rw-r--r-- | testsuite/runtest.main/error/testsuite/error.test/error-al-dbz.exp | 38 | ||||
| -rw-r--r-- | testsuite/runtest.main/error/testsuite/error.test/error-dbz.exp | 29 | ||||
| -rw-r--r-- | testsuite/runtest.main/error/testsuite/error.test/error-undef.exp | 25 | ||||
| -rw-r--r-- | testsuite/runtest.main/error/testsuite/error.test/simple.exp | 21 |
13 files changed, 420 insertions, 20 deletions
@@ -8,6 +8,75 @@ * aclocal.m4, configure.ac, Makefile.am: regenerated GDB too. * doc/dejagnu.texi: Update copyright date. +2020-06-29 Jacob Bachmeyer <jcb62281+dev@gmail.com> + + PR 41824 / PR 41918 + + * NEWS: Add item for repeated error messages. + + * lib/framework.exp (log_and_exit): Add separator lines around the + error dumps repeated at the end of a run. + +2020-06-26 Jacob Bachmeyer <jcb62281+dev@gmail.com> + + PR 41824 / PR 41918 + + * NEWS: Remove items for --keep_going / --no_keep_going options. + + * Makefile.am (CLEANFILES): Adjust for renames. + (TESTSUITE_FILES): Likewise. + + * doc/dejagnu.texi (Invoking runtest): Remove --keep_going and + --no_keep_going options. + (A POSIX Conforming Test Framework): Document that Tcl errors in + test scripts produce UNRESOLVED results as placeholders. + + * doc/runtest.1: Likewise. + + * lib/framework.exp (log_and_exit): Print collected Tcl errors. + + * runtest.exp (dejagnu::error): New internal namespace. + (runtest): Collect Tcl errors caught while executing test scripts. + (dejagnu::opt): Remove internal namespace. + Remove --keep_going / --no_keep_going options. No longer abort on + any Tcl errors; instead store them and repeat the messages at the + end of the run. + + * testsuite/runtest.main/abort.exp: Rename from this ... + * testsuite/runtest.main/error.exp: ... to this and revise tests. + + * testsuite/runtest.main/abort/testsuite/abort.test/abort-al-dbz.exp: + Rename from this... + * testsuite/runtest.main/error/testsuite/error.test/error-al-dbz.exp: + ... to this. + * testsuite/runtest.main/abort/testsuite/abort.test/abort-dbz.exp: + Rename from this... + * testsuite/runtest.main/error/testsuite/error.test/error-dbz.exp: + ... to this. + * testsuite/runtest.main/abort/testsuite/abort.test/abort-undef.exp: + Rename from this... + * testsuite/runtest.main/error/testsuite/error.test/error-undef.exp: + ... to this. + +2020-06-24 Jacob Bachmeyer <jcb62281+dev@gmail.com> + + PR 41824 / PR 41918 + + * NEWS: Revise relevant items. + + * doc/dejagnu.texi (Invoking runtest): Add --no_keep_going option. + * doc/runtest.1: Document --no_keep_going option. Remove + long-obsolete --name option. The --name option had been removed + from runtest before the code was imported into what became the + current repository. + + * runtest.exp: Add option --no_keep_going to stop immediately if a + test script aborts due to a Tcl error. Remove vestiges of + long-obsolete --name option. Make --keep_going option default. + + * testsuite/runtest.main/abort.exp: Adjust tests to use + --no_keep_going and --keep_going instead of assuming a default. + 2020-06-22 Jacob Bachmeyer <jcb62281+dev@gmail.com> Merge patches from Maciej W. Rozycki to fix timeout handling. @@ -67,6 +136,74 @@ * doc/dejagnu.texi (target_compile procedure): Document early_flags. +2020-06-19 Jacob Bachmeyer <jcb62281+dev@gmail.com> + + Merge patches from Tom de Vries <tdevries <at> suse.de> and write + ChangeLog entries. + + * testsuite/runtest.main/abort.exp: Add tests that verify handling + of arithmetic errors (divide-by-zero) in a procedure called within + a test script. + + * testsuite/runtest.main/abort/testsuite/abort.test/abort-dbz.exp: New file. + +2020-06-18 Jacob Bachmeyer <jcb62281+dev@gmail.com> + + PR 41824 / PR 41918 + + Thanks to Tom de Vries for raising these concerns and offering the + initial testsuite patch that led to these changes. + + * NEWS: Add item for consistent abort-on-error handling. + + * lib/framework.exp (unknown): Always link global variables. Tidy. + Silently propagate errors raised in autoloaded procedures and move + the UNRESOLVED result and aborting the test run to... + * runtest.exp (runtest): Report an UNRESOLVED result if a test + script aborts due to a Tcl error. Link global errorCode and + report its value if an error occurs. For consistency, abort the + test run on any Tcl error in a test script instead of only when + calling an undefined procedure. + + * testsuite/runtest.main/abort.exp: Add tests to verify handling + of arithmetic errors (divide-by-zero) in an auto-loaded procedure + called from a test script. Adjust other patterns. + + * testsuite/runtest.main/abort/testsuite/abort.test/abort-al-dbz.exp: + New file. + +2020-06-17 Jacob Bachmeyer <jcb62281+dev@gmail.com> + + PR 41824 + + * NEWS: Add item for --keep_going option. + + * Makefile.am (CLEANFILES): Add abort-init.exp to list. + (TESTSUITE_FILES): Add new testsuite files to list. + + * doc/dejagnu.texi (Invoking runtest): Document new --keep_going + command line option. + * doc/runtest.1: Likewise. + + * lib/framework.exp (unknown): Report an UNRESOLVED result if an + unknown command is invoked. Avoid exiting and propagate the error + from Tcl's "unknown" procedure if --keep_going was + specified. Brace procedure argument list. + * runtest.exp (dejagnu::opt): New namespace. + Add option --keep_going to continue running tests after a test + script aborts due to calling an undefined command. + + * testsuite/runtest.main/abort.exp: New file. + * testsuite/runtest.main/abort/testsuite/abort.test/abort-undef.exp: + New file. + * testsuite/runtest.main/abort/testsuite/abort.test/simple.exp: + New file. + +2020-06-17 Tom de Vries <tdevries <at> suse.de> + + * lib/framework.exp (unknown): Propagate return value of auto-loaded + command. + 2020-06-15 Jacob Bachmeyer <jcb62281+dev@gmail.com> Merge patches from Linaro: diff --git a/Makefile.am b/Makefile.am index 64a7bb4..999b25b 100644 --- a/Makefile.am +++ b/Makefile.am @@ -26,7 +26,7 @@ EXTRA_DIST = ChangeLog-1992 MAINTAINERS dejagnu runtest \ $(commands_DATA) $(TESTSUITE_FILES) $(TEXINFO_TEX)\ $(CONTRIB) -CLEANFILES = options-init.exp stats-init.exp +CLEANFILES = error-init.exp options-init.exp stats-init.exp clean-local: clean-local-check .PHONY: clean-local-check @@ -181,6 +181,11 @@ TESTSUITE_FILES = \ testsuite/runtest.libs/remote.test \ testsuite/runtest.libs/target.test \ testsuite/runtest.libs/utils.test \ + testsuite/runtest.main/error.exp \ + testsuite/runtest.main/error/testsuite/error.test/error-al-dbz.exp \ + testsuite/runtest.main/error/testsuite/error.test/error-dbz.exp \ + testsuite/runtest.main/error/testsuite/error.test/error-undef.exp \ + testsuite/runtest.main/error/testsuite/error.test/simple.exp \ testsuite/runtest.main/options.exp \ testsuite/runtest.main/options/testsuite/null.test/null.exp \ testsuite/runtest.main/stats.exp \ diff --git a/Makefile.in b/Makefile.in index 2757041..fb19047 100644 --- a/Makefile.in +++ b/Makefile.in @@ -383,7 +383,7 @@ EXTRA_DIST = ChangeLog-1992 MAINTAINERS dejagnu runtest \ $(commands_DATA) $(TESTSUITE_FILES) $(TEXINFO_TEX)\ $(CONTRIB) -CLEANFILES = options-init.exp stats-init.exp +CLEANFILES = error-init.exp options-init.exp stats-init.exp bin_SCRIPTS = dejagnu runtest include_HEADERS = dejagnu.h pkgdata_DATA = \ @@ -7,6 +7,13 @@ Changes since 1.6.2: should use this proc. The 'is_remote' proc is deprecated. 2. runtest now accepts --local_init and --global_init options to override the default of reading "site.exp". See the manual for details. +X. runtest now responds consistently to all Tcl errors and generates an + UNRESOLVED result when a test script aborts. Previously, calling an + undefined procedure would cause the test run to abort while other Tcl + errors produced only an easily-ignored message. +X. runtest now collects Tcl errors encountered during a test run and prints + the collected errors a second time at the end of the test run after the + summary. Separator lines containing more than 10 hyphens are included. 3. A utility procedure relative_filename has been added. This procedure computes a relative file name to a given destination from a given base. 4. The utility procedure 'grep' now accepts a '-n' option that diff --git a/doc/dejagnu.texi b/doc/dejagnu.texi index 85b2f58..8f23250 100644 --- a/doc/dejagnu.texi +++ b/doc/dejagnu.texi @@ -385,6 +385,11 @@ A test depends on a previous test, which has failed. @item The test was set up incorrectly. + +@item +A test script aborts due to a Tcl error. In this case, the DejaGnu +framework inserts an @emph{UNRESOLVED} result as a placeholder for an +unknown number of tests that were not run because the script crashed. @end itemize @table @asis diff --git a/doc/runtest.1 b/doc/runtest.1 index d043ee7..d48f992 100644 --- a/doc/runtest.1 +++ b/doc/runtest.1 @@ -56,9 +56,6 @@ Emit Expect output to standard output. .BI --mail \ \'name1\ name2\ ...\' Electronic mail addresses to receive test results. .TP -.BI --name \ HOSTNAME -The network HOSTNAME of the target board. -.TP .BI --objdir \ PATH \fIPATH\fR is a directory containing compiled test code. .TP diff --git a/lib/framework.exp b/lib/framework.exp index 6d7cf4d..24afbed 100644 --- a/lib/framework.exp +++ b/lib/framework.exp @@ -310,21 +310,46 @@ proc isnative { } { # This allows Tcl package autoloading to work in the modern age. rename ::unknown ::tcl_unknown -proc unknown args { - if {[catch {uplevel 1 ::tcl_unknown $args} msg]} { - global errorCode - global errorInfo - global exit_status +proc unknown { args } { + global errorCode + global errorInfo + global exit_status + + set code [catch {uplevel 1 ::tcl_unknown $args} msg] + if { $code != 0 } { + set ret_cmd [list return -code $code] - clone_output "ERROR: (DejaGnu) proc \"$args\" does not exist." - if {[info exists errorCode]} { - send_error "The error code is $errorCode\n" + # If the command now exists, then it was autoloaded. We are here, + # therefore invoking the autoloaded command raised an error. + # Silently propagate errors from autoloaded procedures, but + # complain noisily about undefined commands. + set have_it_now [llength [info commands [lindex $args 0]]] + + if { ! $have_it_now } { + clone_output "ERROR: (DejaGnu) proc \"$args\" does not exist." + set exit_status 2 } - if {[info exists errorInfo]} { - send_error "The info on the error is:\n$errorInfo\n" + + if { [info exists errorCode] } { + lappend ret_cmd -errorcode $errorCode + if { ! $have_it_now } { + send_error "The error code is $errorCode\n" + } } - set exit_status 2 - log_and_exit + if { [info exists errorInfo] } { + # omitting errorInfo from the propagated error makes this proc + # invisible with the backtrace pointing directly to the problem + if { ! $have_it_now } { + send_error "The info on the error is:\n$errorInfo\n" + } + } + + lappend ret_cmd $msg + + eval $ret_cmd + } else { + # Propagate return value. + return $msg } } @@ -409,6 +434,17 @@ proc log_and_exit {} { warning "${tool}_version failed:\n$output" } } + if {[llength $::dejagnu::error::list] > 0} { + # print errors again at end of output + foreach { cell } $::dejagnu::error::list { + clone_output "ERROR: [string repeat - 43]" + clone_output "ERROR: in testcase [lindex $cell 0]" + clone_output "ERROR: [lindex $cell 1]" + clone_output "ERROR: tcl error code [lindex $cell 2]" + clone_output "ERROR: \ + tcl error info:\n[lindex $cell 3]\n[string repeat - 50]" + } + } close_logs verbose -log "runtest completed at [timestamp -format %c]" if {$mail_logs} { diff --git a/runtest.exp b/runtest.exp index ba49e73..21adcf6 100644 --- a/runtest.exp +++ b/runtest.exp @@ -97,6 +97,14 @@ set global_init_file site.exp ;# global init file name set testsuitedir "testsuite" ;# top-level testsuite source directory set testbuilddir "testsuite" ;# top-level testsuite object directory +# +# Collected errors +# +namespace eval ::dejagnu::error { + # list of { file message errorCode errorInfo } lists + variable list [list] +} + # Various ccache versions provide incorrect debug info such as ignoring # different current directory, breaking GDB testsuite. set env(CCACHE_DISABLE) 1 @@ -466,7 +474,6 @@ for { set i 0 } { $i < $argc } { incr i } { "--ig*" - "--loc*" - "--m*" - - "--n*" - "--ob*" - "--ou*" - "--sr*" - @@ -1119,7 +1126,6 @@ for { set i 0 } { $i < $argc } { incr i } { "--ig*" - "--loc*" - "--m*" - - "--n*" - "--ob*" - "--ou*" - "--sr*" - @@ -1550,6 +1556,7 @@ proc runtest { test_file_name } { global bug_id global test_result global errcnt + global errorCode global errorInfo global tool global testdir @@ -1571,7 +1578,7 @@ proc runtest { test_file_name } { } } - if { [catch "uplevel #0 source $test_file_name"] == 1 } { + if { [catch "uplevel #0 source $test_file_name" msg] == 1 } { # If we have a Tcl error, propagate the exit status so # that 'make' (if it invokes runtest) notices the error. global exit_status exit_error @@ -1579,15 +1586,27 @@ proc runtest { test_file_name } { if { $exit_status == 0 } { set exit_status 2 } + set new_error [list $test_file_name $msg] # We can't call `perror' here, it resets `errorInfo' # before we want to look at it. Also remember that perror # increments `errcnt'. If we do call perror we'd have to # reset errcnt afterwards. clone_output "ERROR: tcl error sourcing $test_file_name." + if {[info exists errorCode]} { + clone_output "ERROR: tcl error code $errorCode" + lappend new_error $errorCode + } else { + lappend new_error [list] + } if {[info exists errorInfo]} { clone_output "ERROR: $errorInfo" + lappend new_error $errorInfo unset errorInfo + } else { + lappend new_error [list] } + lappend ::dejagnu::error::list $new_error + unresolved "testcase '$test_file_name' aborted due to Tcl error" } if {[info exists tool]} { diff --git a/testsuite/runtest.main/error.exp b/testsuite/runtest.main/error.exp new file mode 100644 index 0000000..1473ee1 --- /dev/null +++ b/testsuite/runtest.main/error.exp @@ -0,0 +1,81 @@ +# Copyright (C) 1995-2016, 2018, 2020 Free Software Foundation, Inc. +# +# This file is part of DejaGnu. +# +# DejaGnu is free software; you can redistribute it and/or modify it +# under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 3 of the License, or +# (at your option) any later version. +# +# DejaGnu is distributed in the hope that it will be useful, but +# WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with DejaGnu; if not, write to the Free Software Foundation, +# Inc., 51 Franklin Street - Fifth Floor, Boston, MA 02110-1301, USA. + +# This file tests handling of fatal errors in testcases. +# The way we do this is to recursively invoke ourselves on a small testsuite +# and analyze the results. + +load_lib util-defs.exp + +if {![info exists tmpdir]} { + set tmpdir [testsuite file -object -top tmpdir] +} + +set fd [open error-init.exp w] +puts $fd "set srcdir [testsuite file -source -test error]" +puts $fd "set objdir [testsuite file -object -test error]" +puts $fd "set tmpdir $tmpdir" +close $fd + +if {![file isdirectory $tmpdir]} { + catch "file mkdir $tmpdir" +} + +if {![file isdirectory [testsuite file -object -test error]]} { + catch {file mkdir [testsuite file -object -test error]} +} + +set tests { + { "run only simple test" + "simple.exp" + "PASS: simple test.*\ + *expected passes\[ \t\]+1\n" } + { "continue after divide-by-zero, reporting error" + "error-dbz.exp simple.exp" + "PASS: running error-dbz.exp.*\ + *UNRESOLVED: .* aborted.*\ + *PASS: simple test.*\ + *expected passes\[ \t\]+2\n.*unresolved testcases\[ \t\]+1\n" } + { "continue after auto-loaded divide-by-zero, reporting error" + "error-al-dbz.exp simple.exp" + "PASS: running error-al-dbz.exp.*\ + *UNRESOLVED: .* aborted.*\ + *PASS: simple test.*\ + *expected passes\[ \t\]+2\n.*unresolved testcases\[ \t\]+1\n" } + { "continue after calling undefined procedure, reporting error" + "error-undef.exp simple.exp" + "PASS: running error-undef.exp.*\ + *UNRESOLVED: .* aborted.*\ + *PASS: simple test.*\ + *expected passes\[ \t\]+2\n.*unresolved testcases\[ \t\]+1\n" } +} + +foreach t $tests { + if [util_test $RUNTEST \ + "--local_init error-init.exp\ + --outdir $tmpdir -a [lindex $t 1]" \ + "" \ + [lindex $t 2]] { + fail [lindex $t 0] + } else { + pass [lindex $t 0] + } +} + +file delete -force $tmpdir +file delete -force [testsuite file -object -test error testsuite error.test lib] diff --git a/testsuite/runtest.main/error/testsuite/error.test/error-al-dbz.exp b/testsuite/runtest.main/error/testsuite/error.test/error-al-dbz.exp new file mode 100644 index 0000000..52b5bc3 --- /dev/null +++ b/testsuite/runtest.main/error/testsuite/error.test/error-al-dbz.exp @@ -0,0 +1,38 @@ +# Copyright (C) 2020 Free Software Foundation, Inc. +# +# This file is part of DejaGnu. +# +# DejaGnu is free software; you can redistribute it and/or modify it +# under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 3 of the License, or +# (at your option) any later version. +# +# DejaGnu is distributed in the hope that it will be useful, but +# WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with DejaGnu; if not, write to the Free Software Foundation, +# Inc., 51 Franklin Street - Fifth Floor, Boston, MA 02110-1301, USA. + +# Cause a divide-by-zero error in an auto-loaded procedure. + +pass "running error-al-dbz.exp" + +set fd [open [testsuite file -object -test lib foo.tcl] w] +puts $fd {proc throw_arith_error_div_by_zero { } { + expr { 1 / 0 } +} +} +close $fd + +auto_mkindex \ + [testsuite file -object -test lib] \ + [testsuite file -object -test lib/*.tcl] + +lappend auto_path [testsuite file -object -test lib] + +throw_arith_error_div_by_zero + +fail "script did not abort" diff --git a/testsuite/runtest.main/error/testsuite/error.test/error-dbz.exp b/testsuite/runtest.main/error/testsuite/error.test/error-dbz.exp new file mode 100644 index 0000000..463337a --- /dev/null +++ b/testsuite/runtest.main/error/testsuite/error.test/error-dbz.exp @@ -0,0 +1,29 @@ +# Copyright (C) 2020 Free Software Foundation, Inc. +# +# This file is part of DejaGnu. +# +# DejaGnu is free software; you can redistribute it and/or modify it +# under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 3 of the License, or +# (at your option) any later version. +# +# DejaGnu is distributed in the hope that it will be useful, but +# WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with DejaGnu; if not, write to the Free Software Foundation, +# Inc., 51 Franklin Street - Fifth Floor, Boston, MA 02110-1301, USA. + +# Cause a divide-by-zero error. + +pass "running error-dbz.exp" + +proc throw_arith_error_div_by_zero { } { + expr { 1 / 0 } +} + +throw_arith_error_div_by_zero + +fail "script did not abort" diff --git a/testsuite/runtest.main/error/testsuite/error.test/error-undef.exp b/testsuite/runtest.main/error/testsuite/error.test/error-undef.exp new file mode 100644 index 0000000..fa59f4a --- /dev/null +++ b/testsuite/runtest.main/error/testsuite/error.test/error-undef.exp @@ -0,0 +1,25 @@ +# Copyright (C) 2020 Free Software Foundation, Inc. +# +# This file is part of DejaGnu. +# +# DejaGnu is free software; you can redistribute it and/or modify it +# under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 3 of the License, or +# (at your option) any later version. +# +# DejaGnu is distributed in the hope that it will be useful, but +# WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with DejaGnu; if not, write to the Free Software Foundation, +# Inc., 51 Franklin Street - Fifth Floor, Boston, MA 02110-1301, USA. + +# Invoke an undefined command, causing a fatal error. + +pass "running error-undef.exp" + +bogus_command 1 2 3 4 + +fail "script did not abort" diff --git a/testsuite/runtest.main/error/testsuite/error.test/simple.exp b/testsuite/runtest.main/error/testsuite/error.test/simple.exp new file mode 100644 index 0000000..93a03e7 --- /dev/null +++ b/testsuite/runtest.main/error/testsuite/error.test/simple.exp @@ -0,0 +1,21 @@ +# Copyright (C) 2020 Free Software Foundation, Inc. +# +# This file is part of DejaGnu. +# +# DejaGnu is free software; you can redistribute it and/or modify it +# under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 3 of the License, or +# (at your option) any later version. +# +# DejaGnu is distributed in the hope that it will be useful, but +# WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with DejaGnu; if not, write to the Free Software Foundation, +# Inc., 51 Franklin Street - Fifth Floor, Boston, MA 02110-1301, USA. + +# Return a passing result + +pass "simple test" |