From 6a2899bf3d86100162c41efedd923c01b82d8109 Mon Sep 17 00:00:00 2001 From: Jacob Bachmeyer Date: Sat, 25 Jul 2020 21:57:21 -0500 Subject: Add warnings about synchronization with the tested program --- doc/dejagnu.texi | 121 ++++++++++++++++++++++++++++++++++++------------------- 1 file changed, 80 insertions(+), 41 deletions(-) (limited to 'doc') diff --git a/doc/dejagnu.texi b/doc/dejagnu.texi index 8f23250..cd8a179 100644 --- a/doc/dejagnu.texi +++ b/doc/dejagnu.texi @@ -2175,53 +2175,91 @@ built up a family of Tcl procedures specialized for GDB testing. @cindex hints on writing a test case @cindex test cases, writing -It is safest to write patterns that match all the output generated by -the tested program; this is called closure. If a pattern does not -match the entire output, any output that remains will be examined by -the next @code{expect} command. In this situation, the precise -boundary that determines which @code{expect} command sees what is very -sensitive to timing between the Expect task and the task running the -tested tool. As a result, the test may sometimes appear to work, but -is likely to have unpredictable results. (This problem is -particularly likely for interactive tools, but can also affect batch -tools---especially for tests that take a long time to finish.) The -best way to ensure closure is to use the @code{-re} option for the -@code{expect} command to write the pattern as a full regular -expressions; then you can match the end of output using a @emph{$}. -It is also a good idea to write patterns that match all available -output by using @emph{.*\} after the text of interest; this will also -match any intervening blank lines. Sometimes an alternative is to -match end of line using @emph{\r} or @emph{\n}, but this is usually -too dependent on terminal settings. - -Always escape punctuation, such as @emph{(} or @emph{"}, in your -patterns; for example, write @emph{\(}. If you forget to escape -punctuation, you will usually see an error message like: - -@example -extra characters after close-quote -@end example +To preserve basic sanity, no should test ever pass if there was any +kind of problem in the test case. To take an extreme case, tests that +pass even when the tool will not spawn are misleading. Ideally, a +test in this sort of situation should not fail either. Instead, print +an error message by calling one of the DejaGnu procedures +@code{perror} or @code{warning}. Note that using @code{perror} will +cause the next text result to be reported as @samp{UNRESOLVED}, so +printing an error and allowing the test to fail is a good option. If you have trouble understanding why a pattern does not match the program output, try using the @code{--debug} option to @code{runtest}, and examine the debug log carefully. +If you use glob patterns, you will need to escape any @samp{*}, +@samp{?}, @samp{[}, @samp{]}, and @samp{\} characters that are meant +to match literally. If you use regular expressions, see the +@cite{re_syntax(n)} manual page from Tcl for the syntax details, and +be sure to understand what punctuation characters match literally and +what characters have special meanings in regular expressions. + +Tcl has a few options for quoting; the most notable are @samp{@{@}} +and @samp{""}. These quotes behave differently: @samp{@{@}} must +balance, while @samp{""} performs various interpolations. In +@samp{@{@}} quotes, unbalanced @samp{@{} or @samp{@}} characters must +be escaped with @samp{\} and these escapes are @emph{not} removed; +fortunately, backslash-escaped braces match literal braces in Tcl +regular expressions. In @samp{""} quotes, any embedded @samp{"} +characters must be escaped, a literal @samp{$} begins a variable +substitution, and unescaped @samp{[]} introduce a Tcl command +substitution. + +@subsubheading Synchronization with the tested program + +A DejaGnu testsuite executes concurrently with the programs that it +tests. As a result, DejaGnu may see parts of the tested program's +output while the tested program is still producing more output. +Expect patterns must be written to handle the possibility that +incomplete output from the tested program will be considered for +matching. + +Expect reads the output from the tested program into an internal +matching buffer and removes everything from the start of the buffer to +the end of the match when a match is found. Any given character can +be matched at most once, or skipped if a match is found starting later +in the buffer or the buffer reaches its capacity. Anything left in +the buffer after the end of the match remains in the buffer and is +considered for the next @code{expect} command. If @code{expect} is +invoked and no patterns match, Expect waits for more text to arrive. +New text is appended to the buffer as it is read. If the buffer +reaches its capacity, the entire contents of the buffer are discarded +and Expect resumes reading. + +In Expect patterns, the regular expression anchors @samp{^} and +@samp{$} match at the beginning and end of the @emph{buffer}, not at +line boundaries. The @samp{$} anchor must be used with care because +it will match at the end of what Expect @emph{has} read, but the +program may have produced more output that Expect @emph{has not yet} +read. Similarly, regular expressions ending with the @samp{*} +quantifier can potentially match a prefix of the intended text, only +for the rest to arrive shortly thereafter. + +Maintaining synchronization with the tested program is easier if the +patterns match all of the output generated by the tested program; this +is called closure. + +For interactive programs, a prompt is usually a good synchronization +point, provided that the program's prompt can be uniquely recognized. +Since the prompt is usually the last output until the program receives +further input, the @samp{$} anchor can be useful here. + +If the output from the tested program is organized into lines, +matching end-of-line using @samp{\n} is usually a good way to process +one line at a time. Note that terminal settings may result in the +insertion of additional @samp{\r} characters, usually translating +@samp{\n} to @samp{\r\n}. + Be careful not to neglect output generated by setup rather than by the -interesting parts of a test case. For example, while testing GDB, I -issue a send @emph{set height 0\n} command. The purpose is simply to -make sure GDB never calls a paging program. The @emph{set height} +interesting parts of a test case. For example, while testing GDB, a +@samp{set height 0\n} command is issued. The purpose is simply to +make sure GDB never calls a paging program. The @samp{set height} command in GDB does not generate any output; but running any command -makes GDB issue a new @emph{(gdb) } prompt. If there were no -@code{expect} command to match this prompt, the output @emph{(gdb) } -begins the text seen by the next @code{expect} command---which might -make that pattern fail to match. - -To preserve basic sanity, I also recommended that no test ever pass if -there was any kind of problem in the test case. To take an extreme -case, tests that pass even when the tool will not spawn are -misleading. Ideally, a test in this sort of situation should not fail -either. Instead, print an error message by calling one of the DejaGnu -procedures @code{error} or @code{warning}. +makes GDB issue a new @samp{(gdb) } prompt. If there were no +@code{expect} command to match this prompt, the @samp{(gdb) } prompt +will remain in the buffer and begin the text seen by the next +@code{expect} command---which might make that pattern fail to match. @subsubheading Priority of Expect patterns @@ -5968,4 +6006,5 @@ This makes @code{runtest} exit. Abbreviation: @kbd{q}. @c LocalWords: subdirectory prepend prepended testsuite filename Expect's svn @c LocalWords: DejaGnu CVS RCS SCCS prepending subcommands Tcl Awk Readline @c LocalWords: POSIX KFAIL KPASS XFAIL XPASS hostname multitable gfortran api -@c LocalWords: boolean subcommand testcase eval rustc executables +@c LocalWords: boolean subcommand testcase eval rustc executables perror gdb +@c LocalWords: interprocess lookahead -- cgit v1.1