diff options
author | Ben Elliston <bje@gnu.org> | 2016-02-15 18:30:41 +1100 |
---|---|---|
committer | Ben Elliston <bje@gnu.org> | 2016-02-15 18:30:41 +1100 |
commit | 5462bead2a0ecc6907d7981090f89f4cf39ef2e3 (patch) | |
tree | f99e332dc452b7dbea8065389a1d260431cde891 /TODO | |
parent | d506c9a5943ac7789e4178d06c9362a79e055f78 (diff) | |
download | dejagnu-5462bead2a0ecc6907d7981090f89f4cf39ef2e3.zip dejagnu-5462bead2a0ecc6907d7981090f89f4cf39ef2e3.tar.gz dejagnu-5462bead2a0ecc6907d7981090f89f4cf39ef2e3.tar.bz2 |
* TODO: Add ideas produced from Cauldron 2013.
Diffstat (limited to 'TODO')
-rw-r--r-- | TODO | 139 |
1 files changed, 137 insertions, 2 deletions
@@ -1,5 +1,3 @@ -Last updated $Date: 2004/02/08 14:02:07 $ - Bigger items ============ @@ -9,3 +7,140 @@ Bigger items * Add more support for target boards and RTOSes. * Use the new expect terminal support for an "escape codes" API. * Use expectk and write a GUI testing API, complete with record/playback. + + + +Date: Thu, 29 Aug 2013 19:42:07 +0200 +From: Jan Kratochvil <jan.kratochvil@redhat.com> +To: dejagnu@gnu.org +Subject: dejagnu-2.0 feature wishlist (from Cauldron 2013) + +Hi, + +I haven't found any discussion here about the features in hypothetical +dejagnu-2.0, as presented by Rob Savoye at Cauldron 2013. + +I wrote some scripts on top of DejaGNU but I think at least some of the +functionality could be integrated into DejaGNU itself. It depends whether +dejagnu-2.0 scope will remain the same or whether DejaGNU should be used +together with tools like buildbot or whether dejagnu-2.0 will integrate some +of the buildbot-like functionality (multi-note continuous runs). + +Maybe there exists something similar already? Originally I wrote it only for +myself but I see nowadays such tool may be useful for more people. + +Former announcement of my scripts: + https://sourceware.org/ml/archer/2010-q3/msg00194.html +URLs are no longer valid, the files can be found now at: + git clone git://git.jankratochvil.net/nethome + (that is my whole $HOME, not just the testsuite scripts) +The primary script 'hammock' is at: + http://git.jankratochvil.net/?p=nethome.git;a=blob;f=bin/hammock + +Essential fixup of current DejaGNU: +------------------------------------------------------------------------------ +--orphanripper: It is used by default, normal DejaGNU scripts do not track +their spawned children which share fds 0/1/2 (stdio). This means some such +children are due to *.exp code bugs occasionally leftover running forever. As +they have their fds still open the testsuite with output redirected somewhere +will lock up at the end. Some runaway processes also hog CPU for 100%. The +following utility identifies runaway processes by using custom pty for them +and kills them at the end of testsuite run: + http://pkgs.fedoraproject.org/cgit/gdb.git/plain/gdb-orphanripper.c +It sure should be better integrated in DejaGNU somehow. + +Features outside of the current scope of DejaGNU: +------------------------------------------------------------------------------ +--distro: Testing in various OSes. My script implement it based on chroot +(Fedora/RHEL has tool 'mock' for it), it has some performance/management +advantages but it has to (1) run all OSes with the same kernel, (2) mock +supports only Fedora/RHEL OSes, (3) it can run only x86_64/i386 arch this way. +The real solution should be multi-node (so that it can also support non-x86* +testing), for x86* it would be commonly using VMs. But it still could support +even mock/chroot as it runs without the hassle of disk images. + +--component: Pre-set remote repositories for download of gdb/binutils/gcc etc. +I want to run my patches on top of clean tree, not in some existing directory +which may have leftover files forgotten to be checked into repository etc. +Understandably it also supports local repository caches. +--srcrpm is similar, it builds tree from a prepared archive - I should be able +to provide also .src.rpm (or .tar.gz) to run the test for. +--branch asks for example for branch "gdb_7_6-branch" from the repository. + +--file: Provide custom patches for the newly built tree. + +--target: Provide a list of custom configure --target options. This could be +more general such as to provide any custom configure options. + +--parallel: Parallelization of multiple build+testsuite runs, not just +parallelization of the testsuite run part. +If I ask to build 40x binutils with 40 different targets I may want to do +run it in parallel (like with make -j8). + +Convenience: +------------------------------------------------------------------------------ +--gdbserver, --valgrind, --bfd32, --gdbindex, --dwz, --dwarf=X, --stabs: +Various pre-set options. One can configure it by hand but it is too difficult +for daily use, for example for --dwz it means for GDB + runtest CC_FOR_TARGET=/bin/sh\ $PWD/../contrib/cc-with-tweaks.sh\ -m\ gcc CXX_FOR_TARGET=/bin/sh\ $PWD/../contrib/cc-with-tweaks.sh\ -m\ g++ ... + (plus also GNATMAKE_FOR_TARGET, GO_FOR_TARGET and GO_LD_FOR_TARGET) +For --valgrind it means other cryptic options like: + RUNTESTFLAGS=--target_board=valgrind + +Incomplete racy reads ("read1"): +------------------------------------------------------------------------------ +https://sourceware.org/bugzilla/show_bug.cgi?id=12649 +GDB testsuite contains (yes, it still contains them) various racy cases: + gdb_test_multiple "set dprintf-style agent" $msg { + -re "warning: Target cannot run dprintf commands.*" { +It commonly works as when expect does the read() syscall all the GDB output is +ready. But occasionally the next testcase FAILs. This is because +occasionally only part of the output gets read by the read() syscall, regex +gets matched but the final $gdb_prompt is not discarded - and the leftover +$gdb_prompt corrupts the next testcase below. Sure the fix is: + gdb_test_multiple "set dprintf-style agent" $msg { + -re "warning: Target cannot run dprintf commands.*\r\n$gdb_prompt $" { + +There is LD_PRELOAD *.so file in the Bug above to reproduce these cases +reliably. There is also a reproducer of different kind of bugs ("writew") +although those do not happen so often AFAIK. This functionality could be +better integrated into DejaGNU. + +(Sure the primary problem is that the testsuite should not use regex matching +and it should use generic GDB MI output parser. But that is a problem that +only a few GDB features have implemented the GDB MI interface.) + +Diffing of results: +------------------------------------------------------------------------------ +http://git.jankratochvil.net/?p=nethome.git;a=blob;f=bin/diffgdb + * GDB has various known FAILs. They should be but they are not KFAILed or + XFAILed. (On recent Fedora I see there are only 23 of them but on CentOS-5 + there is 1063 of them.) + * One is only interested in introduced regressions so one needs to diff two + *.sum files. Looking again and again at the same known FAILing cases is + not productive. + * During diff one is not interested for example in newly PASSing testcases. + One also is not interested in FAIL->PASS cases. One is definitely + interested in PASS->FAIL regressions. New FAILing testcases are also + interesting. +Therefore the script above does a filtering of the diff results. It parses +DejaGNU *.sum output although DejaGNU did already knew them internally. + +The script also filters out unstable/racy results. This may be outside of the +scope but in fact an unimplemented feature would be to provide statistics on +unstable results (so one can fix those) if I run the same build+testsuite many +times. + +Not yet implemented: Finding a regression common reason: +------------------------------------------------------------------------------ +I run 73 testsuite runs daily - primarily GDB in different OSes, for each OS +its x86_64 and i686 variant, for x86_64 OS also in -m32 mode. +If there happens a general regression I get 73 times PASS->FAIL result. That +is not too convenient to filter out other changes out of the 73 regressions. +Moreover sometimes the regression affects for example only 32-bit OSes +- therefore there will be only about 24 PASS->FAILs and I have to figure out +in which testsuite combinations they happen. +In other cases the regression happens for example only on (older) RHELs and +not on Fedoras but that again means about 6 PASS->FAIL cases. +There would be nice some summary that this PASS->FAIL occured in testsuite +runs in directories rhel* and it did not occur in directories fedora* etc. |