Evaluation reports

Testsuite for the graphic user interface

Regression tests of the surface of the editor should be executed by a capture replay tool. The central issue of an interface test is the verification if the program reacts correctly to user’s inputs and if the editor is integrated correctly in Eclipse. On this purpose first of all an overview on the tools for Java UI tests is given. Two of these tools will be chosen and evaluated. Afterwards a description of the testsuite created with the tool chosen will follow.

Conclusion and selection

In total the commercial tool seems much more comfortable and robust. The remarkably fast and competent support via email and mailing list shouldn’t be neglected. This is something that you cannot expect from an open source tool and that you generally speaken not find. An exception on this are maybe the forums of huge open source products like open office or Eclipse, but the SWTBot’s community isn’t this big. Finally, in table 3.2 all relevant differences between the two tools are summarized.

The result of the evaluation and in consideration of the focus areas that are set by the task was that QF-Test should be used as tool for the UI test. This decision was made in accordance with ubigrade and due to the detected differences - especially regarding to the different linearity in test creation.

Direct comparison between SWTBot and QF-Test

<strong>Criteria</strong> <strong>SWTBot</strong> <strong>QF-Test</strong>
Capture-Replay No Capture component Yes
API knowledge necessary Extensive (SWTBot.*, Hamcrest) Limited (Resolvers)
Injection method Own target platform plugin Instrumented JDK, modified SWT
Test creation effort Very high (Line-by-line and transition between test realization and SUT) Low (Capture, unrecognizable components as worst case)
Weaknesses in test creation Line-by-line scripting, nonlinear work flow Less control on SUT internals
Execution speed Fast (In situ on SUT level Fast (Direct instrumentation of the JDK)
Strength in test execution Introspection Deploy neutrality
Setup/Teardown Just on SUT-level or via ant etc. Procedures, SUT can be restarted between test runs
Customizability A customized Hamcrest-Matcher usable Customized resolvers usable
GEF-support Bad [Mer09] Mature, activatable via import

Table 3.2: Direct comparison between SWTBot and QF-Test

The whole extract from the evaluation report concerning QF-Test can be found here (German PDF only).

Diploma thesis: Conception and realization of a protocol model editor, chapter 3: Testsuite for a Graphic User Interface - QF-Test in comparison with SWTBot - January 2010, Christoph Schmidt, TU Dresden, Germany.

(Original German texts and citations are translated into English.)