Evaluation of QF-Test/web at CoreMedia

Synopsis

This paper documents the evaluation of different tools to test an AJAX-application based on ExtJS. The evaluation started because of strong problems with the currently used test-tool named “Sahi”. Candidates as replacement for Sahi are:

• Windmill Testing Framework
• Selenium web application testing system
• QF-Test

The main focus of this evaluation is capture and replay as this is the common way how new testers will start writing tests. Refactoring of tests for better maintainability is discussed in short side notes.

Purpose

The purpose of this evaluation is to find one common test-tool for all tests of web applications at CoreMedia. This way the knowledge of the usage of this test-tool should be sharable between different projects. Requirements The main requirements for the new test-tool as derived from GUI Testing – Tool Requirements and Test Guidelines:

1. Short time to learn

New testers should soon be able to understand and debug existing tests. Writing first basic tests should be simple.

2. Multiple browser-support

Browsers which should (at least) be supported:

• Internet Explorer 7 and 8
• Firefox 3

3. Easy debugging

This includes that automatic test-runs provide screenshots in case of errors so that the possible error can easily be detected. And it includes that tests can be debugged step by step and that they can be run independently. The latter one of course also poses a requirement to the tests which are written – not only to the tool.

4. Clipboard Interaction

An important part during our tests is interaction with the clipboard. Thus the tools should be able to record clipboard interaction events. Optional and untested during this evaluation is a comparison of screenshots (expected versus actual). Although important for the tests as layout problems are a major threat with multi-browser-support the tests are complicate to develop to keep them robust not only on multiple browsers but also on multiple platforms. It is known that QF-Test offers such a screenshot-comparison which can be easily recorded.

Conclusion

Web applications may be the most challenging task for GUI-Test-Automation. Not only that they render differently in different browsers to get around the quirks of the browsers: Also the element structure is more complex than those of for example a Java Swing application. So it is even more important to find a tool which helps you to get around these obstacles. Taking the free tools out there Selenium seems to be a good choice. Especially because the ExtJS developers are aware of it. Recording is easy and it is a plus that the recorded test cases can be exported to any language (like Java for example). This ensures that your developers can export it to their language they favor most. Of course this also means that the main development and debugging will be done outside of the IDE. If you are willing to pay for a test tool I recommend QF-Test. For CoreMedia it has an additional benefit as we also develop Swing applications and thus can stick to one tool for all. The support is great and fast and QF-Test made great improvements during the past years which makes the future look very promising

Mark Michaelis, Software Engineer Quality Assurance, ISTQB Certified Tester

About the author

Mark Michaelis is Software Engineer Quality Assurance at CoreMedia since 2005. He collected a lot of experiences doing tests with debugging and developing automated GUI-Tests.
These experiences include:

• the pain debugging badly written GUI-Tests,
• working with GUI-Test-Tools which produce random results when tests are replayed and
• the benefit and good feedback for a well written test-framework for GUI-Tests

Mark Michaelis has great knowledge of the tool QF-Test and minor knowledge of other Web-TestTools.

The whole evaluation report can be found here (PDF).