Evaluation of QF-Test at CoreMedia

Comparison QF-Test vs. WinRunner

Introduction

QF-Test is a GUI testtool written in Java and with Jython as scripting language. In December 2005/January 2006 we evaluated this tool if it might serve as new GUI testtool for the SwingEditor of our CMS-product. This document holds the information what caused the decision for or against the new GUI testtool.

Target

The target of the evaluation process is to decide if we want to replace (at least for further development) WinRunner as GUI testtool with QF-Test and to document the reasons for the decision.

Conclusion

The conclusion has not been made during the evaluation phase as also other options have been taken into account for creating more tests for the Swing Editor. But eventually we decided to buy licenses of QF-Test. 

Epilogue

Just as the conclusion which was delayed after the creation of the evaluation report this epilogue also is not part of the original evaluation report. Everything else is as-is, i. e. you just read through the original report from now nearly two years ago.

Today we still use QF-Test – and we still use WinRunner, because we didn’t manage to get time for converting the tests in WinRunner to tests in QF-Test. There is no way for automatically converting those tests and thus every test has to be converted by hand. But we are on our way. Some tests already moved.

I am still happy we made the decision to switch to QF-Test. While I still have to maintain WinRunner Tests and recently had strong problems with Java 1.6 and Windows Vista QF-Test just works as fine on Java 1.6/Windows Vista as on Windows XP with Java 1.5 or 1.4.

It’s always a pleasure to write tests in QF-Test as they are easy to create. To some extent I might say it’s my framework I built meanwhile. It allows me to have tests which read in the code tree just like:
1. Create Document “A”
2. Write some Text into the Document
3. Save
4. Do Checks
Where every single line points to a function/procedure in the framework I built up. But it’s QF-Test which enabled me to build such a framework.

Also CoreMedia developers who don’t write QF-Tests every day feel very comfortable with QF-Test. Most of the times I only need to introduce them for one or two hours and they start right away.

QF-Test still gets better from release to release. So it’s no wonder that some flaws I mentioned I fixed meanwhile. As e. g. the “lazy variable expansion bug” as I call it, which was not a bug but a different approach to handle variable expansion. And great new features were introduced like dependencies I use a lot meanwhile.

And one last word about the company size: Of course I totally misjudged the company size and I am happy to know some more people at QFS now.

Mark Michaelis, Software Engineer Quality Assurance, ISTQB Certified Tester

The whole evaluation report can be found here (PDF).