Logo QF-Test

Finding valuable answers
in the Mailing List Archive.


Free Trial  Download  Buy

Thomas Max, QF-Test training and support

Use the full-text search on our web site to find helpful tips on the mailing list.

Thomas Max, 
Sr. Software Engineer & Trainer, QFS

2017 up to now  | 2016 | 2015 | 2014 | 2013 | 2012 | 2011 | 2010 | 2009 | 2008 | 2007

(older archive entries before 2007 are not shown here, but included in the onsite-search)

Mailing List - Entries of 2012

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [QF-Test] How to perform QF-Test suite (formal) reviews? -- Some ideas...

  • Subject: Re: [QF-Test] How to perform QF-Test suite (formal) reviews? -- Some ideas...
  • From: "Berg, Klaus-Peter" <klaus-peter.berg@?.com>
  • Date: Tue, 24 Jan 2012 15:30:19 +0100

Hi all,
in the following post I have tried to collect and present some ideas on a formal QF-Test suite review for *functional* tests.
1. However, because you can do load testing (and even performance testing) with QF-Test as well, it is necessary to specify which kind of test suite is reviewed first.
Usually this will be a *functional* test suite.
2. You will need a test specification (test spec) store, e.g., HP Quality Center or another tool that is able to handle test specs.
This specification should have a specific outline that addresses all necessary topics.
In Sara Ford's blog I found a good short description of what should be inside such a spec and what is the difference between a test plan and a test spec: http://saraford.net/2004/10/28/developing-a-test-specification/
You can find other good descriptions in the Internet as well but in general, every company has developed its own standard for this kind of document.
3. It would be good to have a kind of "link" between the test spec and your QF-Test suite with test-sets and test-cases.
Naming the test suites, test-sets/cases according to the naming "format" of your test spec manager tool may be an option but this may also lead to quite "poor" or "ugly" test names like "23580_01_02_01_Project Editor". Another option would be to have a tool that can analyze QF-Test suite test-set/test-cases names (.qft files are "only" XML ;-)) and make connections between specs and these names in order to have some kind of "traceability"! (The IEEE Standard Glossary of Software Engineering Terminology defines *traceability* as "the degree to which a relationship can be established between two or more products of the development process, especially products having a predecessor-successor or master-subordinate relationship to one another." [IEEE-610]).
Given a good test specification, now let us take a closer look at the test "implementation" using QF-Test.
4. It is good practise to have a common test layout inside your file system that is also a working copy of your test structure that should be part of a source code control system like Subversion, CVS, or ClearCase.
What should be versioned?
At least your QF-Test suites that contain your actual test-sets/test-cases, corresponding property files, e.g., with login information, etc. (for QF-Test load property nodes), other property files that may be read and analyzed by test scripts in Groovy or Jython,
QF-Test suites that only contain common procedures ("libraries"), (large) Groovy and Jython files that implement QF-Test custom checkers or that are helper classes for your procedures.
Should QF-Test itself be versioned? There are pro and cons. At least for regression tests you need to know what version of QF-Test you were running and what version of SUT you have tested. Because we are testing GUIs here, it is quite common to have the SUT version number available inside the SUT (e.g., for the opening screen and for Help->About). Perhaps with some assistance from your development group you can access this version number by means of a SUT script (quite often this version number is a public final static String).
Besides your scripts you will need to put specific engineering information under version control, if your SUT has to be prepared by means your tests cannot do or will simply require.
When thinking about a file/folder structure it seems reasonable to distinguish between a "base" directory with common files for most or all of your test suites and the "real" test suites that contain your actual tests, and that are usually further subdivided according to your business domain and the "kind" of tests that is executed (functional test/load test/performance test).
There is a special place inside QF-Test install directory to store library files, e.g., in Groovy you will use the place described in chapter "13.7.3 Groovy packages" and chapter "29.1.6 Library path - Directories holding test-suite libraries" of the actual QF-Test manual.
Analyzing the conformance of a "test" to such a structure should also be part of a formal test review.
5. For large Groovy or Jython files it is necessary to have a common header comment, just like for Java files.
This comment should contain a short description of the file's purpose, a copyright remark, @author tag, @since tag describing the specific Groovy/Jython or QF-Test version that must be used (at least) in order to run this script (you know, every QF-Test version is bound to a specific Groovy and Jython version; if you use specific language features in your script that are available only since Version x.y.z+ you will have to document that).
BTW: For Groovy you can just open the Groovy terminal from the QF-Test menu bar and type:
groovy:000>  println "Running Groovy version: " + org.codehaus.groovy.runtime.InvokerHelper.getVersion()
println "Running Groovy version: " + org.codehaus.groovy.runtime.InvokerHelper.getVersion()
Running Groovy version: 1.7.10
===> null
in order to find out the current Groovy version inside QF-Test.
6. Not only your test suites, test sets/cases must have meaningful names, you will also need a naming scheme especially for your procedures and variables. You can either follow the conventions used by the QF-Test base library "qfs.qft" -- including procedure comment conventions! -- or create your own naming scheme; but whatever you do: you should do it in a consistent way.
So, IMO it would make sense to follow the Java Sun naming conventions for procedure and package names as well as for variables and constants (this is also true for your scripts, e.g. Groovy scripts! -- and should be reviewed). On the long run it will provide better readability and maintainability.
When creating libraries youz can use qfs.qft as a template (...qfs/qftest/qftest-3.x.x/include/qfs.qft).
When opening this library you will see that it contains only 3 types of nodes:
- Procedures
- Extras (empty)
- Windows and components
So, during the walkthrough, you should check if your own libraries follow this scheme, i.e., library suites do NOT have any test-sets etc.! However, depending on the kind of procedures they contain, they MAY have a "Windows and components tree" or not. You can express this behavior by naming your libraries like this:
- Libraries WITH "Windows and components tree": <MyLib>ComponentsAndProcedures.qft
- Libraries with EMPTY "Windows and components tree": <MyLib>Procedures.qft .
7. Be *consistent* in your naming: e.g., do NOT call a package "Util" in one suite and "Utils" in another library suite.
8. If your specs tell you that there is some screen content (e.g. text field content) to check, then we should look for adequate "check text" nodes (or rc.checkEqual() etc. in scripts) in your suites!
9. If your test spec requires some windows / dialogs to appear or disappear after mouse button/menu clicks, it is NOT enough to "check" if there are no exceptions during replay! You will have to look for a "Wait for component to appear/disappear) node or some other kind of "check" for existence or non-existence of a window after a timeout! E.g., if your top window is a JDesktop you can check the internal frames that are currently open.
10. If your test spec defines specific pre- and post conditions for a test I would expect to see these conditions expressed as setup/cleanup nodes and dependencies. If you are using QF-Test dependencies (on the long run, you will ;-)) please review if they follow the guidelines expressed in the QF-Test manual (chapter 12) and especially look for proper cleanup/error/exception handling in this area!
11. During a walkthrough you can click on the test-suite top node inside QF-Test and check the test suites include files in the reverse includes (dependencies) - looking at the correct relative names, especially with respect to a common library directory ("library path settings") as mentioned in topic (4), and looking if some suites are missing or out-of-date (not actually used, only copy/paste results ;-))
12. When looking at test steps it is worth to identify LARGE set of steps that use if/then/else/loop nodes together with lots of QF-Test procedure calls to access and evaluate table columns etc. If your company policy is "do not use scripting if ever possible" -- then just leave them untouched -- otherwise check if you can transform the whole set into one nice script that set's a global return variable or even check for the possibility of creating a custom check (chapter 39.3 "Implementing custom checks with the Checker interface"). It could be worth the effort with respect to readability and usability (you can use a custom check like any other QF-Test check, even with the "Record checks" tool).
13. Check your error/exception logging: Here are three rules I recommend for Groovy scripts and QF-Test procedure parameters:
a) In Groovy SUT and server scripts you can use a catch block like this:
        try {
                 float test = 2.0/0.0 // throws Division by zero exception
                 println "Division OK" // should never happen
        catch (Exception e) {
                 def message = "Expected exception was thrown"
                 println("$message: ${e.message}") // write to QF-Test terminal or Java console                  
                 rc.logError("$message: ${e.message}") // write to QF-Test run-log
Perhaps you would expect using "System.err.println" in the catch clause instead of "println", because we are reporting an EXCEPTION.
However, if you execute "binding.variables.each {key,value -> println "$key=$value"}"
you will see that only
is bound, NOT "err". Maybe this is the reason for the behaviour mentioned above... at least in server scripts... but I'm not shure...
b) In CustomChecker Groovy code where the variable "rc" is not fully available you can also write:
        qf.logError("$message: ${e.message}")
        in the catch clause (see QF-Test Manual "37.7 The qf module").
c) In QF-Test procedure call parameters you can use a 'message' parameter like this to log the exception text:
        Exception while running SUT: $[rc.getCaughtException().getMessage()]   
        Exception while running SUT: $[rc.getCaughtException().getLocalizedMessage()]
        but only if the procedure is called inside an catch block where rc.getCaughtException() is available and != NULL!
Example: qfs.utils.writeMessageIntoFile(message, file, true).
14. Variables "from the outside": If you are running QF-Test from the command line, inside an ANT or Maven script, or controlled by a test automation framework, it is necessary to review which variables are handed over, if these variables are located in the right place inside a test suite, and if there names are correct.
Would be nice to share this proposal with ideas/comments from other users of the forum (or with people from the QFS company itself ;-))
Best regards,
Klaus-Peter Berg

Videos Downloads Documentation Buy Free Trial