Creating useful, reliable tests requires more than just recording sequences and playing them back. You can fill a test-suite with lots of sequences in a short time, but you are bound to lose track of what you've got sooner or later. You need to give structure to your tests to make sure that:

• You have a good idea of what you are testing.
• You are testing the right thing.
• Your tests are reliable, repeatable and easy to maintain in case the SUT changes in a way that QF-Test can't adapt to automatically.
• The results of your tests are conclusive.

The essential prerequisite of getting the components right has been discussed in chapter 6. Here we are going to concentrate on structuring the actual sequences, events, checks, etc.

The primary building block of a test is the 'Sequence' node which executes its child nodes one by one in the order they appear. The 'Test' node is very similar but adds semantics for setup and cleanup. Before QF-Test version 2 it was the prime representative of a test-case. In that role it has been replaced by the new 'Test-case' and 'Test-set' nodes which are described in the following section 10.2. However, 'Sequence' and 'Test' nodes are still very useful low-level building blocks. Their difference lies in their semantics.

		Figure 10.1:  'Sequences' and 'Tests'

The concept behind a 'Sequence' node is that its child nodes must be run in exactly that order and that each node depends on the outcome of the previous node to work correctly. A sequence of recorded events is a very good example for what constitutes a 'Sequence' node. Imagine what would happen if the events were rearranged in a random order.

The semantic contract of the 'Test' node on the other hand is that each of its child nodes is independent of all of the other children, so the order in which they are executed shouldn't really matter. Achieving such a state of independence requires that each child of a 'Test' node can rely on the same set of preconditions. It is the 'Test' node's job to ensure that these conditions are met. To that end, a 'Test' node can have two special additional child nodes, the 'Setup' and the 'Cleanup' nodes.

When a 'Test' node is executed, its child nodes are executed one by one in the order they appear. We said this before, but it is a half truth only. Every time it executes one of its normal children, the 'Test' node first runs the 'Setup' node which is responsible for providing the settings that the child node requires. Whenever a child node is done, the 'Cleanup' node is executed to clean up any mess in the SUT the child node may have caused.

This may sound needlessly complex, but implementing all the preconditions of a test-case and isolating test-cases so that they are completely independent are so important that in QF-Test version 2 'Setup' and 'Cleanup' nodes have essentially been replaced by an even more powerful mechanism based on the new 'Dependency' nodes which are described in chapter 12. But as 'Tests' still have their use at lower levels, so do 'Setup' and 'Cleanup' nodes. A 'Test' node with well-matched 'Setup' and 'Cleanup' nodes has the following important qualities:

• You can add child nodes to the 'Test' at any time and any position without disturbing the other child nodes.
• When you want to extend a test or just try out some sequence, you don't need to execute all the previous steps first to get the SUT into the proper state.
• You can run any sub-set of a 'Test' node if you don't want to run the full test.
• If one child of a 'Test' node fails for whatever reason, the rest of its children can still be executed, provided that the 'Setup' and 'Cleanup' node can be relied on to provide the proper settings under these circumstances. In a 'Sequence', where each node depends on the outcome of its predecessor, the failure of one node will invalidate the whole sequence.

The 'Implicitly catch exceptions' attribute of a 'Test' node helps to facilitate error handling of the kind described above.

The 'Test-set' and 'Test-case' nodes provide a small-scale, pragmatic form of test management right inside QF-Test. Their main feature is the smart dependency management described in the following chapter 12 that allows 'Test-cases' to be implemented completely independent from each other. With properly written 'Dependencies', cleanup of the SUT from previously executed tests is handled automatically along with the setup for the next test and all error handling.

Conceptually a 'Test-case' node represents a single elementary test case. As such it is the main link between test planning, execution and result analysis. With the help of 'Dependencies', 'Test-cases' can be isolated from each other so that they can be run in any arbitrary order. QF-Test automatically takes care of the necessary test setup. Cleanup is also automatic and will be performed only when necessary in order to minimize overhead in the transition from one test to the next. This enables things like running subsets of functional test-suites as build tests or retesting only failed 'Test-cases'.

'Test-sets' basically are bundles of 'Test-cases' that belong together and typically have similar requirements for setup and cleanup. 'Test-sets' can be nested. The whole structure of 'Test-sets' and 'Test-cases' is very similar to 'Package' and 'Procedure' nodes. The 'Test-suite' root node can be considered a special kind of 'Test-set'.

'Test-suite', 'Test-set' and 'Test-case' nodes can be called from other places using a 'Test call' node. That way, tests that run only a subset of other tests can easily be created and managed. 'Test call' nodes are allowed everywhere, but should not be executed from inside a 'Test-case' node because that would break the atomicity of a 'Test-case' from the report's point of view. A warning is issued if 'Test-case' execution is nested.

As both 'Test-sets' and 'Test-cases' can be called from a 'Test call' node they each have a set of default parameters similar to those of a 'Procedure'. These will be bound on the fallback stack and can be overridden in the 'Test call' node. A 'Test-case' has an additional set of variable bindings similar to those of a 'Test' node. These are hard bindings for the primary variable stack that will be defined during the execution of the 'Test-case' and cannot be overridden in a 'Test call' node.

The list of 'Characteristic variables' is a set of names of variables that are part of the characteristics of the test for data-driven testing. Each execution of the 'Test-case' with a different set of values for these variables is considered a separate test case. The expanded values of these variables are shown in the run-log and report for improved error analysis.

Another useful attribute is the 'Condition' which is similar to the 'Condition' of an 'If' node. If the 'Condition' is not empty, the test will only be executed if the expression evaluates to true. Otherwise the test will be reported as skipped.

Sometimes a 'Test-case' is expected to fail for a certain period of time e.g. when it is created prior to the implementation of the respective feature or before a bug-fix is available in the SUT. The 'Expected to fail if...' attribute allows marking such 'Test-cases' so they are counted separately and don't influence the percentage error statistics.

For backwards compatibility and to ease transition from old-style 'Test' nodes to 'Test-set' and 'Test-case' nodes QF-Test treats nodes as a 'Test-set' or 'Test-case' for documentation and report if their place in the hierarchy allows it.

Old test-suites with a structure based on 'Test' nodes can by migrated to make use of the new features of 'Test-sets' and 'Test-cases'. To this end, right-click on a 'Test' node to bring up the context menu. If a transformation is allowed, QF-Test will offer to transform the 'Test' node into a 'Test-set' or 'Test-case' node.

3.0+ It is possible to convert a whole hierarchy of 'Test' nodes to a hierarchy of 'Test-set' and 'Test-case' nodes by selecting the recursive conversion option in the popup menu.

Note Both 'Test-set' and 'Test-case' nodes may contain 'Setup' or 'Cleanup' nodes for backwards compatibility. In a 'Test-set', these work just as in a 'Test': 'Setup' and 'Cleanup' are executed for each test contained in the 'Test-set'. In a 'Test-case' however, 'Setup' and 'Cleanup' are only run once at the beginning and end of its execution. If a 'Test-set' or 'Test-case' has both a 'Dependency' and 'Setup'/'Cleanup' nodes, the 'Dependency' will be executed first. 'Setup' and 'Cleanup' will have no impact on the dependency stack described in section 12.2.

In a way, writing good tests is a little like programming. After mastering the initial steps, tests and source code alike tend to proliferate. Things work fine until some building block that was taken for granted changes. Without a proper structure, programs as well as tests tend to collapse back upon themselves at this point as the effort of adapting them to the new situation is greater than the one needed for recreating them from scratch.

The key to avoiding this kind of problem is reuse or avoidance of redundancy. Generating redundancy is one of the main dangers of relying too much on recording alone. To give an example, imagine you are recording various sequences to interact with the components in a dialog. To keep these sequences independent of each other, you start each one by opening the dialog and finish it by closing the dialog again. This is good thinking, but it creates redundancy because multiple copies of the events needed to open and close the dialog are contained in these sequences. Imagine what happens if the SUT changes in a way that invalidates these sequences. Let's say a little confirmation window is suddenly shown before the dialog is actually closed. Now you need to go through the whole suite, locate all of the sequences that close the dialog and change them accommodate the confirmation window. Pure horror.

To stress the analogy again, this kind of programming style is called Spaghetti Programming and it leads to the same kind of maintenance problems. These can be avoided by collecting the identical pieces in one place and referring to them wherever they are needed. Then the modifications required to adapt to a change like the one described above are restricted to this place only.

		Figure 10.2:  'Packages' and 'Procedures'

QF-Test comes with a set of nodes that help to achieve this kind of modularization, namely the 'Procedure', 'Procedure call' and 'Package' nodes. A 'Procedure' is similar to a 'Sequence' except that its 'Name' attribute is a handle by which a 'Procedure call' node can refer to it. When a 'Procedure call' is executed, the 'Procedure' it refers to is looked up and execution continues there. Once the last child node of the 'Procedure' has finished, the 'Procedure call' has completed as well.

'Packages' are just a way to give even more structure to 'Procedures'. A hierarchy of 'Packages' and 'Procedures', rooted at the special 'Procedures' node, is used to group sets of 'Procedures' with a common context together and to separate them from other 'Procedures' used in different areas.

A 'Procedure' that always does exactly the same, no matter where it is called from, is only marginally useful. To expand on the above example, let's say we want to extend the 'Procedure' that opens the dialog to also set some initial values in some of its fields. Of course we don't want to have these initial values hard-coded in the 'Procedure' node, but want to specify them when we call the 'Procedure' to get different values in different contexts. To that end, parameters can be defined for the 'Procedure'. When the 'Procedure call' is executed, it specifies the actual values for these parameters during this run. How all of this works is explained in Variables. Also please take a look at the detailed explanation for the 'Procedure' and 'Procedure call' nodes for a better understanding of how these complement each other.

A test-suite library with a set of commonly useful 'Procedures' is provided with QF-Test under the name qfs.qft. An entire chapter of the Tutorial is devoted to this library and section 17.1 explains how to include it in your test-suites.

If you work with several test-suite libraries you might face a situation, where you define reusable test-steps or sequences, which you only want to use within a dedicated test-suite. If you want to create such local 'Procedures', you can put a '_' as first sign of the procedure's name. This marks a 'Procedure' as test-suite local.

A call of a local 'Procedure' can only be inserted within the test-suite, where it is defined. You can use the same concept for local 'Packages'.

If you call 'Procedures' from other 'Procedures', it could be convenient not to specify the full procedure name all the time.

So called 'relative' procedure calls can only be added to a 'Package', which has the 'Border for relative calls' (see 'Border for relative calls') attribute specified. The structure of that call follows the concept below:

As you can see each dot stands for one level. So calling a 'Procedure' two levels higher requires three dots (Current level also requires a dot.)

As you should organize your tests in separate test steps, which are ideally the same like QF-Test's procedures, QF-Test offers several ways to insert those 'Procedure call' nodes:

1. Via the menu »Insert«-»Procedures«-»ProcedureCall«
2. Via right mouse click and selecting »Insert node«-»Procedures«-»ProcedureCall«
3. Copy a 'Procedure' node and insert it at the location of the 'Procedure call' using the normal Copy/Paste actions
4. Via Drag&Drop operation, i.e. dragging the 'Procedure' node to its target node
5. Via the keyboad shortcut [Ctrl-A]

This approach is also valid for inserting 'Dependency reference' nodes, except the keyboard shortcut.

You can create parameters for a 'Procedure', 'Dependency' or 'Test-case' automatically via opening the context-menu and selecting »Parameterize node«.

The parameter details dialog allows you to define for which actions you want to create parameters, e.g. only text-inputs or check nodes.

This transformation is very useful for developing procedures immediately after recording! Under 'Extras' you can convert a recorded 'Sequence' node into a 'Procedure' and move that to the 'Procedures' node.

3.1+ If you transform a 'Sequence' under 'Test-cases' QF-Test automatically creates a 'Procedure' node and inserts a 'Procedure call' to the previous location of the transformed node.

Like with any programming-related task it is important for successful test-automation to properly document your efforts. Otherwise there is a good chance (some might say a certainty) that you will lose the overview over what you have done so far and start re-implementing things or miss out tests that should have been automated. Proper documentation will be invaluable when working through a run-log, trying to understand the cause of a failed test. It will also greatly improve the readability of test reports.

Based on the 'Comment' attributes of 'Test-set', 'Test-case', 'Package' and 'Procedure' nodes, QF-Test can create a set of comprehensive HTML documents that will make all required information readily available. The various kinds of documents and the methods to create them are explained in detail in chapter 15.

Last update: 01/27/2012 Copyright © 1999-2012 Quality First Software GmbH