The previous chapters all focused on creating and organizing sets of reliable tests in a single test-suite. However, when testing a large application, a single suite may not be enough. There are at least two scenarios where splitting tests into multiple test-suites becomes essential:

• Multiple developers are creating and editing tests. To avoid redundancy and duplication of code, separately developed tests should use common 'Procedures' and 'Components' where possible, but only one person can edit a test-suite at a time.
• Tests are simply getting too large. Run-logs for extensive test-runs may cause the system to run out of memory and organizing a large number of tests in a single suite is difficult. Things may become unwieldy. It may also be desirable to be able to run some of the tests as part of the whole test as well as standalone.

QF-Test provides a set of advanced features that make it possible to split and arrange tests across a set of test-suites. Multiple developers can work on separate parts of a test, then coordinate their efforts, merge the 'Components' of their suites and create libraries of shared 'Procedures'.

This chapter first explains the various mechanisms for distributed test development and how they interact. The final section then summarizes these in concise step-by-step instructions on how to approach large testing efforts with QF-Test.

It is possible to reference 'Procedures' and 'Components' in a test-suite other than the current one. These references can be explicit or implicit through included files:

• Explicit references use a syntax similar to the one used in URLs to specify an item inside a web page. The referenced suite must be prepended to the 'Procedure name' attribute of a 'Procedure call' of the 'Component id' attribute of a 'Component' dependent node, separated by a '#' character. The usual packagepath.procedure becomes suite#packagepath.procedure.
• Implicit references make use of the 'Include files' attribute of the 'Test-suite' node. Whenever a node is not found in the current suite, QF-Test will search for a matching 'Procedure' or 'Component' within all the suite's directly or indirectly included files (a file is considered indirectly included by a suite if it is found as an included file within one of the suite's own included files; for example, if suite A includes B, and suite B includes C, then C is indirectly included by A).

While implicit references are more convenient in most cases, they can make tests harder to understand because it is not immediately obvious where the 'Procedure' or 'Component' referenced by some node is actually located. One possibility to find out is to select "Locate procedure" ([Ctrl-P]) or "Locate component" ([Ctrl-W]) from the context menu. Additionally, QF-Test provides the menu items »Extras«-»Make references absolute« and »Extras«-»Make references implicit« which let you toggle quickly between the two modes without changing the actually referenced nodes.

In both cases, the referenced suite can either be given a relative or absolute filename. Relative filenames will be resolved relatively to the directory of current suite, or - if that fails - relatively to the directories on the library path (see option Directories holding test-suite libraries). Always use the forward '/' as the directory separator, even under Windows. QF-Test will map it to the correct character for the system it runs on. This keeps your test-suites independent from the operating system.

Note Your 'Package' and 'Procedure' names as well as 'Component' 'Ids' should not contain any '\' or '#' characters. If they do, you need to include an escape character in the 'Procedure call' or the 'Component id' id attribute. See section 36.6 for details about escaping and quoting special characters.

When choosing the 'Procedure' for a 'Procedure call' or the 'Component' for some event in the dialog, QF-Test offers a selection of all currently opened test-suites. If a 'Procedure' or 'Component' from another test-suite is selected, QF-Test automatically creates the correct reference, taking included suites into account. When the test is run at a later time, the referenced test-suite is loaded automatically if necessary.

During execution QF-Test keeps a stack of currently executing suites. Whenever a 'Procedure' is called in another suite, the called suite is pushed on to the top of this stack and removed when execution returns to the calling suite. Whenever during the execution of a 'Procedure' a 'Window' or 'Component' is referenced by its id, QF-Test searches through this stack of suites from the top to the bottom, i.e. first in the test-suite of the called 'Procedure' and then in the calling suite, always checking any included files along the way. This process is quite complicated and you should take care to keep your include hierarchies simple. In case you encounter problems anyway, a detailed explanation is given in section 36.7.

As we have emphasized in chapter 6, the 'Components' are the essential part of a test-suite. If the SUT changes between releases, these will likely be affected most. If changes are so massive that QF-Test cannot adapt automatically, the 'Components' will have to be updated manually. This is why you should try to avoid redundancy in the 'Component' hierarchy of your tests more than in any other part.

Therefore, when splitting your tests across multiple test-suites you should try to keep the 'Components' together in one central test-suite and include this suite from the other suites. For very large applications you may want to split the 'Component' hierarchy into parts, each related to a separate part of the SUT's GUI.

Maintaining this central 'Component' library is not trivial. The problems that will arise can be resolved with QF-Test as follows:

• When multiple test developers are recording new 'Components' simultaneously, they cannot be integrated immediately into the central suite, because only one user can edit the central suite at a time. Instead, 'Components' must be integrated later by importing them into the central suite when the new tests have stabilized. This is explained in the following section.
• When the SUT changes, 'Components' in the central suite may need to be updated. If this involves changing any 'Component' 'Ids', this will break any references to these 'Components' from other suites. To avoid that, QF-Test must update those references and it will do so, provided that the suites that depend on the central suite are currently loaded or listed in the 'Dependencies (reverse includes)' attribute of the 'Test-suite' node of the central suite.

Test-suites can be merged by importing one test-suite into another with the »File«-»Import...« menu item.

You can select the areas of the test-suite, which should be imported.

You have to take care about a correct Include/Reverse-Include of your test-suites to ensure, that all calls and component references are still valid. See chapter 25 for details.

During import, all 'Windows' and 'Components' of the imported test-suite are integrated into the component hierarchy of the importing suite. 'Components' that already exist are not copied. An 'Id' conflict (identical components with different 'Ids' or differing components with identical 'Ids') is resolved automatically by changing the 'Id' of the imported component.

Afterwards, all 'Windows' and 'Components' are removed from the imported suite. All nodes in the imported suite that referred to these 'Components' are updated accordingly. Ideally, the imported suite should include the importing suite so no explicit suite references will have to be created.

As you can import 'Components' QF-Test also allows to import 'Procedures', 'Packages', 'Dependencies' and 'Test-cases' as well as 'Test-sets' by choosing 'Procedures' or 'Tests' in the import dialog. You should take care about keeping all calls consistent, e.g. in most cases it does not make sense to import 'Procedures' without their required 'Components'.

In case you only want to import one dedicated 'Procedure' or 'Test-case' you can use the button 'Detailimport' on the importdialog. Here you can choose any node you want to import separately.

There is no single best way of test development or organization, but one approach that works well is the following:

• Start with a central test-suite that has the functionality needed to start and stop the SUT and a basic set of 'Tests' and 'Procedures'. This will become your master suite which will contain all 'Components'.
• Make sure that your developers have understood the importance of assigning names with setName() and that unique names are assigned consistently where needed. Where setName() is not an option, try to implement ComponentNameResolvers to achieve this (see section 39.1). You should be able to record and replay sequences without much ado and without "polluting" the 'Component' hierarchy after trivial changes in the user interface.
• Move as much functionality as possible into 'Procedures', especially commonly-used stuff and the setup and cleanup routines for the SUT.
• To create new tests, start with an empty test-suite. Include the master test-suite by editing the 'Include files' attribute of the 'Test-suite' node of the new suite. Create the 'Setup' and 'Cleanup' nodes to start and stop the SUT by calling the respective 'Procedures' in the master suite.
• Create your tests as required. When recording sequences, the 'Components' of the master-suite will be used if possible. New 'Components' are added to the new suite, so the master suite will not be modified at this stage.
• Where possible, call 'Procedures' in the master suite for common operations.
• When your new set of tests is complete and you are satisfied that they work well, import any required nodes of your new test-suite into the master suite. You have to ensure that all new 'Component' nodes that you recorded are imported into the master suite's 'Component' hierarchy in any case. The master suite's existing 'Components' will not be affected by this, so other suites that depend on the master suite will not need to be modified.
• After importing 'Components' yu can import all or only the required 'Procedures' into the master suite.
• You now have various options of how to arrange the actual sequences of events and checks that form your tests. In any case it is a good idea to move everything to 'Procedures' and 'Packages' structured after your test-plan. Then the top-level 'Tests' nodes of the master suite and your new suite will only contain the required hierarchy of 'Test' and 'Sequence' nodes filled with 'Procedure calls' to the actual test-cases. Such an arrangement has several advantages:
  • All your tests are structured cleanly.
  • You can easily create different sets of tests with varying complexity and run-time.
  • You have the option to keep the test-cases in separate test-suites and have the master suite call them. These "test-case-libraries" must include the master-suite, so they need not contain any 'Components' themselves. You can organize your tests so that the master-suite will run the whole set of tests, while each separate suite can also be run standalone.
  • The tests can be maintained by several developers as long as modifications to the master suite are coordinated.
• If you decide to keep your new tests in the newly created test-suite instead of moving them to the master suite, modify the master suite to tell QF-Test that there is a new test-suite that depends on it. To do so, add the new test-suite to the 'Dependencies' attribute of the master suite's 'Test-suite' node.
• If you need to modify or extend the new test-suite later, proceed as before. You can record new sequences as needed. When you are done, merge any newly created 'Components' back into the master suite.
• If your SUT changes in a way that requires updates or adaptions to the master-suite's 'Component' hierarchy, you must coordinate your test developers. Before you start updating the 'Components', make sure that all suites that directly or indirectly include the master suite are listed in the 'Dependencies' attribute of the master suite's 'Test-suite' node. If modifying the 'Components' of the master suite involves any 'Component id' changes, QF-Test will update the depending test-suites accordingly, so they should not be edited simultaneously by others.
• The file format for QF-Test test-suites is XML and thus plain text. As a result, test-suites can be managed very well by version control systems like CVS. Changes to some 'Component id' attributes of the depending suites can typically be merged with other changes without conflicts, alleviating the need for coordination.

Of course, the above scheme can be extended to have several master suites for testing different parts or aspects of an application. It may be a good idea to ensure that the component hierarchies in these suites don't overlap too much though. This will save you the effort of maintaining all these hierarchies in case the user interface of the SUT changes significantly.

Last update: 04/23/2012 Copyright © 1999-2012 Quality First Software GmbH