24

Distributed test development

The previous chapters all focused on creating and organizing sets of reliable tests in a single test suite. However, when testing a large application, a single suite may not be enough. There are at least two scenarios where splitting tests into multiple test suites becomes essential:

• Multiple developers are creating and editing tests. To avoid redundancy and duplication of code, separately developed tests should use common 'Procedures' and 'Components' where possible, but only one person can edit a test suite at a time.
• Tests are simply getting too large. Run logs for extensive test runs may cause the system to run out of memory and organizing a large number of tests in a single suite is difficult. Things may become unwieldy. It may also be desirable to be able to run some of the tests as part of the whole test as well as standalone.

QF-Test provides a set of advanced features that make it possible to split and arrange tests across a set of test suites. Multiple developers can work on separate parts of a test, then coordinate their efforts, merge the 'Components' of their suites and create libraries of shared 'Procedures'.

This chapter first explains the various mechanisms for distributed test development and how they interact. The final section then summarizes these in concise step-by-step instructions on how to approach large testing efforts with QF-Test.

24.1

Referencing nodes in another test suite

It is possible to reference 'Procedures' and 'Components' in a test suite other than the current one. These references can be explicit or implicit through included files:

• Explicit references use a syntax similar to the one used in URLs to specify an item inside a web page. The referenced suite must be prepended to the 'Procedure name' attribute of a 'Procedure call' of the 'QF-Test component ID' attribute of a 'Component' dependent node, separated by a '#' character. The usual packagepath.procedure becomes suite#packagepath.procedure.
• Implicit references make use of the 'Include files' attribute of the 'Test suite' node. Whenever a node is not found in the current suite, QF-Test will search for a matching 'Procedure' or 'Component' within all the suite's directly or indirectly included files (a file is considered indirectly included by a suite if it is found as an included file within one of the suite's own included files; for example, if suite A includes B, and suite B includes C, then C is indirectly included by A).

A test suite that references a node in another test suite becomes dependent on that suite. If the 'Name' of a 'Procedure' or the 'QF-Test ID' of a 'Component' in the referenced suite changes, the suite with the reference must get updated, otherwise the link is broken and the suite will no longer work correctly. In such cases QF-Test will automatically update references if it knows about them. The best way to ensure that is to have both test suites in a common project because QF-Test automatically tracks all includes and all explicit references within a project. Alternatively you can list the calling suite in the 'Dependencies (reverse includes)' attribute of 'Test suite' root node of the referenced suite.

While implicit references are more convenient in most cases, they can make tests harder to understand because it is not immediately obvious where the 'Procedure' or 'Component' referenced by some node is actually located. One possibility to find out is to select "Locate procedure" ([Ctrl-P]) or "Locate component" ([Ctrl-W]) from the context menu. Additionally, QF-Test provides the menu items »Operations«-»Make references explicit« and »Operations«-»Make references implicit« which let you toggle quickly between the two modes without changing the actually referenced nodes.

In both cases, the referenced suite can either be given a relative or absolute filename. Relative filenames will be resolved relatively to the directory of current suite, or - if that fails - relatively to the directories on the library path (see option Directories holding test suite libraries). Always use the forward '/' as the directory separator, even under Windows. QF-Test will map it to the correct character for the system it runs on. This keeps your test suites independent from the operating system.

Note Your 'Package' and 'Procedure' names as well as 'Component' 'QF-Test IDs' should not contain any '\' or '#' characters. If they do, you need to include an escape character in the 'Procedure call' or the 'QF-Test component ID' attribute. See section 47.5 for details about escaping and quoting special characters.

When choosing the 'Procedure' for a 'Procedure call' or the 'Component' for some event in the dialog, QF-Test offers a selection of all currently opened test suites. If a 'Procedure' or 'Component' from another test suite is selected, QF-Test automatically creates the correct reference, taking included suites into account. When the test is run at a later time, the referenced test suite is loaded automatically if necessary.

During execution QF-Test keeps a stack of currently executing suites. Whenever a 'Procedure' is called in another suite, the called suite is pushed on to the top of this stack and removed when execution returns to the calling suite. Whenever during the execution of a 'Procedure' a 'Window' or 'Component' is referenced by its QF-Test ID, QF-Test searches through this stack of suites from the top to the bottom, i.e. first in the test suite of the called 'Procedure' and then in the calling suite, always checking any included files along the way. This process is quite complicated and you should take care to keep your include hierarchies simple. In case you encounter problems anyway, a detailed explanation is given in section 47.6.

24.2

Managing 'Components'

As we have emphasized in chapter 5, the 'Components' are the essential part of a test suite. If the SUT changes between releases, these will likely be affected most. If changes are so massive that QF-Test cannot adapt automatically, the 'Components' will have to be updated manually. This is why you should try to avoid redundancy in the 'Component' hierarchy of your tests more than in any other part.

Therefore, when splitting your tests across multiple test suites you should try to keep the 'Components' together in one central test suite and include this suite from the other suites. For very large applications you may want to split the 'Component' hierarchy into parts, each related to a separate part of the SUT's GUI.

Maintaining this central 'Component' library is not trivial. The problems that will arise can be resolved with QF-Test as follows:

• When multiple test developers are recording new 'Components' simultaneously, they cannot be integrated immediately into the central suite, because only one user can edit the central suite at a time. Instead, 'Components' must be integrated later by importing them into the central suite when the new tests have stabilized. This is explained in the following section.
• When the SUT changes, 'Components' in the central suite may need to be updated. If this involves changing any 'Component' 'QF-Test IDs', this will break any references to these 'Components' from other suites. To avoid that, QF-Test must update those references and it will do so, provided that the suites that depend on the central suite are currently loaded, belong to the same project or are listed in the 'Dependencies (reverse includes)' attribute of the 'Test suite' node of the central suite.

24.3

Merging test suites

test suites can be merged by importing one test suite into another with the »File«-»Import...« menu item.

You can select the areas of the test suite, which should be imported.

You have to take care about a correct Include/Reverse-Include of your test suites to ensure, that all calls and component references are still valid. See chapter 35 for details.

24.3.1

Importing Components

During import, all 'Windows' and 'Components' of the imported test suite are integrated into the component hierarchy of the importing suite. 'Components' that already exist are not copied. A 'QF-Test ID' conflict (identical components with different 'QF-Test IDs' or differing components with identical 'QF-Test IDs') is resolved automatically by changing the 'QF-Test ID' of the imported component.

Afterwards, all 'Windows' and 'Components' are removed from the imported suite. All nodes in the imported suite that referred to these 'Components' are updated accordingly. Ideally, the imported suite should include the importing suite so no explicit suite references will have to be created.

3.3+24.3.2

Importing Procedures and Testcases

As you can import 'Components' QF-Test also allows to import 'Procedures', 'Packages', 'Dependencies' and 'Test cases' as well as 'Test sets' by choosing 'Procedures' or 'Tests' in the import dialog. You should take care about keeping all calls consistent, e.g. in most cases it does not make sense to import 'Procedures' without their required 'Components'.

In case you only want to import one dedicated 'Procedure' or 'Test case' you can use the button 'Detailimport' on the importdialog. Here you can choose any node you want to import separately.

24.4

Strategies for distributed development

There is no single best way of test development or organization, but one approach that works well is the following:

• Start with a central test suite that has the functionality needed to start and stop the SUT and a basic set of 'Tests' and 'Procedures'. This will become your master suite which will contain all 'Components'.
• Make sure that your developers have understood the importance of assigning names with setName() and that unique names are assigned consistently where needed. Where setName() is not an option, try to implement ComponentNameResolvers to achieve this (see subsection 52.1.7). You should be able to record and replay sequences without much ado and without "polluting" the 'Component' hierarchy after trivial changes in the user interface.
• Move as much functionality as possible into 'Procedures', especially commonly-used stuff and the setup and cleanup routines for the SUT.
• To create new tests, start with an empty test suite. Include the master test suite by editing the 'Include files' attribute of the 'Test suite' node of the new suite. Create the 'Setup' and 'Cleanup' nodes to start and stop the SUT by calling the respective 'Procedures' in the master suite.
• Create your tests as required. When recording sequences, the 'Components' of the master-suite will be used if possible. New 'Components' are added to the new suite, so the master suite will not be modified at this stage.
• Where possible, call 'Procedures' in the master suite for common operations.
• When your new set of tests is complete and you are satisfied that they work well, import any required nodes of your new test suite into the master suite. You have to ensure that all new 'Component' nodes that you recorded are imported into the master suite's 'Component' hierarchy in any case. The master suite's existing 'Components' will not be affected by this, so other suites that depend on the master suite will not need to be modified.
• After importing 'Components' you can import all or only the required 'Procedures' into the master suite.
• You now have various options of how to arrange the actual sequences of events and checks that form your tests. In any case it is a good idea to move everything to 'Procedures' and 'Packages' structured after your test-plan. Then the top-level 'Test set' or 'Test case' nodes of the master suite and your new suite will only contain the required hierarchy of 'Test set', 'Test case', 'Test step' and 'Sequence' nodes filled with 'Procedure calls' to the actual test cases. Such an arrangement has several advantages:
  • All your tests are structured cleanly.
  • You can easily create different sets of tests with varying complexity and run-time.
  • You have the option to keep the test cases in separate test suites and have the master suite call them. These "test case-libraries" must include the master-suite, so they need not contain any 'Components' themselves. You can organize your tests so that the master-suite will run the whole set of tests, while each separate suite can also be run standalone.
  • The tests can be maintained by several developers as long as modifications to the master suite are coordinated.
• If you decide to keep your new tests in the newly created test suite instead of moving them to the master suite, modify the master suite to tell QF-Test that there is a new test suite that depends on it. To do so, either ensure that both test suites belong to the same project or add the new test suite to the 'Dependencies' attribute of the master suite's 'Test suite' node.
• If you need to modify or extend the new test suite later, proceed as before. You can record new sequences as needed. When you are done, merge any newly created 'Components' back into the master suite.
• If your SUT changes in a way that requires updates or adaptions to the master-suite's 'Component' hierarchy, you must coordinate your test developers. Before you start updating the 'Components', make sure that all suites that directly or indirectly include the master suite belong to the same project as the master suite or are listed in the 'Dependencies' attribute of the master suite's 'Test suite' node. If modifying the 'Components' of the master suite involves any 'QF-Test component ID' changes, QF-Test will update the depending test suites accordingly, so they should not be edited simultaneously by others.
• The file format for QF-Test test suites is XML and thus plain text. As a result, test suites can be managed very well by version control systems like CVS. Changes to some 'QF-Test component ID' attributes of the depending suites can typically be merged with other changes without conflicts, alleviating the need for coordination.

Of course, the above scheme can be extended to have several master suites for testing different parts or aspects of an application. It may be a good idea to ensure that the component hierarchies in these suites don't overlap too much though. This will save you the effort of maintaining all these hierarchies in case the user interface of the SUT changes significantly.

3.1+24.5

Static validation of test suites

Working in a project over time will cause modifications, refactoring or deletion of steps in your test suite structure, e.g. you may consider renaming 'Procedures' or simply removing them once they are not required anymore.

24.5.1

Avoiding invalid references

In such cases it is quite important that you adapt all references of the according 'Procedure' in order to guarantee that the remaining tests keep running. For this purpose QF-Test automatically updates all references during the process of renaming or moving elements on demand.

If you want to ensure that your created test structure doesn't contain any call of non-existing 'Procedures' anymore, you can also use the "Analyze references" command of QF-Test in order to perform a static validation of your test suite. This command will open a dialog showing all references and whether they are still okay or something is missing.

You can trigger the analysis via a right mouse-click and selecting »Additional node operations«-»Analyze references...« or selecting the according entry from the main menu under »Operations«. This method is also available in batch mode.

		Figure 24.1:  Result of analyzing references

3.5+ QF-Test also provides features to search through your test suites for duplicate nodes, empty 'Packages' or 'Procedures' or to analyze for nodes having invalid characters in their names.

This kind of static validation is available for 'Procedures', 'Dependencies', 'Test cases', 'Test sets' and 'Components' and their references.

4.0.3+24.5.2

Unused procedures

During test development it could happen that procedures, which were used in the first version of your tests will not be used in newer versions due to re-factoring of tests. If those procedures won't get deleted immediately they will stay in the test suite and the test suite will grow and grow. Sometimes you could get the feeling that you have too many procedures or that you have lost the overview of your procedures. In order to check for such unused procedures or dependencies in your test suite you can open the context menu via a right mouse click at 'Test suite' or 'Procedures' and select »Additional node operations«-»Find unused callables...«. This operation creates a report showing any procedures or dependencies which had been created but haven't been used yet. Now you could decide what you want to do with those.

Sometimes you might simply remove all of those unused nodes immediately via »Additional node operations«-»Remove unused callables«.

Last update: 3/13/2024 Copyright © 1999-2024 Quality First Software GmbH