In addition to functional and system tests, QF-Test can also be used to perform small-to-medium sized load tests, stress tests or performance tests. The idea is to test the performance of some server by running a number of clients concurrently.

There are many different ways for setting up and performing load tests, most of which do not work with real clients. Instead they directly make use of the protocol between the client and server, e.g. by sending HTTP request or performing RMI or other kinds of remote procedure calls.

There are a number of pros and cons for protocol-based or GUI-based load testing:

• Protocol-based testing uses very little resources at the client side, so it can easily scale up to the breaking point of the server without requiring too much hardware. GUI-based tests incur the full memory and performance overhead for each client, which can be quite significant, especially in case of Swing-based rich clients. Besides QF-Test itself, a typical PC can sustain between 5 and 20 parallel SUT clients without significant loss of performance on the client side, depending on available RAM and the size and activity of the clients.
• Rich clients typically represent a complex user interface which correlates with a certain complexity of the client/server API. Creating protocol-based tests that cover most of that API can be quite an effort. On the other hand, GUI-based tests that have already been implemented for functional testing may be available for reuse. If not, it is still much easier to automate complete use-cases with QF-Test than at the protocol level.
• With GUI-based testing, actual end-to-end response times (the time span from user action until the result shows up) are measured, while protocol-based tests measure only the times for the server call. Either can be useful, depending on the situation.

In summary, GUI-based load testing can be very useful and efficient - especially if functional tests can be reused - provided that either the number of clients that need to be simulated is not too high, or that sufficient hardware is available for the client side.

QF-Test is probably unique with its ability to drive several SUT clients in parallel on a single desktop without these clients interfering with each other. To that end, QF-Test filters hard events coming from the operating system and prevents them from reaching the SUT at the wrong time, most notably when popup menus are shown or when a component must not lose keyboard focus. Without that functionality, interference between SUT clients on the same desktop would quickly break any test.

To avoid the additional overhead of having to run one QF-Test instance per SUT client, the command line argument -threads <number> can be used to run QF-Test in batch mode with multiple threads for testing. The same test-suite will be executed in each thread. One runtime license is required per thread. If insufficient runtime licenses are available, full development licenses will be used unless the command line argument -runtime is given which forces use of runtime licenses only. Please see the documentation for the -runtime command line argument in section 32.2 for further information about runtime licenses.

All threads are independent and each thread will create and control its own set of processes and SUT clients. To that end, QF-Test appends the thread identifier to the 'Client' attribute of any node that needs to refer to a SUT client. Thus it is impossible to access one thread's SUT client from another thread.

Two variables have special values when running with multiple threads: ${qftest:threads} expands to the number of threads given with -threads <number> (default is 1 when running single-threaded) and ${qftest:thread} is the index of the current thread (default is 0).

Running multiple clients on one desktop can sometimes lead to delays longer than usual, especially during startup of the SUTs. Therefore, timeouts should be increased generously, especially in 'Wait for client to connect' and 'Wait for component to appear' nodes. Also, the options under Replay->Timeouts (see subsection 29.3.5) should be increased to at least twice their default values. It is a good idea to use a separate set of system options for load testing by specifying the configuration file with the command line argument -systemcfg <file>.

To get consistent results, it may sometimes be necessary to coordinate the tests in the parallel threads, either to make sure that all clients access the server simultaneously, or to prevent just that.

Threads can be synchronized with the help of a 'Server script' node with the following script:

rc.syncThreads("identifier", timeout)

The identifier is a name for the synchronization point and timeout is the maximum time in milliseconds to wait for all threads to reach the given synchronization point. If not all threads are expected to reach that point because they follow different execution paths, the additional parameter count can be given to identify the number of threads to expect, e.g.

rc.syncThreads("case1", 120000, count=3)

If the timeout is exceeded without the expected number of threads reaching the synchronization point, a TestException is thrown. To log an error instead of raising an exception, set the optional parameter throw to 0 (default value 1), e.g.:

rc.syncThreads("case1", 120000, count=3, throw=0)

If more than one machine is required to sustain the desired number of clients, tests can be run in parallel on multiple machines. Each QF-Test instance on each machine can use multiple threads as long as sufficient licenses for the total number of threads are available.

Currently QF-Test has no special support for coordinating the start of QF-Test instances on different machines. This has to be done manually, through a script or through some third party tool. However, QF-Test does support synchronization of threads across multiple machines. The syncThreads method has one additional parameter called remote which defaults to 0 (no remote synchronization) and represents the number of QF-Test instances to synchronize. Each instance will first synchronize its running threads internally, then use network calls to synchronize with other running instances. Only when all threads in all requested machines have reached the synchronization point the test will continue.

For example, to synchronize all threads in tests on 4 machines after SUT startup, add a 'Server script' node at the end of the main setup sequence with something like:

rc.syncThreads("startup", 300000, remote=4)

It is a good idea not to hard-code the number of remote instances to expect. Instead, use a variable defined at test-suite or system level that defaults to 0 so the tests will run fine on a single machine. When running on multiple machines, override that variable on the command line as explained in chapter 8.

Last update: 04/23/2012 Copyright © 1999-2012 Quality First Software GmbH