When talking about test execution, there are two aspects to be considered. On one hand you need to run tests while they are developed to check them for proper operation. This situation has already been described in section 5.2. Basically all you have to do to run a test interactively is invoking »Run«-»Start« from the main menu.

On the other hand you want to run your tests periodically to ensure the stability of the system under test, for example in nightly regression tests. Instead of launching QF-Test, loading the test-suite and running it from the graphical user interface, it is much more convenient here to execute tests from the command line in batch mode. This kind of running tests is explained in the first section of this chapter (Test execution in batch mode).

Sometimes, for instance when you want to run the test on a remote computer, a second variant comes into play: the daemon mode. This type of test execution, which uses a running QF-Test instance to execute tests, is the topic of the second section (Executing tests in daemon mode).

For integration of QF-Test with build tools like ant, maven or Hudson/Jenkins, please refer to chapter 20.

There are a lot of command line arguments when running QF-Test in batch mode; an overview can be found in chapter 32. Here we will present examples showing the most important of them.

The examples are written for the Windows operating system, but you may easily adapt them for the Linux platform. What is different is the path specification and also the syntax for placeholders (subsection 32.2.4): On Linux you can use +X as well as %X. On Windows there's a separate console application qftestc.exe. In contrast to its GUI variant qftest.exe, it waits until the execution of QF-Test has terminated and also displays print output from a 'Server script'. You can use qftestc.exe in place of qftest.exe wherever you'll find it convenient.

Let's start with the most simple QF-Test command to execute a test:

qftest -batch -run c:\mysuites\suiteA.qft

The argument -batch makes QF-Test start without a graphical user interface. The second argument, -run, is the specifier for test execution. Finally, at the end of the command line, you find the test-suite to be executed.

Note The argument -run is optional, i. e. the test execution is defined as default for the batch mode.

When running the above command, all top-level 'Test', 'Test-case' and 'Test-set' nodes of suiteA.qft will be executed one after another. After the test-run you will find a run-log file in the current directory; it has the same name as the test-suite (except from the extension, which can be .qrl, .qrz or .qzp). The run-log file shows the result of the test-run.

By specifying -nolog you can suppress the creation of a run-log. Probably this only makes sense, if you have extended your test by your own log output (written to a file). Otherwise you'd have to check the result code of QF-Test, whereas 0 means that everything is alright. A positive value in contrast indicates that warnings, errors or exceptions occurred during the test-run (see section 32.3). That's why in most situations you'll probably prefer to create a run-log and save it at a fixed place in the file system. This can be achieved with the parameter -runlog:

qftest -batch -compact -runlog c:\mylogs\+b c:\mysuites\suiteA.qft

A run-log file suiteA.qrz will now be created in the specified directory c:\mylogs. The placeholder +b is responsible for its name being identical with that of the test-suite. The additional switch -compact prevents the run-log from growing too large: Only the nodes needed for a report and those immediately before an error or an exception are kept in the run-log. Especially in case of very long test-runs this may help to reduce the amount of required memory. In the meantime however, QF-Test offers split run-logs for the same reason. For more information about this topic see section 9.1.

Note Whether the file is indeed created as compressed run-log (to be distinguished from the above "compact") with extension .qrz, depends on the system settings. To force the creation of a particular format you can set the file extension explicitly. With -runlog c:\mylogs\+b.qrl, for example, an uncompressed XML file will be produced.

Sometimes you may want to execute not the whole test-suite but only parts of it. By using the parameter -test you can run a specific node of the test-suite:

qftest -batch -runlog c:\mylogs\+b -test "My test-case" c:\mysuites\suiteA.qft

The parameter -test expects the 'Id' attribute of the node to follow or the qualified name of a 'Test-case' or 'Test-set'. If you want to execute several nodes, you can define -test <Id> multiple times. Apart from the node's 'Id', -test accepts also the numerical index of a top-level node. For example, -test 0 will run the first child of the 'Test-suite' node.

The run-log provides a rather technical view of the test-run; it is helpful mainly when analyzing errors (cf. section 9.1). The report in contrast contains a summary of the executed test-cases and errors (cf. chapter 15) in XML or HTML format. It is created from the run-log either in a separate step after running the test or automatically with the test-run:

qftest -batch -runlog c:\mylogs\+b -report c:\mylogs\rep_+b_+y+M+d+h+m c:\mysuites\suiteA.qft

In this example the XML and HTML files are saved in a directory which name consists of the test-suite and a timestamp like c:\mylogs\rep_suiteA_0806042152. When replacing the argument -report with -report.xml or -report.html respectively, only an XML or HTML report will be created.

Test-cases often uses variables to control the execution of the test. For example, you may have defined the variable myvar in the 'Test-suite' node of the suite. You can overwrite its default value when running the test-suite from the command line:

qftest -batch -variable myvar="Value from command line" -runlog c:\mylogs\+b c:\mysuites\suiteA.qft

If needed, you can specify -variable <name>=<wert> multiple times to set values for different variables.

Running tests from the command line is fundamental for integrating QF-Test in test management systems (see Interaction with Test Management Tools). Otherwise, living without such a tool, you may find it convenient to embed the command for the test execution into a script. A simple Windows batch script (qfbatch.bat) looks like this:

@echo off
setlocal
if "%1" == "" (
    echo Usage: qfbatch Testsuite
    goto end
) else (
    set suite=%~f1
)
set logdir=c:\mylogs
pushd c:\programs\qftest\qftest-3.4.7\bin

@echo on
.\qftest -batch -compact -runlog %logdir%\+b %suite%
@echo off

if %errorlevel% equ 0 (
  echo Test terminated successfully
  goto end
)
if %errorlevel% equ 1 (
  echo Test terminated with warnings
  goto end
)
if %errorlevel% equ 2 (
  echo Test terminated with errors
  goto end
)
if %errorlevel% equ 3 (
  echo Test terminated with exceptions
  goto end
)
if %errorlevel% leq -1 (
  echo Error %errorlevel%
  goto end
)

:end
popd

Now you can simply run that script with only the file name of the test-suite as parameter. Everything else is done automatically: The test-suite will be executed, the run-log file stored in logdir and finally the script will print out the state of the test-run (depending on the QF-Test result code).

Since version QF-Test 3 the language Groovy is part of the release (cf. chapter 13). It is meant mainly for scripting inside QF-Test (Server and SUT scripts), but it can, like Jython, also be used outside of QF-Test. Groovy is probably well suited to create a little test execution management system by yourself. By the way, Groovy simplifies working with Ant, too: Instead of dealing with bulky XML files, which makes it hard to define conditions, you can work with the Groovy AntBuilder. However, that's out of scope here, the following example doesn't rely on Ant but only on the basic Groovy features:

def suite = ''
if (args.size() == 0) {
    println 'Usage: groovy QfExec Testsuite'
    return
}
else {
    suite = args[0]
}

def qftestdir = 'c:\\programs\\qfs\\qftest\\qftest-3.4.7'
def qftest = qftestdir + '\\bin\\qftest.exe'
def command = "$qftest -batch -compact -runlog c:\\mylogs\\+b \"$suite\""
def printStream = { stream ->
    while (true) {
        try {
            stream.eachLine { println it }
        } catch (IOException) {
            break
        }
    }
}

println "Running command: $command"
def proc = command.execute()
new Thread().start() { printStream(proc.in) }
new Thread().start() { printStream(proc.err) }
proc.waitFor()

switch (proc.exitValue()) {
    case '0': println 'Test terminated successfully'; break
    case '1': println 'Test terminated with warnings'; break
    case '2': println 'Test terminated with errors'; break
    case '3': println 'Test terminated with exceptions'; break
    default: println "Error ${proc.exitValue()}"
}

If you have Groovy installed on your computer independently of QF-Test, you can run the example test-suite simply via groovy QfExec c:\mysuites\suiteA.qft. Otherwise you can use the Groovy jar file from the QF-Test installation, preferably again with help of a batch script:

@echo off
setlocal
if "%1" == "" (
    echo Usage: qfexec Testsuite
    goto end
)
set qftestdir=c:\programs\qftest\qftest-3.4.7
set scriptfile=QfExec.groovy
java -cp %qftestdir%/lib/groovy-all.jar groovy.ui.GroovyMain %scriptfile% %*
:end

Now execute the test-suite with qfexec c:\mysuites\suiteA.qft.

In daemon mode QF-Test listens to RMI connections and provides an interface for remote test execution. This is useful for simplifying test execution in a distributed load-testing scenario (chapter 21), but also for integration with existing test-management or test-execution tools (chapter 19).

!!! Warning !!! When run in daemon mode, QF-Test will accept connections from anyone that can access the host and port. No checking of user name and access rights is performed. Don't use this feature outside a secure environment.

To work with a daemon, you must first launch it on any computer in your network (of course, this host can also be localhost):

qftest -batch -daemon -daemonport 12345

If you omit the argument -daemonport, the daemon will listen on QF-Test's standard port 3543. You may check whether the daemon is running by means of the netstat utility:

Windowsnetstat -a -p tcp -n | findstr "12345"

Linuxnetstat -a --tcp --numeric-ports | grep 12345

If you want to launch a daemon on a remote host, you may use for instance ssh or VNC. Your network administrator knows whether and how this works. To follow the examples below, a local daemon will be sufficient.

The easiest way to get in touch with a daemon is running QF-Test from the command line in the calldaemon mode. The following example checks if a daemon is listening at the specified host and port:

qftestc -batch -calldaemon -daemonhost localhost -daemonport 12345 -ping && echo Daemon is running

In contrast to the netstat command from above -ping also works between different computers (if you check the daemon on your local computer, you can omit the argument -daemonhost).

What you actually want from a daemon is executing your test-case(s) and getting back a run-log file. It sounds and indeed looks quite similar to what you have seen before when running a test in batch mode:

qftest -batch -calldaemon -daemonhost somehost -daemonport 12345 -runlog c:\mylogs\+b -suitedir c:\mysuites suiteA.qft#"My test-case"

Note In contrast to the batch mode, a 'Test-case' or a 'Test-set' node is always referenced here by its qualified name, for instance "My Test-set.My Test-case" (just to remember: -test <Id> may be used in batch mode). To execute the complete suite suiteA.qft, you can simply omit the test-case or write suiteA.qft#..

If the daemon is running on a remote host, you have to specify it explicitly via -daemonhost (default is -daemonhost localhost). Note that the parameter -suitedir refers to the remote host (where the daemon is running) while -runlog defines a local file.

3.4+ In case you cannot easily observe the test running on a remote host, you may find it convenient to add the argument -verbose to get status output in the console (on Windows, use qftestc to see the output).

A running daemon, no matter whether local or remote, can be terminated with the calldaemon command -terminate:

qftest -batch -calldaemon -daemonport 12345 -daemonhost localhost -terminate

A complete list of the calldaemon parameters can be found in the chapter Command line arguments and exit codes.

Using the QF-Test command line to control a daemon was quite easy. On the other hand, to get all capabilities of a daemon, you have to deal with the daemon API. In this section we will concentrate on some basic examples, the whole interface is described in chapter 40.

To get started with the daemon API, insert a 'Server script' node with the following code:

from de.qfs.apps.qftest.daemon import DaemonRunContext
from de.qfs.apps.qftest.daemon import DaemonLocator

host = "localhost"
port = 12345
# Leading r means raw string to allow normal backslashes in the path string.
testcase = r"c:\mysuites\suiteA.qft#My test-case"
timeout = 60 * 1000

def calldaemon(host, port, testcase, timeout=0):
    daemon = DaemonLocator.instance().locateDaemon(host, port)
    trd = daemon.createTestRunDaemon()
    context = trd.createContext()
    context.runTest(testcase)
    context.waitForRunState(DaemonRunContext.STATE_FINISHED, timeout)
    result = context.getResult()
    log = context.getRunLog()
    rc.addDaemonLog(log)
    context.release()
    return result

result = calldaemon(host, port, testcase, timeout)
rc.logMessage("Result from daemon: %d" %result)

The script shows the basic mechanisms to control a daemon:

• First find a running daemon with locateDaemon.
• Provide an environment for test-runs by calling createTestRunDaemon.
• To run a test, you need a context object (createContext). The creation of that object requires a QF-Test run-time license.
• Now the context enables you to start a test-run (runTest) and to query about its current state. waitForRunState waits during the defined timeout (in milliseconds) until the specified state has occurred. In the example above, we wait for the test to terminate within one minute.
• Finally, when the test-run has terminated, the context can query the test result with the method getResult (cf. Exit codes for QF-Test).
• Moreover, you can use the context to get the run-log of the daemon test-run. It can be included in the local run-log by means of the rc method addDaemonLog.

Note To keep it small and simple, the example script does not contain any error handling. However, particularly when working with a daemon, you should check every method call.

Note Driving a daemon from a 'Server script' has the disadvantage of consuming an additional QF-Test license to run the script node interactively or in batch mode. However, this doesn't apply nor for the above-mentioned calldaemon mode neither for the case when controlling a daemon outside QF-Test (see below).

The usage of the daemon API is not restricted to 'Server scripts'. Outside QF-Test a daemon can be contacted by means of a Java program or, more easily, a Groovy script. The following Groovy script works with several running daemons and may serve as a starting point for load tests. Suppose we have started some daemons in our network, each on a separate machine. We want to execute a test-case simultaneously by all of the daemons and we want to save a run-log for every single test-run (daemon1.qrl, ..., daemonN.qrl). The test-suite containing the test-case to be executed may be available for all daemon instances via the network drive z:).

import de.qfs.apps.qftest.daemon.DaemonLocator
import de.qfs.apps.qftest.daemon.DaemonRunContext

def testcase = "z:\\mysuites\\suiteA.qft#My test-case"
def logfile = "c:\\mylogs\\daemon"
def timeout = 120 * 1000

def daemons = DaemonLocator.instance().locateDaemons(10000)
def contexts = []
// Start tests
for (daemon in daemons) {
    def trd = daemon.createTestRunDaemon()
    trd.setGlobal('machines', daemons.size().toString())
    def context = trd.createContext()
    contexts << context
    context.runTest(testcase)
}
// Wait for tests to terminate
for (i in 0..<contexts.size()) {
    def context = contexts[i]
    context.waitForRunState(DaemonRunContext.STATE_FINISHED, timeout)
    byte[] runlog = context.getRunLog()
    def fos = new FileOutputStream("$logfile${i + 1}.qrl")
    fos.write(runlog)
    fos.close()
    context.release()
}

To run that Groovy script, you need the QF-Test libraries qftest.jar, qfshared.jar, and qflib.jar as well as the Groovy library, which is also part of the QF-Test installation. The following batch script shows how it works:

@echo off
setlocal
set qftestdir=c:\programs\qftest\qftest-3.4.7
set qflibdir=%qftestdir%\qflib
set classpath=%qftestdir%\lib\groovy-all.jar
set classpath=%classpath%;%qflibdir%\qftest.jar;%qflibdir%\qfshared.jar;%qflibdir%\qflib.jar
java -cp %classpath% groovy.ui.GroovyMain CallDaemon

To make the daemon example a load test (cf. chapter 21), you have to synchronize the test-runs inside of "My test-case" (e. g. after starting the SUT). This can be done by means of the rc method syncThreads:

def machines = rc.getNum('machines')
rc.syncThreads('startup', 60000, -1, machines)

The variable machines denotes the number of hosts with a daemon running on them. Best define it in the 'Test-suite' node of the test-suite with a default value of 1. When running the Groovy script, it will be overwritten with the correct value.

Last update: 04/23/2012 Copyright © 1999-2012 Quality First Software GmbH