Automated GUI testing is just one step in the build process of a software development project. The basic need for coordinating all those steps of compiling, testing, documenting and deploying of software has lead to a variety of build tools (e.g. make, ant, maven) and so called continuous integration tools (e.g. Jenkins/Hudson, Cruise Control, Continuum).

In general, by use of QF-Test's command line interface as documented in chapter 16 and chapter 32 a straight forward integration with those tools should be possible.

The following sections contain examples for integrations with a some of the tools mentioned above.

People who are using Apache Ant (http://ant.apache.org) as build system may easily integrate QF-Test in their build file:

<project name="QF-Test" default="runtest">
      
    <property name="qftest" location="c:\Program Files\qfs\qftest\qftest-3.4.7\bin\qftest.exe" />
    <property name="logdir" value="c:\mylogs" />
      
    <target name="runtest" description="Run a test in batchmode">
        <echo message="Running ${suite} ..." />
        <exec executable="${qftest}" failonerror="false" resultproperty="returncode">
            <arg value="-batch" />
            <arg value="-compact" />
            <arg value="-runlog" />
            <arg value="${logdir}\+b" />
            <arg value="${suite}" />
        </exec>
        <condition property="result" value="Test terminated successfully.">
            <equals arg1="${returncode}" arg2="0" />
        </condition>
        <condition property="result" value="Test terminated with warnings.">
            <equals arg1="${returncode}" arg2="1" />
        </condition>
        <condition property="result" value="Test terminated with errors.">
           <equals arg1="${returncode}" arg2="2" />
        </condition>
        <condition property="result" value="Test terminated with exceptions.">
            <equals arg1="${returncode}" arg2="3" />
        </condition>
        <echo message="${result}" />
    </target>
      
</project>
        

The above example assumes the test-suite to be defined as property when running ant: ant -Dsuite="c:\Program Files\qfs\qftest\qftest-3.4.7\demo\carconfig\carconfig_en.qft".

People who are using Apache Maven (http://maven.apache.org) as build system may easily integrate QF-Test in their build. This can be achieved by using the antrun plugin of Maven. A demo pom.xml file, where QF-Tests tests are executed in the test phase could look like this:

<project xmlns="http://maven.apache.org/POM/4.0.0"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
    http://maven.apache.org/maven-v4_0_0.xsd">

    <modelVersion>4.0.0</modelVersion>
    <artifactId>testant</artifactId>
    <packaging>jar</packaging>
    <name>testant</name>
    <groupId>de.qfs</groupId>
    <version>1</version>

    <properties>
        <qf.exe>"C:\Program Files\qfs\qftest\qftest-3.4.7\bin\qftest.exe</qf.exe>
        <qf.reportfolder>qftest</qf.reportfolder>
        <qf.log>logFile.qrz</qf.log>
        <qf.suite>"c:\path\to\testsuite.qft"</qf.suite>
    </properties>

    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-antrun-plugin</artifactId>
                <executions>
                    <execution>
                        <phase>test</phase>
                        <configuration>
                            <tasks>
                                <exec executable="${qf.exe}">
                                    <arg value="-batch"/>
                                    <arg value="-report"/>
                                    <arg value="${qf.reportfolder}"/>
                                    <arg value="-runlog"/>
                                    <arg value="${qf.log}"/>
                                    <arg value="${qf.suite}"/>
                                </exec>
                            </tasks>
                        </configuration>
                        <goals>
                            <goal>run</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>

</project>
        

In your project it might become required to run the tests during another build phase, than the configured test phase in the example. In this case you have to configure the plugin accordingly, like described in the Maven documentation.

Jenkins/Hudson (http://jenkins-ci.org or http://hudson-ci.org) are continuous integration build tools. Therefore they are meant to control and monitor the build process within a software project. One important step in this build process is automated testing.

There are number of benefits to be gained when integrating QF-Test with Jenkins/Hudson:

• In case Jenkins/Hudson is already used for the continuous integration process, integration of automated GUI tests can be easily achieved.
• Easy-to-use administration of scheduled test runs and notification of results via email or RSS.
• Jenkins/Hudson's web based GUI provides good overview and control of test results.
• By use of the HTML Publisher Plugin it is possible to embed QF-Test's HTML reports directly into the Jenkins/Hudson GUI.
• Results generated during the test run such as run-logs and reports can be archived automatically. Therefore maintaining an own directory structure is not needed anymore.

As Jenkins and Hudson share a common history, following chapters just use Hudson as representative.

In general, current information on download and installation of Hudson can be found at http://hudson-ci.org.

Installing Hudson just means to download the war Archive and start it e.g. via java -jar hudson.war.

Another option without installation is to run Hudson by use of Java Webstart directly via https://hudson.dev.java.net/hudson.jnlp.

As soon as Hudson is started it's web interface can be accessed via http://localhost:8080. It should look like the following:

		Figure 20.1:  Hudson after start-up

To create a new job you just need to follow the respective link in Hudson (see above figure) and to select "Build a free-style software project" which leads you to a form containing all the build and report option. All options are documented ("?" buttons right beside each option), so filling the form is very easy.

Please note the following specifics related to QF-Test:

• QF-Test test execution can be defined as Hudson build step, e.g. as Windows batch file or shell script. Please see chapter 16 how to execute QF-Test from the command line. The QF-Test demo project, as described in subsection 20.3.3, contains respective example scripts also handling Exit codes for QF-Test to show test-runs with the correct passed/failed state in Hudson.
  3.4.2+The usage of command line argument -exitcodeignoreexception now directly ignores exeptions, errors and warnings for the exit code.
• In case you want to use your own custom workspace, this can be configured in the Advanced Project Options.
• The option "Archive the artefacts" allows to keep given files, helpful e.g. for generated run-logs to allow a later error analysis.
• Another important option is "Publish JUnit test result report". JUnit reports, as QF-Test is able to create when being invoked with the -report.junit <directory> parameter, allow Hudson to generate a graphical trend chart as well as a short result summary for a test-run. Note that only results of 'Test-cases' are taken into account for JUnit reports.
• To be able to embed the comprehensive HTML reports QF-Test generates by use of the -report.html <directory> parameter, the HTML Publisher plugin in Hudson needs to be installed. This can be easily achieved via »Hudson«-»Manage Hudson«-»Manage Plugins«. After reloading Hudson there should be a respective option available in the configuration form of your job.

NoteHaving the HTML Publisher Plugin installed, as described above, is recommended before starting with the demo.

When Hudson is successfully installed and run once, there should be a directory .hudson (note the leading dot) available in you user directory (e.g. C:/Documents and Settings/Your User Name) in which all Hudson related files are stored. Among others it contains a folder called jobs holding all data related to a certain project i.e. configuration files and build results.

To use activate the demo project in Hudson you simply need to extract the Zip archive QF-Test_Demo_Job.zip from qftest-3.4.7/ext/hudson of your QF-Test installation into your .hudson/jobs directory. To make the project now visible in Hudson you might need to restart it or you simply use »Hudson«-»Manage Hudson«-»Reload Configuration from Disk«.

After choosing the now available QF-Test demo job you should see a page like the following:

		Figure 20.2:  QF-Test Demo Job in Hudson

The test result trend chart as well as the latest test result summary are generated from the QF-Test JUnit reports.

Thanks to the HTML Publisher Plugin also HTML reports are directly available. You may now have a look at the various builds or start a new one yourself via »Build Now«.

The »Configuration« link allows you to inspect and possibly adapt the build options. They might prove useful as basis for your own projects.

NoteThe batch file runtest.bat and the shell script runtest.sh, located in the qftest_demo directory, contain handling of the Exit codes for QF-Test to show test-runs with the correct passed/failed state in Hudson.
3.4.2+The usage of command line argument -exitcodeignoreexception now directly ignores warnings for the exit code.

For troubleshooting the consol output within the build details is a good point to start.

NoteOn Linux you can replace the "Execute windows batch command" by a "Execute shell" build step with following content:

cd "$WORKSPACE/qftest_demo"
./runtest.sh
          

Last update: 04/23/2012 Copyright © 1999-2012 Quality First Software GmbH