Video We created a video showing the integration of QF-Test into Jenkins. This video can be found on our video page

Jenkins/Hudson ( or are continuous integration build tools. They are meant to control and monitor the build process within a software project. One important step in this build process is automated testing.

There are number of benefits to be gained when integrating QF-Test with Jenkins/Hudson:

  • In case Jenkins/Hudson is already used for the continuous integration process, integration of automated GUI tests can be easily achieved.
  • Easy-to-use administration of scheduled test runs and notification of results via email or RSS.
  • Jenkins/Hudson's web based GUI provides good overview and control of test results.
  • By use of the HTML Publisher Plugin it is possible to embed QF-Test's HTML reports directly into the Jenkins/Hudson GUI.
  • Results generated during the test run such as run-logs and reports can be archived automatically. Therefore maintaining an own directory structure is not needed anymore.

As Jenkins and Hudson share a common history, following sections will use Jenkins as representative.

Install and start Jenkins

Note For GUI tests, Jenkins must not be configured to run as a service but within a real user session. On Windows the .msi installer unfortunately directly installs Jenkins as service without any further inquiry. Please beware of it therefore and ensure Jenkins is started as real user process as described below.

To install Jenkins download the war Archive (which can be found here) and start it via java -jar jenkins.war.

As soon as Jenkins is started its web interface can be accessed via http://localhost:8080. It should look like the following:

Jenkins Start Screen
Figure 25.4:  Jenkins after start-up.
Requirements for GUI tests

GUI testing requires an unlocked, active desktop. That is the only way to ensure that the SUT behaves the same as if a normal user interacts with it. Chapter Hints on setting up test-systems contains useful tips and tricks to set-up the Jenkins/Hudson process.

Jenkins allows execution of tasks on remote machines. This is of course also relevant for GUI testing. Due to its nature GUI tests are typically not intended to run on the central build server. In addition, tests might need to be executed for different environments, operating systems and SUT versions.

On a remote machine, a Jenkins agent needs to be launched in order to connect to the Jenkins server and wait for jobs to be processed. As described in the Jenkins documentation, there are several options to launch this agent, but for the GUI tests to properly work the only possible launch method is to use Java Web Start.

For GUI tests it is vital to have an active, unlocked user session. Therefore it is not possible to start the agent via a windows service but a real (test) user must be logged in (e.g. via auto login) using Windows Autostart to launch the Jenkins agent. Furthermore screen locking needs to be disabled.

NotePlease see also FAQ 14 for more technical background details.

Install QF-Test Plugin

Video Video explanation:

The QF-Test Plugin enables QF-Test to interact with Jenkins. To install the plugin open the Jenkins dashboard and navigate via "Manage Jenkins" to "Manage Plugins". Select the QF-Test Plugin from the "Available" tab. When installing the QF-Test Plugin the JUNIT and HTML-Publisher Plugin will also be downloaded automatically, in case they were not already installed. Finally restart Jenkins to complete the installation. Now the QF-Test Plugin will show up under the Installed tab, as shown in Figure 20.2.

NoteFor an in-depth look at the plugin and all of its available features, please have a look at the Youtube Video about the installation and configuration of the plugin.

NoteJenkins will automatically use the latest installed version of QF-Test. In case you want to use a different version, you can provide its path under the QF-Test section in the Jenkins configuration (Manage Jenkins -> Configure System).

QF-Test Plugin installed
Figure 25.5:  Install QF-Test Plugin.
Configure a job for QF-Test

Video Video explanation:

To create a new job open the Jenkins dashboard and click "New Item". The standard project type is "Freestyle project". Once the job is created it has to be configured.

The first thing that needs to be done is to change the workspace of this job to the directory that contains your test-suites. It can be changed in the "Advanced Project Options" under "Advanced..." where you check "Use custom workspace" and specify the test-suites' directory.
In the "Display Name" field you have the option to provide a name to be shown instead of your job name, which is typically left empty.

Custom Workspace
Figure 25.6:  Set custom workspace.

Now proceed to the "Build" section and add the "Run QF-Test" build step by choosing it from the "Add build step" drop down. In the left field you can either enter the name of a test-suite or a folder containing test-suites (e.g. your workspace directory being "C:\MySuites\" and the path of the test-suite you want to run being "C:\MySuites\qftest\mysuite.qft", simply enter qftest/mysuite.qft in the first field). In case you enter a directory, every test-suite in this directory will be run. In the right field you can enter additional Command line arguments.

Build Step
Figure 25.7:  Add build step to run QF-Test

The advanced options of the "Run QF-Test" build step are optional.
The first option allows you to provide a dedicated QF-Test version for this job.
The next option lets you change the directory for the temporary reports. The QF-Test Plugin will create a folder called "qftestJenkinsReports" in the selected workspace and save the reports and logs for the post-build actions.
Finally, there is an option to run your test-suites on a daemon. You need to enter the deamonhost and port to do so. To learn more about the daemon mode please see section 21.2 of the manual.

Advanced Options
Figure 25.8:  Configure build step advanced options.

Next the post build actions need to be configured to process test results and archive QF-Test run-logs and HTML reports.
To archive the run-logs add the post-build action "Archive the artifacts" and for "Files to archive" enter this path: qftestJenkinsReports/$JOB_NAME/$BUILD_NUMBER/logs/*.q*

To publish the HTML Reports add the post-build action "Publish HTML reports". In the HTML directory enter this path: qftestJenkinsReports/$JOB_NAME/$BUILD_NUMBER/html/ . Also change the index page from index.html to report.html. You can keep past HTML reports if you check the respective box.

Finally add the post-build action "Publish JUnit test result report" and for "Test report XMLs" enter this path: qftestJenkinsReports/$JOB_NAME/$BUILD_NUMBER/junit/report_junit.xml

NoteThere will be warnings that the path you entered doesn't match anything, because of the environment variables in the path. These warnings can be ignored.

Post Build Actions
Figure 25.9:  Configure post build steps.

NoteThis plugin supports concurrent builds, as long as your licence allows enough QF-Test instances.

NoteYou can use the additional command line argument "-dbg" to get more debug information in the console output of your build.