Logo QF-Test

Tutorial as a practical
introduction to QF-Test


Free Trial  Download  Buy

Yann Spöri, QF-Test development & support

The QF-Test examples, demos and practical exercises give you a quick start.

Yann Spöri,
Software Engineer, QFS

Uwe Klüh, Senior Sales Manager, QFS

Search for the information you need in the whole documentation (Manual, Tutorial, Mailing list, Standard library) by using the onsite-search.

Uwe Klüh, Sr. Sales Manager, QFS


Working with a Sample Test-suite [30-45 min]
Before you start

On first startup of QF-Test and/or the System Under Test (SUT) via QF-Test you might get a security warning from the Windows firewall asking whether to block Java or not. As QF-Test communicates with the SUT by means of network protocols, this must not be blocked by the local firewall in order to allow automated testing.

Starting QF-Test and Loading the Test-suite

After starting up QF-Test, you can immediately bring up our first example test-suite. You can locate the test-suite file by bringing up the open-file dialog window with the »File«-»Open« menu option. Select the directory qftest-4.2.0-dev/doc/tutorial from your QF-Test installation. From this directory, select the file Options.qft. QF-Test will then load the indicated test-suite which should look as follows:

Figure 1.1:  The Test-suite Options.qft

The left part of the main window contains the tree structure that represents the test-suite. The right side displays the details of a selected node. (In case you don't see the details view, you can switch it on with the »View«-»Details« menu option.) At the bottom of the main window you'll see the terminal output area, which displays standard messages and communications in between your test-suite and the client application you are testing. Displaying the terminal output can be toggled on or off using the »View«-»Terminal«-»Show« menu option.

With the help of the tree structure in the main window, you can navigate and select individual nodes of the test-suite. When a node is selected, you'll see its properties displayed in the details area on the right.

Double click on the node "Test-set: Options" to expand the nodes underneath it. You'll see that basically three test-cases are contained in this test-set: "Clickstream", "Text check" and "Selected test". Those test-case nodes are enclosed by a "Setup"/"Cleanup" pair to ensure a proper system as base to run the test-cases on.

Figure 1.2:  The Content of the Node "Test-set: Options"

In the following sections we'll describe the purpose and function of these individual nodes.

Starting the Application

Our first step will be to examine the "Setup" node. Expand this node now, so that you see the contents as shown in the following diagram:

Figure 1.3:  The Setup Node

Within the "Setup" node you'll see three child-nodes:

  • "Start Java SUT client" - starts the application for the client you will be testing, called the System Under Test (SUT).
  • "Wait for client to connect" - waits until the new Java Virtual Machine, in which your SUT will run, binds itself to QF-Test.
  • "Wait for component" - waits for the appearance of a certain component of the SUT. For this example, we wait for the main window of the SUT itself. Don't worry if this step isn't completely clear at the moment, we'll explain later about components and their meaning.

Now click on the "Start Java SUT client" node so that it is selected as seen below:

Figure 1.4:  The Start Java SUT Client Node

Let's take a closer look at the properties of this node, as displayed in the details window on the right side:

  • In the first field, labeled "Client," is an unambiguous name for the SUT application. You can give the SUT any name you wish, but the same name will be used as a reference to the SUT in other nodes. In other words, this name must be used consistently.
  • The "Executable" is the second field, in which the executable name of the program is specified. For our example, the variable "${qftest:java}" has been given. This variable represents the Java program you established during the installation of QF-Test. You could alternatively manually enter the entire path to your Java Virtual Machine, along with the java executable. With the assistance of the "Select executable file" button Select executable file you can enter in an executable with a standard file-browser.
  • The field "Directory" indicates the working directory for the SUT when it is started.
  • The fourth field, labeled "Class name," specifies the Java class used to implement the SUT. This class must contain the main() method, which will be called by Java when the SUT is started.
  • Under "Parameters" you see the command line arguments for the SUT. In this example we specify the argument to set the application language to English. In addition, the classpath is set appropriately, which is necessary because a class of QF-Test itself is used as SUT.
  • The remaining property fields can be left empty and are not required for our example. You will note, however, that a comment has been filled in (the last field). All nodes have an optional comment field which you as a test-suite developer may find helpful for documenting a node's purpose.

At this point, we're ready to start the SUT. Click on the "Setup" node again (so that it is selected), but still expanded (the child-nodes still visible). Now click on the "Start test run" button Play. This button causes the selected node - "Setup" in our case - to be executed.

When a node executes, QF-Test will use the tree structure to show you what is happening. An active node will be marked with an arrow pointer "->" that indicates which node is being executed. You'll also see messages from the SUT client in the terminal display.

When the "Setup" sequence is completed, our demo application "Options Demo" will appear on your screen. This is the SUT, fully under the control of the watchful eye of QF-Test.

Note If the Options demo is not visible on your screen or only appears for a short moment, it is probably covered by the main window of QF-Test. The best solution is to place both windows side by side so you can watch them at the same time. In case you get an error dialog indicating that QF-Test is not able to connect to the SUT, please take a look at manual chapter 3.1.3.

Figure 1.5:  The Options Demo
Clickstream Test-case

Our next task is to take a first look at how a test-case for the SUT in QF-Test is structured. The test-case node labeled "Clickstream" contains several sequences of operations that perform functions within the SUT, such as mouse-clicks and populating text-fields. Through this test we'll demonstrate several simple but very important features of QF-Test.

When you expand the "Clickstream" test-case node, you'll see the two sequences as shown below:

Figure 1.6:  The Clickstream Test-case Node

To execute the test, select the "Clickstream" test-case node and click the replay button Play. The sequences will then be carried out directly in the SUT, as you'll see during execution and are supposed to pass through without problems.

The test result is indicated during and after the test run in the status line at the bottom of the QF-Test main window and says "No errors". Next to it there are counters for the numbers and results of the executed test-cases. In our case it was just one that performed error-free which means a success rate of 100%.

Hint: Let the mouse cursor hover above a counter icon to see its description. A listing of all possible counters can be found in chapter Capture and replay of the manual.

For a better understanding of what you saw during replay, let's take a look at the sequence "Table" within the "Clickstream" test-case:

Figure 1.7:  The "Table" Test Sequence

Here we see that the sequence starts with several mouse clicks. Next to the "Mouse click" label you'll see the actual coordinates used to implement the mouse click relative to the component listed in the brackets next to the coordinates. The first node is, for example, a click on the element of the SUT labeled "Table" in the tree-structure "OptionsGroup-tree-Tree". The meaning of the syntax in detail will be explained later.

After the first few mouse-clicks in the sequence there is a "Wait for component" node that causes the execution of the sequence to wait for the appearance of the SUT's edit dialog. After the dialog window appears, execution of the sequence will continue, in which the "Input" node is activated to enter a string into a field in the dialog window. The remaining nodes of the sequence have similar functions to those mentioned above and should be comprehensible from the descriptions you see.

In case you're feeling curious, expand the other sequence ("Tab") too. Inside this sequence you'll find similar nodes to those you saw in "Table" with the addition of one node used to control the keyboard.

A Few Tips

We take a short break in this section to give a few tips that might prove helpful once you continue on with the tutorial.

While working with various nodes in QF-Test you may, for example, require some assistance in remembering what purpose they are used for. To serve this need, QF-Test comes with context-sensitive help, driven simply by your mouse. Move the mouse pointer over an element you would like to have assistance with, then click the right mouse-button. In the now appearing popup menu, you'll see an entry labeled "What's this?" By selecting this option from the popup menu, the appropriate section in the user's manual will be brought up and displayed in your standard browser. You'll hopefully find the manual quite helpful and complete in answering questions that may arise.

We should also mention that there is a PDF Version of the user's manual available in the installation directory of QF-Test. Within the PDF version you may find it easier (in comparison to the HTML version) to search for specific terms and concepts.

Text Check

One of the most important concepts in QF-Test is that of a check; that is, a query of certain elements in the SUT. A 'text check' queries the existence or absence of text within a component of the SUT, such as a text field.

For example, let's make a check of a field in the SUT client. Bring up the window of the "Options Demo" and click on the "Miscellaneous" node listed in the left-hand side tree structure so that the node expands to display its child nodes. Within the child nodes you'll see the node "Numbers." Click on this node, which will bring up on the right side a new window which includes the text field labeled "May be negative." For our example, we'll do a check of this field.

Figure 1.8:  The Text Field to Check

Now return to your test-suite and expand the "Text check" test-case node. Within you will see the following test sequence:

Figure 1.9:  The Text Check Test-case

It consists of two mouse clicks, aiming to open the "Misc" node and select the node "Numbers", and a check of the label text.

With the "Check text" test-case node selected, click on the replay button to execute it. When the sequence completes, a dialog-box will appear indicating that an error occurred:

Figure 1.10:  Error in Check Text Node

What happened? Whenever such incidents occur, taking a look at the QF-Test run-log can prove very useful for diagnosis. You can open it directly via the "Open run-log" button on the error dialog. Alternatively the latest run-log can be opened from the test-suite via the »Run«-»1. ...« menu item or by pressing [Control-L]. A new window will then appear displaying the recorded contents of the sequence you just executed:

Figure 1.11:  Run-Log for the Check Text Sequence

The run-log is similar in structure to the tree structure you are already familiar with from the normal test-suite view. The tree on the left side represents the time-recorded events of the test run, with the first event at the top, the last at the bottom. When you click on one of the event nodes on the left side, the properties of the event will be displayed on the right.

As you expand the nodes in the tree structure on the left, you'll notice immediately that some nodes are surrounded by a suspicious-looking red box. As you may suspect, this is an indication by QF-Test locating where a problem occurred in a child-node. If you keep expanding the red nodes, you'll eventually come to the actual problem node. An easier way to find the error is to simply use the »Edit«-»Find next error« menu option of the run-log's menu. This leads to the following:

Figure 1.12:  Diagnosis of Error in the Check Text Sequence.

When you click on the final red node, you'll see in the details of this node that there was a deviation between the text which was expected in our SUT field and the one which was actually found. The typical next step is to jump to the corresponding node in the test-suite and correct it. Provided that the appropriate node in the run-log is selected, you can use »Edit«-»Find node in test-suite« or simply type [Control-T]. One step beyond goes a functionality provided by the run-log node titled "Failed: Check text" in figure 1.12 available from the context menu, to be opened via a right click - »Update check node with current data« or [Control-U].

Another useful feature for error diagnosis is the tree node labeled "screenshot" in the run-log. Its details contain a full screenshot taken at the time when the error happened. Being able to see the state of the SUT at that moment is a valuable aid in determining the cause of the error. The following image shows a screenshot node:

Figure 1.13:  Screenshot node showing the error situation

Beside a screenshot of the whole screen, QF-Test also saves independent images of all SUT windows. For our example it is the run-log node "Screenshot of window: Option Demo". Such can be helpful when the SUT is hidden by another application window at the time when the error occurs.

Note: The information gathered from a long test-run accumulates and can devour enormous amounts of memory. Therefore QF-Test is typically configured to create a special form of compact run-log. Herein only the last 100 protocol nodes are kept, the rest is discarded, except for information relevant for report generation and for error diagnosis which is always retained. This functionality is configurable through the option "Create compact run-log" within »Edit«-»Options «-»Run-logs«-»Content«. The type of a run-log is also shown in its root node. The number of screenshots stored in the run-log can be configured as well.

As a simple exercise, you may wish to modify the expected text in the "Check text" node of your test-suite. Now when you execute the sequence, no error will occur. Please reverse the text to its previous state by »Edit«-»Undo« or [Control-Z] once the test has finished, because we will need the error for demonstration purpose later on.

Checking a Radio Button

The third test sequence "Selected test" performs queries about the state of a specific radio button in the SUT in order to determine if that radio button is selected. Under the "Choices" section of the "Miscellaneous" area in the "Options Demo" application, you can interactively select any one of four radio buttons. We'll use these radio buttons to implement a test that includes a successful check as well as a check that leads to an error.

Figure 1.14:  The "Selected test" Node

The node "Selected test" contains three mouse click events and two checks. The first two clicks cause the nodes in the tree structure of the SUT to be expanded until the "Choices" node is visible and its properties displayed on the right. Afterwards, you see a "Check boolean: selected" node that queries the state of the fourth radio button to determine that it is not selected. The third mouse event then performs a click on this very same fourth radio so that it is now selected. Finally, another "Check boolean: selected" is performed just as before. However, now that the radio button is selected, you'll see that this last check leads to an error.

Figure 1.15:  Radio Buttons in the Options Demo

Try running the "Selected test" sequence now so that you can see the actual execution of the description we just provided. After running the test-sequence, you may want to collapse the nodes in the SUT so that it is back in its original state (the "Miscellaneous" child-nodes no longer visible).

Stopping the Application

The function of the "Cleanup" sequence is important to understand. Inside a test-set a "Cleanup" node will be executed after every test-case located in the same level of the tree-structure, just as the "Setup" sequence will be executed before each test. The "Cleanup" node serves the purpose of leaving the SUT in a specific or 'clean' state so that further tests can be executed deterministically. This step is equivalent to the "tear-down" method within unit testing.

Note however that the "Cleanup" node, just like the "Setup" node, will only be executed when the top-level node is executed or when you explicitly execute it. In the previous sections, you specifically selected and carried out one specific test or sequence (such as "Clickstream"), which did not lead to the "Setup" or "Cleanup" nodes being run. In the next section, however, you'll see visually how this all works when you execute the top-level "Test: Options" node.

Expand the "Cleanup" node now to take a look at its child-nodes. For this "Cleanup" node, we attempt to stop and close the SUT application in an orderly fashion by clicking the "Cancel" button of the SUT. In case this desired result does not occur (the SUT does not close), a more aggressive attempt is made.

Figure 1.16:  The Cleanup Sequence

Our "Cleanup" sequence is organized as follows:

  • The first node clicks the "Cancel" button of the SUT.
  • The remaining nodes are built around the frame of a try/catch construct, which is used to handle the erroneous case of the SUT failing to stop when "Cancel" is clicked. The first node in the try/catch block performs a "Wait for client to terminate" check.
  • In the case where the SUT client fails to terminate within the given timeout period, an exception is thrown - called "ClientNotTerminatedException" - which will then be caught within the "Catch" node.
  • If the exception is indeed thrown, a "Stop client" node is then executed under the "Catch" node. The "Stop client" node causes the actual process running the SUT application to be terminated.
The Complete Test

In the previous sections we've taken you step-by-step through many elements of our example test-suite. At this point, you're now ready to execute the entire test in one fell swoop.

First, close the "Options Demo" in case it is still running. You can do this by executing the "Cleanup" node described in the previous section. Then select the "Test-set: Options" node and execute it with the replay button. The entire suite of tests will last a few minutes, due to the delays we built into certain nodes of the test-suite so that you can better follow what's going on. If you look in the details of the "Test-set: Options" node, you'll see a variable called "delay" defined in the parameter default values section under "Defaults". You can change this value if you wish to shorten or lengthen the total run time.

When the tests complete, you should end up with two failures, both of which we discussed in the previous sections. Open up the run-log again to take a closer look:

Figure 1.17:  Run-Log for the Completed Test-set "Options"

Here you'll see in a visible fashion the process we attempted to explain in the last section concerning the behavior of the test-set node in which execution of the "Setup" and "Cleanup" nodes take place before and after each test. Thus a clearly defined "clean" state is ensured.

Report Generation

In the world of quality assurance the documentation and archiving of test results are of vital importance. In order to achieve this, QF-Test offers an automated report-generation feature. Since you've just finished a complete test run, we're at a good point to show you the capabilities of this feature.

Make sure the run-log for your test-run is open. Now use the »File«-»Create HTML/XML report...« option from the run-log's menu to bring up the dialog window used to specify the nature of your desired report.

Figure 1.18:  Report Generation Properties

In the first field, you can specify the file-name of the report. Following this, you can decide what type of report you want. QF-Test offers two kinds of reports, HTML or XML format. An XML report is useful if you've written your own XSLT stylesheets to shape the report into the style you'd prefer.

There are further options to control handling of HTML and doctags used inside comment fields as well as viewing the report in your browser after creation. You can just leave those fields unchanged. Further details can be found in chapter Reports and test documentation of the user manual.

Let's try generating a simple HTML report from the results of the last test-run. After possibly adapting the fields we indicated above and confirming the generation with the "OK" button the report will then be generated and generally look as follows in your browser:

Figure 1.19:  An HTML Report

The report begins with a summary containing informational data from your system on the top left side, a legend describing the meaning of icons used in the report on the top right side and the overall test result below. In our case as result we see the two well-known errors again.

Following the summary, an error overview lists the errors with its location where they occurred (the test-case) and the message describing the error details. Part three is a test-set overview just containing the "Options" test-set, as our test-suite only contains that one. Finally all test-cases contained in the "Options" test-set are listed with details like description, result and execution time.

The report generation feature proves very useful for creating an overview over the test-run and a document for presentation and archiving purposes.

Videos Downloads Documentation Buy Free Trial