FAQ - Frequently Asked Questions

Evaluation and licensing
1
Is an evaluation version available for download?
2
Do I need anything else?
3
So how do I get a license?
4
How much does QF-Test cost?
5
Does QF-Test need an additional license server?
Support, training and feedback
6
Where do I get help troubleshooting?
7
What about training for QF-Test?
8
How can I request an enhancement to QF-Test?
9
Where do I report a QF-Test bug?
Test execution
10
Why do tests fail today that were OK yesterday though nothing changed in the meantime?
11
How do I run a test automatically from the command line, a test management tool or some other kind of script?
12
Is it possible to test two applications running at the same time in two different JVMs?
13
I've got a long-running test and QF-Test runs out of memory. How can I prevent that?
14
Hard mouse events and drag'n'drop operations do not work flawlessly, components cannot be found, the run log contains black or corrupt screenshots.
What do I have to take care about for test execution?
Scripting
15
How can I access objects in my application that are not components?
16
Fine, but how do I use these from Jython, Groovy or JavaScript respectively?
17
How can I access additional Java classes from a script?
18
How can I throw an exception from a script?
19
Which external editor should I use?
Web
20
How do I know which web UI toolkit is used for my web application and what do I do if it is not directly supported by QF-Test?
21
Why is this small file-upload/file-download dialog showing up before the real file selection dialog gets displayed?
22
Why does the replay of an already recorded file-upload-/file-download-sequence fail if I use another browser and how can I bypass that issue?
23
I get an error page notifying me about untrusted certificates. Unfortunately the standard dialog to add an exception is not working properly. How can I solve this problem?
24
I'm getting an OutOfMemoryError for the browser. How to increase memory for the QF-Test browser?
25
The web-application opens a popup-window. If I try to close this window using a 'Window event' "WINDOW_CLOSING" the main-window is closed instead of the popup-window at times. How can I ensure that the correct browser-window is closed ?
Evaluation and licensing
1
Is an evaluation version available for download?

Yes. Please visit www.qfs.de/en/qf-test/download.html.

2
Do I need anything else?

Normally a license file is required to run QF-Test. It will run without a license, but it will not let you save any files or load any files that were not provided by Quality First Software GmbH. This is sufficient for getting a first impression, working through the tutorial and making a first attempt at running your application under QF-Test. To go beyond that, you'll need a license file.

3
So how do I get a license?

You can obtain a free trial license valid for 4 weeks by filling in the request form at www.qfs.de/en/qf-test/free-trial.html.

4
How much does QF-Test cost?

License types and prices for QF-Test are listed at www.qfs.de/en/qf-test/pricing.html.

5
Does QF-Test need an additional license server?

No, not necessarily. QF-Test handles multi user license management for local networks by itself, provided that IP multicast works. For floating licenses across multiples sites and in case of restricted networks, a dedicated license server is available. The license server itself is free of charge and server licenses are very reasonably priced. For further information about the license server, please get in touch with QFS via <sales@qfs.de>.

Support, training and feedback
6
Where do I get help troubleshooting?
7
What about training for QF-Test?

QF-Test trainings for beginners and advanced users in German and English language take place regularly here at QFS. There is also the option for webinar-based or on-site consulting and training. Details can be found at www.qfs.de/en/qf-test-support/training-consulting.html.

8
How can I request an enhancement to QF-Test?

Enhancement requests are welcome anytime at support@qfs.de.

9
Where do I report a QF-Test bug?

Simply send an email to qftest-bug@qfs.de. and we will have a look. Please be sure to provide as much information as possible, especially test suites and run logs.

Test execution
10
Why do tests fail today that were OK yesterday though nothing changed in the meantime?

As the first step, please ensure that really nothing has changed as automatic updates of Java or browser versions may happen without being recognized.

Irrespective of changes there can be tests that fail only occasionally for no apparent reason. This may sound like a bug in QF-Test, but that's rarely the case. In complex, multi-threaded environments, many actions and interactions depend on timing. The first thing to try is to introduce delays at critical points. If that helps you can focus on minimizing the delay by using 'Check' nodes with a timeout or 'Wait for component to appear' nodes to wait for a certain condition.

If delays don't help you need to dig deeper and try to understand what's happening. It's not unlikely that the root cause is a bug in your application - typically a tricky one that shows only occasionally depending on timing or other circumstances. The blatant, obvious bugs are typically found earlier - these tricky ones are part of what testing is all about. The detailed logs and screenshots that QF-Test creates help analyzing such situations. Our support can help you interpret the data and isolate the relevant information to forward to development.

11
How do I run a test automatically from the command line, a test management tool or some other kind of script?

You can run QF-Test in batch mode through the command line argument -batch. Many other command line arguments can be used to configure the test run. The exit code of QF-Test reflects the outcome of the test. See Test execution, Command line arguments, Exit codes for QF-Test and Interaction with Test Management Tools for details.

12
Is it possible to test two applications running at the same time in two different JVMs?

Yes, just start two SUT clients with different names. You can then control both of them.

13
I've got a long-running test and QF-Test runs out of memory. How can I prevent that?

To increase the available memory, start QF-Test with the argument -J-Xmx1280m (or an even greater value; QF-Test uses up to 1024 MB by default). On Windows you can alternatively use the QF-Test Java Configuration tool, available from the Windows system menu. On Linux rerunning the QF-Test setup script (setup.sh) also lets you adapt memory usage. Of course the amount of memory you can use depends on your computer. Please refer also to chapter 1 for further details.

There are a number of ways to reduce the memory use of QF-Test:

  • Make sure that the option Create compact run log is checked so that all irrelevant nodes are removed from run logs.
  • QF-Test keeps 4 run logs accessible from the »Run« menu by default. Keep the option Automatically save run logs active so that QF-Test can save these run logs to files and release their memory.
  • Close run log windows that you no longer need so that the memory for these run logs can be reclaimed.
  • For long-running tests the best option is to create split run logs (which QF-Test uses as default) so QF-Test can save partial run logs to files instead of holding the entire run log in memory. See subsection 7.1.6 for details.
  • If the option Don't create run log is checked, no run log will be generated at all. This should also be used with caution since it can be extremely difficult to interpret what happened without the help of the run log. Use split run logs instead.
  • If your SUT prints lots of output you can reduce the number of old clients that are kept around by changing the option Number of terminated clients in menu.
14
Hard mouse events and drag'n'drop operations do not work flawlessly, components cannot be found, the run log contains black or corrupt screenshots.
What do I have to take care about for test execution?

GUI testing requires an unlocked, active desktop. That is the only way to ensure that the SUT behaves the same as if a normal user interacts with it.

To make sure that your test environment complies with this requirement, you'll probably need to tweak its setup. This holds true notably for continuous integration and build tools like Jenkins (c.f. chapter 28). Otherwise you can run into serious trouble during Test execution, for example black screenshots in the run log (c.f. section 7.1), failing drag'n'drop operations, non-working hard mouse events or even problems during component recognition (c.f. chapter 5). Java WebStart applications fail to start up. Chapter Hints on setting up test systems contains useful tips and tricks to set-up your test systems.

Before running a GUI test, check whether the following conditions are met:

  • QF-Test and the SUT have to run within an active, unlocked user session.
  • The Test must not be executed within the Windows service session. It also must not run without an user session.
  • During the Test execution with Jenkins you have to ensure that the Jenkins Windows node does not get started as service. It has to be started either via Windows Autostart or Windows Task Scheduler within a real user session. Please ensure that you select a valid user account at the 'Security options' and that you have 'Run with highest privileges' disabled. The selected user must be logged in during test execution, e.g. by automatically logging in, and the desktop must not be locked in any way.
  • RDP connections must not be minimized or closed, that would result in a locked session. Instead of RDP you should use VNC, Teamviewer or similar tools to observe the running tests. RDP must not be used for the initial user login and start of the related session.

    NoteOn Windows 10 or Windows Server 2016 systems you can make use of RDP if you modify the Registry. Therefore navigate to HKEY_CURRENT_USER\Software\Microsoft\Terminal Server Client or HKEY_LOCAL_MACHINE\Software\Microsoft\Terminal Server Client and add a new value RemoteDesktop_SuppressWhenMinimized as DWORD having the value 2. Once that setting has been set you are allowed to minimize RDP connections, but you have to keep the connection alive. The tests will still fail if you disconnect or close the session You can find further details about setting up your test systems in the manual at chapter 38 and chapter 24.

Technical background:
The keywords are 'session 0 isolation'. This means that every user has its own session ID, starting with 1. The session with ID 0 is reserved for services and applications without user context. It is restricted in its functionality and applications which run in this session are isolated from other sessions. Applications with GUI cannot be displayed within this session. When running a GUI application in the service session, it will not be rendered correctly and thus may not behave as expected. If you search for 'session 0 isolation' with your preferred search engine you will get extensive information, especially for Windows Vista and newer.
Due to security reasons, the well-known workaround for Windows XP and Windows 2000 via tscon.exe and redirection of session 0 is not working anymore since Windows Vista.

To get round all that problems you should consider to work with virtual machines, especially from a security point of view. If you execute the tests on a virtual machine, the above mentioned requirements apply for this virtual machine only, not for the host. You can lock the host and don't care about session management on the host.

Scripting
15
How can I access objects in my application that are not components?

You cannot get an object out of the blue, some kind of registry must exist that returns the object from a class static method. Typical examples in the standard Java API are java.lang.Runtime.getRuntime() or java.awt.Toolkit.getDefaultToolkit().

16
Fine, but how do I use these from Jython, Groovy or JavaScript respectively?

This is standard Jython stuff: Simply import the class and call its methods, e.g.
from java.lang import Runtime
runtime = Runtime.getRuntime()

In Groovy the package java.lang gets imported even automatically:
def runtime = Runtime.getRuntime()
You can access any class of your application the same way, provided the class is declared public. Note that you must use an 'SUT script' node, not a 'Server script' node.

17
How can I access additional Java classes from a script?

To make additional Java classes available to Jython, Groovy and JavaScript, put them in a jar file and place that in QF-Test's plugin directory (see section 49.2).

18
How can I throw an exception from a script?

There are two ways to do that:

  • Jython:
    raise UserException("Some arbitrary message")
    Groovy:
    import de.qfs.apps.qftest.shared.exceptions.UserException
    throw new UserException("Some arbitrary message")

    JavaScript:
    import {UserException} from 'de.qfs.apps.qftest.shared.exceptions'
    throw new UserException("Some arbitrary message")

  • rc.check(condition, "Message", rc.EXCEPTION)
    will raise an exception only if the condition is false.
19
Which external editor should I use?

That's a matter of taste, to some even religion. A comprehensive list of editors for all kinds of operating systems that support Python syntax highlighting and other goodies is available at https://wiki.python.org/moin/PythonEditors. There are probably dozens of suitable editors with syntax highlighting for Jython, Groovy and JavaScript - jEdit (www.jedit.org) is only one of them.

Web
20
How do I know which web UI toolkit is used for my web application and what do I do if it is not directly supported by QF-Test?

If possible, please ask your developers about UI toolkits or JavaScript components used.
Alternatively, activate the auto-detection mode in the Setup sequence creation when creating your start sequence. Then QF-Test recognizes supported toolkits automatically and prints a respective message in the terminal.
If the toolkit is unsupported or remains unknown you might want to ask our support to take a look at your web application's HTML code. New or custom toolkits can be integrated with the help of a CustomWebResolver with little effort - either by yourself (Improving component recognition with a CustomWebResolver) or using our services.

21
Why is this small file-upload/file-download dialog showing up before the real file selection dialog gets displayed?

Before a file selection dialog is displayed the QF-Test browser is showing up a small dialog with an OK-/Cancel-button. This helper dialog is needed by QF-Test to get the data out of the file selection dialog since this dialog is created natively by the operating system. After clicking the OK-button in the helper-dialog the native file selection dialog is displayed and you can enter the filename or select the file directly.

22
Why does the replay of an already recorded file-upload-/file-download-sequence fail if I use another browser and how can I bypass that issue?

Depending on the implementation of the file-upload-/file-download on a specific page the replay might get a bit complicated and even vary between different browsers. The QF-Test standard library qfs.qft contains a special procedure qfs.web.input.fileUpload to handle this. Please use this procedure instead of your recorded sequence if you encounter problems during replay.

23
I get an error page notifying me about untrusted certificates. Unfortunately the standard dialog to add an exception is not working properly. How can I solve this problem?

This problem won't occur in QF-Test version 3.5.1 and higher, because SSL certificates are now accepted automatically. Please use the following workaround if you are using an older version of QF-Test.

There are different approaches to add a certificate using firefox.

Solution 1:

  • Open the URL chrome://pippki/content/certManager.xul in a QF-Test browser window.
  • This will show the certificate manager where you can add the certificate.
  • After pressing OK the whole browser window will close.
  • When loading the URL which needs a certificate again the page will load without a certificate error.
Solution 2:
  • You can run a normal firefox from the command line with the following parameters
    firefox -profile "[path to your userprofile]/.qftest/mozprofile"
    (e.g.: firefox -profile "c:/Users/user1/.qftest/mozprofile")
    (Please close all other running firefox instances before executing this command.)
  • Load the URL in the browser and add the certificate to the trusted certificates.
  • When loading the URL which needs a certificate again the page will load without a certificate error.

24
I'm getting an OutOfMemoryError for the browser. How to increase memory for the QF-Test browser?

In general it is recommended to create the setup sequence by use of the quickstart wizard. In the resulting sequence within the step "Start browser without window" there is a "Start browser" node in which you can specify the maximum memory as part of the Java VM parameters via e.g. -Xmx384m, which means 384 MB maximum memory. The current default is 256 MB.

Set browser maximum memory
Figure A.1:  Set browser maximum memory
25
The web-application opens a popup-window. If I try to close this window using a 'Window event' "WINDOW_CLOSING" the main-window is closed instead of the popup-window at times. How can I ensure that the correct browser-window is closed ?

In order to distinguish those windows QF-Test requires additional information. This information specifies which window should be used for replaying events. It has to be set at the 'Wait for document to load' node as well as for the recorded 'Web page' in the attribute 'Name of the browser window'. You can reach the recorded 'Web page' quite fast via right mouse click at the 'Wait for document to load' node and selecting »Locate component«.

We recommend to set that attribute to ${default:windowname:}.

Specifying the attribute 'Name of the browser window' allows QF-Test to evaluate the content of that attribute in addition to the web page's URL for recognizing the web page itself and all underlying components. That's why QF-Test is able to distinguish between both windows during replay. Setting that value for the attribute at the 'Wait for document to load' node assigns a new name to that window from QF-Test's perspective. You can see that name in the titlebar of the browser as well.

Before replaying the respective 'Wait for document to load' and events for that popup window you need to set the variable windowname to an arbitrary value, e.g. "popup". You could also use a dedicated 'Set variable' node in order to set that variable value. A more convenient way might be to pack all events into a sequence and define the variable windowname at the "Variable definitions" table of that sequence.

Now you can replay those events.

It is recommended to reset the variable windowname after executing those event nodes again. Otherwise QF-Test will try to replay all subsequent events on that popup window again. Therefore you can use another 'Set variable' node for windowname with an empty "Default value". In case of specifying the variable at the sequence node the variable will disappear after that sequence.

If you have already recorded some tests, you should update your test suite using a global replacement action for the attribute 'Name of the browser window'. This action should set all values of that attribute from empty to ${default:windowname:}. Therefore choose »Edit«-»Replace« from the menu and change to the advanced replacement mode via pressing the two golden arrows at the dialogs's toolbar. Now leave the "Search for" field blank and specify ${default:windowname:} at the "Replace with" field. Then select 'Name of the browser window' for "attribute" and check the checkbox "Whole attribute".

You need to repeat that replacement action for any test suite which contains event and component nodes for that web page.