Run log options

These options let you control which information is collected in a run log, if and when a run log is shown and how to locate errors.

General run log options

Run log options
Figure 39.37:  General run log options
When to show run logs (User)

A run log is created for every execution of a test. A number of recent run logs are available from the »Run« menu, the most recent run log can also be opened by pressing [Control-L]. Additionally the run log can be shown during execution or after an error as follows:

At start
This choice causes QF-Test to open the run log when it begins executing a test sequence. The nodes of the log will be added as execution proceeds.

After finish
With this choice the run log is shown after replay is finished.

On exception
The run log is shown only if an uncaught exception is thrown.

Don't show
The run log is not displayed automatically. You have to open it via the »Run« menu or by pressing [Control-L].

6.0+Show relative duration indicators (User)
Server script name: OPT_LOG_DURATION_INDICATORS

To analyze the run-time behaviour of a test it is helpful to see at a glance which branches are taking up the most time. To that end, display of duration indicators can be activated via this option, which is also directly accessible via the »View« menu of a run log.

The length of the bars shown represents the percentage of the time taken for execution of the respective node relative to the time taken by its parent node.

6.0+Duration indicator kind (User)
Server script name: OPT_LOG_DURATION_INDICATORS_KIND
Possible Values: VAL_LOG_DURATION_INDICATORS_KIND_DURATION, VAL_LOG_DURATION_INDICATORS_KIND_REALTIME, VAL_LOG_DURATION_INDICATORS_KIND_BOTH

This option determines whether duration indicators show duration, real time spent or both.

NoteThe difference between the values of "Duration" and "Real time spent" are explicit delays introduced in nodes via the 'Delay before/after' attribute or user interrupts.

Number of run logs in menu (User)

A limit for the number of menu items for recent run logs kept in the »Run« menu.

3.0+Automatically save run logs (User)

To prevent excessive memory use through run logs and also to make the most recent run logs persistent between QF-Test sessions, the recent run logs kept in the »Run« menu are saved automatically to the user configuration directory or the directory defined in the option Directory for run logs. The filename for the run log is based on a timestamp. QF-Test uses file locks to prevent collisions and accidental removal in case of parallel sessions and automatically keeps the user configuration directory clean by removing unreferenced logs, so there should be no reason to disable this feature. Still, you can do so by disabling this option.

4.0+Directory for run logs (User)

By default, run logs created during interactive use of QF-Test are stored in the user configuration directory. This option can be used to specify a different target directory.

Note This option is interpreted by QF-Test whenever a test is started. At that point, test suite and global variable bindings are already in place and in contrast to other options it is possible to use QF-Test variable syntax here. This includes special variables like ${env:HOME} to look up an environment variable or even ${qftest:suite.dir} to save the run log next to the test suite. If the directory is dynamic, like in the latter case, QF-Test may not be able to clean up old run logs regularly. Errors in variable expansion are silently ignored and the user configuration directory is used instead.

3.1+Show expanded variable values in tree nodes (User)

The nodes in the tree view of a run log can either be displayed with variables expanded to the value they had at run-time or with the original variables. Both views have their use, so you can toggle between them via this option or more quickly via the menu item »View«-»Show nodes expanded«.

7.0+Show return values of procedures (User)

If this option is active, return values of 'Procedures' are displayed in the tree after the respective 'Procedure call' node. In a run log you can also toggle this option's value via the menu item »View«-»Show procedure return values«.

Skip suppressed errors (User)

Like the previous one, this option controls the search for errors in a run log. If activated, warnings, errors or exceptions that have not propagated to the top, are not found. Thus exceptions caught by a 'Try'/'Catch' clause or messages suppressed through the 'Maximum error level' attribute are skipped.

This option is also accessible through the »Edit«-»Skip suppressed errors« menu item.

3.0+Cleanup tree when jumping to next or previous error (User)

When repeatedly jumping to errors in a run log the tree can easily get cluttered with many expanded nodes. If this option is activated, QF-Test will automatically clean up the tree each time you navigate to an error so that only the parents of the error node are expanded.

Note When viewing split run logs, partial run logs containing an error will remain in memory as long as their nodes are expanded. Keeping this option activated will ensure that partial run logs will be released as soon as possible, keeping memory use manageable even viewing the errors of a very large run logs.

Save compressed run logs (*.qrz) (System)
Server script name: OPT_LOG_SAVE_COMPRESSED

Run logs can either be saved as plain or as compressed XML files. For large run logs without screenshots the compression factor can be as high as 10, so it's advisable to use compressed logs where possible. The only reason not to use compression is if you want to transform the XML run log afterwards. But even then compressed run logs are an option because the compression method used is standard gzip format, so converting to and from compressed run logs can easily be done using gzip.

When saving a log-file interactively you can always switch between compressed or non-compressed format by choosing the appropriate filter or by giving the file the extension .qrz or .qrl.

In batch mode, the default run log format is compressed. To create an uncompressed run log, simply specify the extension .qrl in the parameter for the -runlog [<file>] command line argument.

7.0+Save run log in current XML format with UTF-8 encoding (System)
Server script name: OPT_XML_LOG_NEW

Starting with QF-Test version 7.0, run logs are saved in the current XML format with UTF-8 encoding, no indentation and arbitrarily long lines. If you need to fall back to the old format (ISO-8859-1 encoding, 2 characters indentation, line length 78) for use in external tools you can do so by deactivating this option.

Options for splitting run logs

Split run log options
Figure 39.38:  Options for splitting run logs
3.0+Create split run logs (User)
Server script name: OPT_LOG_SAVE_SPLIT

A run log can be split into several parts by setting the 'Name for separate run log' attribute of a 'Data driver' or any of the various test nodes. By turning this option off you can temporarily disable support for split run logs in order to get a normal, single run log without having to modify any 'Name for separate run log' attributes.

See subsection 7.1.6 for further information about split run logs.

3.0+Save split run logs as ZIP files (*.qzp) (User)
Server script name: OPT_LOG_SAVE_SPLIT_ZIP

Split run logs can either be saved as a single ZIP file with the extension .qzp, containing the main run log and all partial logs together, or as a normal .qrl or .qrz run log that is accompanied by a directory with the same base name and the suffix _logs, e.g. the file runlog.qrz plus the directory runlog_logs. This option determines the format in which split run logs are created in interactive mode. It has no effect if the option Automatically save run logs is turned off.

See subsection 7.1.6 for further information about split run logs.

3.4+Minimum size for automatic splitting (kB) (System)
Server script name: OPT_LOG_AUTO_SPLIT_SIZE

Note This option exclusively applies to 'Test case' and 'Test set' nodes. At other places run logs are split only when 'Name for separate run log' has explicitly been set.

Split run logs are the only reliable way to prevent running out of memory during very long running tests or when the run log grows quickly due to screenshots or output from the SUT. They are also more efficient when transforming run logs into reports. However, explicit setting of 'Name for separate run log' attributes requires an understanding of the issues involved and either making decisions about where best to split a run log or tedious typing when trying to split into small pieces.

As a compromise, QF-Test makes a very rough calculation about the size of a run log during executing, taking screenshots and program output into account. When execution of a 'Test case' or 'Test set' has finished and the approximate size of the run log pertaining to that node is larger than the threshold specified in this option, the run log is split off and saved automatically. A value of 0 prevents automatic splitting.

See subsection 7.1.6 for further information about split run logs.

3.4+Name for automatically split 'Test case' run log (System)
Server script name: OPT_LOG_AUTO_SPLIT_TESTCASE_NAME

This option specifies the name to use for an external log when it is split off automatically after execution of a 'Test case' has finished as described in the previous option. Variables can be used as well as the '%...' placeholders documented for the attribute 'Name for separate run log'.

The special variable ${qftest:testcase.splitlogname} is a good base. It expands to a path name created from the name of the 'Test case' with possible parent 'Test set' nodes as directories.

See subsection 7.1.6 for further information about split run logs.

3.4+Name for automatically split 'Test set' run log (System)
Server script name: OPT_LOG_AUTO_SPLIT_TESTSET_NAME

This option specifies the name to use for an external log when it is split off automatically after execution of a 'Test set' has finished as described in the option Minimum size for automatic splitting (kB). Variables can be used as well as the '%...' placeholders documented for the attribute 'Name for separate run log'.

The special variable ${qftest:testset.splitlogname} is a good base. It expands to a path name created from the name of the 'Test set' with possible parent 'Test set' nodes as directories.

See subsection 7.1.6 for further information about split run logs.

Options determining run log content

Run log content options
Figure 39.39:  Options determining run log content
Log variable expansion (System)
Server script name: OPT_LOG_CONTENT_VARIABLE_EXPANSION

If this option is activated, every variable expansion is logged.

Log parent nodes of components (System)
Server script name: OPT_LOG_CONTENT_PARENT_COMPONENTS

Setting this option will cause all direct and indirect parent nodes to be logged in addition to the target component node for every event, check, etc.

Log level for the SUT (System)
Server script name: OPT_LOG_CONTENT_SUT_LEVEL
Possible Values: VAL_LOG_SUT_LEVEL_MESSAGE, VAL_LOG_SUT_LEVEL_WARNING, VAL_LOG_SUT_LEVEL_ERROR

The level for automatically generated messages in the SUT during replay, e.g. details for component recognition. Only messages with the respective level, i.e. plain messages, warnings or errors will be logged. This option has no effect on messages created explicitly via the 'Message' node, rc.logMessage or qf.logMessage.

Number of events to log for error diagnosis (System)
SUT script name: OPT_LOG_CONTENT_DIAGNOSIS

During replay of a test QF-Test logs various events and other things going on behind the scenes. This information is quickly discarded except when an error happens. In that case the most recent events are written to a special run log node. The information may also be useful to developers but is mostly required for error diagnosis when requesting support from Quality First Software GmbH.

This option determines the number of recent internal events to keep. Setting it to 0 disables the feature altogether. You should not set this value to less than about 400 without a good reason. Because the information is logged only for errors, the cost for gathering it is minimal.

Maximum number of errors with screenshots per run log (System)
Server script name: OPT_LOG_CONTENT_SCREENSHOT_MAX

The maximum number of screenshots that QF-Test takes and stores in the run log during a test run on situations of exception or errors. Setting this value to 0 disables taking screenshots entirely, a negative value means unlimited screenshots.

Count screenshots individually for each split log (System)
Server script name: OPT_LOG_CONTENT_SCREENSHOT_PER_SPLIT_LOG

If this option is set, each partial log of a split run log may contain the maximum number of screenshots defined above without affecting the count for the main run log. Otherwise the limit applies for the sum of all parts belonging to the same main run log.

See subsection 7.1.6 for further information about split run logs.

Create screenshots of the whole screen upon error (System)
Server script name: OPT_LOG_CONTENT_SCREENSHOT_FULLSCREEN

Activating this option causes QF-Test to take an image of the whole screen and save it in the run log when a screenshot is triggered by an exception or error.

6.0+Limit screenshots to relevant screens (System)
Server script name: OPT_LOG_CONTENT_SCREENSHOT_RELEVANT_ONLY

If multiple monitors are connected when taking screenshots it may not be desirable to take screenshots of all screens, especially on personal workstations where this might expose private or confidential information.

If this option is active (the default), QF-Test will try to determine the screens on which SUT or QF-Test windows are showing and exclude the rest.

Create screenshots of the client's windows upon error in client (System)
Server script name: OPT_LOG_CONTENT_SCREENSHOT_WINDOW

Activating this option causes QF-Test to record images of all windows and dialogs of the SUT and store them in the run log when screenshots are triggered due to exceptions or errors coming from that SUT. In many cases this will work even for windows that are covered by other windows or in cases where a full screenshot is not possible, for example when a screen is locked.

4.3+Create screenshots of all client windows upon error (System)
Server script name: OPT_LOG_CONTENT_SCREENSHOT_ALL_WINDOWS

If this option is active QF-Test will log images of all windows of all connected SUTs for any exception or error, regardless of where the exception or error is coming from. This option also affects which screens are considered relevant, depending on where SUT windows are showing.

6.0+Create screenshots for warnings (System)
Server script name: OPT_LOG_CONTENT_WARNING_SCREENSHOTS

If this option is active, screenshots will also be saved when warnings are logged in the run log. Otherwise this is done for errors and exceptions only.

3.4+Log successful advanced image checks (System)
Server script name: OPT_LOG_CONTENT_SCREENSHOT_ADVANCED_ALWAYS

If this option is activated, QF-Test will store the expected and actual images as well as the transformed images for successful advanced image checks in the run log. Otherwise these details are kept for failed image checks only.

Activating this option can raise the size of the run log drastically so be sure to use it in combination with compact run logs and/or split run logs.

4.5+Compress images in run logs and test suites (System)
Server script name: OPT_LOG_CONTENT_COMPRESS_SCREENSHOTS

If this option is activated, QF-Test will store new images in test suites and run logs losslessly compressed.

This option can reduce the memory consumption of protocols and test suites on disk and in memory significantly. On the other hand, compression and decompression requires some CPU time, so the option can be deactivated for very time-critical test executions.

Create compact run log (System)
Server script name: OPT_LOG_CONTENT_COMPACT

Activating this option causes QF-Test to discard every node from a run log that is neither relevant for error diagnosis, nor for the XML/HTML report. After an error or exception, as well as at the end of a test run, the 100 most recent nodes are not discarded, so the most relevant information should remain available.

Even large tests should not cause memory issues, provided the option Create split run logs is turned on and used as described in Split run logs. But if you do run out of memory, activating this option can be useful.

This option is used only when QF-Test is run in interactive mode. It is ignored in batch mode (see section 1.7) to avoid accidental loss of information. To create a compact run log in batch mode, use the -compact command line argument.

Don't create run log (System)
Server script name: OPT_LOG_CONTENT_SUPPRESS

For very-long-running tests or demos that are run in an endless loop, memory consumption of the run log is an issue, but split run logs are an ideal solution. Before split run logs were available, turning run logs off completely via this option was sometimes the only way to get long-running tests to work. Now this option is only retained for backwards compatibility.

Note In batch mode this option is ignored. To suppress the run log, use the argument -nolog.

3.0+Log SUT output individually (System)
SUT script name: OPT_LOG_CONTENT_SPLIT_IO

If set, any text that an SUT client prints to its stdout or stderr stream is also logged in the run log. For each interaction with the SUT QF-Test collects text printed before the event and after the event during synchronization. This makes it possible to associate output like an exception stacktrace that is triggered by an event with the event itself, something that is impossible if all output is kept in a single piece.

3.0+Compactify individually logged SUT output (System)
Server script name: OPT_LOG_CONTENT_COMPACTIFY_SPLIT_IO

Output from an SUT client tends to accumulate and can consume a lot of memory. If this option is activated, individually logged SUT output for events that are no longer of interest can be removed along with the events in compact run logs. Please see the option Create compact run log for further information about compact run logs.

5.0+Log comments to run log (System)
Server script name: OPT_LOG_COMMENTS_TO_RUNLOG

If this option is activated, 'Component' steps are added to the run log.

5.3+Wrap lines in exception messages (System)
Server script name: OPT_LOG_WRAP_EXCEPTION_MESSAGE

If activated, display of exceptions messages in the run log or an error dialog are displayed using word-wrap to break long lines.

4.0+39.9.4
Options for mapping between directories with test suites

Test-suite directory options
Figure 39.40:  Options for mapping between directories with test suites
Directory map for test suites (System)

When analyzing a run log it is often necessary to quickly switch between the run log and the respective test suite. However, when running automated tests on different systems like Windows and Linux, the directories from which test suites are loaded during the test vary and there is no automatic way to map between different directory layouts.

With this option you can assist QF-Test in locating test suites. The 'From' column is a glob pattern that must match from the beginning of the path of the test suite stored in the run log to the end of some directory in that path. The 'To' column is the respective replacement for the matching part and can also contain a glob pattern. When searching for the test suite, QF-Test processes this list top to bottom, performing the replacement for every match found and the first match leading to an actual test suite is used.

Note A glob patterns is a simpler form of a regular expression often used by development tools: An '*' stands for any number of characters up to the next file separator while '**' means 0 or more characters of any kind, including '/'. Some examples:

**/test-suites
All directories named test-suites at any depth.
T:/test/sut_*
All directories starting with sut_ in the T:/test directory.