Manual

5.0+15
Testing native Windows applications (preview)
15.1
Getting started

This chapter covers automation and testing of Windows desktop applications, namely

  • Classical Win32 applications,
  • .NET applications based on Windows Presentation Foundation (WPF) or Windows Forms,
  • Universal Windows Platform (UWP) applications using XAML controls.

All these kinds of applications support Microsoft UI Automation or Microsoft Active Accessibility (MSAA) for software test automation, joined together in the Windows Automation API, please see section 15.2 for background information.

As a precondition of testing, QF-Test first needs a connection with the respective process. An appropriate 'Setup' sequence can be created by help of the Setup sequence creation from the »Extras« menu. Choose 'A native Windows application' as application type then.

If your application is already up and running, just specify a regular expression for the title of its (main) window, for example .*- Editor for the Windows Notepad application. Otherwise give the path to your Windows executable (.exe) so that QF-Test can start the program, please see Launching/Attaching to an application for details.

Once the application is connected to QF-Test (as 'GUI engine' win), capture and replay can be performed as described in chapter 4. However, due to the nature of UI Automation, you should observe the recording rules listed in section 15.4.

A few example test-suites can be found in the QF-Test installation folder, namely

  • qftest-4.7.1/demo/carconfigForms/winDemoForms_en.qft
  • qftest-4.7.1/demo/carconfigWpf/winDemoWPF_en.qft
  • qftest-4.7.1/demo/windows/*.qft

Also have a look at the (Current) Limitations, most of which are expected to be fixed or improved in future releases of QF-Test.

15.2
Technical background

A common framework for all Windows-based applications is the Windows Automation API consisting of Microsoft's Active Accessibility and its successor, Microsoft UI Automation. Being it the heart of the win engine, QF-Test is now able to control virtually all kinds of Windows applications.

A Windows application has to expose so-called Providers in order to follow the rules of UI Automation. This is done automatically when a framework like WPF is used to develop a program. This is also done for Win32 applications via proxy providers. That means, how good an application can be controlled and tested depends on the quality of the respective providers, i.e. usually on the framework used in application development. For example, as UI Automation was introduced along with WPF, these kinds of applications are supposed to be testable quite well. And when you wonder what cannot be tested via UI Automation, try out a Java Swing application. However, there is already a pretty good tool available when you want to deal with Swing ...

A test application that wants to control a program via UI Automation can get hold of so called Automation Elements which represent the actual UI elements in the SUT (System Under Test). Though every automation element has a control type (like Button, MenuItem, etc.), its actual functionality - for example, setting a value in a text field - depends on Control Patterns implemented by the respective providers.

To deal with the UI Automation framework, QF-Test starts a special Java program which serves as UI Automation client application. That program can access all UI Automation elements in a given process and handle them according to the rules of QF-Test (e.g. create a snapshot of an element as 'Component').

15.3
Launching/Attaching to an application

Testing a native Windows application does not require you to launch that application from QF-Test. You can also connect to a running process and that way even control parts of the operating system, for example the Windows Taskbar.

In order to connect to a process you can specify a regular expression for the window title or the respective process ID or the window's UI Automation class name. Strictly speaking, that window must not be a Window but could also be a Pane or a Menu in terms of UI Automation control types. Whatever feature is used for attaching, QF-Test will eventually determine the respective process ID and treat exactly that process as the actual client application (SUT).

To connect just define the attribute 'Window title (RegExp)' in the 'Start Windows client' node and this can be

  • a regular expression for the title
  • -pid <process ID>
  • -class <class name>

For example, by specifying .*- Editor you can attach to a running Windows Notepad application, while -class Shell_TrayWnd will address the Windows Taskbar.

In order to find out the titles, process IDs and class names of running programs, you can run the procedure qfs.autowin.logUIAToplevels in qfs.qft, cf. The standard library.

Besides attaching to a running process, it is also possible to launch a program from the 'Start Windows client' node. To this end, specify the path to the respective executable in the 'Windows application' attribute.

In some cases, it can also be useful to define both the 'Windows application' and the 'Window title (RegExp)' attribute. QF-Test will then first try to attach. If that fails, the given program will be started and connected via its process ID. If that fails too - the process may launch a child process and terminate itself or may not display a (graphical) user interface - another attempt to attach is made.

When you terminate a win client in QF-Test (either via the 'Stop client' node or from the »Clients« menu), the respective UI Automation client process will be stopped along with its sub-processes. That is, your actual SUT will terminate as well, if you started it from QF-Test. On the other hand, the SUT will not be stopped when it was running before you attached to it.

When you close the SUT, the UI Automation client will terminate as well.

To attach to an elevated processes (presenting the UAC prompt), you have to launch QF-Test as administrator.

15.4
Recording

After connecting QF-Test with the SUT, you can record events (section 4.1), checks (section 4.3) and components (section 4.4).

However, as the communication between the SUT and the QF-Test UI Automation client is handled by Windows (the UI Automation core), accessing elements is not quite as fast as you may know it from the QF-Test Java automation. Furthermore, in contrast to Java and Web testing (QF-Driver), events are processed asynchronously, i.e. you cannot expect that an application's dispatch thread is blocked while QF-Test is handling an event.

That makes recording more difficult, because a target element might be destroyed by the action to be recorded, for example when selecting an entry from a ComboBox or clicking on a button that closes its parent window.

So you'd best get into the habit of following a few recording rules:

  • Activate the recording mode and move the mouse over the element for which you want to record an event.
  • As QF-Test may take a little time to retrieve information about the element below the mouse cursor, a red pane is displayed until it is done; the little 'QF-Test Element Information' window will then show which automation element was found.
  • Now perform the mouse click to be recorded.
  • When a mouse click will close a dialog or window (might also be a popup displaying a list), make sure to perform the click slowly, i.e. do not release the mouse button immediately after pressing it so that QF-Test has the opportunity to gather information before the window will disappear (when the mouse release is done).
  • When recording checks or components, the respective frame around the element is drawn almost immediately when the mouse is hovering over an automation element. Before recording a check, you should wait until the frame disappears.

Sometimes check recording (and transforming the node afterwards) may work better than event recording, for example when a click on a button (like OK, Cancel) closes the respective dialog or when a mouse down event recreates elements (for example the accessory table in the CarConfiguratorNet WPF demo application). In check recording mode QF-Test covers the SUT with an (almost) invisible window to prevent mouse clicks from triggering an action in the client application.

15.5
Components

In QF-Test an automation element is recorded (or can be inserted manually, of course) as 'Window' or 'Component' and stored within the 'Windows and components' node. The QF-Test (generic) 'Class name' often corresponds to the type of the UI Automation element, for example Button. To be able to differentiate between the UI Automation type and the generic class name, QF-Test adds a prefix Uia. to the type. Similarly, the UI Automation framework specifier is used as prefix for the automation element's class. So you may for example see a classname: WPF.DataGrid in the 'Extra features' of a Table component recorded in a WPF application.

QF-Test does not strictly follow the hierarchy of the UI Automation elements. That is often the case with dialogs (like Notepad's Font dialog) which are usually listed below the main application window in the UI Automation tree. From the Win32 perspective as well as what QF-Test users would expect, such dialogs are also top-levels and thus listed as a sibling of the main window below 'Windows and components'. On the other hand, a context menu can be a top-level in the UI Automation tree, but may be a window's child in QF-Test.

15.6
Playback and Patterns

The QF-Test win engine usually records mouse clicks with the 'Replay as "hard" event' attribute. That is the safe way, i.e. a "hard" mouse click is likely to actually trigger the desired action.

However, UI Automation supports various "soft" actions which do not rely on mouse events. For example, to trigger a button's action you can play back

  +Select: invoke [myButtonID]

The effect should be the same as with

  +Mouse click [myButtonID]

but no mouse is involved when using the 'Selection' node. Instead, the UI Automation core will trigger the execution of a provider's Invoke() method in the SUT.

The 'Selection' node does support the following actions in its 'Detail' attribute:

  • invoke: Usually equivalent with a mouse click.
  • expand, collapse: Should expand/collapse a ComboBox, MenuItem or TreeItem.
  • select: Should select an item in a list (also use 0 as 'Detail'). 1 and -1 are meant to extend or reduce the selection when multi-selection is allowed.
  • toggle[:on|off]: Change the state of a CheckBox element.
  • scroll:horiz%,vert%: Values between 0 and 100 are possible, defining the position of the scroll location in percent; specify -1 when you do not want to change a position (horizontally or vertically).

The actions actually supported depend on an automation element's patterns. They are recorded among the 'Extra features' of a component or can be determined in an 'SUT script' (see below).

What exactly a pattern means can vary from application to application. If, for example, both SelectionItem and Invoke patterns are supported, Invoke might be preferable because

  +Select [list@item]

may only highlight the element but not trigger the respective action (e.g. Notepad Fonts).

Even worse, the formal support of a pattern does not mean that applying it has any effect, for example scrolling an (invisible) entry in the list of Windows Calculator's modes into view (ScrollItem pattern). To get around the problem, you can simply play back select here, whether or not the entry is currently visible.

As already mentioned above, because "soft" playback may simply not work (due to the provider implementation) or even block when a modal dialog is to be displayed (due to COM method invocation), QF-Test records "hard" mouse clicks by default. If you deactivate the 'Replay as "hard" event' attribute, invoke is played back instead (only if the respective pattern is supported by the element).

Regarding 'Key events', a text can only be set directly by a 'Text input' node if the Value pattern is supported. Otherwise single key events have to be played back.

15.7
Scripting

Internally, the win engine represents automation elements by a class WinControl. To obtain an element in a Groovy 'SUT script' node, run

def ctrl = rc.getComponent("myComponentID")
println ctrl
Example 15.1:  Retrieving a WinControl in a Groovy 'SUT script'

with the respective 'QF-Test component ID'. The most important WinControl methods are

  • getType(), getClassName(), getFramework(), getName(), getId(), getHwnd(), getLocation(), getSize(), getLocationOnScreen(), getPatterns(), hasPattern() to retrieve UI Automation properties of the element
  • getChildren(), getParent(), getChildrenOfType(), getAncestorOfType(), getElementsByClassName() to traverse the element hierarchy
  • getControl() to retrieve the respective control from the mmarque ui-automation library. A snapshot of that library is part of the QF-Test releases.
15.8
Options

The behavior of the win engine can be influenced via a set of QF-Test options and additionally by defining preferences which affect the native part of the UI Automation Client. Those options and preferences can be set in an 'SUT script' node via

  rc.setOption(<name>, <value>)

or

  rc.engine.preferences().setPref(<name>, <value>)

respectively. To reset an option, use

  rc.unsetOption(<name>)

15.8.1
Windows scaling

As the display resolution increased over the years, Windows allows to define a scale factor so that application windows and controls are enlarged and text becomes more readable. Usually, UWP, WPF and Windows Forms application do scale automatically, but especially Win32 programs may retain the size of its controls or scale differently.

QF-Test works with physical display coordinates by default so that geometry values will change when an application is scaled. Say the scale factor is set to 125%, a button which originally (100%) resides at location (24, 40) with size (100, 20) will be moved to location (30, 50) within its container and grow to an area of 125 times 25 pixels. The consequences are

  • different geometry when recording the component anew
  • geometry mismatch when QF-Test tries to identify an element which was recorded at 100% now in the scaled environment
  • a (hard) mouse click onto a given region within an element may fail because the scaled region is farther away from the element's top-left corner.

To make QF-Test work with logical coordinates (as seen with a scale factor of 1), you can set Options.OPT_WIN_USE_SCALING to true. QF-Test then uses the scale factor of the primary connected monitor to adapt the geometry of components and mouse event coordinates. Note that rounding errors may occur when calculating new integer coordinates so that the mouse may not hit a given point exactly.

15.8.2
Visibility

You sometimes may want to play back an event on an element that is actually not visible (it may not be scrolled into view). To perform an invoke event then, you may need to get rid of the visibility test which is usually part of the component recognition.

This can be achieved by setting Options.OPT_WIN_TEST_VISIBILITY to false. After playing back the event, you should reset the option to re-enable the visibility test.

15.8.3
Attaching to a window with a given class

If you attach to an application via -class <class name>, QF-Test by default ignores all toplevels in the application which do not have the given class name. That way, you can for example deal solely with the Windows Taskbar (and set apart the desktop and its icons which run in the same process).

To be able to access all toplevels in the process, you can set the preference "windriver.restrict.tops.to.class" to "false".

15.9
(Current) Limitations

There are a number of limitations in the current implementation status of the Windows testing functionality. We will try to further improve things within the next versions, but possibly not all of the following points will be resolved soon.

As the support for UI Automation depends on the framework used for application development, the recording in QF-Test may not always be consistent. For example, a 'Wait for component to appear' node may or may not be recorded when opening a dialog.

Dealing with applications consisting of several processes requires several win clients and can be tricky.

Further limitations / not yet implemented features (August 2019) are among other things

  • Component recording does work only for a single component (not for children or the full window). This is supposed to be fixed in a future release of QF-Test.
  • Supported check types are incomplete. 'Text', 'Image', 'Geometry' and 'Visible' are implemented for all WinControls. Additional checks may be available for Tables, Lists and other types. This is supposed to be fixed in a future release of QF-Test.
  • Elements in the title bar of a Windows app cannot be accessed (easily), because they live in a different process. This might be improved in a future release of QF-Test.
  • MenuItems in .NET Forms application are not properly recorded, you only get a mouse click on a menu popup with the respective location. Replay should work though. This might be fixed in a future release of QF-Test. As a workaround you may create yourself a generic MenuItem as described in section 33.6.
  • Redirection from a Button's Text element to the Button element is done when recording a mouse click, but may be missing elsewhere. This is supposed to be improved in a future release of QF-Test.
  • Big hierarchies of automation elements may cause performance problems. This is supposed to be improved in a future release of QF-Test.
15.10
Links

The Windows Automation API is described here: https://docs.microsoft.com/en-US/windows/desktop/WinAuto/windows-automation-api-portal.

More about Mark Humphrey's ui-automation Java library can be found on https://github.com/mmarquee.