Introduction

Software technologies have developed so quickly and extensively in recent years that developers have hardly had any time left to additionally take care of software quality. Therefore, testing has become an increasingly important part of software development to relieve developers.

Since in today's world people's lives depend on computer software, it is necessary if professional implementation is done. This is one reason why software testing is of utmost importance. Testing must be done before the software is deployed, as this helps to detect bugs in time and ensure that the functionalities of the software work as required. Software testing is an important part of software quality assurance, an activity to evaluate and improve software quality (Cf. F. Okezie et al 2019 p. 1).

In recent years, it often happened that software was almost finished before it was tested because many companies did not yet have defined testing departments. After some negative experience gained from this, there is now a consensus on how significant training and awareness of the importance of software testing during development has become. Software testing is necessary to meet the demand for high delivery quality, whether the software is delivered to customers or internal users (Cf. Witte (2019), p. 4.).

In the context of this work, the question is pursued as to where the differences, similarities, problems and limitations lie between the individual test tools. In summary, the test tools are compared with each other. Thereby the achievement of the goals, quality, and time must be possible, whereby also the economic aspects must be considered.

In addition, different test tools are compared with each other using different criteria. In chapter 2 the terms relevant for this work, like software quality, tests, errors, test tools and test expenditure are exactly defined. In chapter 3 the basic test procedures and test types are presented. The test procedures include the static and the dynamic test. In static testing, tool-assisted analysis and structured group testing - also called reviews - are considered in more detail. In dynamic testing, black box and white box procedures are explained. Test types include unit testing, integration testing, system testing, acceptance testing, functional testing, non-functional testing, negative testing, regression testing, and end-to-end testing. In chapter 4, defect management is explained in detail. Thereupon, a categorization of the test tools is presented in chapter 5. A distinction is made between tools for static testing and tools for dynamic testing. Chapter 6 compares the common tools with each other. Thereby it is defined, what is understood by these test tools and how they function. Since the license of three tools was not accessible, among them "Ranorex", "TestComplete", "UFT (Unified Functional Testing)", it is dealt in this work only on the description of these tools. Furthermore, in chapter 7 it is shown where the test tools can be used by system type. Finally, recommendations are presented to users in Chapter 8, taking into account programming skills and cost effectiveness considerations.

Results

After extensive research in the literature on the following testing tools, the table below summarizes the required information about the tools based on certain selected criteria. This analysis can help testers to select the best tool for testing software. The table once again summarizes the features of the tools.

Criteria/ToolsTestCaféSelenium Web-DriverAppiumSelenium IDEQF-TestRanorexTestCompleteUFT
Programming
language
JavaScript, Type-Script, CoffeeScriptJava, C#, Perl, Python, JavaScript, Ruby, PHPJava, C#, Perl, Python, JavaScript, Ruby, PHPJava, C#, Perl, Python, JavaScript, Ruby, PHPJava, Python, Groovy, JavaScriptC #, Python, C++, VB.net and XMLVbScript , Delphi, C++, C# und JScriptsVbScri pt
Supported platformsWebWebMobile, WindowsWebJava, Web, Desktop (Win), Mobile (Android)Mobile, Web, Desktop (Win)Mobile, Web, Desktop (Win)Web, Desktop (Win)
Reports on test resultsyesyesyesyesyesyesyesyes
Re-
reusability
yesyesyesnoyesyesyesyes
Recording/ Playbacknononoyesyesyesyesyes
easy to learnyesyesnoyesyes   
Programming skillsyesyesyesyesnononono
Tools test typesEnd-to-End TestsFunktional Unit Test, End-to-End TestGUI Test, Functional Unit TestFunctional Unit TestGUI TestGUI TestFunkcional Unit Test, GUI Test, Unit TestFunctional Unit Test, Regression Test
LicenseFreeFreeFreeFreeNot freeNot freeNot freeNot free
Sources

DevExpress - TestCafe 
Documentation (2012-2021)

Thought-Works - Selenium Documentation (2021)Tom Christie Appium - Documentation (2014)Thought-Works - Selnium Dokumentation (2021)QF-Test Documentation (2020)Ranorex GmbH - Documentation (2021)Smart-Bear Software - Documentation (2021)MicroFocus - Documentation (2021)

 

Recomendations

In software development, quality is the most important goal of any project, so when choosing a tool, it is recommended to take into account the scope of the project and the costs estimated for testing, Also, the platform on which the project will be deployed should be reflected in the criteria for selecting a testing tool. Based on the results obtained in this work, Ranorex, TestComplete, UFT and QF-Test are recommended for testing on all platforms and for large projects. Since these are licensed tools, the budget for testing should be considered. Appium is recommended for mobile applications, while TestCafe, Selenium Webdriver, Selenium IDE are available for web testing (Cf. F. Okezie et al. 2019 p. 9).

  • Target group without programming knowledge
    For users without programming knowledge, it is recommended to use QFTest, Ranorex, TestComplete, UFT and Selenium IDE, as these software do not require programming knowledge. These tools are designed for recording automated test cases and have the same goal.
    These tools are user-friendly and easy to use and the user interface is clear and self-explanatory. This makes it easy for beginners and inexperienced testers. In Selenium IDE, for example, the possible inputs are suggested by means of a popup in case commands have to be inserted by hand in the "Table" tab. In addition, the possible inputs are well documented on the website of each tool. In addition, the source code can also be changed later, which makes the tool interesting for testers. Furthermore, the tests can be re-executed immediately after recording, so that the correct function of the tests can be easily checked even by inexperienced users.
  • Target group with programming knowledge
    For users with programming knowledge it is recommended to use TestCafe, Selenium Webdriver, Appium. Here it is left to the personal decision of the user, as this is a matter of their own interests.
    It is highly recommended to request a free trial version to be familiar with the process before starting the official tests. As seen in this work, not all vendors offer a wide range of scripting languages to choose from, so the tools that do not offer the preferred technology can quickly be rejected 66. Since the market for Mac users is particularly small, the purchase of a virtualization tool should be considered.
  • Investing in testing tools
    The introduction of a new tool involves costs for selection, acquisition, and maintenance of the tools. In addition, there may be costs for hardware and staff training. Depending on the complexity of the tool and the number of workstations to be equipped, the investment can quickly reach six figures. Of course, as with any investment, it is important to define the time frame for the amortization of the new test tool.
    In the case of tools for automating test execution, it is easy to estimate how much effort can be saved by an automated test run compared to manual execution. The new effort for programming the test must still be deducted, so that after only one automated test run, the cost-benefit balance is usually negative.
    The investment must also take into account the extent to which the test quality increases as a result of using the new tool, leading to more defects being found and corrected. The costs for development, support and maintenance are reduced as a result. However, the savings potential is higher here and therefore more interesting (Cf. Spillner & Linz. (2019) p. 223 f.).
    If the software to be tested is a small project, open source tools can be used instead of investing more costs for the purchase of a licensed tool. For those with a very small budget, the free software TestCafe, Selenium Webdriver, Selenium IDE, Appium can be recommended.

Conclusion

A goal of this work was it to compare individual test tools on the basis different criteria within the range of the software test with one another. For the comparison test tools were selected from the Internet and tested by means of a cinema web page, which was programmed in the context of the module "Web technology" during the study.

It is essential for a tester to know some important terms from the field of software testing and their definitions. This includes the term software quality, because quality and its assurance are also part of the tester's tasks. Likewise, a tester must master the definition of tests in order to be able to do the job with knowledge. Also the concept of error, which can have different definitions depending on what the error refers to, must be known. Since a tester has only a certain amount of time to test in a project, he must be able to estimate how long the test will take. In addition, the main types of tests that are suitable for implementation differ in their objectives. These include the objectives and methods of functional testing, where the functionality of the system as described in a specification is tested. Conversely, there are also the non-functional tests, which deal with aspects such as usability and user acceptance. In order for testing to be performed, procedures are needed to create test cases or to be able to get a view of the test object. Procedures such as the black box procedure, which performs testing based on specifications, and the white box procedure, which takes the program text as a reference for the tests, must be considered. In both procedures, different views of the test object can be considered, which differ greatly and provide a good overview of the system.

Software tests are an important part of the software development life cycle. They ensure that the software made available on the market is free of errors. Therefore, a tester must be able to distinguish the different software testing methods as well as the categories of tools and their areas of application. Tester should always be aware of the use of these tools in the development process.

During the work it was found that some tools are suitable only for testing web applications and others can be used for testing web, mobile, desktop - applications. Also most of the tools were "capture/replay tools" that can be used by users without programming skills.

What makes a good tester is the accumulated experience in software testing, because only through a lot of experience a tester can develop a good feeling for finding bugs in software. Another requirement for a good tester is having a lot of programming skills because the scripts are generated in different programming languages. Since a tester is the one who delivers the bug caused by another to the developer, he must have a certain sensitivity to convince the developers to fix the bug he found.

Because of the points discussed in this paper, software testing has become indispensable in software development. As a software tester, one is responsible for ensuring the quality of the product as well as the documentation. This allows developers to focus on programming, designing and correcting errors in the program. They no longer have to search for the errors themselves, but they are presented with them by the testers. Therefore, the cooperation and mutual support between them must work well.

This work would be useful for testers and developers to know which tool is best for a particular project, more tools can be compared and more criteria can be highlighted for comparison. This would allow software testers to choose perfect tools for testing applications. This will save more time and reduce costs if the right testing tool is selected. It can be concluded that there is no one perfect tool for testing, but for a particular testing purpose, compromises can be made depending on the size of the project to select the best tool that takes into account the planned cost of testing and allows for the platform of the application and also the language in which the project is developed.

The whole bachelor's thesis concerning QF-Test can be found here (German PDF only).

Bachelor's thesis: Software Testing: Common test tools in comparison, October 2021 - Fachhochschule Südwestfalen - University of Applied Sciences, Iserlohn, Germany.

(Original German texts and citations are translated into English.)