We would like to thank José Campos for again choosing QF-Test for his course "Verification and Validation". We are happy to publish the students' feedback here.
It is our pleasure to provide free trial licenses for research, teaching, open-source & charitable projects
It was easy to learn how to start using the tool by making simple tests.
The tool was really useful for Model-based testing, but there were some problems along the way that were mainly due to bugs. The application we were testing had some features that involved the icon tray menu. Recorded actions that involved interaction with the tray icon were not reliable since they would only work sometimes.
The main pain-point of the using QF-Test is the difficulty in sharing the test with a teammate. The jar file location is hard-coded as an absolute path and we haven't found a way to make this dynamic between systems/computers. As such, when coding in a different machine, we had to reconfigure these paths. Overall, the experience was positive with the exception of the application's jar file path problem.
The QF-Test tool appeared to be promising when automating tests of user interfaces. One problem that is not directly related to the test creation, but we ran into, was that with high DPI monitors, some workarounds were needed to run the application decently.
We felt that the QF-Test tool is pretty good and the installation process was straightforward. We didn't manage to add the tool to PATH with the setup tool, so we resorted to finding the executable in the project's files and executing it using the terminal. We feel that this part of the user experience could be improved upon.
The tool felt very intuitive to use. However, we could not figure out how to check if the search bar was invisible since the variable visible still appeared as true even when there was no bar on the screen.
We can easily test transitions and states, and every single detail about what we do during recording (clicks, checks, inputs, etc.). Our group didn't use all the features of the tool, but the ones we used looked perfect for what we needed to do.
We would like to share a suggestion about the tool: we believe it’s a symbiotic improvement if your QF-Test tool can be integrated in an IDE. We`re not saying the tool is hard to understand and work (it's super intuitive by instance), but we think that might facilitate and incentive its use for many other people that might not even know the existence of the tool.
After using QF-Test to create test suites for the project, we consider it very well-prepared and useful. Despite some initial confusion regarding the setup, by using the Quickstart Wizard, we managed to use the tool to its full potential in the scope of our project. The only issue we encountered was that we had to change the path of the executable file every time we switched between different devices, due to using the complete path, because we were unaware how to use a relative path.
The initial setup and integration with our project were easy and the wizard worked as intended. We could not find a way to use a relative path for the jar path. This meant that on the different PCs, there was a need to run the wizard every time someone changed the tests. After that, step recording was intuitive, and we did not have any issues. We found that dragging-and-dropping test steps on the left menu worked as intended as well as copying, cutting and pasting them.
Overall, QF-Test proved to be very useful when creating test cases that require interaction with the UI. However, one problem that we found is that the configuration file used makes it harder to collaborate with other people when using the tool. The configuration file contains information that is specific to each developer's machine. We edited this file in an attempt to use it in a different machine. Unfortunately, QF-Test_ signs the configuration file and checks its integrity upon loading, making it impossible to use a file that's been edited. Perhaps, if the test suite data was stored in a separate file, it could be used with VCS, while the file with user specific information is ignored.
In general, the tool is really intuitive, however:
Our experience with the QF-Tool was generally positive. We will now highlight some features that we think would be excellent additions to the tool, as they would make the user's life easier in some cases that we came across throughout this work:
Test portability is an issue. Since QF-Test Setup Wizard generates a dependency on the local filesystem, we were forced to manipulate this dependency every time we committed information towards Github since another team member was running in a different filesystem.
While the tool seems interesting for developing UI testing, it also comes with some difficulties for developers, such as:
We have worked with similar tools in the past (like cypress) and there are a few differences to note. To start, cypress has a cleaner/more modern GUI, and I felt like it has a better user experience. The learning curve for the QF-Test seems much steeper than other tools, but it looks super powerful. The manual / tutorials seem very helpful and complete, which is something that is necessary when there isn't much community online to get support from.
One feature that we noticed that it was missing is a toggle button for quickly disabling/enabling test case steps. Sometimes when developing a test, we implement verifications that aren't finished, as a placeholder, and it's a bit annoying to get errors for those each time we do a test run.
Otherwise seems a slick tool. A must-have for desktop java applications!
The QF-Test tool allowed us to fulfil the basic usage of writing simple GUI tests for a Java application. In a couple of hours with the help of QF-Test tool beginner tutorials we were able to understand how to use it within our basic application. However, we found the controls not so intuitive and the UI outdated and confusing. A better UI and tooltips would greatly improve beginner level user experience.
By watching the tutorials we learned that this tool enables much more powerful testing with integration in CI/CD pipelines, multiplatform testing (not specific to Java applications), video recording, etc. We didn't test these features (Out of the scope of the assignment). Also, the platform from our experience appears to really stable and ready for professional/production environments. We tested on Windows and MacOS without any issue. Really curious to see how the QF-Test runner would perform on docker/Linux machines for the CI/CD integration (not tested).
For this project the QF-Tool for Windows platform, desktop version was used. The installation process was very simple and very intuitive, the use is also simple, with a graphical interface where the elements are well distributed, which makes the learning curve to work with the tool very flat. In terms of tests it covered all the scenarios we identified during the project. One negative aspect of the tool is that it is necessary to invest time to explore and read the documentation to be fully acquainted with the tool, because the community is really small, so any problems could take a lot of time to resolve.