This post is frequently updated. Last updated: 27 Mar, 2020
Although the demand for desktop app testing is not growing as fast as mobile and web app testing, it’s still a crucial day-to-day duty of many testers, especially those who work on sophisticated legacy systems in complex enterprise environments. Ignoring desktop app automation is simply a luxury that they can’t afford.
Desktop applications are computer programs that must be installed, configured and executed on a desktop operating system such as Windows, Linux and macOS. End-users need a desktop computer or laptop with sufficient hardware resources to access the functionalities of these apps.
Common technologies to build desktop apps include Java (Swing, AWT, SWT, JavaFX), .NET (UWP, WPF, WinForms, VB.NET), Windows API (win32), GDI, MFC, Visual Basic, Visual C++, PowerBuilder, SAPGUI, Oracle Forms, Telerik, DevExpress, Infragistics, Xlib, Cocoa, etc.
This article lists out the best automation tools for automating functional testing of desktop apps. If you’re looking for an automation tool, you’ll find this list convenient to explore, evaluate, compare and eventually adopt the best tool for your automation needs.
- Tricentis Tosca
- Micro Focus UFT (QTP)
- Robot Framework
- White Framework
• The tools are not ranked.
• You may notice that many of the tools on the list are commercial. This is due to the complex nature of automating desktop apps. Most tools that are able to automate desktop apps are usually proprietary, and there are limited open-source options.
When assessing the pros and cons of a specific tool, I used the following criteria.
|1||Cross Operating Systems||Can the tool run on multiple OS (Windows, Linux, macOS)?|
|2||Cross Automation Platforms||Can the tool automate desktop apps built by various technologies, e.g. Java, WPF, UWP, SAPGUI, Oracle Forms, etc. in a single end-to-end test flow?|
|4||Supported Test Runners||Does the tool support various test runners or unit test frameworks, e.g. Junit, xUnit, Pytest, TestNG, Mocha, etc.?|
|5||Supported Automation Technologies||What kind of automation technologies does the tool support? Does it provide its own proprietary technology or build on top of open-source solutions like Selenium WebDriver?|
|6||Element Locating Technique||What types of element locators does the tool support? Does it allow testers to inspect the internal properties of UI objects? Does it support its proprietary technique?|
|7||Built-in UI Inspection Tool||Does the tool provide a built-in tool to inspect the internals of UI objects, or does it rely on a third-party tool?|
|8||Page Object Model (POM) Support||Does the tool separate the automation tests from the UI element locators as in the form of the Page Object Model (POM)?|
|9||Coding Skills Required||How much coding proficiency does the tool require?|
|10||Record & Playback||Does the tool provide built-in record & playback capabilities?|
|11||Built-in DevOps Integrations||Does the tool integrate with CI/CD tools such as Jenkins, TFS, Azure DevOps, AWS, CircleCI, etc.?|
|12||Price||Is the tool free and open-source? If it’s commercial, does it provide a Freemium option?|
WinAppDriver is built on top of the WebDriver protocol – the gold standard for web & mobile app testing. You can use WinAppDriver to drive any Windows apps such as classic Windows controls (Win32), WPF, WinForms, and UWP using a wide variety of language bindings (C#, Java, Python, Ruby), and test runners (MSTest, JUnit, Nunit, etc.).
|◘ You can use any language and test runner you prefer to test your Windows apps thanks to WebDriver||◘ Only runs on Windows 10, not macOS or Linux|
|◘ It’s WebDriver-based so Selenium/Appium testers feel right at home||◘ Cannot automate apps written by other tech stacks like Java, DevExpress, Infragistics, SAPGUI and Oracle Forms, etc.|
|◘ You can effortlessly integrate with Selenium and Appium projects in the same language||◘ No built-in spy tool. You must use inspect.exe (Windows SDK) and mentally map inspect.exe properties to WinAppDriver properties|
|◘ You can perform advanced gestures such as pressure, tilt, barrel button (Surface pen), etc.||◘ You must know how to code & architect a maintainable test automation framework|
|◘ You can use XPath and other properties to locate Windows UI elements||◘ No record & playback|
|◘ You can use the industry-standard POM||◘ No built-in support for ALM integrations|
|◘ It’s FREE, open-source and backed by Microsoft|
If you’re only interested in automating Windows apps with WebDriver, and you know how to code, you should go with WinAppDriver. If you’re not confident with coding, adopting WinAppDriver may not be the best option for you. If you’re not sure how to build a well-structured WebDriver-based framework, check out this post for ideas.
Winium.The desktop is an open-source test automation framework for WinForms and WPF app testing. Just like WinAppDriver, it’s based on WebDriver protocol. However, the tool is considerably weaker than WinAppDriver in several areas.
|◘ You can use any test runner (JUnit, TestNG, NUnit, PyUnit, etc.)||◘ Only runs on Windows 10, not macOS or Linux|
|◘ You can automate WPF and WinForms||◘ Cannot automate apps written by other tech stacks such as Java, DevExpress, Infragistics, SAPGUI, Oracle Forms, etc.|
|◘ It’s also WebDriver-based so using the tools is easier if you’re familiar with Selenium & Appium||◘ The API is not comprehensive enough|
|◘ You can use XPath to locate Windows UI elements||◘ No built-in spy tool. You must use inspect.exe or any other third-party tools|
|◘ You can use the industry-standard Page Object Model (POM)||◘ You must know how to code & architect a maintainable test automation frameworks|
|◘ It’s FREE and open-source||◘ No record & playback|
|◘ No built-in support for ALM integrations|
For open-source projects, active maintenance is key. Winium has not been actively maintained since 2016 so if you go with it, you have to fix bugs or add new improvements on your own. Your team needs extensive technical knowledge and skills. Ultimately Winium is a less desirable choice than WinApp driver.
TestArchitect is a Keyword-Driven Testing tool. Testers with modest programming skills can learn quickly thanks to an extensive keyword library. TestArchitect also features a modern Keyword-Driven Testing method called Action-Based Testing. TestArchitect appears in Forrester Research reports in recent years.
|◘ TestArchitect runs on multiple Operating Systems (Windows, Linux and macOS)||◘ Besides ABT, you can only use 3 languages: Java, Python and C# to write actions. No other languages supported.|
|◘ Complete feature set for test development, test execution, and test management||◘ Only one test runner provided: TestArchitect Controller (proprietary).|
|◘ You’re able to write cross-platform tests jumping from desktop to web, mobile, API, database, and more platforms.||◘ You cannot push TestArchitect test scripts into Git for version control because TestArchitect has its own built-in version control system (proprietary)|
|◘ You can automate apps written by different tech stacks e.g. .NET, Java, classic Windows controls, Oracle Forms, DevExpress, etc.||◘ The commercial product so you have to pay (flexible pricing). However, you can use TestArchitect Team for free with a limited number of test cases.|
|◘ You can locate UI elements using various techniques including XPath, CSS selector, ID, name, class, etc.||◘ You cannot run desktop tests in parallel on one machine but you can distribute tests to a network of bare-metal machines, VMs or Docker containers|
|◘ You can easily integrate TestArchitect with many ALM tools like Team Foundation Server, Azure DevOps, Zephyr, Jenkins, HP QC, etc.||◘ Automation technologies are proprietary|
|◘ Action-Based Testing is a relatively more modern Keyword-Driven Testing method than other keyword-driven approaches|
|◘ Built-in record-playback|
|◘ Built-in object spy tool|
|◘ TestArchitect has its own standalone IDE|
|◘ You don’t have to learn coding since tests are keyword-driven|
TestArchitect is relatively stronger than open-source solutions in terms of usability, width, and depth of support. If you don’t want to build a framework from scratch, you should go with this well-rounded framework. Besides, the tool is well maintained by a professional engineering team, unlike Winium. The caveat? You need a budget for it (perpetual or annual). Luckily, the tool does offer a Freemium version with a limited number of test cases.
Ranorex is a test automation tool that has been in the market for a long time. They offer proprietary automation technologies to automate a wide range of desktop, web, and mobile apps. Ranorex also appears in Forrester Research reports in recent years.
|◘ Ranorex runs on Windows, macOS, and Linux||◘ Only one proprietary test runner supported|
|◘ You can compose tests in their standalone Ranorex Studio instead of using a third-party IDE such as Visual Studio Code||◘ The UI element locating technique called RanorexPath is their own invention. Nobody else adopts this technique so you can’t use any third-party tools besides the built-in Spy tool.|
|◘ Tests are keyword-driven so it’s easier to maintain in the long run.||◘ In the Omni-channel Functional Test Automation Tools 2018 report conducted by Forrester, Ranorex was ranked as the weakest offering|
|◘ Keyword-Driven Testing eliminates most of the coding needs so you could develop tests faster without coding skills||◘ Automation technologies are proprietary.|
|◘ Keywords can be scripted using VB.NET and C#. You could mix record-and-playback keywords with scripted keywords in one test||◘ The commercial product so you have to pay (~$5,800 per floating license)|
|◘ Built-in ALM integrations such as Bamboo, HP Quality Center, TFS/MTM, Jenkins, TeamCity, etc.|
|◘ Built-in object spy tool|
|◘ Built-in record-playback|
Ranorex’s presence in the Japan market is strong thanks to their Japanese localization and reliable Record & Playback. For other markets, Ranorex is not that strong. Invest in Ranorex if you love Record & Playback as the primary method of test development.
#5. Tricentis Tosca
Tricentis Tosca provides wide support for virtually all test activities. They offer customers a Continuous Testing platform that covers test design, UI testing, API testing, service virtualization, and test data management. With the recent acquisition of QASymphony (qTest), they further expand their scope to test management.
|◘ Tosca is easier to adopt for non-technical testers thanks to its model-based method||◘ Cannot run on macOS and Linux|
|◘ Tosca can automate a wide range of platforms like .NET, Java, browsers, mobile apps, etc.||◘ Element locators are stored in “modules” which are similar to page objects but it’s harder to use than page objects|
|◘ Cross-platform end to end tests jumping from desktop to mobile, web, SAP and API in one flow.||◘ Cannot integrate with Git for version control of test assets since Tosca has its own repository. However, you can link GitHub revisions to Tosca revisions.|
|◘ Locating elements using various properties including ID, text, and anchor.||◘ Model-based approach (noun-first keywords, learn more) create unnecessary friction in test creating and maintenance problems|
|◘ Built-in object spy tool called XScan||◘ Too much mouse clicking when writing tests|
|◘ Built-in record-playback||◘ Proprietary test runner|
|◘ Built-in ALM integrations and test management tools, especially QASymphony qTest||◘ Tosca only supports C# and Java|
|◘ Automatically generate test data (the feature is actually called “generate instances for a test case”).||◘ Commercial options only, no Freemium|
|◘ Gartner ranked the tool as Leader in their recent automation testing reports|
Tricentis Tosca is very friendly to manual testers who want to transition to test automation engineers. The company claims that given 3 weeks, they can train 85 out of 100 manual testers to become automation engineers using Tosca. On the contrary, only 15 out of 100 manual testers became Selenium engineers given 3 months of training (yes it’s months, not weeks). However, albeit easier to adopt at first, Tricentis Tosca will gradually become more complex and difficult to use in the long run due to the weight of Model-Based Testing.
#6. SmartBear TestComplete
SmartBear TestComplete has been on the automation market since 1999 so it’s obviously one of the veterans. TestComplete (TC) offers a comprehensive solution that covers desktop, web, and mobile. They’ve recently added a hybrid UI matching method combining visual recognition and object recognition.
|◘ TC is easier to adopt for non-technical testers thanks to its Keyword-Driven Testing method||◘ Cannot run on macOS and Linux|
|◘ TC can automate a wide range of platforms like .NET, Java, browsers, mobile apps, etc.||◘ Proprietary test runner|
|◘ Cross-platform end-to-end tests jumping from API to desktop, mobile, and web in one flow.||◘ Noun-first keyword approach creates unnecessary friction in test creating|
|◘ Locating elements using various properties including ID, text, XPath.||◘ Cannot integrate with Git for version control of test assets since TC has its own repository.|
|◘ Built-in object spy tool||◘ Commercial options only, no Freemium.|
|◘ Built-in record-playback that runs reliably|
|◘ Built-in ALM integrations such as Jenkins, HP Quality Center, etc.|
|◘ Gartner ranked the tool as Leader in their 2018 report|
TestComplete is indeed one of the best tools on this list. Consider checking it out.
#7. Micro Focus UFT (formerly HP QTP)
UFT/QTP is one of the oldest tools that still survives in the test automation market since the very early days. UFT is a powerful UI testing tool. It allows you to test a wide range of platforms including desktop, web, web services, SAP, and mobile. It also integrates seamlessly with HP Quality Center.
|◘ UFT is easier to adopt for non-technical testers thanks to its Keyword-Driven Testing method||◘ Cannot run on macOS and Linux|
|◘ UFT can automate a wide range of platforms like .NET, Java, browsers, mobile apps, web services, SAP, ComponentOne, DevExpress, UIA, databases, etc.||◘ Proprietary test runner|
|◘ Cross-platform end to end tests jumping from API, to desktop, mobile, and web in one flow.||◘ UFT supports only VBScript|
|◘ Locating elements using various object properties and XPath.||◘ Noun-first keyword approach creates unnecessary friction in test creating & maintenance|
|◘ Ability to integrate with Git, Subversion and TortoiseSVN for version control of test assets||◘ Too much mouse clicking when writing tests|
|◘ Built-in object spy tool||◘ Commercial so you must pay, no Freemium|
|◘ Built-in record-playback that runs reliably|
|◘ Built-in ALM integrations, especially with HP Quality Center/ HP ALM, etc.|
If you already have HP QC as your test management system, UFT is your natural choice for test automation.
SikuliX is an open-source tool that automates anything you see on screen using image recognition and OCR. The name actually means “God’s Eye” in Huichol Indian culture. Check it out here.
|◘ Java-based so it can run on Windows, Linux and macOS||◘ Since SikuliX is image-based testing, it needs a real computer screen connected. That means you cannot run tests in headless mode.|
|◘ Since SikuliX is image-based you can automate any platforms (Java, .NET, SAP, etc.)||◘ Image-based automation is slower and more fragile than the object-based technique|
|◘ You don’t have to know the internals of UI objects on-screen since those UI controls are recognized by images||◘ You cannot run tests in parallel on one machine since SikuliX takes away the entire control of the mouse and keyword interactions|
|◘ You can write keyword-driven tests that are relatively more readable using Robot Framework.||◘ No built-in record-playback capabilities|
|◘ Not much coding required.||◘ No built-in ALM integrations|
|◘ You can automate mobile apps via mobile emulators and web apps on desktop browsers||◘ Poor documentation (user guides) and community support|
|◘ No need for object spy tool|
|◘ FREE and open-source|
SikuliX is suitable for small projects since it was designed to automate various mundane tasks on Windows. However, once you scale up, you’ll find unsolvable problems due to the nature of the image recognition technique. Additionally, image recognition is way slower than object recognition.
#9. Robot Framework
Robot Framework is another open-source tool that is keyword-driven. The robot is flexible since you could use it with many desktop automation libraries such as SwingLibrary (Java Swing GUI), SikuliLibrary (SikuliX), SapGuiLibrary, JavaFXLibrary (JavaFX apps), EclipseLibrary (Eclipse RCP/SWT), and AutoItLibrary (AutoIt).
|◘ Can run on multiple OS: Windows, Linux, macOS||◘ You have to use Eclipse or RIDE to write tests (no built-in IDE)|
|◘ Write easy to understand tests in keyword-driven format so not much coding required||◘ Tests are tab-delimited so it’s hard to view|
|◘ Support many platforms such as desktop, web, and mobile||◘ No built-in object spy tool|
|◘ Support many test runners||◘ No built-in ALM integrations|
|◘ Multiple element locator types||◘ No built-in record-playback|
|◘ Can use page objects||◘ Out-of-date documentation & user guides|
|◘ Can run multi tests in parallel using Pabot|
|◘ Strong community support|
|◘ It’s FREE|
Robot Framework is an excellent alternative to expensive commercial tools. If your budget is tight, consider adopting Robot FW. However, beware of the not-so-obvious investment in learning and maintaining the framework in the long run.
AutoIt is a freeware designed to automate Windows GUI. Although the initial purpose of AutoIt was to “roll out” and configure thousands of PCs, you can use AutoIt for test automation too. Fun fact: AutoIt became popular in part thanks to Selenium’s inability to interact with Windows dialogs that you manually use to upload files to a web page.
|◘ AutoIt scripts can be compiled into standalone executables and executed from a Selenium-Java program||◘ Compatible with many Windows versions from XP to Windows 10 but cannot run on Linux and macOS|
|◘ Provides its own IDE to write scripts based on the free SciTE source code editor||◘ You cannot automate newer apps written by UWP, WPF or other platforms such as Java, DevExpress, etc.|
|◘ Built-in tool to inspect UI elements||◘ Only support its own BASIC-like scripting language|
|◘ Can use many object properties to identify UI controls||◘ You cannot run tests in parallel|
|◘ The project is actively maintained||◘ Only one unique test runner|
|◘ It’s FREE but closed source||◘ Coding required|
|◘ You must handle timing on your own through hard-coded sleeps|
|◘ No built-in record-playback|
|◘ No built-in ALM integrations|
|◘ Not popular so it’s harder to get community support|
AutoIt works for some simple automation scenarios. If you want a free and lightweight solution to get through simple automation challenges, use AutoIt. Also note that if you like Selenium, you ought to use AutoIt to work around pop-up dialogs.
#11. White Framework
White framework helps you automate rich client applications written by Win32, WinForms, WPF, Silverlight and JavaSWT. It’s based on .NET and depends on Microsoft’s UIAutomation (UIA) to drive desktop apps.
|◘ Support multiple .NET languages||◘ Cannot run on Linux and macOS|
|◘ Built on top of UIAutomation and Windows API so it’s powerful for its niche||◘ No built-in object spy tool must use Windows SDK Inspect, UIAutomation Verify or Snoop|
|◘ FREE and open-source||◘ No built-in IDE|
|◘ Not actively maintained by the community (the last commit was 2017)|
|◘ Poor documentation and sample code|
White is not a strong contender on this list due to its lack of capabilities and narrow target automation platforms. Try to avoid using White if you want a maintainable tool that can help scale up your test project. That said, small quick and dirty projects can still leverage White thanks to its free cost.
Conformiq is not strictly an automation tool. Its core offerings (Creator and Transformer) are tools that help you model the business flows of the app under test, generate tests and transform those tests to executable scripts. Conformiq totally relies on 3rd party automation libraries such as QTP/UFT for desktop testing and Selenium for web testing.
|◘ Supports desktop, web and mobile apps depending the automation tool||◘ Java-based so Conformiq runs across Windows and Linux but not macOS|
|◘ Provides its own Eclipse-like IDE||◘ No built-in record playback|
|◘ Model-based testing (MBT) so you can visualize your test flows using a model||◘ More expensive since you have to purchase UFT on top of Conformiq’s cost|
|◘ Auto-generate test cases from a model|
|◘ Keyword-Driven Testing / BDD test scripts so it’s more friendly to non-developers|
|◘ Can work with Selenium, UFT, Tricentis Tosca, Parasoft SOAtest, Eggplant, Appium, Cucumber, etc.|
|◘ Built-in tool to inspect UI elements|
|◘ Built-in ALM integrations with HP QC|
Conformiq works really well for test teams with more business expertise and less technical. It helps you visualize the test flows as a whole through a model so you don’t miss test cases and bugs. However, having to purchase additional UFT licenses is a big entry barrier.
This list was composed as of early 2019 and has been continuously updated ever since. The tool market for automation testing is very competitive. So you can expect radical changes and innovative features being introduced every day. If you spot some outdated information, please leave a comment. I’ll fix ASAP.
Have you seen better desktop app testing tools not listed above? Comment the tool names and why you think they are the best below. I’ll add them as soon as possible.
Disclaimer: This article is not endorsed by, directly affiliated with, maintained, authorized, or sponsored by any of the companies mentioned in this article (Microsoft, Ranorex, Tricentis, SmartBear LLC, HP, Micro Focus, and the listed open-source project teams). All product and company names are the registered trademarks of their original owners. The use of any trade name or trademark is for identification and reference purposes only and does not imply any association with the trademark holder or their product brand.