Pushing the Boundaries of Test Automation: an Overview of How to Automate the UX with Heuristics

One of my current responsibilities is to find ways to automate, as much as practical, the ‘testing’ of the user experience (UX) for complex web-based applications. In my view, full test automation of UX is impractical and probably unwise; however, we can use automation to find potential UX problems, or undesirable effects, even in rich, complex applications. I, and others, am working to find ways to use automation to discover these various types of potential problems. Here’s an overview of some of the points I have made. I intend to extend and expand on my work in future posts.

In my experience, heuristic techniques are useful in helping identify potential issues. Various people have managed to create test automation that essentially automates different types of heuristics.

EXAMPLES OF PUSHING THE BOUNDARIES

  • Dynamic Usability / Accessibility Testing—See the following article I wrote that describes some of my work in the area. The code is available here. You’re welcome to use and experiment with it.
  • Fighting Layout Bugs—This is by Michael Tamm. He described the work in a public “tech talk” at Google’s Test Automation Conference (GTAC) in 2009. The link is available on his project’s homepage.
  • Crawljax—This is another open-source project which applies actions. It uses patterns to determine when to apply the actions. I’ve seen it used for significant, global web applications. There is a video online which describes some of that work.
  • BiDi Checker—This software helps to identify problems related to bi-directional content on web sites and web applications. It successfully finds and identifies a wide range of potential issues.

You might notice that all the examples I’ve provided are available as free open-source software (FOSS). I’ve learnt to value open source because it reduces the cost of experimentation and allows us to extend and modify the code, e.g. to add new heuristics relatively easily. (You still need to be able to write code, however the code is freely and immediately available.)

AUTOMATION IS (OFTEN) NECESSARY, BUT NOT SUFFICIENT

Automation and automated tests can be beguiling, and paradoxically increase the chances of missing critical problems if we chose to rely mainly, or even solely, on the automated tests. Even with state of the art (the best we can do across the industry) automated tests, I still believe we need to ask additional questions about the software being tested. Sadly, in my experience, most automated tests are poorly designed and implemented, which increases the likelihood of problems eluding the automated tests.

Here are two articles which describe some key concerns:

The first describes how people can be biased into over-reliance on automation. It is called “Beware of Automation Bias by M.L. Cummings, 2004. The article is available online.

The second helped me understand where testing helps us work out which questions to ask (of the software) and that we need to use a process to identify the relevant questions. The article is called “5 Orders of Ignorance,” by Phillip G Armour, CACM 2000.

 

Julian Harty

Julian has been working in technology since 1980 and over the years has held an eclectic collection
of roles and responsibilities: He was the first software test engineer at Google in Europe, the Tester at Large for eBay group, and has consulted and helped lots of companies and projects globally. He’s also been a company director for a mix of companies and startups. Currently, Julian combines commercial work, parttime Ph.D. studies, and helping with improving education, teaching and learning using low-cost mobile devices particularly for disadvantaged schools globally. He has authored several books, most recently the Mobile Analytics Playbook which can be downloaded for free at: http://www.themobileanalyticsplaybook.com/. You can find lots of his work, including opensource projects, online.

Julian Harty
Julian has been working in technology since 1980 and over the years has held an eclectic collection of roles and responsibilities: He was the first software test engineer at Google in Europe, the Tester at Large for eBay group, and has consulted and helped lots of companies and projects globally. He’s also been a company director for a mix of companies and startups. Currently, Julian combines commercial work, parttime Ph.D. studies, and helping with improving education, teaching and learning using low-cost mobile devices particularly for disadvantaged schools globally. He has authored several books, most recently the Mobile Analytics Playbook which can be downloaded for free at: http://www.themobileanalyticsplaybook.com/. You can find lots of his work, including opensource projects, online.

The Related Post

From cross-device testing, to regression testing, to load testing, to data-driven testing, check out the types of testing that are suitable for Test Automation. Scene: Interior QA Department. Engineering is preparing for a final product launch with a deadline that is 12 weeks away. In 6 weeks, there will be a 1 week quality gate, ...
Recently while teaching a workshop on Testing Dirty Systems, I uttered this “Randyism” off the top of my head, “Test automation is not automatic.” I realized immediately that I had just concisely stated the problem in making test automation a reality in many organizations. Most testers know that test automation is not automatic. (Wouldn’t it be great?) However, ...
September Issue 2019: Advancing Automation
“Testing Applications on the web” – 2nd EditionAuthors: Hung Q. Nguyen, Bob Johnson, Michael HackettPublisher: Wiley; edition (May 16, 2003) This is good book. If you test web apps, you should buy it!, April 20, 2001By Dr. Cem Kaner – Director of Florida Institute of Technology’s Center for Software Testing Education & Research Book Reviews ...
Introduction As a consultant and trainer, I am often asked by my clients and students how to deal with automated acceptance tests. One common question is whether automated acceptance tests should use the graphical user interface (GUI) provided by the application.
Regardless of your current state of tools, building an effective Continuous Integration suite of significant automated regression tests is the key to moving to a higher level of confidence in today’s development world. In the evolution timeline of software development tools, new tools have recently proliferated. We have all been sold on collaboration, transparency and ...
Automated Testing is a huge part of DevOps, but without human-performed quality assurance testing, you’re increasing the risk of  lower-quality software making it into production.  Automated Testing is an essential DevOps practice to increase organizations’ release cadence and code quality. But there are definitely limits to only using Automated Testing. Without human quality assurance (QA) ...
An automation framework is a way to organize your code in meaningful manner so that any person who is working with you can understand what each file contains. Automation frameworks differ based on how you organize your code – it can be organized based on your data, so that any person who wants to use ...
When Netflix decided to enter the Android ecosystem, we faced a daunting set of challenges: 1. We wanted to release rapidly (every 6-8 weeks). 2. There were hundreds of Android devices of different shapes, versions, capacities, and specifications which need to playback audio and video. 3. We wanted to keep the team small and happy. ...
People who know me and my work probably know my emphasis on good test design for successful test automation. I have written about this in “Key Success Factors for Keyword Driven Testing“. In the Action Based Testing (ABT) method that I have pioneered over the years it is an essential element for success. However, agreeing ...
For this interview, we talked to Greg Wester, Senior Member Technical Staff, Craig Jennings, Senior Director, Quality Engineering and Ritu Ganguly, QE Director at Salesforce. Salesforce.com is a cloud-based enterprise software company specializing in software as a service (SaaS). Best known for its Customer Relationship Management (CRM) product, it was ranked number 27 in Fortune’s 100 ...
The Cloud demands that we be as nimble as possible, delivering features and fixes in almost real-time fashion. Both customer and provider rely on software development that can maintain quality while being light on its feet and constantly moving. In addition, Cloud-oriented systems tend to be highly complex and dynamic in structure — more than ...

Leave a Reply

Your email address will not be published. Required fields are marked *

Stay in the loop with the lastest
software testing news

Subscribe