An important but often underestimated part of software development is testing. Testing is, by definition, challenging. If bugs were easy to find, they wouldn’t be there in the first place (although it should be noted that early in the SLDC various trivial bugs might show up as well of course). A tester has to think outside the box to find the bugs that others have missed. In many cases, understanding the business domain of an application is more crucial for effective testing, as is detailed knowledge of the application itself.
In open source projects, quality is typically addressed by contributors and coordinating architects. Tests of units, components, and services are often done effectively and automated well. This allows a project to move forward even when many contributions are made. Comprehensive automated testing with sufficient range and depth helps keep the product stable.
While some open source projects develop from accumulating contributions of dispersed people, DevOps oriented projects may follow a Scrum or Kanban approach that includes simultaneous development and release. This process also relies heavily on comprehensiveness of tests and their seamless automation. Whenever there is a new version (this can be as small as a check in of a single source file), tests should be able to verify that the system didn’t break. At the same time, those tests shouldn’t break themselves either, which for UI-based tests is not trivial.
The testing pyramid, proposed by Mike Cohn in his book, Succeeding with Agile, positions the UI as the smallest part of testing. Most of the testing should focus at the unit and service or component levels. It makes it easier to design tests well, and automation at unit or component/service level tends to be easier and more stable.
I agree that this is a good strategy. However, from what I’ve observed on various projects, the UI testing remains an important part. In the web world, for example, the availability of techniques like Ajax and AngularJS allow designers to create interesting and highly interactive user experiences in which many parts of the application come together under test. An ultimate example of UI right web applications are single-page applications, where all or much of the application functionality is presented to users in a single page. The complexity of a UI can rival that of the more traditional client-server applications.
I therefore like to leave some more room in the top of the picture, making it look like this.
Even for UI automation, the technical side can be fairly straightforward. There are simple open source tools like Selenium that can take care of interfacing with the UI, mimicking the user’s behavior toward the application under test. Tests through the UI are often mixed with non-UI operations as well, such as service calls, command line commands, and SQL queries.
The problems with UI tests come in maintenance. A small change in a UI design or UI behavior can knock out large amounts of the automated tests interacting with them. Common causes are interface elements that can no longer be found or unexpected waiting times for the UI to respond to operations. UI automation is then avoided for the wrong reason: the inability to make it work well.
Let me describe a couple of steps you can take to alleviate these problems. A good basis for success automation is test design. How you design your tests has a big impact on their automation. In other words, successful test automation is not as much a technical challenge as it is a test design challenge. As I see it, there are two major levels that come together in a good test design:
- Overall structure of the tests
- Design of individual test cases
For the structure of tests we follow a modularized approach, which is a similar approach to how applications are designed. Tests cases are organized in test modules. Think of them like the chapters in a book. We have some detailed templates for how to do that, but at the very minimum you should try to distinguish between business tests and interaction tests. The business tests look at the business objects and business flows, hiding any UI (or API) navigation details. Interaction tests look at whether users or other systems can interact with the application under test, and consequently care about UI details. The key goal is to avoid mixing interaction tests and business tests, since the detailed level of interaction tests will make them hard to understand and maintain.
Once the test modules have been determined, they can be developed whenever it is convenient. Typically business tests can be developed early, because they depend more on business rules and transactions than on how an application implements them. Interaction tests can be created when a team is defining the UIs and APIs.
One other major factor to define automation success is known as testability. Your application should facilitate testing as a key feature. Agile teams are particularly well suited to achieving this, since product owners, developers, QA people, and automation engineers cooperate. However, open source projects do not necessarily have such teams, and the product ownership will have to define testability.
For more on testability, check out this article I wrote for TechWell Insights as well. The main aspects of testability, as described there, are:
- Overall design of the application, with clear components, tiers, services, etc.
- Specific features for testing, like API hooks or “ready” properties to help in timing, clear and unique identifying properties for UI elements, and white box access to data and events before they’re even displayed
Automation can be challenging, especially via the UI. However, it cannot be avoided—nor should it be—because it is difficult. Cooperation between all participants in a project can lead to good results that are stable, maintainable, and not inefficient to achieve. That’s where LogiGear can help. Our Selenium Automation solution is specifically designed to help QA/Test Engineers and teams to leverage the full benefits of Selenium and Appium. To learn more about our quickstart and custom development services packages visit logigear.com today
This article originally appeared on opensource.com View it here: https://opensource.com/business/16/4/what-makes-test-automation-successful
Author: Hans Bulwada, Chief Technology Officer
Hans Buwalda is an internationally recognized expert in test development and testing technology management and a pioneer of keyword-driven test automation. He was the first to present this approach, which is now widely used throughout the testing industry. Originally from The Netherlands, Hans now lives and works in California as CTO of LogiGear Corporation, directing the development of what has become the successful Action Based Testing™ methodology for test automation and its supporting TestArchitect™ toolset. Prior to joining LogiGear, Hans served as project director at CMG (now CGI) in the Netherlands. He is co-author of Integrated Test Design and Automation and a frequent speaker at international conferences.