The Automation of Mobile Testing – Should We or Shouldn’t We . . .

6.3 -Sep-2014-03Manual testing teams may not be able to test all the processes with each build

Test automation of applications has been around for many years. There are many of us in the automated testing field that started very early in the test automation phase, but the introduction of mobile devices has brought on a new angle in test automation of applications. Mobile device testing introduces new technical challenges along with new processes and testing tools. But should we use test automation for mobile applications that are on mobile devices? Many of us have mobile devices and we get our applications from the stores on Android or iOS and we go about our business.

Quality, performance and business rules?

As we know, there are at least three kinds of applications for Mobile Devices; Native, Mobile Web, and Hybrid. Like any application, developers need to build it, and testers test it. Now, when we introduce methodology into the mix, we add challenges. This agile development process introduces tight timelines and tight delivery dates. Developers are challenged to get the business processes into the applications as quickly as possible. This adds multiple builds daily, sometimes hourly. In the desktop and standard web testing process, whether automated or manual, we would expect to regression test each build, right? How many testers would you need to do that when builds are being delivered so quickly? Can manual testers do that and keep up with what they have tested, along with making sure that the business processes have been tested each time?

When there is a time crunch, usually there will be decisions made to stick to the critical business processes or new functionality and leave it at that. It sounds great, but is that the best way to test and make sure a quality product has been delivered to production? Users are accustomed to application pages being rendered within a few seconds, which means response times need to be verified before going into production. Users are accustomed to the application functionality being consistent each and every build. If there is a new build, users are not expecting to lose functionality or any visual effects. Our eyes are accustomed to application pages looking great each and every build. How is a small test team going to do that? Honestly, a manual testing team will probably not be able to test all the processes each time and will revert to the 80/ 20 rule, or something close. Now, there are some teams out there that will claim to have an army of testers that they will call up and have the start testing the build. That is awesome, however, even that army of testers may take the common path or the happy path. They may not have all the business rules or the defects to look at to see what the challenges or critical areas of the app are. How do they report back to a central point and collaborate on what they have tested and what the results were? How would they also add requirements trace-ability into the process? Unless this is a seasoned group, that will be a challenge.

Where test automation comes in.

From wherever you are in the world, using test automation can make testing more complete and add requirements coverage, test coverage, trace-ability, defect tracking, and overall test reporting easier to manage. From one report, management can see what percentage of the application has been tested, what defects have been introduced and what defects have been fixed. With test automation, when a defect is found, it is very easy to recreate that defect and the steps for it. When manual testers test and find a defect, if the steps are complex enough, then chances are, the tester will not follow the same path and the defect may not be reproduced. That may be a big issue.

With test automation, test scripts can be run off hours if needed. Now with mobile device testing, that will add complexity because some environments and devices will need to be configured to run the test. That may or may not be able to be automated. Either way, the scripts can be run and tests can be run without manual intervention once set up. With test automation, when builds are done multiple times during the day, we can find the changes to the application much faster than a manual tester.

How fast can scripts do that?

Well, that would depend upon the environments being used, the complexity of the application under test, and the performance of the application.

For the most part, running automated scripts will always test more functionality than a manual tester. If there is a critical path that needs to be tested, the scripts can be run specifically for that and they will be executed the same each and every time. For many teams, cost comes into the picture. Most departments have budgets and test automation tools are usually pretty expensive. Once research is done, it will be found that there are many tools out there that can create scripts for a fairly inexpensive cost with even cheaper renewal costs. For the teams that do not have much experience with test automation, it would be better to try a small project with one of the less expensive tools. And remember, cheaper cost does not mean cheaper in the way the tool works, but it may mean that you don’t get all the bells and whistles. Compromises will always need to be added.

Should you use test automation for mobile device testing?

We can ask the same question for the test automation of a desktop application. It does not just need to be answered for mobile device testing. If your team plans to create subsequent updates to the app, or create the application in phases, I would highly recommend some sort of test script automation. If the plan is to just create a quick application one time and that is all, then it is up to the team to decide whether or not it is cost effective to automate the testing process. There will always be a benefit to test automation as a whole because test automation is not just script execution. Test automation is the whole testing process. That is where the real benefit for the whole project comes in. But for development and testers, script automation will help by allowing the manual testers to move onto some other critical tasks and leave the regression to the automation team.

Whether you have 4 or 400 testers, there will always be something that will be missed. Adding test automation will add test coverage and higher requirements coverage to your testing results. Mobile Device test automation will use a lot of the same processes that standard test automation uses. The tools may be different, but the rest will be similar.

(This article was originally published on


Sam Mullin

Sam Mullin is a specialist in Mobile Device Test Automation. He has contributed to test planning, design, and execution of automated testing efforts within the healthcare, electricity, and banking industries. He continues to work on providing specialized support for mobile device test automation using various mobile device testing tools. Currently, Sam is working for Olenick and Associates located in Chicago Illinois (

Sam Mullin
Sam Mullin is a specialist in Mobile Device Test Automation. He has contributed to test planning, design, and execution of automated testing efforts within the healthcare, electricity, and banking industries. He continues to work on providing specialized support for mobile device test automation using various mobile device testing tools. Currently, Sam is working for Olenick and Associates located in Chicago Illinois

The Related Post

What you need to know for testing in the new paradigm This two part article analyzes the impact of the Internet of Things (IoT) product development on traditional testing. Part one of this series starts with a wide view on the IoT, embedded systems and device development aspects of testing. Part two, to be published ...
Don’t make the mistake of assuming too many similarities. It is common knowledge that mobile applications don’t function in the same way as their web-based counterparts. The user experience is affected by a few other factors such as device and network capability. If you are building out a performance testing strategy for your mobile website ...
  Mobile analytics experts Julian Harty and Antoine Aymer have teamed up to deliver a 161-page handbook designed to help you “enhance the quality, velocity, and efficiency of your mobile apps by integrating mobile analytics and mobile testing”.
A sampling of some free, online, and easy-to-use mobile device emulators that can help get you started with testing. ScreenFly A free, customizable tool to test your website on any screen size, including desktops, tablets, televisions, and mobile phones.
To start with, we need a Test schedule. The same is created in the process of developing the Test plan. In this schedule, we have to estimate the time required for testing of the entire Data Warehouse system. There are different methodologies available to create a Test schedule. None of them are perfect because the ...
25% of Americans own a tablet. Up from 11% of U.S. adults in July of 2011 to 18% in January of 2012. – Pew Internet & American Life Project Nigeria has close to 100 million mobile phone lines, making it Africa’s largest telecoms market. – Nigerian Communications Commission Google plans to sell 200 million Android ...
Great mobile testing requires creativity to find problems that matter. I’d like to take you through the thought process of testers and discuss the types of things they consider when testing a mobile app. The intention here is to highlight their thought processes and to show the coverage and depth that testers often go to.
To ensure the success of an app, QA must be involved in all stages of development. Quality Assurance (QA) plays a vital role in the development of mobile applications, but many overlook the critical nature of this piece of the app development process. To ensure the success of an app, QA must be involved in ...
What you need to know in order to have effective and reliable Test Automation for your mobile apps I realized that Test Automation interfaces are pivotal to effective and efficient Test Automation, yet very few people trying to test their mobile apps seemed to know how their automated tests connected with the apps they wanted ...
What you need to know to get started Introduction to mobile application testing Gone are the days when the telephone was an appliance that sat in a corner and had to ring to get our attention, or a computer was a machine only few people used — these devices are now an extension of our ...

Leave a Reply

Your email address will not be published. Required fields are marked *

Stay in the loop with the lastest
software testing news