Options for Efficient Mobile Testing

By focusing on test design, analyzing test requirements and optimizing the approach to testing, it’s possible to maximize mobile test automation cost effectively.

In a previous article we outlined the importance of understanding the mobile ecosystem and test design for planning and executing mobile testing. The focus of this article is about efficient mobile test execution and test coverage maximization. 

Despite the growing number of mobile testing solutions the market is far from mature, which is one of the reasons why the majority of mobile device testing is still done manually. The primary issues with manual testing for mobile are scalability and cost. Outsourcing is always an option to increase the scale of testing, and then there is crowdsourcing. Crowdsourced mobile testing is being promoted as an ultra low-cost method of outsourced mobile testing. Crowdsourcing proponents tout flexibility, multilingual testing and a broad cross section of devices, but crowdsourcing is performed by a collection of freelancers so keep in mind the oft-used adverting disclaimer: “Results may vary.”

To adequately perform testing that assures mobile applications will meet customer expectations, test automation is necessary, and the need will only grow as mobile applications increase in complexity and importance. A high degree of test coverage can be achieved using both emulators and actual devices. How much testing to perform using emulators versus actual devices will depend on the budget, the automation tool, the application and the type of testing.

We have approached mobile test automation to make it possible for testers to employ a combination of testing assets—emulator/device/cloud testing resources—to efficiently achieve testing goals. Our automation tools provide the features and functionality to scale mobile test automation along with project needs and resources.

Emulators are ideal for testing new functionality or a new component design, and have some advantages over using actual devices. Logging faults and capturing screenshots are much simpler when working from a desktop, and you can simulate conditions that are hard to duplicate on real devices, like low battery power. A nice feature of TestArchitect is the ability to perform multilingual testing with emulators. This feature allows running identical tests in multiple languages in a short period of time.

Emulators can be accessed from the respective OS (Android, iOS, Windows Phone, Blackberry etc.) vendor’s SDK for free. Unlike iOS, Android isn’t the same for every device. For the best device approximation emulators from each Android device provider should be used. Then there are tools to augment the device emulators like Fiddler, NetLimiter and Charles Proxy that simulate GSM/GPRS networks to include testing of near real-world network conditions.

The main thing to keep in mind is that emulators and simulators only provide an approximation of device performance. From a QA perspective, emulators and simulators are fine for testing base functionality, but critical application sign-off should be done by live testing.

Testing a representative sample of actual devices is always ideal. The choice here is acquiring and updating an inventory of devices or using cloud base services. For overall maximum test coverage a public cloud option will provide broadest device selection. There are a number of cloud based options, each with their pros and cons. The big considerations are avoiding the need to recreate tests and security of the devices.

The accepted norm is to perform tests on 30 to 40 different devices currently available in the market. In cases where it’s essential to test actual devices, but less than the ideal number, applying Pareto’s Principle—80% of your customers will use just 20% of the devices available—should enable you to select the minimum number of devices that will provide the maximum test coverage.

Our approach to mobile cloud test automation was to create a host/agent set-up. Agents are downloaded onto the mobile devices (via wi-fi) to be tested, and tests are run remotely from the host machine. TestArchitect allows up to 40 devices to be tested simultaneously on a host or agent equipped machine to test within accepted norms. This approach eliminates API programming and provides the potential for regional testing—important for testing apps/devices on various operators’ networks, and in multiple languages—to validate applications for use on a global scale. The additional advantage of using TestArchitect is that tests used for any initial emulator testing can be run without modification on the actual devices, whether in the cloud or physically connected.

A well thought out combination of emulators, actual devices and cloud testing can provide broad test coverage and a high degree of automation. New applications can first be tested using emulators to test base functionality and/or multiple languages and identify major bugs. Next, targeted and thorough testing can be performed on actual devices.

Segregating testing in this manner can speed testing, and significantly help separate OS issues from device issues to minimize debugging time. Similarly, current applications can be tested on emulators of new or upgraded OS’s first, and then on a representative sample of next generation devices. Regression testing can be performed in the cloud across a wide range of devices with various hardware and OS versions.

Mobile devices are quickly becoming preferred to the desktop. To assure the best user experience, it is important to test mobile apps and websites with actual devices, and to keep this up requires automation. Mobile test automation is easily achievable. By focusing on test design, analyzing test requirements and optimizing the approach to testing, it’s possible to maximize mobile test automation and to do it cost effectively.

John Kane
John Kane oversees LogiGear’s Product Program Management to develop, productize and deliver LogiGear’s Testing Automation solutions to the marketplace. John has more than thirty years of experience in the software industry in the areas Electronic Design Automation (software for designing computer chips), Software Licensing, Chip Design, Government ARPA Programs, and Software Quality Consulting and Services. John Kane holds a Bachelor of Science in Computer Science from Oregon State University and a mini-MBA from Santa Clara University Leavey School of Business.
John Kane
John Kane oversees LogiGear’s Product Program Management to develop, productize and deliver LogiGear’s Testing Automation solutions to the marketplace. John has more than thirty years of experience in the software industry in the areas Electronic Design Automation (software for designing computer chips), Software Licensing, Chip Design, Government ARPA Programs, and Software Quality Consulting and Services. John Kane holds a Bachelor of Science in Computer Science from Oregon State University and a mini-MBA from Santa Clara University Leavey School of Business.

The Related Post

To help testers gain an edge, here’s a list of free resources Mobile testing is making leaps and bounds of progress in the overall testing space. As this field is highly dynamic, a tester must constantly evolve and improvise his or her knowledge of mobile testing. To help software testers gain an edge, I have compiled the following list ...
To start with, we need a Test schedule. The same is created in the process of developing the Test plan. In this schedule, we have to estimate the time required for testing of the entire Data Warehouse system. There are different methodologies available to create a Test schedule. None of them are perfect because the ...
I am not a big fan of concepts which moves industry standards to IT. I am rather a Agile and Scrum guy. Managing multiple projects at once and trying to set a highest quality standard is a challenge and this book shows how industrial language can be translated into software development. I do not think that it ...
In today’s mobile-first world, a good app is important, meaning an effective Mobile Testing strategy is  essential.  
Will testers be among the first IT professionals to shift their toolset and workflows from desktops and laptops to tablets and smartphones? As I’m sure you already know, a monumental shift from desktop to mobile is upon us. Not only have consumer applications started leaving the desktop behind, but B2B applications are also starting their ...
The mobile application ecosystem is very dynamic. OEMs are launching new devices and new customization, and new OS versions are delivered every now and then. This is the constant challenge that most enterprises face.
Gal Tunik explains the necessity of test automation in mobile applications, citing the unique challenges that that platform raises, including the intricate differences between mobile and desktop interfaces, and performance test results for applications as well as mobile networks. It is no surprise in this day and age of instant-on gratification that mobile devices are ...
I’ve spent the last six months or so testing mobile apps for both iOS and Android. Here’s eight of my key lessons learned: Automated UI testing tools for mobile apps are immature: whilst tools like WebDriver for automated UI testing of web apps are very mature, automated UI testing of native mobile apps is the ...
What you need to know to get started Introduction to mobile application testing Gone are the days when the telephone was an appliance that sat in a corner and had to ring to get our attention, or a computer was a machine only few people used — these devices are now an extension of our ...
Steps that will enable you to identify the weaknesses of your new app, its vulnerabilities and strengths. So you’ve just finished developing a nifty, customisable app that can help farmers track their produce from source to market via their mobile phone. You’re elated and want to get started marketing it right away. Not to burst ...
The outbreak of smartphones and tablets forces us to be digitally available with speed. Keeping pace with communication tool developments, Lindiwe Vinson defines the methods used at Organic, Inc. where she leads her team discovering bugs using various key programs for both PC and Mac platforms.
Users aren’t likely to forgive and forget buggy apps. Mobile has big implications for business. The mobile experience is the customer experience, and you don’t get many second chances.

Leave a Reply

Your email address will not be published. Required fields are marked *

Stay in the loop with the lastest
software testing news

Subscribe