Options for Efficient Mobile Testing

By focusing on test design, analyzing test requirements and optimizing the approach to testing, it’s possible to maximize mobile test automation cost effectively.

In a previous article we outlined the importance of understanding the mobile ecosystem and test design for planning and executing mobile testing. The focus of this article is about efficient mobile test execution and test coverage maximization. 

Despite the growing number of mobile testing solutions the market is far from mature, which is one of the reasons why the majority of mobile device testing is still done manually. The primary issues with manual testing for mobile are scalability and cost. Outsourcing is always an option to increase the scale of testing, and then there is crowdsourcing. Crowdsourced mobile testing is being promoted as an ultra low-cost method of outsourced mobile testing. Crowdsourcing proponents tout flexibility, multilingual testing and a broad cross section of devices, but crowdsourcing is performed by a collection of freelancers so keep in mind the oft-used adverting disclaimer: “Results may vary.”

To adequately perform testing that assures mobile applications will meet customer expectations, test automation is necessary, and the need will only grow as mobile applications increase in complexity and importance. A high degree of test coverage can be achieved using both emulators and actual devices. How much testing to perform using emulators versus actual devices will depend on the budget, the automation tool, the application and the type of testing.

We have approached mobile test automation to make it possible for testers to employ a combination of testing assets—emulator/device/cloud testing resources—to efficiently achieve testing goals. Our automation tools provide the features and functionality to scale mobile test automation along with project needs and resources.

Emulators are ideal for testing new functionality or a new component design, and have some advantages over using actual devices. Logging faults and capturing screenshots are much simpler when working from a desktop, and you can simulate conditions that are hard to duplicate on real devices, like low battery power. A nice feature of TestArchitect is the ability to perform multilingual testing with emulators. This feature allows running identical tests in multiple languages in a short period of time.

Emulators can be accessed from the respective OS (Android, iOS, Windows Phone, Blackberry etc.) vendor’s SDK for free. Unlike iOS, Android isn’t the same for every device. For the best device approximation emulators from each Android device provider should be used. Then there are tools to augment the device emulators like Fiddler, NetLimiter and Charles Proxy that simulate GSM/GPRS networks to include testing of near real-world network conditions.

The main thing to keep in mind is that emulators and simulators only provide an approximation of device performance. From a QA perspective, emulators and simulators are fine for testing base functionality, but critical application sign-off should be done by live testing.

Testing a representative sample of actual devices is always ideal. The choice here is acquiring and updating an inventory of devices or using cloud base services. For overall maximum test coverage a public cloud option will provide broadest device selection. There are a number of cloud based options, each with their pros and cons. The big considerations are avoiding the need to recreate tests and security of the devices.

The accepted norm is to perform tests on 30 to 40 different devices currently available in the market. In cases where it’s essential to test actual devices, but less than the ideal number, applying Pareto’s Principle—80% of your customers will use just 20% of the devices available—should enable you to select the minimum number of devices that will provide the maximum test coverage.

Our approach to mobile cloud test automation was to create a host/agent set-up. Agents are downloaded onto the mobile devices (via wi-fi) to be tested, and tests are run remotely from the host machine. TestArchitect allows up to 40 devices to be tested simultaneously on a host or agent equipped machine to test within accepted norms. This approach eliminates API programming and provides the potential for regional testing—important for testing apps/devices on various operators’ networks, and in multiple languages—to validate applications for use on a global scale. The additional advantage of using TestArchitect is that tests used for any initial emulator testing can be run without modification on the actual devices, whether in the cloud or physically connected.

A well thought out combination of emulators, actual devices and cloud testing can provide broad test coverage and a high degree of automation. New applications can first be tested using emulators to test base functionality and/or multiple languages and identify major bugs. Next, targeted and thorough testing can be performed on actual devices.

Segregating testing in this manner can speed testing, and significantly help separate OS issues from device issues to minimize debugging time. Similarly, current applications can be tested on emulators of new or upgraded OS’s first, and then on a representative sample of next generation devices. Regression testing can be performed in the cloud across a wide range of devices with various hardware and OS versions.

Mobile devices are quickly becoming preferred to the desktop. To assure the best user experience, it is important to test mobile apps and websites with actual devices, and to keep this up requires automation. Mobile test automation is easily achievable. By focusing on test design, analyzing test requirements and optimizing the approach to testing, it’s possible to maximize mobile test automation and to do it cost effectively.

John Kane
John Kane oversees LogiGear’s Product Program Management to develop, productize and deliver LogiGear’s Testing Automation solutions to the marketplace. John has more than thirty years of experience in the software industry in the areas Electronic Design Automation (software for designing computer chips), Software Licensing, Chip Design, Government ARPA Programs, and Software Quality Consulting and Services. John Kane holds a Bachelor of Science in Computer Science from Oregon State University and a mini-MBA from Santa Clara University Leavey School of Business.
John Kane
John Kane oversees LogiGear’s Product Program Management to develop, productize and deliver LogiGear’s Testing Automation solutions to the marketplace. John has more than thirty years of experience in the software industry in the areas Electronic Design Automation (software for designing computer chips), Software Licensing, Chip Design, Government ARPA Programs, and Software Quality Consulting and Services. John Kane holds a Bachelor of Science in Computer Science from Oregon State University and a mini-MBA from Santa Clara University Leavey School of Business.

The Related Post

Don’t make the mistake of assuming too many similarities. It is common knowledge that mobile applications don’t function in the same way as their web-based counterparts. The user experience is affected by a few other factors such as device and network capability. If you are building out a performance testing strategy for your mobile website ...
Great mobile testing requires creativity to find problems that matter. I’d like to take you through the thought process of testers and discuss the types of things they consider when testing a mobile app. The intention here is to highlight their thought processes and to show the coverage and depth that testers often go to.
A sampling of some free, online, and easy-to-use mobile device emulators that can help get you started with testing. ScreenFly A free, customizable tool to test your website on any screen size, including desktops, tablets, televisions, and mobile phones.
To ensure the success of an app, QA must be involved in all stages of development. Quality Assurance (QA) plays a vital role in the development of mobile applications, but many overlook the critical nature of this piece of the app development process. To ensure the success of an app, QA must be involved in ...
Removing the barriers to move and better mobile testing. Over the last decade, application testing has continually proved itself to be an important concern. When done well, testing can drastically reduce the number of bugs that make it into your release code (and thus actually affect your users). In addition, good testing approaches will help your ...
Test engineers face a rapidly changing mobile application landscape, making mobile test automation a necessity. We know that mobile apps are becoming increasingly complex along with the technological advances in tablets and smartphones. Test engineers have to address multiple challenges while testing data-centric mobile apps, including the growing diversity of device features, platforms, and technologies. Fortunately, ...
To help testers gain an edge, here’s a list of free resources Mobile testing is making leaps and bounds of progress in the overall testing space. As this field is highly dynamic, a tester must constantly evolve and improvise his or her knowledge of mobile testing. To help software testers gain an edge, I have compiled the following list ...
Drawing from the Greek mythology of the lotus eaters, Anne-Marie Charrett warns testers to be weary of enjoying early success too soon upon finding high impact bugs.
In today’s mobile-first world, a good app is important, meaning an effective Mobile Testing strategy is  essential.  
Here’s some good news: Jonathan Kohl has a new book out titled Tap Into Mobile Application Testing. This is a Leanpub release and is still in Beta, but the beauty of buying through Leanpub – you get all the author updates as they make them! However I will be keeping an eye out for the print-on-demand ...
Strategies to Approach Mobile Web App Testing Mobile web technology has been continuously changing over the past few years, making “keeping up” challenging. In this article, Raj Subramanian covers the latest trends and changes happening in the mobile web and how testers can prepare for them.
The outbreak of smartphones and tablets forces us to be digitally available with speed. Keeping pace with communication tool developments, Lindiwe Vinson defines the methods used at Organic, Inc. where she leads her team discovering bugs using various key programs for both PC and Mac platforms.

Leave a Reply

Your email address will not be published. Required fields are marked *

Stay in the loop with the lastest
software testing news

Subscribe