8 Lessons Learned in Mobile App Testing

I’ve spent the last six months or so testing mobile apps for both iOS and Android. Here’s eight of my key lessons learned:

  1. Automated UI testing tools for mobile apps are immature: whilst tools like WebDriver for automated UI testing of web apps are very mature, automated UI testing of native mobile apps is the opposite. Whilst Appium shows some promise as a cross-platform UI testing tool for mobile apps, in my opinion, the maintenance overhead is still high as the tests are quite flaky. I recommend more thorough automated unit and integration testing, possibly supplemented with WebView tests (see below) and some exploratory manual app testing of the UI.

Münster, Germany - August 25, 2011: A close up of an Apple iPhone 4 screen showing various social media apps, including Skype, Google+, Facebook, Twitter, LinkedIn, Myspace, tumblr, foursquare, and Messenger.

  1. Base your mobile OS testing on projected usage, not current usage. Mobile hardware and OS upgrade cycles are shorter than desktop hardware, and if you look at trends and test against those, chances are you will have more realistic usage based testing.

This chart from mixpanel shows iOS7 adoption across a very large number of devices. You can see iOS6 is trending downwards as iOS7 is broadly adopted, so you could put less focus on testing iOS6.

  1. Test on real devices: I prefer to test on real devices like phones and tablets so that I can get a realistic feel for how the app runs. There’s also some things that can’t be tested on emulators/simulators like push notifications which require a device id, and Apple VoiceOver for accessibility testing which isn’t available in the simulators.

I have found that iPod Touch’s make excellent test iOS devices as they’re a lot cheaper than iPhones, and you can buy them refurbished on the Apple store, although if you want to test in the wild (see below), you’ll need a cellular iOS device. If you’re after an iOS6 specific test device you’re only option is to buy a 4th generation iPod touch refurbished as all new Apple iOS devices now come with iOS7 pre-installed.

It’s easy to find cheap Android pre-paid phones for testing, and you can still buy Android 2.3 phones with lower-res screens which is good for older version compatibility testing.

  1. Enable wireless distribution/installation of your test application locally: to enable quick installation of updates to your app on both iOS and Android.

For wireless iOS distribution this involves have a build machine (which has to be a Mac) compile an IPA file which is copied to a web-server, along with an XML manifest (PLIST) file that describes the app. You then have a simple web page with a itms-services link to the manifest, which can be opened in Safari on iOS, prompting the user to automatically install. If you sign your iOS app using an enterprise account it will run on an unlimited number of iOS devices without registering them as test devices. Full details are here.

For wireless Android distribution it is much easier. Simply have your build server compile an APK file and copy it to a web-server along with a page linking to the APK directly. Any Android phone set to allow installation of apps outside the Play Store will download the file and prompt to install it. Voila.

  1. Test in the wild: I like testing during my bus/train ride home to see how my app handles with intermittent cellular network coverage, especially in underground tunnels. If your backend services aren’t publically available you can use a VPN client on your phone to allow you to test against local infrastructure.

The one thing you need to be aware of is other people on your bus/train stickybeaking if your app is commercially sensitive.

  1. Test WebView/bridged app HTML content independently of your app: we use a service that returns HTML5 content that is displayed and executed using a JavaScript bridge in both iOS and Android native apps. This content can be tested independently of the native apps which is both faster and easier than testing via the apps themselves.
  1. Fake back-end dependencies for quicker UI testing. Back-end dependencies of your application such as a database can be faked so you can quickly test your app’s UI is functioning as designed initially without having to worry about setting up an entire application stack for your testing.
  1. Test for accessibility on real devices: using iOS VoiceOver & Android TalkBack. These are both excellent screen readers which use very similar gestures to control the operating system and apps on your phone. The most common accessibility issues I discover are missing accessibility labels for UI controls and embedded WebViews not being marked accessible to the screen reader tools.

Summary

There we have it: eight tips for testing mobile apps. What lessons have you learned in the mobile testing space?

 

Alister

Alister is an Excellence Wrangler for WordPress.com at Automatic. He has extensive experience in automated software testing and establishing quality engineering cultures in lean cross-functional software development teams. He lives in Brisbane, Australia with his wife and three sons, and writes a popular software testing blog at watirmelon.com.

Alister Scott
Alister is an Excellence Wrangler for Wordpress.com at Automatic. He has extensive experience in automated software testing and establishing quality engineering cultures in lean cross-functional software development teams. He lives in Brisbane, Australia with his wife and three sons, and writes a popular software testing blog at http://watirmelon.com.

The Related Post

This is the second part of a two part article that analyzes the impact of product development for the internet of things (IoT) on software testing.  Part one of this article (LogiGear Magazine, Sept 2014) gave a wide view on the IoT, embedded systems, and the device development aspects of testing on these projects. This ...
Devices matter. We don’t yet trust the mobile devices like we trust desktops and laptops. In the course of testing traditional web applications, rarely do you have to think about the model of the actual machine. In mobile, however, the behavior of an application can vary from device to device. You can no longer just ...
LogiGear Magazine – November 2011 – Mobile Application Testing Issue
To help testers gain an edge, here’s a list of free resources Mobile testing is making leaps and bounds of progress in the overall testing space. As this field is highly dynamic, a tester must constantly evolve and improvise his or her knowledge of mobile testing. To help software testers gain an edge, I have compiled the following list ...
Will testers be among the first IT professionals to shift their toolset and workflows from desktops and laptops to tablets and smartphones? As I’m sure you already know, a monumental shift from desktop to mobile is upon us. Not only have consumer applications started leaving the desktop behind, but B2B applications are also starting their ...
A sampling of some free, online, and easy-to-use mobile device emulators that can help get you started with testing. ScreenFly A free, customizable tool to test your website on any screen size, including desktops, tablets, televisions, and mobile phones.
The mobile application ecosystem is very dynamic. OEMs are launching new devices and new customization, and new OS versions are delivered every now and then. This is the constant challenge that most enterprises face.
The outbreak of smartphones and tablets forces us to be digitally available with speed. Keeping pace with communication tool developments, Lindiwe Vinson defines the methods used at Organic, Inc. where she leads her team discovering bugs using various key programs for both PC and Mac platforms.
Gal Tunik explains the necessity of test automation in mobile applications, citing the unique challenges that that platform raises, including the intricate differences between mobile and desktop interfaces, and performance test results for applications as well as mobile networks. It is no surprise in this day and age of instant-on gratification that mobile devices are ...
Organizations need to implement automated testing strategies designed specifically for mobile applications. Mobile device usage continues to gain momentum at increasing speed. Enterprises that delay mobile adoption face the danger of becoming competitively disadvantaged. But, before jumping in headlong, you need to be fully aware of the unique challenges that can arise when developing and implementing ...
A fully integrated test automation platform for mobile testing Mobile application testing is taking on greater significance as just about everything done on the desktop is transitioning to mobile. It’s taken a while for mobile testing in the consumer space to be made much of a priority. Now that sensitive data is being passed back ...
To start with, we need a Test schedule. The same is created in the process of developing the Test plan. In this schedule, we have to estimate the time required for testing of the entire Data Warehouse system. There are different methodologies available to create a Test schedule. None of them are perfect because the ...

Leave a Reply

Your email address will not be published. Required fields are marked *

Stay in the loop with the lastest
software testing news

Subscribe