Beware of the Lotus Eaters: Exploratory Testing

Drawing from the Greek mythology of the lotus eaters, Anne-Marie Charrett warns testers to be weary of enjoying early success too soon upon finding high impact bugs.

 

Like many professions, the lure of the “low hanging fruit” in software testing is very appealing. In this case, fruit refers to finding high impact bugs quickly.
Exploratory Testing (ET) combined with Risk based Testing is a useful strategy for finding these bugs. Of course ET can deliver a whole lot more than that, but the opportunity to demonstrate an immediate impact is often a compelling reason to use Exploratory Testing.

Testers are not the only ones who like this fruit; it is also welcomed by the rest of the team, in particular project managers. Some bugs found early in the testing phase are, in general, welcomed by most team players.

Consequently, exploratory testers often tailor strategies to find and deliver these types of bugs. This means cheap and easy tests are run first. There is nothing wrong with this. It’s a good and effective strategy. There’s nothing like a few serious bugs at the start of testing to lull a manager out of a false sense of security.
It becomes a problem though if it becomes the only model used to perform testing. Testers eat the lotus flower of early success and are seduced and satiated with its bewitching nectar and stop testing. Any inner doubts are squashed with arguments such as “there is no such thing as 100% coverage” or “exhaustive testing is expensive and impractical, so why try?”

This is especially true when faced with complex and hard to understand software. It can be a big problem, because Exploratory Testing is tester centric—making the tester in control of how much testing to perform. In general, there are few team members who will object to the decision to stop testing and jump ship.
So, testing is declared as “Done Enough.” To have “Done Enough” testing is the poor man’s cousin to “Good Enough” Testing. “Done Enough” testing stops when the tester has had enough. Some bugs have been found and any more testing either requires imagination, technical expertise or some hard work on the tester’s part. “Done Enough” Testing is bad, folks. It reflects poorly on test teams and does little to educate the wider community on the benefits of Exploratory Testing.
Exploratory testers (well, any tester for that matter) need to be aware of the lotus flower and continue to test. And while testers may not be able to perform exhaustive testing, it shouldn’t preclude extensive testing.

Extensive testing goes beyond early success. Testers don’t dwell on the satisfaction of finding bugs early. They test beyond that. Project managers might be reasonably content (yes, they may have been seduced by the lotus flower, too) but testers need to wake up and keep testing until testing is “Good Enough.”
It’s more expensive to continue testing and it is harder to achieve at a reasonable price. This is where automation or tools come in. A good toolsmith can change where tools used effectively will reduce the cost of testing to allow more testing to take place. This results in a better standard of “Good Enough.”
I had a firsthand experience of the importance of tools in exploratory testing.

Eusebiu Blindu, a Czech tester, recently came up with an interesting online challenge. He created a testing exercise in finding bugs and patterns. The application drew a polygon with the number of sides determined by a number you entered into the field. You can see it here: http://www.testalways.com/2010/07/05/find-bugs-and-patterns/

There was no indication of the limit of numbers this field could take. I decided to test using a variety of numbers as testing every number seemed beyond my capabilities. I took the following numbers: 0 ~ 10, then I jumped to 15, tried a few random numbers to 99, 100, 999 and then 1000 and finally I threw in a couple of negative numbers to see what would happen. I noticed some relationships in the colors, a few irregularities, but beyond that nothing else stood out.

So what did I do? Did I read books to find out more? Did I seek outside knowledge, ask someone, or question. Did I try and change my model? Did I examine my assumptions?

No, instead I limited my testing. I put the puzzle aside. I decided I had “Done Enough”

Until I saw the following tweet:

@jamesmarcusbach “A good exercise for tools. RT @charrett: Eusebiu sets a challenge. Anyone care to answer the call?”

I was surprised; I didn’t think that this puzzle merited a tool. It was a polygon puzzle with only one input. What was the benefit from automating one field? Anyhow, how could you automate the testing of something like this?

I contacted James Bach on Skype and asked him what he meant by the tweet. We had a coaching session on the topic. Incidentally, James and I do a lot of IM coaching. In fact, we’re in the process of writing a book on the subject.

James explained to me he had tested 1200 inputs. I was impressed; regardless of the time it took to enter so many numbers, how could he remember the differences between each of the polygons?

It turned out his solution was to write a Perl script to automatically enter in a number, then take a snapshot of the result and save it in a directory. This was then followed by some deft blink testing. I followed his approach and very quickly, patterns and bugs became apparent.

I realised how limiting my approach to testing was in this scenario. When faced with a problem beyond my immediate capability, instead of figuring out how to fix the problem, I narrowed my model.

I allowed myself to be seduced by the deadly combination of early success and fear of complexity.

I reduced the scope because of the complexity of inputting large inputs. Rather than solve the problem of entering large amounts of inputs, I decided to test less. This was a major flaw in my testing strategy.

The lesson I learned was not to be limited by what seems to be impossible or hard. If it’s not possible to achieve (or if it’s too tedious to achieve) through manual means, then find another way of doing it. In this case automate it!

Testers need to be able to suspend belief of the impossible and put aside a skewed view of what can and cannot be done. They need to shine torches not only where others don’t want to look but also where we fear to tread

Beware the lotus eaters!


 

Anne-Marie Charrett

Anne-Marie Charrett is a test consultant and runs her own company Testing Times. An electronic engineer by trade, software testing chose her, when in 1990 she started conformance testing against European standards. She was hooked and has been testing since then. Anne-Marie blogs at http://mavericktester.com and her twitter id is @charrett .
Anne-Marie Charrett
Anne-Marie Charrett is a test consultant and runs her own company Testing Times. An electronic engineer by trade, software testing chose her, when in 1990 she started conformance testing against European standards. She was hooked and has been testing since then.

The Related Post

To help testers gain an edge, here’s a list of free resources Mobile testing is making leaps and bounds of progress in the overall testing space. As this field is highly dynamic, a tester must constantly evolve and improvise his or her knowledge of mobile testing. To help software testers gain an edge, I have compiled the following list ...
By focusing on test design, analyzing test requirements and optimizing the approach to testing, it’s possible to maximize mobile test automation cost effectively. In a previous article we outlined the importance of understanding the mobile ecosystem and test design for planning and executing mobile testing. The focus of this article is about efficient mobile test ...
A sampling of some free, online, and easy-to-use mobile device emulators that can help get you started with testing. ScreenFly A free, customizable tool to test your website on any screen size, including desktops, tablets, televisions, and mobile phones.
Mobile is no longer an area that a few UX people specialize in, and we need to start designing and testing everything for smartphones and tablets as well as computers. If you’re new to mobile usability testing, fear not. It is not as hard as you might think but there are some key differences from ...
I’ve spent the last six months or so testing mobile apps for both iOS and Android. Here’s eight of my key lessons learned: Automated UI testing tools for mobile apps are immature: whilst tools like WebDriver for automated UI testing of web apps are very mature, automated UI testing of native mobile apps is the ...
In today’s mobile-first world, a good app is important, meaning an effective Mobile Testing strategy is  essential.  
CEO and founder of mVerify Corporation, Robert V. Binder tackles questions from field testers regarding such issues as strategic considerations when dealing with single stack apps versus globalized enterprise mobile apps, and methods and tools that developers and testers should be aware of. He also offers his own advice from lessons learned from experience. 1. ...
Will testers be among the first IT professionals to shift their toolset and workflows from desktops and laptops to tablets and smartphones? As I’m sure you already know, a monumental shift from desktop to mobile is upon us. Not only have consumer applications started leaving the desktop behind, but B2B applications are also starting their ...
Here’s some good news: Jonathan Kohl has a new book out titled Tap Into Mobile Application Testing. This is a Leanpub release and is still in Beta, but the beauty of buying through Leanpub – you get all the author updates as they make them! However I will be keeping an eye out for the print-on-demand ...
To start with, we need a Test schedule. The same is created in the process of developing the Test plan. In this schedule, we have to estimate the time required for testing of the entire Data Warehouse system. There are different methodologies available to create a Test schedule. None of them are perfect because the ...
LogiGear Magazine December 2012 – Mobile Test Automation  
Strategies to Approach Mobile Web App Testing Mobile web technology has been continuously changing over the past few years, making “keeping up” challenging. In this article, Raj Subramanian covers the latest trends and changes happening in the mobile web and how testers can prepare for them.

Leave a Reply

Your email address will not be published. Required fields are marked *

Stay in the loop with the lastest
software testing news

Subscribe