Why Exploratory Testing Should Be Part of the Test Plan

Brian
Brian is a delivery-focused manager with a wealth of multi-disciplinary IT experience, the last fifteen years of which have been mainly in quality assurance and testing. He has been fortunate enough to work for some of the world’s largest companies using some of the latest technologies and tools.As well as being responsible for the day-to-day running of the business, Brian keeps a hand in the industry by being regularly involved in projects as an interim manager. He also runs The Naked Tester, a regularly updated blog. sharing curated and created content about testing and quality assurance and all other fun things in life.

A test plan should always include exploratory tests as part of the approach. Without them, the number of defects that find their way into production will always be higher.

Exploratory testing is a key testing technique that is often left out of formal test plan phases such as system testing, system integration, and regression. Instead, these phases favor planned, scripted tests that are easily repeatable and measurable.

While sometimes labelled as ‘ad-hoc’, and frowned upon in some circles because of the more unstructured nature, the truth is exploratory testing can be extremely fruitful in finding elusive bugs that may not otherwise have been discovered until user acceptance testing (UAT) or beyond.

There should therefore always be an allowance in the testing budget for conducting unplanned, exploratory tests – and the earlier the better.

Encourage Learning

According to the Wikipedia entry:

Exploratory testing … is concisely described as simultaneous learning, test design and test execution.

The key word is learning. One of the problems with scripted tests and script reuse is that the same functionality is covered, over and over again. This inhibits learning by narrowing the tester’s focus only to known areas, discouraging them from going off on their own and exploring other features or other ways of using the software.

People like to learn, and exploratory testing encourages that, making the process of testing more challenging and enjoyable, and leading to more productive testing.

“The real problem is not that the software hasn’t been tested, but that only key parts have been tested.”

Uncover Defects Earlier

A big problem with only running scripted tests is that they cover the same ground. While this may be fine for regression testing of core functionality, such script reuse does not fully test software changes in the way thorough exploratory testing can.

Relying only on scripted tests results in a larger than desired number of defects being missed during UAT.

As a tester, I have lost track of the number of times I have heard business users complain during UAT that “nobody has tested this”, when the real problem is not that the software hasn’t been tested, but that only key parts have been tested.

End-users are often very effective at finding bugs – in part this is probably due to their tendency to take a more exploratory approach as a result of their non-familiarity with the application.

Suggested Metrics for Exploratory Testing

Exploratory testing is sometimes left out of test plans because of uncertainty over what testing is being carried out, and therefore how to report progress. Many test managers focus on numerical progress reporting because of expectations from project and program managers who like to see a dashboard-style report that clearly shows the status of testing.

When there is no set number of planned exploratory tests to be carried out it can be difficult to show numerically whether or not you are on track, or whether you are achieving sufficient coverage of functionality.

One solution is to allocate a number of days for exploratory testing, and report a percentage complete based on the time elapsed. The level of coverage could be reported alongside that by completing a coverage matrix as tests are executed in defined areas of functionality.

Summary

However progress is reported, the key takeaway here is that a test plan should always include exploratory tests as part of the approach. Without them, the number of defects that find their way into UAT and production will always be higher.

By scheduling a number of days into the test plan for exploratory tests, more defects can be uncovered earlier in testing, sometimes dramatically improving the perceived quality of what gets handed over to the users.

Brian Heys
Brian is a delivery-focused manager with a wealth of multi-disciplinary IT experience, the last fifteen years of which have been mainly in quality assurance and testing. He has been fortunate enough to work for some of the world’s largest companies using some of the latest technologies and tools. As well as being responsible for the day-to-day running of the business, Brian keeps a hand in the industry by being regularly involved in projects as an interim manager. He also runs The Naked Tester, a regularly updated blog. Sharing curated and created content about testing and quality assurance and all other fun things in life.

The Related Post

A few months ago, Dr. Rebecca Fiedler and I published BBST—Test Design. This third course completes the Black Box Software Testing (BBST) set. The other two courses are BB ST Foundations and BBST Bug Advocacy. This article offers some information about the series, the design of the series (and the underlying instructional theory) and why you might be ...
Gondola by TestArchitect is a low-code Test Automation solution for End-to-End Testing across Web, API, and Mobile Applications. Gondola builds upon the TestArchitect family’s already-powerful testing capabilities in many ways, including faster mobile and web testing as well as heightened support for mobile application testing. LogiGear Product Manager Thuc Nguyen delves into why LogiGear created ...
Incorporate Exploratory Testing into your Test Strategy and find better bugs faster! Description: This two-day course is designed to give test engineers a global understanding of exploratory testing. From why we do it and its uses to how we do it and the value of measurement, Exploratory Testing will be examined and practiced to empower ...
According to Associate Press (February 9, 2010, Yuri Kageyama, AP Business Writer), “Toyota is recalling 437,000 Prius and other hybrid vehicles worldwide to fix brake problem.” Toyota is the world’s largest automaker with impeccable quality reputation up to now.
LogiGear Magazine September 2012 – Integrated Test Platforms
If you want to enjoy your job and not worry about lack of resources, or have old, outdated strategies, with failing or meaningless test automation – get help!   We all know about globalization. Markets are global, products are global, mobile is global and software development is a global. As a result, the workforce is ...
LogiGear Magazine – February 2011 – The Exploratory Testing Issue
Lab team brainstorming session Whether you work in engineering/product, operations, or even marketing, keeping your team trained and engaged with their work is a challenge that is universal for all managers. This is hard enough when your team is in-house, but what are you to do, when you have multiple teams to manage across different ...
Exploring key competences that endow a good games tester In this article, I will explore what I feel are the most important skills and attributes of a good game tester, and what type of mindset is needed for games testing.  I believe a good game tester should have. This is based on my experience and ...
Most software engineers intuitively perform BVA to some degree. By applying these guidelines, boundary testing will be more complete, thereby having a higher likelihood for error detection. Software is tested from two different perspectives: 1. Internal program logic is exercised using “white box” test case design techniques. 2. Software requirements are exercised using “black box” test case ...

Leave a Reply

Your email address will not be published. Required fields are marked *

Stay in the loop with the lastest
software testing news

Subscribe