Spotlight Interview with Jonathan Kohl

Based in Alberta, Canada, Jonathan Kohl takes time out of his busy schedule to discuss his views on software testing and automation.

1. Can you tell us a little bit about your take on Exploratory Testing (ET)? What do you do differently than James Bach, for example?

I’m influenced by Cem Kaner and James Bach, and I have used a lot of their work as a base to build from. In my work, you’ll often hear me cite their prior work, but one of the interesting things with an exploratory approach is that it is dependent on the individual. That makes it hard to compare between one or the other tester talking about ET. Sure, the ideas are similar, and you’ll hear common themes from many of us, but the individual experiences and knowledge collected over time is very different.

Rather than try to compare, I will tell you what I am focusing on in my work. I’m passionate about bringing ET to the masses, by making it approachable:

  • I actively work with teams to help them apply ET or complement their scripted testing with it
  • I write articles addressing questions that come up the most in my public work, often providing alternate explanations for an ET approach
  • I try to explain concepts in ways that people can understand and apply in their own work
  • share experiences, tell stories and provide live testing demonstrations
  • I explore areas where scripted testing and ET can work together
  • I explore how I can use automation tools within my exploratory test work, as described in my Man and Machine article
  • I hate it when a tester feels stupid. I want ET to help empower them and help them feel energized and excited about their work
  • I try to make my ET training as real-world and approachable as possible, and I constantly work at improving it

I’ve done some work as a commentator: there are a lot of approaches to ET out there. Some of them are well thought out, thorough and cite prior work. Others, not so much. However, it is fascinating to see how there are different ideas and approaches, and personalities with different ideas.

I have done some work categorizing different approaches and strategies that I see. You have people like James Bach who is constantly coming up with different ideas and approaches and is often pushing the envelope. You have that with others now too, particularly the Thoughts from the Test Eye blog authors. Rikard Edgren, Henrik Emilsson, Martin Jansson are a great recent example. I call a lot of these ideas “abstract strategies.”

They spur testing thinkers on to get better and better and expand knowledge. At the other end of the spectrum, you have that unknown tester who just discovered how to apply a new idea to their testing, and pragmatically records it on their “cheatsheet” wiki page. “Copy this data and paste it in the app in various places” or something along those lines. That’s a very concrete idea or strateg, and different personality types reach for different strategies at different times. There are others – I introduced some of these ideas here:

I’m also slowly trying to develop productivity tools to help aid ET. Why should the programmers get the benefits from cool technology advances while we still use stone-age tools? It’s taking more time than I would like, but I hope I’m at least contributing ideas in that space.

I’ve tried to bridge the gap between the theory that we, who talk about ET often like to indulge in and the testers on the front-lines with their fingers on the keyboards. That has been surprising. For example, I have been working with others on a free open source tool to help record and structure session-based testing called Session Tester. I started out with a simple design for capturing session notes, and then I looked at the Session-Based Testing Management article by Jon and James Bach.

I’d tried out SBTM as described in the article several times in the past, but I had adapted it for different reasons. Some of it was too heavyweight for an Agile team, or some testers got hung up on the word “charter” and so on. I always kept the basics of SBTM sessions: a clear goal or charter, a time box, a reviewable result, a debrief with others, and the use of heuristics and other thinking tools to help with test idea generation during test sessions.
However, I tracked different things and recorded different things with different teams. When we started going through the design, we hit a crossroads: do we follow the SBTM document to the letter, or do we start basic and get feedback from real users on how they have implemented SBTM? We decided to go with the latter and let the community dictate what they wanted in the tool.

The results were fascinating. Once I got feedback on a half-baked tool pre-release, I found that there were many variations of SBTM out there. I had very few people chastise me for not supporting a “by the book” implementation of SBTM in the tool. Instead, I got a lot of encouragement to make the tool as flexible as possible to allow for variations.
I think this is great – people taking a tool, trying it, and adapting it to suit them. We have so much restrictive attitudes in software development: “do this, do that and only until you are deemed a master should you try to adapt” and it is refreshing to see what people are doing to adapt ET tools on their own without permission from a guru. Wonderful!

I find that space fascinating. There can be an amazing contrast between what people like me are talking about versus what is actually going on in the industry inside shops that we don’t have visibility into. I have to admit I don’t always like what I see, but if it is working for that team, then that’s really all that matters. They don’t have to enjoy ET at all the levels I do.

2. How do you run your sessions in ET? Are they 120 minutes long or do you have a different approach?

I follow the basics of SBTM sessions: a clear goal or charter, a time box, a reviewable result and most importantly, a debrief. How that looks, what we report on, or what terms we use depends on the team, and how I can get the concept working with that group of individuals. Often, just getting people comfortable with note-taking is the first hurdle. I’ve often replaced the term “charter” with “goal” or more often “mission.” Like “heuristic” the word “charter” can invoke a negative reaction in some people. To me it’s just a word, so let’s get past the word and get on to the concept and make it work.

The time box varies in length. I like to start people out with an hour with an appropriate setup and cool down period afterwards and work up from there. It is amazing how exhausting doing full concentrated testing work with note taking for an uninterrupted period of time can be. Once people get used to that, we ramp up or down depending on energy levels and other factors. I rarely go over 120 minutes in my own work, but I have had testing sessions in pairs or trios that went for much longer because we were experiencing flow, we were discovering all sorts of interesting information and we fed off each other’s energy.

3. How do you use heuristics in your ET process?

When I first started leading test projects, I found that my team was overwhelmed with paperwork stemming from test plans and test case management tools. I adjusted to try to meet the needs of various stakeholders in the team, and I thought: “our mission as testers is to provide important information, like bug reports to stakeholders. Hardly anyone reads our test plans, and no one else looks at our test cases. What are we in business to do? Find and report information quickly or maintain and create heaps of paper?”

I and the testers I worked with decided we should focus on testing and do as little documentation as required. We adapted quite naturally and organically, but we broke some of the rules in the QA department. People liked our results, but they were uncomfortable with how we got there. My manager at the time had me read Testing Computer Software by Kaner, Falk and Nguyen and suddenly I had words to describe what we were doing: exploratory testing. Cool! Now I had a term and respected people in the testing community to back up my approach. That was powerful.

As I moved on and began to lead another team, I had trouble training them. They didn’t have the benefit of the experience and gradual adjustments my other team had. I needed to get them up to speed quickly, so I started creating these “test philosophy” documents, and how to guides, bug taxonomies and so on to accompany our test checklists.

I was used to using whiteboards as communication devices, so I got the biggest one I could find and stuck it up outside my office to help guide and track our testing work. This was all quite crude and rudimentary, but it worked. I started digging into Cem’s work to see what other options were out there, and from Cem I discovered James Bach’s work, and suddenly I had a new word to describe what we were doing: heuristic.

I was quite familiar with the term from both my undergrad work in logic and philosophy and from having my Dad as a teacher in elementary school. He was big into using heuristics to solve word problems in mathematics. When I read James’ work on heuristics, I started using them to help us plan and structure our testing. It wasn’t until I met James in person and we tested software together that I saw a heuristic being used in real time. A big lightbulb went off over my head.

I can use these much more in real time as well as planning and everything in-between. Wow! How did I miss that? I know about heuristics and they worked great with those word problems in real time. In fact I was already doing much of that, but using James’ work as a springboard really kick-started me. Why reinvent the wheel when there is such great prior work that you can draw on?

So I use heuristics for several purposes. One is to help plan and structure our testing, and how to take a seemingly endless world of possibilities, and consciously optimize that work into something manageable and most likely to reveal important information. I also use them in real-time and I have a few of my own I like to pull out in different situations.

Heuristics are a memory aid and a possible way to solve a problem, so they help me in all sorts of situations. Anywhere from hearing a trigger when interviewing a team member about a design such as “persistence” – my heuristic is the question: “what gets persisted where?” to using something like San Francisco Depot in a live testing demonstration or during my own planning work.

4. Do you use Oracles? If yes, what was your experience in using the program?

Of course, that’s another word I got from James and Cem. It used to drive me crazy when I’d see a scripted test case with one expected result specified. My brain would immediately leap to different options and combinations. How ludicrous is it to just specify one result when there are lots of things we can use? How can I possibly pass this test case by looking at one tiny aspect? The work Cem and James described on oracles really resonated with me. Thankfully I wasn’t alone.

The oracle problem – finding out sources of information that let you know if your tests are passing or not, can be subtly difficult. My rule of thumb is to get a second source of information to verify things by. Sometimes that’s relatively simple: “There is a free government service that provides the currency exchange rates once a day that we can compare our calculations with.”

Sometimes it is complex, like working with data visualization techniques when doing performance testing. Why on earth are we seeing a spike in page load times every 20 minutes? That sort of oracle determination requires a combination of experience, knowledge and a willingness to research.

Some things that have surprised me:

  • Validating data sent from embedded devices in a test environment – discovering that the real thing worked far differently than a new design and tests let on. The (passing) tests were using mocked up data, and I decided to use the real thing to contrast with the data used in the tests. An engineer unlocked a storage space, pulled out some dusty devices and helped me get them going. I needed them to generate real data, so we played with a temperature probe, just to get some sort of data generated. To generate the first batch of real data, I grabbed a soldering iron and put a can of Coke in the freezer. Once the soda pop was sufficiently cold, I alternated touching the hot soldering iron and the cold pop can on the probe. Real data was generated, immediately uncovering real problems discovered in the system.
  • A tester’s gut feel, or intuition – faced with overwhelming evidence to the contrary, including passing tests, massive project inertia, great testers seem to be able to stand up and say: “Wait a minute. Something feels wrong. Give me a couple of hours to investigate.” Those are always surprising because they start with a suspicion or hunch, and investigation and a good investigative structure around their testing leads to real evidence.

Countless examples of the wrong oracle in a test case document. Either the document was out of date, or just plain wrong to begin with. Using simple means to come up with an independent check on that oracle revealed the test case was just wrong. Even a little creativity within test cases or using them as exploratory guidance references can reveal interesting information.

5. How do you find the best testers with the best domain knowledge? What are some of the ways you go about finding this information out?

I find out if someone has a testing aptitude by asking about their interests, and their experiences in the past. I worked with a test manager who had worked for many years in a restaurant, and I was able to map some of the lessons they had learned there to their success as a tester. When you personalize and validate prior experience that the individual tester may not associate with technology, interesting things happen.

I’ve had sports coaches, history researchers, accountants, short-order cooks, programmers, database administrators, systems administrators, and all sorts of roles excel in testing. Two of the best who come to mind were an executive assistant who was brought in to help test a new system part-time, and a technical support analyst. These were both people with superb analytical, reporting and investigative skills, and you’d never have known it without giving them real testing problems to solve.

6. Lastly, can you tell us about a recent tour that you’ve completed or done, and what you found in the way of bugs or problems using ET that you would not have found using traditional testing methods?

I feel that way about most of the past 13 years of doing exploratory testing. It’s hard to pick just one. Often, this occurs when there are intermittent bugs that people are struggling to reproduce. Hold on – one is coming to mind now …
Recently, I worked with a team that was transitioning from a highly clerical scripted testing approach to an exploratory testing approach.

They were in a regulated environment, so still required documentation. We had a transition period where we were using both systems – the emerging lightweight documentation system for ET, and the older scripted system. The test lead asked me to help get some coverage on some of the test cases that hadn’t been converted over to checklists and guidance docs yet, so I happily obliged. It’s fun to do something a little different once in a while.
One of the test cases felt wrong. There wasn’t a lot of information beyond: “go here, put in these inputs and these outputs should show up” and a note to use some tool to validate the results. I installed the tool, looked over the test case and while it was passing I was still confused. I tracked down the test case author and they were a bit taken aback.

The test didn’t make much sense to them either. So I moved on and talked to a programmer, the Product Owner and a subject matter expert. They all felt that the test was out of date and not valid anymore. I had a hunch that there was something to it. Why write the test case in the first place? Also, I can’t just mark a test as passed or failed without understanding it, so I decided to spend a few minutes exploring.

Now for the embarrassingly simple part of this story: I had no idea what this feature did, or what its purpose was, so I started off on a simple tour. What were all the functional aspects of this feature? I moved from area to area on the screen clicking, typing, moving the mouse around, but I still felt lost. So, in desperation, I used a “click frenzy” heuristic. The mouse started slowing down when I was in a certain region of the screen, so I honed in on that. All of a sudden: Bang! The application crashed. I went back to the programmer: “Is there any reason why clicking here over and over would cause a crash?”

That suddenly jogged his memory about that feature, and all these details from months ago rushed back. Now he knew what the test was about, and we brought in the Product Owner and Subject Matter Expert and started sharing knowledge. Of course, the feature had nothing to do with a click frenzy + crash, but that was enough of a primer to get the ideas and memories going. The scary part of all of this?

The test had been reported as passing for months, but the functionality had been broken all that time. That’s what happens when testers get bored of following the same test scripts over and over – they start to game the system instead of using their powerful brains.

Jonathan Kohl
Jonathan Kohl is an internationally-recognized consultant. As a thought leader in mobile application development, developing policy and strategy, and helping teams adjust to methodology changes, Jonathan doesn’t just write and talk about developing software, he actively helps teams deliver the best products they can. Jonathan has contributed to three books as a chapter author, has written over thirty articles for industry publications, and has presented at many conferences as a tutorial and workshop trainer, track talk and keynote speaker. He is the author of the book “Tap Into Mobile Application Testing”.

The Related Post

An automation framework is a way to organize your code in meaningful manner so that any person who is working with you can understand what each file contains. Automation frameworks differ based on how you organize your code – it can be organized based on your data, so that any person who wants to use ...
How to do UI test automation with the fewest headaches I’m currently interviewing lots of teams that have implemented acceptance testing for my new book. A majority of those interviewed so far have at some point shot themselves in the foot with UI test automation. After speaking to several people who are about to do ...
This article was developed from concepts in the book Global Software Test Automation: Discussion of Software Testing for Executives. Introduction There are many potential pitfalls to Manual Software Testing, including: Manual Testing is slow and costly. Manual tests do not scale well. Manual Testing is not consistent or repeatable. Lack of training. Testing is difficult ...
LogiGear Magazine September Test Automation Issue 2017
What is the Automation ROI ticker? The LogiGear Automation Return on Investment (ROI) ticker, the set of colored numbers that you see above the page, shows how much money we presumably save our customers over time by employing test automation as compared to doing those same tests manually, both at the design and execution level.
How lagging automotive design principles adversely affect final products. Cars are integrating more and more software with every model year. The ginormous screen introduced by Tesla in their flagship Model S a few years ago was seemingly unrivaled at the time. Nowadays, screens of this size are not only commonplace in vehicles such as the ...
Looking for a solution to test your voice apps across devices and platforms? Whether you’re new or experienced in testing voice apps such as Alexa skill or Google Home actions, this article will give you a holistic view of the challenges of executing software testing for voice-based apps. It also explores some of the basic ...
An Overview of Four Methods for Systematic Test Design Strategy Many people test, but few people use the well-known black-box and white-box test design techniques. The technique most used, however, seems to be testing randomly chosen valid values, followed by error guessing, exploratory testing and the like. Could it be that the more systematic test ...
We’ve scoured the internet to search for videos that provide a wealth of knowledge about Test Automation. We curated this short-list of videos that cover everything from the basics, to the more advanced, and why Test Automation should be part of part of any software development organization. Automation Testing Tutorial for Beginners This tutorial introduces ...
The following is a transcript of a May 7, 2008 interview with Hung Q. Nguyen, founder and CEO of LogiGear Corporation and coauthor of the best selling textbook Testing Computer Software. Interviewer: When it comes to software testing, what concerns or issues are you hearing from software developers? Hung Q. Nguyen: The most pressing concern ...
Developers of large data-intensive software often notice an interesting — though not surprising — phenomenon: When usage of an application jumps dramatically, components that have operated for months without trouble suddenly develop previously undetected errors. For example, the application may have been installed on a different OS-hardware-DBMS-networking platform, or newly added customers may have account ...
“Happy About Global Software Test Automation: A Discussion of Software Testing for Executives” Author: Hung Q. Nguyen, Michael Hackett, and Brent K. Whitlock Publisher: Happy About (August 1, 2006) Finally, a testing book for executives!, November 17, 2006 By Scott Barber “Chief Technologist, PerfTestPlus” Happy About Global Software Test Automation: A Discussion of Software Testing ...

Leave a Reply

Your email address will not be published. Required fields are marked *

Stay in the loop with the lastest
software testing news