2010 – 2011 LogiGear Global Testing Survey Results – Test Methods


M1. The test cases for your effort are based primarily on:

Response percentResponse count
Requirements documents61.3%46
Discussions with users on expected use2.7%2
Discussions with product, business analysts, and marketing representatives9.3%7
Technical documents4%3
Discussions with developers8%6
My experience and subject or technical expertise12%9
No pre-writing of test cases, I test against how the application is built once I see it2.7%2
Guess work0%0

Result analysis: This confirms conventional wisdom. Over 60% of the respondents list requirements documents as the basis of their test cases. This brings up two important discussion points: 1) test cases can only be as good as or as thorough as the requirements and 2) past survey results exhibit that most testers are hired because of their subject matter expertise. This subject matter expertise is the primary basis for test case development for a far-distant 12%.

Some test teams complain regularly about requirements documents they receive. I would assume that reliance on subject matter expertise would have been a more common basis for test case development.

M2. Are you executing any application-level security and/or usability testing?

Response percentResponse count
Yes for both42.1%32
Yes for usability23.7%18
No for both23.7%18
Yes for security10.5%8


M3. Are you conducting any performance and scalability testing?

Response percentResponse count


M4. Does your group normally test for memory leaks?

Response percentResponse count


M5. Does your group normally test for buffer overflows?

Response percentResponse count


M6. Does your group normally test for multi-threading issues?

Response percentResponse count

M7. Do you do any API testing?

Response percentResponse count

Result analysis M2- M7: As test engineers, you need to know what types of testing need to be executed against your system even if you are not the person or team executing the tests. For example, performance tests are often executed by a different team separate from those that execute functional tests. However, your knowledge of test design and data design will help those executing these tests.

Knowledge in other tests can also help cut redundancy and shorten a test cycle. Most importantly, it becomes a serious problem if you are not executing these tests because you don’t know how or hope someone else will take over. If you are not doing memory tests because you think or hope the developers are, this is also a mistake.

API testing can find bugs earlier in the development cycle and has easier defect isolation. If API testing is not being practiced, you should have a good reason.

M8. What percentage of testing is exploratory/ad hoc and not documented?

Response percentResponse count
10 – 33% (important part of our strategy)46.7%35
Less than 10% (very little)26.7%20
33 – 66% (very important, approximately half, more than half)13.3%10
66% – 99% (our most important test method)10.7%8
0% (none)2.7%2
100% (all our testing is exploratory)0%0

Result analysis: With almost half responding that 10 -33% of exploratory testing plays an important part of their strategy, my biggest concern is that the project team knows and understands your use of exploratory testing and the difficulty in measuring it.

The high percentage calling it important and the difficulty measuring exploratory testing often leads to incorrectly sizing a test project or increasing risk in cutting schedule time allotted for exploratory testing.

I expected a bigger reliance on exploratory testing with only over a quarter responded, “less than 10 %.” Most team teams say they find most of their bugs running exploratory tests as opposed to validation test cases. This may still be true, but many test teams may lack the time to do exploratory tests.

M9. How effective is your exploratory/ad hoc testing?

Response percentResponse count
Somewhat effective, it is useful54.8%40
Very effective, it is our best bug finding method39.7%29
Not effective, it is a waste of time/money testing5.5%4

Result analysis: Almost 40% said their exploratory/ad hoc testing very effective and the best bug finding method. I agree and hope you communicate to your teams how effective it is and your reliance on it.

M10. How is exploratory/ad hoc testing viewed by project management?

Response percentResponse count
Somewhat effective, it is useful58.6%41
Essential, necessary for a quality release20%14
Very effective, it is our best bug finding method11.4%8
Not effective, it is a waste of time/money testing10%7

Result analysis: Close to 60 % of respondents say management views the strategy as somewhat effective. In the previous questions, nearly the same percentage saw the testing as useful. This surprises me. Very often testers see ad hoc testing as more valuable than management who often see it as immeasurable and unmanageable.

M11. What is the goal of testing (test strategy) for your product?

Response percentResponse count
Validate requirements34.20%25
Find bugs26%19
Improve customer satisfaction23.30%17
Cut customer support costs/help desk calls8.20%6
Maximize code level coverage5.50%4
Improve usability1.40%1
Improve performance1.40%1
Comply with regulations0%0
Test component interoperability0%0

Result analysis: The number one answer for what is the goal of testing is validating requirements. This is a surprise. Typically, finding bugs is seen by testers as the goal and management, business analysts or marketing see validating requirements as the main goal and job of test teams.

Even with this, about half the respondents see finding bugs and improving customer satisfaction as the goal of testing. We see a few times in this survey a large reliance on requirements as the basis of testing. This can be a problem with anything less than great requirements!
M12. Which test methods and test design techniques does your group use in developing test cases? (You can choose more than one answer.)

Response percentResponse count
Requirements-based testing93.20%69
Regression testing78.40%58
Exploratory/ AdHoc testing68.90%51
Scenario-based testing56.80%42
Data driven testing40.50%30
Equivalent class partitioning and boundary value analysis27%20
Forced error testing25.70%19
Keyword/Action-based testing17.60%13
Path testing16.20%12
Cause-effect graphing12.20%9
Model-based testing10.80%8
Attack-based testing9.50%7

M13. How do you do regression testing?

Response percentResponse count

Result analysis: For 33% of respondents, regression testing is purely manual. I see this commonly in development teams. There are so many good test automation tools on the market today that can be used on more platforms that teams not automating their regression tests ought to re-examine test automation. For all testers, test automation is a core job skill.
M14. Is your test environment maintained and controlled so that it contributes to your test effort?

Response percentResponse count

M15. Are there testing or quality problems related to the environments with which you test?

Response percentResponse count

M16. Are there testing or quality problems related to the data with which you test?

Response percentResponse count

Result analysis M14- M16: Controlling test environments and test data is essential. Environments and data leading to testing problems is very common and very problematic! Building and maintaining great test environments and test data needs time and investment.

These answers confirm what I see in companies regularly─ problems in environments and data not getting resolved. With some time and perseverance, fixing these would greatly improve the effectiveness of testing and the trust of the test effort.

Michael Hackett
Michael is a co-founder of LogiGear Corporation, and has over two decades of experience in software engineering in banking, securities, healthcare and consumer electronics. Michael is a Certified Scrum Master and has co-authored two books on software testing. Testing Applications on the Web: Test Planning for Mobile and Internet-Based Systems (Wiley, 2nd ed. 2003), and Global Software Test Automation (Happy About Publishing, 2006). He is a founding member of the Board of Advisors at the University of California Berkeley Extension and has taught for the Certificate in Software Quality Engineering and Management at the University of California Santa Cruz Extension. As a member of IEEE, his training courses have brought Silicon Valley testing expertise to over 16 countries. Michael holds a Bachelor of Science in Engineering from Carnegie Mellon University.

The Related Post

As part of my on-going series on Agile for Testers – see this month’s article on People and Practices, I wanted to include the data I collected Agile development and testing and give you a chance to view them.
This survey takes an in-depth look at teams that practice DevOps and compares it to teams that don’t practice DevOps. For 2017, LogiGear is conducting a 4-part survey to assess the state of the software testing practice as it stands today. This is a 4-part series to mirror LogiGear Magazine’s issues this year.
Few people like to admit team dynamics and project politics will interfere with successful completion of a software development project. But the more projects you work on, the more you realize it’s very rare that technology problems get in the way. It’s always the people, project, planning, respect, communications issues that hurt development teams the ...
Process The objective of this survey and analysis is to gather information on the actual state-of-the-practice in software testing today. The questions originate from software development team assessments I executed over the years. A process assessment is an observation and questioning of how and what you and your team does.
In this installment of the 2010-2011 Global Testing Survey, we analyze the demographics of the more than 100 respondents from 14 countries. The next and final installment will analyze the “For Managers” section of the survey.
Michael Hackett looks at questions posed to managers in the final installment of our 2010-2011 Global Survey results.
TOOLS T1. What testing-support tools do you use? (Please check all that apply.) Response percent Response count Bug tracking/issue tracking/defect tracking 87.70% 64 Source control 54.80% 40 Automation tool interface (to manage and run, not write automated tests) 52.10% 38 Test case manager 50.70% 37 Change request/change management/change control system 47.90% 35 A full ALM ...
Data was compiled and analyzed by Michael Hackett, LogiGear Senior Vice President. This is the first analysis of the 2010 Global Testing Survey. More survey results will be included in subsequent magazine issues.
LogiGear strives to keep its finger on the pulse of the latest trends in Software Testing. During this defining moment in history, we want to hear from you about how your work has been impacted by the pandemic in this quick 5 minute survey.
Complete 2010 – 2011 Global Survey Results LogiGear Corporation LogiGear Corporation provides global solutions for software testing, and offers public and corporate software-testing training programs worldwide through LogiGear University. LogiGear is a leader in the integration of test automation, offshore resources and US project management for fast and cost-effective results. Since 1994, LogiGear has worked ...
This survey on modern test team staffing completes our four-part 2017 state of software testing survey series. We’ll have more results and the “CEO Secrets” survey responses to publish in 2018.
The target audience of the survey were black box testers. Please note that to these respondents, test automation is mainly about UI level automation, not unit, performance or load testing.

Leave a Reply

Your email address will not be published. Required fields are marked *

Stay in the loop with the lastest
software testing news