An Interview with Computer Scientist – D. Richard Kuhn About Applications Development of Combinatorial Research

D. Richard KuhnComputer Scientist, National Institute of Standards & Technology

LogiGear: How did you become interested in developing applications for combinatorial research? What led you to it personally, and what did you find fascinating about it?

Mr. Kuhn: About 12 years ago Dolores Wallace and I were investigating causes of software failures in medical devices. About that time, Raghu Kacker in our math division introduced me to some work on the design of experiments and testing, which had been done by a colleague of his at Bell Labs. The idea behind these methods was that some failures only occur as a result of an interaction between components. For example, a system may correctly produce error messages when space is exhausted or when input rate exceeds some limit, but crashes only when these two conditions are both true at the same time. Pairwise testing has been used for a long time to catch this sort of problem.

I wondered if we could determine what proportion of the medical device failures were caused by interactions, and just how complex the interactions would be – in other words, how many failures were triggered by just one parameter value, and how many only happened when 2, 3, 4, etc. conditions were simultaneously true. We were surprised to find that no one had looked at this question empirically before. It turned out that all of the failures in the FDA reports appeared to involve four or fewer parameters. This was a very limited sample in one application domain, so I started looking at others and found a similar distribution. So far we have not seen a failure involving more than 6 parameters. This does not mean there aren’t any, but the evidence so far suggests that a small number of parameter values are involved in failures. In general, most problems are caused by one or two input values, fewer by an interaction among three, and still fewer at each step as we increase interactions above three.

What’s interesting about this research is that it suggests we can significantly improve the efficiency and the effectiveness of software testing – if we can test all 4-way to 6-way interactions, we are likely to catch almost all errors, although there are a lot of caveats to that statement and we don’t want to oversell this. In particular, selection of test values is critical unless a variable has a small set of discrete values, but this problem exists for all test methods. With the Automated Combinatorial Testing for Software (ACTS) project and tool, we hope to make these methods practical for real-world testing.

LogiGear: What has been your own personal contribution to this research and development? What sort of problems are you attracted to and how have you been able to solve them?

Mr. Kuhn: The series of papers that I did on the number of parameters involved in interaction failures has helped establish an empirical basis for the work and seems to have encouraged research on combinatorial testing by others. I also developed a parallel algorithm for generating combinatorial tests, although, frankly, there hasn’t been much need for it since Jeff Lei’s IPOG algorithm in the ACTS tool is so fast. More recently I’ve worked out a way to apply combinatorial methods to testing event sequences – instead of n! Tests for n events, the number of tests is proportional to log n. This method was motivated by interoperability testing, and it has already been used successfully.

I’ve always been fascinated by large collections of data – whether there might be some interesting or useful properties hiding in a mass of numbers. This is one area where that interest seems to have paid off. Another is some work I did showing that there’s a hierarchy of fault classes for boolean expressions.

LogiGear: What do you think is the current state of this development? You’re working on developing an open-source testing tool. How practical is it now? What successes have you experienced with it and what does it need? Also, how would you like its development to proceed in the future -would you like to see it take off and be developed as a sort of ‘Linux OS’ of combinatorial testing? What would you tell those who seek to get involved in this work?

Mr. Kuhn: The ACTS tool development is being led by Jeff Lei at U. of Texas Arlington, who has been an integral part of this project since the beginning. It’s very robust and has been acquired by several hundred organizations, and so far the feedback is very positive. It appears to be easier to use and more efficient than other tools out there, but we want to integrate lessons learned and make it better for real-world projects. In particular, the constraint handling feature is being strengthened. This is what you need, for example, to prevent the generation of tests that specify Internet Explorer on a Linux system. It already has this ability, but we are learning new needs from users all the time.

I like your suggestion that ACTS could be like Linux, in that Linux has become both a freely available tool and the basis for successful commercial products and services. It would be great if ACTS followed a similar path. In fact we are already seeing commercial extensions of it.

LogiGear: And finally, where do you see your own contributions and questions heading in the future? Where do you see this group – this “movement” if you will – headed?

Mr. Kuhn: Two things I really want to work on are integrating combinatorial testing with formal specifications and model based testing, and investigating the combinatorial coverage of tests developed with other methods. Both of these efforts should help to make this kind of testing more practical and cost effective. We have a good start on both, but we need better automation and tool support. The evidence so far suggests that we can make testing more efficient, but we need real-world data to prove it. I also want to understand the relationship between test value selection and combinatorial testing – are combinatorial methods more or less sensitive than other test methods to the choice of input values? But our primary goal is to collect and analyze data on real-world projects.

LogiGear: Thank you, Mr. Kuhn.

LogiGear Corporation   LogiGear Corporation provides global solutions for software testing, and offers public and corporate software-testing training programs worldwide through LogiGear University. LogiGear is a leader in the integration of test automation, offshore resources and US project management for fast and cost-effective results. Since 1994, LogiGear has worked with hundreds of companies from the Fortune 500 to early-stage startups, creating unique solutions to exactly meet their needs. With facilities in the US and Vietnam, LogiGear helps companies double their test coverage and improve software quality while reducing testing time and cutting costs. For more information, contact Joe Hughes + 01 650.572.1400
LogiGear Corporation
LogiGear Corporation provides global solutions for software testing, and offers public and corporate software testing training programs worldwide through LogiGear University. LogiGear is a leader in the integration of test automation, offshore resources and US project management for fast, cost-effective results. Since 1994, LogiGear has worked with Fortune 500 companies to early-stage start-ups in, creating unique solutions to meet their clients’ needs. With facilities in the US and Viet Nam, LogiGear helps companies double their test coverage and improve software quality while reducing testing time and cutting costs.

The Related Post

 A clear practice with the support of tools will help teams be more successful in delivering done products. It is always challenging to create a piece of a software system that fulfills a customer need, ready for use. Especially when it should be realized in an iteration of just three weeks, from an idea to ...
In today’s retail marketplace, digital channels hold a prominent and often central position. Modern software practices emphasize the need for rapid development and deployment. The retail world is no exception to this need for speed. Continuous Testing is a great strategy for getting increasing velocity.
The pervasive use of Agile methodologies has changed the way that development teams work. With this change, new tools and vendors have entered the traditional application life-cycle arena. A recent evaluation of this trend was published by Forrester Research, the “Agile Development Managements Tools Forrester Wave” report. Key observations of the report: Tools have changed ...
A tool empowering you to reach your automation successIn case you haven’t heard, earlier this month we launched TestArchitect Team, the new, free, test automation tool that empowers testers to achieve automation success.
What is Automated Test and Retest (ATRT)? IDT’s ATRT Test Manager addresses the complex testing challenges of mission critical systems by providing an innovative technical solution that solves the unique testing problems associated with this domain.It provides an integrated solution that can be applied across the entire testing lifecycle.
To adapt with the fast software delivery pace in today’s world, the same automation test suites should be able to run on a range of various platforms: different Operation System versions, different Application under test versions, different localization versions, etc. All while keeping the test development and maintenance effort at the minimum.
With the new release of Selenium late last fall, it was viewed as a saving grace to many test teams. The launch of the upgrade came with the bonuses and testing tools most had been waiting for, but unfortunately there was still a kink in Selenium’s armor: the Firefox browser.
Check out the top 12 Automation tools with pros and cons–like Cross-Operating Systems, Cross-Automation Platforms, Programming Language Support, and more – for desktop Automation Testing. Although the demand for desktop app testing is not growing as fast as mobile and web app testing, it’s still a crucial day-to-day duty for many testers, especially those who ...
Budget-friendly planning and management tools—plus a few others. I have been researching sources of test tools and other resources that are inexpensive or free. In some cases, you get what you pay for. However, it is possible to automate some or a lot of your testing using these tools. The good news is that the ...
Much has been written over the past few years about SMAC—Social, Mobile, Analytics and the Cloud—as the new platform with exponential growth. In this article we will answer the question: what is this and what does it mean to software testing? What is SMAC?
December Issue 2019: Tester Profile – Real World Stories

Leave a Reply

Your email address will not be published. Required fields are marked *

Stay in the loop with the lastest
software testing news

Subscribe