September 2006 - Using Oracles in Testing and Test Automation (Part 1 of 3)
Douglas Hoffman, Software Quality Methods, LLCPart 1: Introduction Summary Testing requires more than just creating an exercise for the software under test (SUT). When we design a test we must identify what needs to be verified and a set of expected result values are needed in order to check the actual results. The mechanism for confirming whether or not the SUT behaves as expected is called an oracle (Note 1). In manual testing we may not be conscious of the need because we are able to recognize expected or unexpected outcomes. This is more difficult for automated tests since we must specify what to check and the expected results for comparison. The generation of expected results is often an obstacle for doing anything more than simple regression tests. We end up hard coding the expected values or comparing log files. This series of articles describes some different purposes and uses of oracles in automated software verification and validation that significantly expand what we may automatically test. Several types of oracles, some of their most relevant characteristics, the advantages, disadvantages, and implications for test automation are included. Real world oracles can vary widely in their characteristics. Although the mechanics of specific oracles may be vastly different, a few classes can be identified, and are categorized based upon the strategy for verification using the oracle. Thus, a verification strategy using a lookup table to compare expected and actual results can use the same type of oracle as one that uses an alternate algorithm implementation to compute them (a True Oracle). In both cases we have a source of trusted results and can simply compare actual with expected to determine whether the result is correct. The series of articles will cover the different types of oracles, with a particular emphasis on the oracles that enable automated tests to go much further than simple regression tests. Properly applied oracles allow automated tests to perform high volume tests (billions or trillions of actions), randomize data or track different paths each run, and a variety of other activities impractical or impossible to do manually. These are tests which can find errors a human cannot. The table below summarizes the oracles that future articles will cover: .abc td ul{ margin-left:0px; padding-left:18px; } .abc td ul li { padding-left:5px; margin-left:0px; }
Table 1: Types of Oracles In part 2 of this article series, Doug will discuss modeling testing with an oracle. In part 3 he will discuss characteristics of all oracles. About the Author Douglas Hoffman has over 30 years experience as a consultant, manager, and engineer in the computer and software industries. He has held numerous quality-related positions including the position of Vice President of Quality for a 500+ person company, as well as Manager of Software Quality Assurance for several other companies. He has been a registered ISO Lead Auditor, holds Certificates in Software Quality Engineering and Quality Management from ASQ and is an ASQ Fellow. He has also taught Computer Science at the college level at the University of San Francisco, UC Santa Cruz, and Howard University in Washington, DC. Douglas holds a BA in Computer Science, an MSEE, and an MBA. Copyright © 2006, Software Quality Methods, LLC. All rights reserved. Reprinted here with permission. Note 1: The term "oracle" is often used to describe the generation of expected results (data) or the comparison of actual with expected results. My use of the term combines the two. The reasons should become clear as the oracle mechanisms are described. |
