Testing Computer Software Second Edition

Common Software Errors - Testing Errors

TESTING ERRORS

This section deals with technical, procedural, and reporting errors made by testers and Testing Groups. Even though these aren't problems in the programs per se, you'll run into them when testing programs. Our focus is on suggestions for dealing with these problems, since they're under your control.

MISSING BUGS IN THE PROGRAM

You will always miss bugs because you can't execute all possible tests. However, you'll probably miss more bugs than you have to. When a bug is discovered in the field or late in testing, ask why. Not to assign blame but to look for ways to strengthen your test procedures.

Failure to notice a problem

You may miss a bug that a test exposes because:

• You don't know what the correct test results are. Whenever possible, include expected results in the test notes. In automated tests, display them beside test results on the screen and on printouts.
• The error is buried in a massive printout. People scan long outputs quickly. Keep printouts as short as you can, and make errors obvious at a glance. Patterned outputs are good. If possible, redirect long outputs to a disk file; have the computer check this against a known good file.
• You don't expect to see it exposed by this test. While a test may be designed to focus on one small part of a program it can still reveal other, unexpected, bugs. Beware of tunnel vision.
• You are bored or inattentive. Rotate tasks across testers. Try not to have the same person run the same test more than three times.
• The mechanics of running the test are so complicated that you pay more attention to them than to the test outputs. You will be distracted by files or printouts with erroneous comparison data, poorly organized checklists, procedures that require you to swap disks or tapes frequently, and tasks that you have to redo from the start if you make an entry error.

Misreading the screen

You can easily miss errors like spelling mistakes, missing menu items, and misaligned text. Reserve some time exclusively for scrutinizing the screen. It's just like proofreading manuscripts: unless you're consciously looking for spelling and layout errors, you'll see what you expect to see, filling in gaps and correctly spelling mistakes unconsciously.

Failure to report a problem

You may find a problem and not report it because:

• You keep poor notes
• You're not sure if it's a bug and are afraid to look silly
• You think it's too minor or you don't think it will be fixed
• You're told not to report bugs like this any more

These are not acceptable reasons. If you're not sure whether something is a problem, say so in the report. Appeal to higher management to relieve criticism for reporting minor or politically inconvenient bugs. It is your responsibility to report every problem you find. Deliberate suppression of bug reports leads to confusion, poorer tester morale, and a poorer product. It can also bring you into the middle of nasty office politics, possibly as a scapegoat.

Failure to execute a planned test

You may not execute a planned test because:

• Your test materials or notes are disorganized. You've lost track of what has been tested.
• You are bored. The test series is repetitive. You take shortcuts by skipping tests that are similar to others. To reduce this, rotate tasks among testers. Reduce repetition by combining cases, cutting some out, or running some tests only on every second or third cycle of testing.
• You have combined too much into one test. If one test is buried inside another, or depends on another, then if that other test fails, this test probably won't be executed. Overly complex combinations of test cases can lead to missed tests because they confuse you.

Failure to use the most "promising" test cases

If two test cases cover essentially the same code, you should use the one most likely to reveal an error (see Chapter 6).

Ignoring programmers' suggestions

The programmer knows better than anyone else which areas of the program he tested least, and which ones proved least stable under his testing. He knows which areas he coded quickly. He knows which special types of tests have exposed bugs so far. Rigid test plans and bad politics are problems in their own right, but they are not excuses for ignoring programmers' tips.

FINDING "BUGS" THAT AREN'T IN THE PROGRAM

You report an error. Eventually, the problem is traced to a flaw in your test procedure, a misunderstanding of the program, or to something else that you did. This wastes time and does your credibility no good.

Errors in testing programs

When you automate tests, you write programs to drive test cases. Your test programs will have bugs. Some will abort your tests, or skip them. Others will make the program appear to fail tests that it can actually pass. It is common to compare test data against incorrect "known good" results. Your disk files and printed constants are no more likely correct than the program's output.

You should manually reproduce any automated tests that reveal errors. This doesn't take all that much time because you only redo tests that the program fails. Unless the program is in disastrously bad shape, it won't fail many tests.

Corrupted data file

Some apparent bugs are due to a bad data file that you're using while testing. Programs will trash input, output, and comparison files. Your files may be corrupted at any time, even by program segments that, if they were error-free, wouldn't read or write these files. When a program is in testing, it doesn't matter that it isn't supposed to touch a file. If the program worked the way it was supposed to, you wouldn't have to test it. It is wise to keep three backup copies of test files on separate disks or tapes. Before reporting an error, check your working copies of the input and comparison files against the backups.

Misinterpreted specifications or documentation

You think the program works incorrectly because you've misunderstood the documentation. This is unavoidable. Specifications are outdated, and early versions of the documentation are rough. You rarely have much time to read before starting to test the program.

When you find an error, unless you're sure you understand what's happening in this part of the program, reread the relevant sections of the documentation and specifications. If you're not sure whether what you've got is an error, write your report as a Query. If the manual's unclear, file a Problem Report on that part of the manual too.

POOR REPORTING

It's not enough to find a bug. You have to communicate it to someone who can fix it, in a way that makes it as easy as possible for that person to figure out what went wrong and what to do about it. How well you describe the problem will directly affect how easily it is resolved.

Illegible reports

If the programmer finds it hard to read a report, he will ignore it for as long as possible. Many reports are hard to read because you pack too much information into them. Put separable problems on separate report forms. If a single problem requires a long description, type it on a separate page and attach it to the Problem Report.

Failure to make it clear how to reproduce the problem

You report a problem without outlining, step by step, what the programmer must do to see it. This is the most common error in problem reporting. It saves time to skip the details, but realize that the first thing that the programmer is going to do with your report is sit at the machine and try to see the problem himself. If he can't reproduce the problem, he won't fix it.

For anything complicated, attach a copy of any data files you were using, a keystroke by keystroke list of things you did, a printed dump of the screen if your operating system supports this, or any other comments or materials that will make the programmer's job easier. The nearer you are to the development deadline, the more important this is.

Failure to say that you can't reproduce a problem

If you can't consistently reproduce the problem, say so. This tips off the conscientious programmer that he should try variations on the conditions you describe. A nonconscientious programmer might ignore your report as soon as she sees that you can't replicate it, but she'd toss away the report anyway after trying exactly what you say you did and, like you, failing to see the bug.

Failure to check your report

After writing a report, but before submitting it, follow it step by step to reproduce the problem. This costs a few moments but it catches transcription and other reporting errors that you make. It is all too easy to omit or misdescribe important details, especially if you're writing the report from notes or memory, long after seeing the bug.

Failure to report timing dependencies

You might not notice that to reproduce a bug you have to press two keys within milliseconds of each other, or that you have to wait at least 5 minutes between keystrokes. Sometimes you will just not realize that you're dealing with a race condition or other time-dependent bug. If you do notice a time dependence, say so. Clock it as well as you can. If you didn't notice a time dependence, look for one when a report comes back to you as irreproducible.

Failure to simplify conditions

You will often use complex test cases, combining many different tests into one, for speed of testing. If all goes well, you've gotten through many tests quickly. When a bug does show up, spend time looking for the simplest series of steps possible to reproduce it. Try not to lay out a long and complicated series that includes irrelevancies. Complex reports are disheartening to read and tempting to ignore.

Concentration on trivia

Don't make big issues over small problems. Don't get too far drawn into long arguments over wording, or style of presentation. Don't exaggerate the severity of bugs. Be wary of getting a reputation as a nitpicker.

Abusive language

If you refer to work as "unprofessional," "sloppy," or "incompetent," expect the programmer who did it to get angry. Don't bet that he'll fix the bug, even if it's serious. It can be useful to shock a programmer occasionally, but be conscious of what you're doing. Do it rarely (once a year).

POOR TRACKING OR FOLLOW-UP

It's not enough to just report a bug. You've got to make sure that it's noticed and not forgotten. Otherwise, bugs will "slip through the cracks" and make it into the shipping product.

Failure to provide summary reports

Don't assume that just because you gave it to a programmer, it's being dealt with. Some programmers lose reports. Others use them to make paper airplanes or wrap fish. Some also hide reports from their managers. Every week or two, you should circulate a brief description of unfixed bugs. Make this a standard procedure, for all bugs, to keep it impersonal and uncontroversial.

Failure to re-report serious bugs

If the bug is serious, don't automatically accept a response of Deferred or Works to Spec. Figure out a way to make it look a little worse and report it again. If it's an ugly, horrible bug, make it sound that way the second time. If that doesn't work, send a copy of the third report to a more senior manager.

Failure to verify fixes

A programmer reports that he fixed a problem. Don't take his word for it without retesting. Up to a third of the fixes either won't work or will cause other problems. Further, some programmers only address the exact, reported symptoms. Instead of investigating the causes of a problem, they write special-case code to handle the precise circumstances reported. If you skimp on regression testing, you will assuredly miss bugs.

Failure to check for unresolved problems just before release

Just before the product is released for use or sale, check for problems that are neither fixed nor deferred. It's good practice, which we recommend highly, to make sure that all Problem Reports are resolved one way or another before the product is released. At a minimum, make sure no one's sitting on anything serious. This is your last chance to remind people of serious bugs.