Tracking Down a Defect Management Tool

How to select the right defect tracking solution for your business

By Hung Quoc Nguyen

C o n g r a t u l a t i o n s … you've been nominated by your manager or team to select a defect tracking system for your company. Where do you begin? Of course, you'll be expected to conduct your research while carrying out your regular responsibilities-so you need to come up with a quick strategy that enables you and all parties involved to objectively evaluate various options and select a system in a timely manner. You want to make sure that the tool you select is the best fit for your organization-a tool that offers the features you need, a reasonable response time at normal and even large load levels, a price that's within your budget, and integration with your organization's existing systems.

It's not a casual decision, because you know that once the tool is deployed it will be difficult and costly to change.

If you are doing this for the first time, it's a challenge to know where to begin and what to look for in an effective defect tracking and management solution. If you have done this before, you may find that several new commercially available defect tracking solutions offer many advanced feature sets.

Making the right choice requires a methodical planning and evaluation process. First, treat this assignment as a real project. As with any project, you need to know who the stakeholders are. You'll need to identify who has influence over your organization's defect tracking and management process, and the selection of the proposed tool. Ideally, one person will have ownership of all these issues. Realistically, however, several people (testers, QA staff, test managers, engineering managers, project managers, and even upper management) will have an interest in these requirements. Determine how you can best gain the consensus of these individuals. Ask yourself, too, who will make the ultimate buying decision. Even when the selection process is a group effort that's built on consensus, it's wise to know ahead of time who will sign the check and what their potential objections might be.

Who will be your system's users? While some of the users won't have direct input about the requirements, identifying and understanding who the users are helps assess the suitability of the products you're evaluating.

Consider what resources-people, time, and equipment-you'll require to complete the selection process. To collect meaningful assessments, you'll need to allocate some of these resources for hands-on product evaluation. You may need to schedule vendor visits or group presentations. Organizing such events can be a significant scheduling and coordination chore. And, as you would with other projects, factor in your deadline. Know when you have to make your recommendation.

The most important step in your process is defining the business and technical requirements of the system. Identifying these requirements allows you to generate a features list, which in turn will help you evaluate and trim down your list of tool options. In this article we'll discuss all the steps toward the selection of the most appropriate solution for your company. We'll begin with an examination of your organization's business and technical requirements for the tool.

Business Requirements

What business requirements will you need this tool to meet? That is, how is it going to support your organization in your software development effort? It's my belief that the main objective of a defect tracking solution is to track every identified issue so that valid defects that affect customer satisfaction, business financial success, and reputation can be fixed in a timely manner.

But a tool is never, by itself, going to be your defect tracking solution-for that, you need a defined and effective process. That is, you must understand your company's tracking process before you can identify a tool that will support it. Once you've ensured that a process is defined and agreed upon by all stakeholders, you can generate a list of features required to facilitate and manage the bug reporting and resolution cycle.

Taking a look at your company's defect management process with these considerations in mind will assist you in identifying your company's business requirements:

• people and productivity
• defect management and resolution process
• project management and metrics
• security
• administration

(And in the event that your company doesn't already have a defect management process in place, the following considerations will help you to establish one.)

People and Productivity Considerations

Let's start with your most unique resources- the people who will be using the tool you select, and their individual needs and productivity issues (see Table 1). Different staff members have varying responsibilities and will require specialized functionality of the defect management system. Internally, you have your product team-testers report issues, developers fix bugs, project managers review and monitor progress, and tech support engineers require access to deferred bug information. You want a system that can easily generate reports to support the decision-making process. Externally, you may have partners, customers, or beta testers-and since each group has different access needs for information, you'll have to grant different access privileges. Some users may access the system more frequently than others. Some users may be more technically savvy than others.

Consider what the important design factors are for your organization. Ease of use encourages involvement; the system should be easy to use and non-intrusive so that users can focus on their tasks rather than on fighting the system. Frequently used features should be accessible within one or two clicks.

To evaluate ease of use, you'll have to rely on more than reading the packaging; you won't find any tool that tells you it's difficult to use. Pay attention to how accessible the most frequently used features are. Also check out the number of navigation commands (such as mouse clicks) that are required to access a certain feature; the most frequently used queries should be easily accessible. That being said, the more advanced query features that enable the building of complex queries should also be readily available.

Defect Management and Resolution Process Considerations

You may require certain features in your defect tracking tool that enforce ownership and accountability of submitted reports. Otherwise, errors may go unfixed if someone forgets about them or tries to hide them.

You may require a notification mechanism, such as email notification, so that all responsible team members learn about defects in a timely manner.

Identify what decisions must be made during the bug reporting and resolution cycle (e.g., labeling issues as New, Fixed, To Be Fixed, and Not Reproducible). Identify who has the privilege to make certain decisions, and to redefine those resolutions. Consider who will be responsible for carrying out corrective actions for each resolution. You might want a tool that offers a resolution customizability feature.

At its most basic level, "workflow" is the path that a report follows through an organization, from initial submission to final resolution. It's a map of your bug's lifecycle. What are the various activities within the bug reporting and resolution cycle that make up the workflow in your organization? When reports are submitted or modified, where do they go, and who should be alerted? How will they be alerted? The tool you select will require features that enforce such workflow processes.

Consider what you'll want your issue reports to include. You may require report forms that allow for customized text entry fields. You may want some fields to be required while other fields remain optional.

Project Management and Metrics Considerations

Graphical reports, or "metrics," are invaluable for gaining perspective over a defect management project. More than likely, your organization will require some metrics feature for your projects. Generally, there are two types of metrics: distribution metrics and trend metrics. Distribution metrics present a snapshot in time, detailing how reports are presently distributed. Trend metrics track changes over a period of time.

Also, consider whether you'll need to track changes that reports undergo. Some tools maintain report histories, allowing project managers to track the history of metrics.

Security Considerations

You'll want to look for a tool that ensures that authorized individuals are able to submit issues while unauthorized individuals are denied access. The tool may require security controls so that access and data security are enforced at different levels. At the same time, you'll want the flexibility to grant different levels of access to different groups of users.

You'll need a tool that has the ability to allow everyone who is part of a project to submit reports against the project. This feature will likely increase the number of defect reports submitted against a project-meaning that fewer bugs will be missed.

Although you want as many people as possible to submit reports against a project, you probably won't want to grant everyone access to report querying and modification features. Staff who work with reports will require privileges for changing such attributes as resolution, priority, and severity. Those who don't have responsibilities with submitted reports shouldn't have access to these features.

You may want to grant specific external users (beta testers, for example) only limited privileges for checking the status of submitted reports-perhaps only the privilege to check reports they have submitted.

Administration Considerations

Who will be responsible for deploying, managing, and maintaining the system? What will the skill set of this person be? Will you need to hire such a person or do you already have the resource available? Normally there are two types of administrators: project administrators and network/database administrators.

If your organization will require remote administration, then Webbased administration features may be helpful.

Other Considerations

The number of people who will use the system may affect its price; the cost of many products is usually based on the number of supported users. And are those users in different geographical locations? Then you'll require a system that supports easy access for distributed team members.

Technical Requirements

Unlike business requirements, which focus on solutions to business problems, technical requirements focus on reliability, robustness, security, supportability, programmability, and scalability.

1. Reliability The system is stable and provides consistently accurate results.
2. Robustness The system is able to handle erroneous conditions without compromising results.
3. Programmability The system offers extensibility through exposing its features to other components or products through application programming interface (API).
4. Security The system's data is safe from both intruders and unauthorized users.
5. Supportability The system is maintainable at reasonable cost and convenience.
6. Scalability The system is adaptable to your company's growing needs.

In addition to examining these characteristics, you'll want to ask yourself a few questions about your specific environment. If the tool is a client-server system, which targeted client platform do you want the tool to support (e.g., Windows, UNIX, Linux, Macintosh)? If the client is to be browser based, which browser and respective browser versions will the tool need to support? If your company already has a back-end database solution (whether it's Microsoft SQL, Oracle, or something else), you'll want to make sure that the tool supports it. Depending on your existing server-side platform infrastructure, you may have specific preferences about the tool platform. If your organization will be interacting with the tool programmatically to submit and query reports through the user interface of another system, look for a tool that extends its features through a set of APIs

Generating a Feature List

Now that you've invested a lot of thought in the requirements for the system, it's time to document them as a "feature list." The prime objectives of the list are to enable you to (1) communicate with your potential vendors and team members about the targeted requirements, and (2) evaluate potential tools by grading their most important capabilities, feature by feature.

Many of the tools included on your initial list can be eliminated by reviewing the information we've already covered here. The tool candidates that you are still considering should be evaluated against carefully defined and measurable requirements. Note that measurable requirements are concrete enough that when evaluating a product you can clearly determine whether a requirement is "not supported," "works well," "works okay," or "barely works."

Table 2 is a simple example of a feature list that could be sent to potential vendors to let them know what you're looking for in your defect tracking solution. This example is not intended to suggest requirements; you will have to determine which requirements are most important for your own organization.

Once you've assembled your list of requirements, you'll need to prioritize them. Group the requirements in their order of importance, as shown in Table 3. For example, most important features are "must have" features; important features are features that you want, but may be willing to compromise on; the least important features are those features that would be nice to have but are not required.

Once you have a feature list, you can draft a request for proposal (RFP) and send it to prospective tool vendors. Your objective in this process should be to narrow the list of prospects down to the most promising systems that fit your needs-while avoiding the prospects that you can't afford or that don't have the features you require. This process will also provide you with enough information to refine your feature list. Build in adequate time for this part of the process; you'll want to allow about five business days to collect information from vendors.

Evaluating the Tools

There are several evaluation options you can use in your decision-making process. You can bring in sales people from the vendor to give you and your team a demo, or ask them for an online demo. If you feel daunted by the prospect of face-to-face sales pressure, you can download, install, and evaluate the products with your team members.

While your team's impressions are crucial, use other people's experiences to your advantage as well. Ask the vendor for references. Ask your peers for references. One way to compare notes is through forums such as the newsgroup comp.software. testing, in which you can ask your colleagues what they know about various defect tracking products, and determine how the products stack up side by side. Use the combination of options that makes the most sense for your situation, and that best enhances the success of your selection process.

In any case, it's crucial that you always try before you buy. If you don't try out a system before deploying it, you may make a costly mistake that will affect your organization for years to come. Allow enough uninterrupted time to evaluate from two to five of your most promising candidates. You'll need enough time to assess each vendor's product individually, and then time to compare the products.

Prior to the evaluation process, create several scripts of activities that can be executed against each candidate. User scenarios you might consider creating include:

• setting up the system
• submitting reports
• retrieving and working with submitted reports
• obtaining metrics and other reports
• creating a workflow to emulate your organization's process

Using a Weighted List to Grade Tools

Once you have prioritized requirements into categories of importance (Most Important, Important, and Least Important), you can assign weights to the requirements and include them in a weighted evaluation table such as the example shown in Table 4. Weighted tables offer a powerful means of grading and comparing products-feature to feature.

		Vendor A			Vendor B
Requirement	Weight	Grade	Points	Subtotals	Grade	Points	Subtotals
1.a	3		0		enter a grade value here	0 points = grade x weight
1.b	3		0			0
1.c	3		0			0
...	3		0			0
				0			0 subtotal for each group of requirements
2.a	2		0			0
2.b	2		0			0
...	2		0			0
				0			0
3.a	1		0			0
3.b	1		0			0
3.c	1		0			0
...	1		0			0
				0			0
Total Points				0			0 total points for each vendor

To use the weighted table, list your requirements in the left-hand column grouped by importance. Most Important ("must have") requirements are assigned a weighting of three; Important requirements are assigned a two; while Least Important requirements are assigned a weighting of one.

During your evaluation process, determine how well each vendor's product addresses your features list. Grades of effectiveness include High, Medium, Low, and No Support. Multiply each requirement's weight by the grade you've assigned it to calculate points, as shown in Table 5 on the following page.

Each requirement group's points are subtotaled, while total points for each vendor are also calculated. You may decide to make your buying decision solely on a vendor's total score, or you may decide to base your buying decision on how well your Most Important requirements are addressed, as shown in that group's subtotal.

Making the Final Decision

After you've completed your weighted list analysis, you're ready to make a final buying decision, based on the following criteria:

• Vendor's total score A vendor's total score indicates how well the vendor's product addresses your particular needs. From the information in Table 5, Vendor B, with an overall score of 55, appears to be the better option.

		Vendor A			Vendor B
Requirement	Weight	Grade	Points	Subtotals	Grade	Points	Subtotals
Workflow Support	3	3	9		3	9
Customizable Fields	3	2	6		1	3
Metric Reports	3	2	6		2	6
Ease of Use	3	3	9		1	3
				30			21
File Attachment	2	2	4		3	6
Web-Based Client	2	3	6		3	6
Audit Trail Support	2	3	6		3	6
Security Implementation	2	2	4		3	6
				20			24
Configuration Management Support	1	1	1		3	3
Report Cross- Referencing	1	0	0		3	3
Email Notification	1	2	2		3	3
Remote Administering	1	1	1		1	1
				4			10
Total Points				54			55

• Most important requirements subtotal The overall score may not be your primary objective. You may wish to make your buying decision based on how well your Most Important requirements are addressed. In Table 5, for example, Vendor A has a lower overall score, but a significantly higher subtotal than Vendor B in the Most Important requirements category-and is likely the better prospect.
• Short-term, intermediate, and long term needs Consider where your company will be a few years from now. How many people, projects, and reports do you envision the tool will have to support? Will the selected tool be able to scale with your company growth?
• Your budget or best-value proposition Clearly, you would not want to recommend a system with a price tag that exceeds your budget. On the other hand, the focus should not be on saving money at the expense of functionality and quality. The goal might be to identify your "best-value" solution. In calculating this value, don't forget that there are hidden costs that are often difficult to quantify, such as supporting, training, and productivity. Remember, too, that spending a little more in the near term can sometimes save you money in the long term.

To sum up, you have to know what you want in your defect tracking and management solution before you can effectively evaluate the available tools. The solution must work well within your organizational structure and process-factoring in both policies and people. The solution should support your business and team members-rather than vice versa.

It's essential that you are able to communicate your requirements in descriptive and measurable terms, and that you use metrics in your evaluation of various options to ensure objectivity in your assessment. Finally, if you do not find a solution that meets your needs, you still have two options:

• Use a public domain tool, such as GNATS. If your volume is small, you don't have money to spend, and you don't mind spending time setting the system up, this may be a workable solution for you.
• Build a system yourself. If done right, this approach assures you of getting a system that meets your needs. But this option can be expensive; it should be treated as a serious project with proper requirement analysis, design analysis, specifications, prototyping, implementation, and testing.

Regardless of which option you choose, be methodical in your selection process. The decisions you make will have a significant impact on your business. With thoughtful planning and systematic execution, the defect tracking and management solution you choose should prove effective for years to come.

About the Author

Hung Q. Nguyen is the President and CEO of LogiGear Corporation, a full-service consulting firm offering outsourced testing, QA training, and TRACKGEAR (a Web-based defect tracking solution). He is the author of Testing Applications on the Web and co-author of Testing Computer Software.

Other Articles by this Author

1. A Primer in Software Security Testing (Part 1)
2. A Primer in Software Security Testing (Part 2)
3. A Primer in Software Security Testing (Part 3)
4. Effective Management of Test Automation Failures
5. The Four Drivers for Delivering Automation Success
6. Leadership in Software Testing
7. Tests Built-to-Last: Software Tests as an Asset