Software Testing 3.0: Case Study #1
Software Testing 3.0 is a strategic end-to-end framework for change based upon a strategy to drive testing activities, tool selection, and people development that finally delivers on the promise of software testing. For more details on the evolution of software testing and Software Testing 3.0 see:
- The Early Evolution of Software Testing
- Software Testing 3.0: Delivering on the Promise of Software Testing
Following is a case study of Software Testing 3.0 principals in action.
MX Logic Inc.:
Automation Expands Testing Coverage, Reduces Testing Time
|Environment Prior to LogiGear |
MX Logic is a leading provider of email and Web managed security services. The company was running a suite of 423 software tests manually to test their security software. Manual testing was so time consuming that:
- Manual testing of their software’s functionality was taking approximately 240 person-hours to run
- Build acceptance testing was done on an ad-hoc, time-permitting basis
LogiGear implemented an automated testing program for MX Logic that made use of:
- Low-cost off-shore LogiGear testing resources in Vietnam. These resources are all trained by LogiGear University in Software Testing 3.0 concepts and tools.
- TestArchitectT, a tool set that integrates the latest methodologies and technologies in one easy-to-use package.
The automated software testing solution implemented by LogiGear allowed the company:
- To automate existing tests and expanded test coverage to 945 automated test cases that run in 150 to 160 machine hours (the same tests run manually would consume 480 to 500 man hours).
- To expand testing coverage from 50% to 90%
- To use a subset of these tests to implement regular automated build acceptance testing that runs in 2 machine hours (the same would take 1 person day manually)
|“LogiGear has delivered an innovative turnkey solution that is fully automated, low cost and has expanded our testing coverage by 90%. |
While reviewing any test automation initiative it is important to understand that a successful automation project requires care and maintenance. Not only does LogiGear provide MX Logic with an excellent automation tool in TestArchitect, it provides me with the cost effective resources to enhance and maintain our automation projects. With a comparable feature set to more expensive licensed tools, my automation budget can be utilized on both the tool and the engineers to support it for the same price. And this guarantees the success of our automation, where others fail!”
– Jamie Tischart, Director,
Quality Assurance, MX Logic
Coverage expanded from 50% to 90%
Automated test coverage expanded from 10% to 80%
Twice as many tests tested in less time (150 machine hours vs. 240 person hours)
Added regular build-acceptance testing
Added production portal performance testing
- To reuse the automated test suite release after release with only the addition of a small number incremental test cases to test new functionality
- To avoid staff growth in their domestic operations
|Using the difference in cost between fully-burdened U.S. based resources and Vietnam based resources, LogiGear estimates that this level of automated testing effort done manually would have cost the company an additional $1 million or more. LogiGear’s automated software testing efforts have expanded testing coverage and effectiveness while avoiding significant additional expense.|
LogiGear’s automated software testing efforts expanded testing coverage and effectiveness while avoiding significant additional expense.
In addition, LogiGear created performance testing to help MX Logic monitor their customer experience on their production portal. They run this daily to ensure that their customer experience is not negatively affected by any new features or Internet traffic spikes.
LogiGear will be producing an additional 300 automated test cases to cover new functionality that MX Logic will be adding to their software. This will bring the total number of test cases to nearly four times the company’s original manual test suite.