Tools for Integrated Test Platforms

Regardless of your current state of tools, building an effective Continuous Integration suite of significant automated regression tests is the key to moving to a higher level of confidence in today’s development world.

In the evolution timeline of software development tools, new tools have recently proliferated. We have all been sold on collaboration, transparency and traceability – but that is not enough. Today, test tools are more often taking center stage. Integrated test platforms can push us deeper technically, help test faster, add more tools for our use and help us provide better, faster appraisals of the product under test.

In this article, I will discuss the introduction and integration of test tools into project wide tools, called ALM (application lifecycle management) and go into greater details about eXtreme Programming’s important Continuous Integration practice.

Remember: not too long ago, test teams had bug tracking tools. That was it. Then we added automation tools that pushed us to get test case managers. For years, that was where we stood. Now, imagine your user stories, test cases, automation, source control, unit test automation, API test automation, UI test automation and bug tracker all in the same tool. For many software development teams, this is already taken for granted. If you have not moved to this situation yet – get ready.

Team tools often get brought into a company where the main purpose is communication and transparency. Just as important to testers are – speed, ease, notification and access to information. Transparency and centralized communication are great for management and project progress. Using integrated test platforms, especially those inside ALM, give test teams access to information and should make your job easier.

Stand-alone tools no longer serve us in today’s fast-paced world. I have too much experience with frustrated test teams using a stand-alone tool like Quality Center where the requirements section is empty because the marketing team or business analysts would not use “the test tool.” And the bug data was piped out to Bugzilla or another tool because developers would not go into “the test tool.”

This won’t happen with the test tools in an ALM platform, for example, like TFS/Visual Studio, Rally or a custom built suite of tools like Jira, Bamboo, Bugzilla, nUnit, Subversion, etc. The tasks, storyboards, user stories, test cases, bugs, code repository and automation are all visible and accessible from one interface. The platform belongs to the product owner, designer, developer, tester, release engineer, database admin – everyone. The test team’s tool is everyone’s tool. Transparency and visibility are obvious, but there is no withholding of information due to “it’s not my tool” or lack of access.  It’s a big step forward.

ALM—Application Lifecycle Management: A Brief History

The main forerunner of this ALM, all-in-one, tool explosion is Rational Software’s suite, now Rational/IBM: Requisite Pro (requirements), Rational Rose (code design), ClearCase (source control), Rational Test manager, Rational Robot (test automation), and ClearQuest (bug tracking).  We now have a wide variety of tools – from enterprise wide tools to open source individual tools – so you can put all the pieces together to have the same impact.

The use of these tools has grown in the past couple of years as teams have become more distributed, faster and more dynamic while at the same time, coming under increased pressure from executives to be more measurable and manageable. When Agile, as a development framework, grew explosively, the anti-tool development framework got inundated by tools. Now every tool professes to support Agile practices and be Lean.

ALM tool suites can generally manage user stories/requirements, call center/support tickets and also help track a sprint’s planning game, store test cases associated with user stories, include source control, facilitate Continuous Integration, coverage analysis and track bugs.

There are many tools that do this, but the big leap forward with current ALM tools is the ability to be linked to source control, unit testing frameworks, and GUI test automation tools. Lite ALM tools satisfy the need for speed required by the rapid nature of software development – it’s a communication tool.

There are great reasons for using ALM tools, just as there are great reasons not to use them! Let’s start with great reasons to use them:

  • If you’re working on products with many interfaces to other teams and products, ALM tools can be very beneficial in communicating what your team is doing and where you are in the project.
  • ALM tools are essential if you’re working with distributed teams.
  • If you are working on projects with large product backlogs, or with dynamic backlogs often changing value, a management tool can be very helpful.
  • ALM tools are useful if you have teams offshore or separated by big time zone differences.
  • You are scaling Agile – for highly integrated teams, enterprise communication tools may be essential.
  • If, for whatever reason, the test team is still required to produce too many artifacts – like test cases, bug reports, build reports, automation result reports and status reports (ugh!) – most ALM tools will have all the information you need to report readily available.

Some reasons not to use them:

  • The Agile manifesto states people and their interactions are more important than processes and tools. Clearly, tools cannot dictate process – that is a mistake.  People and their interactions must be supported by tool use or do not use one.
  • Bad tool use is not Lean. Tools have to be easy, fast and not create waste.
  • The tools can be expensive or the open source ones have a high build/integrate/maintain cost. I have to say, if this is the problem, the cost to the team of not using an ALM will be greater.

Development and Test Tools

In the old days, developers had their IDEs (Integrated Development Environment). These tools have progressed to be more than text editors and compilers. They’ve grown into task tracking, build management, debuggers, coverage analyzers, syntax checkers, unit test harnesses, dependency analyzers, etc. These tools now make programming easier, give access to many useful built-in tools — some with a variety of libraries – lots of “stuff” to help programmers work faster, more efficiently and produce a better product faster. An IDE is now a unified platform for development, production and white box testing of code.

Testing tools developed more slowly. Test teams started to have access to sets of integrated tools in the mid-1990s mainly because teams began using multiple automation tools. The early integrated tools included a GUI/UI test tool, a performance test tool and perhaps an API test tool. Teams also needed a single bug database, maybe automatic bug creation, and, to reduce the maintenance headache, a unified interface for test case creation, execution and management. These tools tended to be “owned” by test teams. Rarely, if ever, were they used by other roles on the project team and there was no avenue to integrate the test team into the wider project.

The days of separate platforms are going the way of the dinosaur. It can be hugely beneficial for test teams to have easy access to source code, developer tools, coverage analyzers, debuggers and make their own builds as needed. It can also be convenient for developers to run test automation suites or execute test cases specifically linked to pieces of code. In addition to traceability for compliance or coverage measurement, ALM tools support tracing user stories to chunks of code to test cases to bugs. The benefits of linking or tracing items this way is a big benefit for targeted regression, gated builds and resolving issues. These are all the benefits that can be realized from a unified platform.

Continuous Integration- Not just autobuild!

The XP practice of Continuous Integration (CI) is revolutionizing software development. With the need for speed, rapid releases and very compressed development cycles, an automated build process is a no-brainer. This is not rocket science and not specific to Agile/XP. Continuous Integration tools have been around for years; there are many of them and they are very straightforward to use. It is also common for test teams to take over the build process. Implementing an automatic build process by itself is a step forward, but a team will realize more significant gains if they add to automated builds with Continuous Integration.

Continuous Integration includes the following:

  • Automated build process.
  • Re-running of unit tests.
  • Automated smoke test/build verification.
  • Automated regression test suite.
  • Build report.

The ability to have unit tests continually re-run has significant advantages:

  • It can help find integration bugs faster.
  • Qualifying builds faster will free-up tester time.
  • Testing on the latest and greatest build will save everyone time.

The positives for Continuous Integration far outweigh any resistance to their implementation. Test teams can take on more responsibility here: they can be more in control of the testing process — on how many builds they take and when they take them.

There’s no reason why a test team can’t take over making a build. There is also no reason a test team does not have someone already technically skilled enough to manage Continuous Integration. With straightforward Continuous Integration tools such as Bamboo, Cruise/CruiseControl, Git, Jenkins, Hudson—the list goes on— test teams can do it. Integrated platforms make this possible with easy access, easy notification, sharing information, linking requirements or stories to code, to test cases, to automation to bugs as needed.

I suggest you visit Wikipedia to see a very large list of Continuous Integration tools for integration into various platforms at this here.

Test teams can use Continuous Integration tools to make a very effective build validation process:

  • Scheduling new builds.
  • Automatic re-running of the unit tests. Note: this does not mean write the unit tests, it means re-run—for example, the JUnit harness of tests.
  • Re-running automated user story acceptance tests and whatever automated smoke tests are built.

Test teams taking over the Continuous Integration process with high volume automation is, from my experience, what separates consistently successful teams from teams that enjoy only occasional success.

Significant Automation: Automation is the rule, not optional

It’s a forgone fact that significant, sophisticated, yet simple automation is mandatory. If you do not have significant, high volume automation, you are doing the product team a disservice. All the integrated development/test platforms are including significantly more test tools. Many have GUI test recorders, limited performance tools and database test tools. To have these automation tools integrated into the test and development platform is essential.

High-volume test automation is the only path to rapid development and confident deployment. Making it happen can still be a challenge. It requires perseverance, support, cooperation, expertise and more! But at the same time, having an integrated test platform will make it more effective than ever.

From TestArchitect for Visual Studio, to Selenium and Eclipse integrated with Git or Hudson, there are a wide variety of possibilities for having all your test tools integrated into a development platform beneficial to both dev and test teams. Integrating testing tools with development tools and project wide tools allows test teams to provide the maximum benefit to the whole product team.

Project Management

The project management aspects of software development are facilitated by having integrated tools. For test teams, the obvious reasons are traceability, reporting, communication and transparency. The most important reasons are: speed, ease of work, being aware of other team members’ work, more automation, more technical testing, smarter testing, form more grey box knowledge, leveraging other team members’ work, knowledge and artifacts.


Stand-alone test tools, although beneficial, are not leveraging easy technologies we have available today. To keep pace with current software development practices, efficient tools to support rapid release and confident deployment are essential.

Having a unified test platform is the first step. Integrating the test platform with the development platform and the whole ALM is the most effective infrastructure for software development. Most importantly, regardless of your current state of tools, building an effective Continuous Integration suite of significant automated regression tests is the key to moving to a higher level of confidence in today’s development world.

Michael Hackett
Michael is a co-founder of LogiGear Corporation, and has over two decades of experience in software engineering in banking, securities, healthcare and consumer electronics. Michael is a Certified Scrum Master and has co-authored two books on software testing. Testing Applications on the Web: Test Planning for Mobile and Internet-Based Systems (Wiley, 2nd ed. 2003), and Global Software Test Automation (Happy About Publishing, 2006). He is a founding member of the Board of Advisors at the University of California Berkeley Extension and has taught for the Certificate in Software Quality Engineering and Management at the University of California Santa Cruz Extension. As a member of IEEE, his training courses have brought Silicon Valley testing expertise to over 16 countries. Michael holds a Bachelor of Science in Engineering from Carnegie Mellon University.

The Related Post

When Netflix decided to enter the Android ecosystem, we faced a daunting set of challenges: 1. We wanted to release rapidly (every 6-8 weeks). 2. There were hundreds of Android devices of different shapes, versions, capacities, and specifications which need to playback audio and video. 3. We wanted to keep the team small and happy. ...
The growing complexity of the Human-Machine Interface (HMI) in cars offers traditional testers an opportunity to capitalize on their strengths. The human-machine interface (HMI) is nothing new. Any user interface including a graphical user interface (GUI) falls under the category of human-machine interface. HMI is more commonly being used to mean a view into the ...
TestArchitect TM is the name we have given to our automation toolset. It reflects the vision that automated testing requires a well-designed architectural plan allowing technical and non-technical elements to work fluidly in their capacity. It also addresses the continual missing link of all test automation tools of how to design tests. In TestArchitect the test ...
This article was developed from concepts in the book Global Software Test Automation: A Discussion of Software Testing for Executives, by Hung Q. Nguyen, Michael Hacket and Brent K. Whitlock Introduction The top 5 pitfalls encountered by managers employing software Test Automation are: Uncertainty and lack of control Poor scalability and maintainability Low Test Automation ...
LogiGear Magazine – September 2010
Framework: An abstraction in which software providing generic functionality can be selectively changed by additional user written code, thus providing application specific software. A software framework is a universal, reusable software platform used to develop applications, products and solutions. Harness: A collection of software and test data configured to test a program unit by running it under varying conditions and monitoring ...
With the new year just around the corner, here’s a look at the Test Automation trends that have the potential to dominate. DevOps is being relied upon more than ever. With there being strong Market Drivers for the adoption of DevOps, the need for Test Automation has also never been greater. But what’s next after ...
Understanding the benefits and challenges of Automating ERP is critical. According to SAP, ERP (Enterprise Resource Planning) “is the core processes that are needed to run a company: finance, human resources, manufacturing, supply chain, services, procurement, and others. At its most basic level, ERP integrates these processes into a single system. But new ERP systems ...
In order to make the right choices among tools, you must be able to classify them. Otherwise, any choice would be at best haphazard. Without functioning classification, you would not be able to understand new tools fast, nor come up with ideas of using, or creating new tools.
The following is a transcript of a May 7, 2008 interview with Hung Q. Nguyen, founder and CEO of LogiGear Corporation and coauthor of the best selling textbook Testing Computer Software. Interviewer: When it comes to software testing, what concerns or issues are you hearing from software developers? Hung Q. Nguyen: The most pressing concern ...
LogiGear Magazine – The Big Testing Issue – April 2012
Utility: A program that performs a specific task related to the management of computer functions, resources, or files, as password protection, memory management, virus protection, and file compression. Tool: A program or application that software development teams use to create, debug, maintain, or otherwise support other programs and applications. The term usually refers to programs that can be combined together ...

Leave a Reply

Your email address will not be published. Required fields are marked *

Stay in the loop with the lastest
software testing news