The Cure for Continuous Integration Testing

Do you want to speed up your automated tests by a factor of 10 and deploy your application continuously? In this article we share how the JIRA development team at Atlassian has accomplished this using Stages in Bamboo.

Stages have allowed the JIRA Development team to take a week’s worth of testing and condense it to one day.

Additionally, we have taken functional testing time from 8 hours to 40 minutes and use it as an “inner loop” continuous integration feedback build.

There are two primary feedback loops that the JIRA development team utilizes:

  • Generic “CI” Build – developers get feedback on their code changes quickly.
  • Tested Platform Matrix – feedback about how our release is validated against different types of distributions and the different environments they can run on, which include different operating systems and database configurations.

What Did We Do Before Bamboo Stages?

Running our tests in serial, the CI build would take 8 hours to run our functional test suite, which consisted of around 3000 tests. However, we only did this to deploy the JIRA artifacts to Maven because it is important to run all of our tests prior to deployment. This is key for JIRA artifact consumers to make sure they can pin point true integration failures instead of tracking down failures within JIRA. In addition to this build, in order to get faster feedback, we had 15 different build plans to run our functional tests but had no way to act upon those results. Finally, we also ran a unit test build as a fast CI loop that ran in 15 minutes, but only provided unit level testing. We had two builds for functional testing, one for validating deploying artifacts to Maven and the other to give us a faster feedback cycle with those tests. This meant we were potentially testing the same changeset twice and burdening our build hardware. For the Maven deployment functional test build that took 8 hours, we could deploy our artifacts once a day. Sometimes this deployment would be delayed two or three days if those tests stayed in a failed state. The Tested Platform Matrix build would take an entire week and might not pass within that week. For most, if not all development teams, those feedback times are unacceptable.

What are we trying to run that took 8 hours of serial execution?

  • 12,567 JUnit Tests
  • 3020 Functional Tests
  • 328 QUnit Javascript Tests
  • 639 Selenium UI Web Browser Tests

Staging a Revolution in JIRA’s Testing Cycles

By relying on an 8 hour development testing cycle and a week release testing cycle, we added higher risk in our ability to release JIRA on time. A lot of time was spent figuring out what commit broke what test, and sometimes even what day it happened. We needed help with running our tests in parallel and we needed to do it on a massive scale in order to shorten these long feedback loops. Bamboo stages allow us to run more than one build “job” within a stage and more than one stage in a build plan. The JIRA development team has done work to break the test suite into smaller parts, called batches, to launch each of these batches as a job within a Stage. Our functional testing Stage, for example, now includes 24 batches, or “jobs” that run simultaneously. Of course, we need somewhere to execute all of these jobs in parallel. This is where another important capability of Bamboo helps us out: the ability to run multiple remote agents on different types of hardware.

But the build pipeline is not done yet. Once our CI Build passes and “goes green”, what if we wanted to…

1. deploy these tested artifacts to Maven?
2. deploy our tested JIRA distribution to QA environments for manual testing?
3. build other JIRA Plugins so they can consume and validate against these tested artifacts?
4. run more tests, specifically our Tested Platforms Matrix?

We’d like to ensure that we can build and promote through these phases in this diagram going from the center left in development to release validation on the outside of the circles. The two primary loops are signified in yellow for development testing in our CI build and blue for release testing in the Tested Platforms Matrix and other important consumers of JIRA.

How Stages Help Us Reach Our Goals

Like many other development teams, the JIRA team wants to do all of these great tasks and accomplish these goals. The focus is to have our development CI loop as the producer bridge to launch into other outer loop CI activities. However, we can’t act until we can aggregate all of the jobs that run our functional tests into one result, to decide if we want to proceed further into greatness. This is another key feature in Bamboo Stages.

Previously there were 15 functional test build plans all running on their own after a dependent parent build triggered from commit detections. However, it was impossible to launch other build plans against the results of these 15 builds.

Because of this, we would not be able to deploy our tested Maven artifacts from those results. Now, with Stages, our CI build plan can not only have these same 15 jobs but we also get the ability to deploy the artifacts from that commit revision to Maven if and only if all of the functional test jobs pass. Additionally, Bamboo Stages also makes adding Jobs to each Stage an easy process. This is important because we plan on growing our functional test batches from our current set of 20 to 40 with a goal of having each job execute in less time.

Next we modified our other child build plans—that needed to consume JIRA artifacts—to depend upon the result of the CI build plan. For example, the final Stage in the CI plan is to deploy our JIRA artifacts to Maven only if all the previous Stages’ jobs passed, then build and validate important plugins like the JIRA Fisheye, JIRA Workflow and Activity Streams plugins. The final step was to build against the JIRA artifacts that were just deployed in the parent CI plan only if all functional tests passed. All this was done with the use of Bamboo Stages and additionally, another child plan of our CI plan, then deploy the JIRA Standalone distribution to a clean test remote environment for QA manual testing. The best news of all is that the CI build now executes all functional and unit tests on average in 40 minutes and we have ambitions to make this a 10 to 20 minute build in its entirety.

Our Tested Platforms Matrix is the second set of longer running tests that will drive us closer to release readiness. We run exactly the same functional tests in this set of builds. However, the difference here is we run against each distribution that we allow customers to download against most of our supported environments.

What does this mean?

We have:

  • 4 distributions – Standalone, War, Source, and a Windows Installer.
  • 3020 functional tests that were previously defined
  • MySQL, Postgres, MS SQL Server, and Oracle databases that, except SQL Server, run on two operating system platforms, Linux and Windows.

Adding up all of the cells in the Tested Platforms Matrix shows us that we execute 24,160 functional tests.

These 24,160 functional tests are executed against the same Maven artifacts that were deployed from any previously passing CI Build. We can use staging again to help organize the number of jobs launched against the matrix and also aggregate results.

Because of our ability to deploy JIRA artifacts through our development CI build, we can be assured our Tested Platforms Matrix builds run at least once a day. By using Stages, the execution of the matrix can help us complete the entire set of build plans in 5 hours. This allows us to have faster assurance from the Tested Platforms Matrix that we are ready promote what we created from a passing build in CI to make these same artifacts be release candidate artifacts for customer consumption.

Stages for the Win!

Bamboo Stages have been the ultimate win in condensing the amount of time to execute our functional tests in JIRA. It has given us the ability to not only help us launch our functional tests in parallel but also to importantly aggregate the results of those tests into a single pass fail result. This result has enabled us to then, with confidence, deploy and publish our JIRA artifacts for further testing in validation in JIRA and other products that depend on JIRA.

There are other elements to what I have shown you. We use Bamboo’s artifact passing to build once and re-use many times so we can pass the JIRA artifacts from one Stage to the next. This means that all of the functional test batches I described don’t have to checkout JIRA source and compile it. They can re-use the passed artifacts and give us the purity of testing against the exact same artifacts that were built in the compile Stage. When we started using this feature were able to trim even more time from both our CI build and our Tested Platforms Matrix –nearly 45 minutes!

Bryce
Bryce is a Lead Build Engineer at Decree in Sydney Australia. As a build engineer he is a quality gateway between development and QA/Release and provides services and processes to ensure that quality. He has been working in the software industry since 1996 and specializes in SVN, Mercurial, Bamboo, Maven2, Ant, Ivy, Java, Unix, Linux, Shell Scripting, Continuous Integration and Puppet. To read more from Bryce, visit his blog here: http://blogs.atlassian.com/author/bjohnson/

To keep up-to-date with the latest features released into Bamboo, as well as get tips for tricking out your CI pipeline, check out the Bamboo blog at http://blogs.atlassian.com/blog-cat/bamboo/.

You can also follow Atlassian Dev Tools on Twitter: @AtlDevTools. Happy building, everyone!

Bryce Johnson
Bryce is a Lead Build Engineer at Decree in Sydney Australia. As a build engineer he is a quality gateway between development and QA/Release and provides services and processes to ensure that quality. He has been working in the software industry since 1996 and specializes in SVN, Mercurial, Bamboo, Maven2, Ant, Ivy, Java, Unix, Linux, Shell Scripting, Continuous Integration and Puppet.

The Related Post

If you haven’t already caught a glimpse, we announced in January the Testing in DevOps and Continuous Testing videos series, available on YouTube!
Introduction Everything changes. It’s the only constant. The landscape of software testing is undergoing a fast and dramatic change driven by societal, business and technology changes that have placed software everywhere. Computing is ubiquitous. There is hardly anything today that doesn’t contain a CPU or information transmission capability, with software to drive it. From smart toasters ...
From adopting the culture, to implementing Continuous Delivery With the relative newness of DevOps, there are not yet a ton of DevOps books. That’s why we’ve assembled a list of the 7 best DevOps books based on four criteria: the number of ratings from Amazon, the average Amazon rating, number of ratings from GoodReads and the ...
Throw away clunky hyper-visors, and stop thinking about computer hardware and software license during your development projects. The first thing you think about when you hear “The Cloud” may not be development and testing. The Cloudy market is filled with SaaS applications, hosting, and cloud-based file systems. All are very useful, and offer a clear ...
DevOps has been described as Agile on Steroids; DevOps has also been described as Agile for Operations/IT. I like both of those descriptions. Many organizations want Development, Test, and Operations teams to move to DevOps now. DevOps is a big topic, but DevOps is not the focus of this article. We will not be talking ...
Making the leap to CT is easier than you think— follow this guide to transform your testing process No pain, no gain! Achieving Continuous Testing shouldn’t take a “Hans and Franz” attitude. It should be painless, more like a natural progression from implementing certain practices over time.
How Halliburton leveraged outsourcing to achieve their goals. Organizations are focusing on speed, both in Continuous Integration and rapid deployment as a competitive advantage. Many software development organizations can significantly shorten development cycles by implementing one or a combination of Agile practices, continuous integration & deployment methods, and feature branches. While these frameworks and techniques ...
Eric Minick brilliantly demonstrates how automation works in Continuous Delivery using this real-world “case study”. As a product manager at IBM working on our UrbanCode products, I think a lot about Continuous Delivery, how to do it and what it does for teams. Continuous Delivery techniques enable teams to deliver software faster and with less ...
How to Adopt the “Third Way” in the Dojo’s Method to Master CD In The DevOps Handbook: How to Create World-Class Agility, Reliability, and Security in Technology Organizations the authors describe “The Three Ways” – the underlying principles forming the basis for all DevOps practices. 
…On what you need to know before making the transition to EaaS 1. What are the main differences between cloud-based environments and cloud infrastructure? An environment is a collection of infrastructure elements working in conjunction to enable an application stack to work. For example, a simple 3-tier application, with a web front-end component, a business logic ...
  LogiGear_Magazine_June 2016_Testing in the New World of DevOps  
Times have changed, the tools have improved, and with books like this available you have no reason to not give CI a go. I still remember the first time I was on a project that used NAnt and CruiseControl.NET. It was years ago and both were new tools with plenty of bugs. The project manager ...

Leave a Reply

Your email address will not be published. Required fields are marked *

Stay in the loop with the lastest
software testing news

Subscribe