The Cure for Continuous Integration Testing

Do you want to speed up your automated tests by a factor of 10 and deploy your application continuously? In this article we share how the JIRA development team at Atlassian has accomplished this using Stages in Bamboo.

Stages have allowed the JIRA Development team to take a week’s worth of testing and condense it to one day.

Additionally, we have taken functional testing time from 8 hours to 40 minutes and use it as an “inner loop” continuous integration feedback build.

There are two primary feedback loops that the JIRA development team utilizes:

  • Generic “CI” Build – developers get feedback on their code changes quickly.
  • Tested Platform Matrix – feedback about how our release is validated against different types of distributions and the different environments they can run on, which include different operating systems and database configurations.

What Did We Do Before Bamboo Stages?

Running our tests in serial, the CI build would take 8 hours to run our functional test suite, which consisted of around 3000 tests. However, we only did this to deploy the JIRA artifacts to Maven because it is important to run all of our tests prior to deployment. This is key for JIRA artifact consumers to make sure they can pin point true integration failures instead of tracking down failures within JIRA. In addition to this build, in order to get faster feedback, we had 15 different build plans to run our functional tests but had no way to act upon those results. Finally, we also ran a unit test build as a fast CI loop that ran in 15 minutes, but only provided unit level testing. We had two builds for functional testing, one for validating deploying artifacts to Maven and the other to give us a faster feedback cycle with those tests. This meant we were potentially testing the same changeset twice and burdening our build hardware. For the Maven deployment functional test build that took 8 hours, we could deploy our artifacts once a day. Sometimes this deployment would be delayed two or three days if those tests stayed in a failed state. The Tested Platform Matrix build would take an entire week and might not pass within that week. For most, if not all development teams, those feedback times are unacceptable.

What are we trying to run that took 8 hours of serial execution?

  • 12,567 JUnit Tests
  • 3020 Functional Tests
  • 328 QUnit Javascript Tests
  • 639 Selenium UI Web Browser Tests

Staging a Revolution in JIRA’s Testing Cycles

By relying on an 8 hour development testing cycle and a week release testing cycle, we added higher risk in our ability to release JIRA on time. A lot of time was spent figuring out what commit broke what test, and sometimes even what day it happened. We needed help with running our tests in parallel and we needed to do it on a massive scale in order to shorten these long feedback loops. Bamboo stages allow us to run more than one build “job” within a stage and more than one stage in a build plan. The JIRA development team has done work to break the test suite into smaller parts, called batches, to launch each of these batches as a job within a Stage. Our functional testing Stage, for example, now includes 24 batches, or “jobs” that run simultaneously. Of course, we need somewhere to execute all of these jobs in parallel. This is where another important capability of Bamboo helps us out: the ability to run multiple remote agents on different types of hardware.

But the build pipeline is not done yet. Once our CI Build passes and “goes green”, what if we wanted to…

1. deploy these tested artifacts to Maven?
2. deploy our tested JIRA distribution to QA environments for manual testing?
3. build other JIRA Plugins so they can consume and validate against these tested artifacts?
4. run more tests, specifically our Tested Platforms Matrix?

We’d like to ensure that we can build and promote through these phases in this diagram going from the center left in development to release validation on the outside of the circles. The two primary loops are signified in yellow for development testing in our CI build and blue for release testing in the Tested Platforms Matrix and other important consumers of JIRA.

How Stages Help Us Reach Our Goals

Like many other development teams, the JIRA team wants to do all of these great tasks and accomplish these goals. The focus is to have our development CI loop as the producer bridge to launch into other outer loop CI activities. However, we can’t act until we can aggregate all of the jobs that run our functional tests into one result, to decide if we want to proceed further into greatness. This is another key feature in Bamboo Stages.

Previously there were 15 functional test build plans all running on their own after a dependent parent build triggered from commit detections. However, it was impossible to launch other build plans against the results of these 15 builds.

Because of this, we would not be able to deploy our tested Maven artifacts from those results. Now, with Stages, our CI build plan can not only have these same 15 jobs but we also get the ability to deploy the artifacts from that commit revision to Maven if and only if all of the functional test jobs pass. Additionally, Bamboo Stages also makes adding Jobs to each Stage an easy process. This is important because we plan on growing our functional test batches from our current set of 20 to 40 with a goal of having each job execute in less time.

Next we modified our other child build plans—that needed to consume JIRA artifacts—to depend upon the result of the CI build plan. For example, the final Stage in the CI plan is to deploy our JIRA artifacts to Maven only if all the previous Stages’ jobs passed, then build and validate important plugins like the JIRA Fisheye, JIRA Workflow and Activity Streams plugins. The final step was to build against the JIRA artifacts that were just deployed in the parent CI plan only if all functional tests passed. All this was done with the use of Bamboo Stages and additionally, another child plan of our CI plan, then deploy the JIRA Standalone distribution to a clean test remote environment for QA manual testing. The best news of all is that the CI build now executes all functional and unit tests on average in 40 minutes and we have ambitions to make this a 10 to 20 minute build in its entirety.

Our Tested Platforms Matrix is the second set of longer running tests that will drive us closer to release readiness. We run exactly the same functional tests in this set of builds. However, the difference here is we run against each distribution that we allow customers to download against most of our supported environments.

What does this mean?

We have:

  • 4 distributions – Standalone, War, Source, and a Windows Installer.
  • 3020 functional tests that were previously defined
  • MySQL, Postgres, MS SQL Server, and Oracle databases that, except SQL Server, run on two operating system platforms, Linux and Windows.

Adding up all of the cells in the Tested Platforms Matrix shows us that we execute 24,160 functional tests.

These 24,160 functional tests are executed against the same Maven artifacts that were deployed from any previously passing CI Build. We can use staging again to help organize the number of jobs launched against the matrix and also aggregate results.

Because of our ability to deploy JIRA artifacts through our development CI build, we can be assured our Tested Platforms Matrix builds run at least once a day. By using Stages, the execution of the matrix can help us complete the entire set of build plans in 5 hours. This allows us to have faster assurance from the Tested Platforms Matrix that we are ready promote what we created from a passing build in CI to make these same artifacts be release candidate artifacts for customer consumption.

Stages for the Win!

Bamboo Stages have been the ultimate win in condensing the amount of time to execute our functional tests in JIRA. It has given us the ability to not only help us launch our functional tests in parallel but also to importantly aggregate the results of those tests into a single pass fail result. This result has enabled us to then, with confidence, deploy and publish our JIRA artifacts for further testing in validation in JIRA and other products that depend on JIRA.

There are other elements to what I have shown you. We use Bamboo’s artifact passing to build once and re-use many times so we can pass the JIRA artifacts from one Stage to the next. This means that all of the functional test batches I described don’t have to checkout JIRA source and compile it. They can re-use the passed artifacts and give us the purity of testing against the exact same artifacts that were built in the compile Stage. When we started using this feature were able to trim even more time from both our CI build and our Tested Platforms Matrix –nearly 45 minutes!

Bryce
Bryce is a Lead Build Engineer at Decree in Sydney Australia. As a build engineer he is a quality gateway between development and QA/Release and provides services and processes to ensure that quality. He has been working in the software industry since 1996 and specializes in SVN, Mercurial, Bamboo, Maven2, Ant, Ivy, Java, Unix, Linux, Shell Scripting, Continuous Integration and Puppet. To read more from Bryce, visit his blog here: http://blogs.atlassian.com/author/bjohnson/

To keep up-to-date with the latest features released into Bamboo, as well as get tips for tricking out your CI pipeline, check out the Bamboo blog at http://blogs.atlassian.com/blog-cat/bamboo/.

You can also follow Atlassian Dev Tools on Twitter: @AtlDevTools. Happy building, everyone!

Bryce Johnson
Bryce is a Lead Build Engineer at Decree in Sydney Australia. As a build engineer he is a quality gateway between development and QA/Release and provides services and processes to ensure that quality. He has been working in the software industry since 1996 and specializes in SVN, Mercurial, Bamboo, Maven2, Ant, Ivy, Java, Unix, Linux, Shell Scripting, Continuous Integration and Puppet.

The Related Post

The book is an incredibly effective and valuable guide that details the risks that arise when deploying cloud solutions. More importantly, it provides details on how to test cloud services, to ensure that the proposed cloud service will work as described. It is a great start to the topic. The 6 chapters detail a paradigm ...
Continuous Testing and Continuous Monitoring What is the goal of Continuous Integration? Is it to enable Continuous Delivery of the code developers’ produce out to users? Yes, eventually. But first and foremost it is to enable ongoing test and verification of the code. It is to validate that the code produced and integrated with that ...
LogiGear Magazine June Issue 2020: Transform Your SDLC With Continuous Testing
DevOps may be the next big buzzword, but Test teams really need to focus on its little sister, Continuous Delivery If you pay attention to trends in software development—from the perspective of what some sophisticated teams are doing, what articles and books are being written, to conference topics, you may have noticed the tools being ...
From adopting the culture, to implementing Continuous Delivery With the relative newness of DevOps, there are not yet a ton of DevOps books. That’s why we’ve assembled a list of the 7 best DevOps books based on four criteria: the number of ratings from Amazon, the average Amazon rating, number of ratings from GoodReads and the ...
It is a fundamental role for testing teams to align their test design, test automation, and test case development with DevOps–not only to verify that code changes work but that the changes do not break the product. A key differentiator of DevOps is testing maturity. An organization can automate their integration, testing, delivery, and monitor, ...
Introduction Everything changes. It’s the only constant. The landscape of software testing is undergoing a fast and dramatic change driven by societal, business and technology changes that have placed software everywhere. Computing is ubiquitous. There is hardly anything today that doesn’t contain a CPU or information transmission capability, with software to drive it. From smart toasters ...
  LogiGear_Magazine_June 2016_Testing in the New World of DevOps  
LogiGear University announces the launch of a new, free video series on Testing in DevOps and Continuous Testing which is available today.
Over the years we’ve provided an extensive number of articles, videos, and infographics that provide a wealth of knowledge about Continuous Delivery.
LogiGear Magazine – December 2013 – Cloud Testing
Aligning the Dev and Ops Teams DevOps as a philosophy has had as its centerpiece the principle that Dev and Ops teams need to align better. This is a people and organizational principle, not a process centric principle. To me this is more important when adopting DevOps than any other capability or tool. My last post ...

Leave a Reply

Your email address will not be published. Required fields are marked *

Stay in the loop with the lastest
software testing news

Subscribe