The Cure for Continuous Integration Testing

Do you want to speed up your automated tests by a factor of 10 and deploy your application continuously? In this article we share how the JIRA development team at Atlassian has accomplished this using Stages in Bamboo.

Stages have allowed the JIRA Development team to take a week’s worth of testing and condense it to one day.

Additionally, we have taken functional testing time from 8 hours to 40 minutes and use it as an “inner loop” continuous integration feedback build.

There are two primary feedback loops that the JIRA development team utilizes:

  • Generic “CI” Build – developers get feedback on their code changes quickly.
  • Tested Platform Matrix – feedback about how our release is validated against different types of distributions and the different environments they can run on, which include different operating systems and database configurations.

What Did We Do Before Bamboo Stages?

Running our tests in serial, the CI build would take 8 hours to run our functional test suite, which consisted of around 3000 tests. However, we only did this to deploy the JIRA artifacts to Maven because it is important to run all of our tests prior to deployment. This is key for JIRA artifact consumers to make sure they can pin point true integration failures instead of tracking down failures within JIRA. In addition to this build, in order to get faster feedback, we had 15 different build plans to run our functional tests but had no way to act upon those results. Finally, we also ran a unit test build as a fast CI loop that ran in 15 minutes, but only provided unit level testing. We had two builds for functional testing, one for validating deploying artifacts to Maven and the other to give us a faster feedback cycle with those tests. This meant we were potentially testing the same changeset twice and burdening our build hardware. For the Maven deployment functional test build that took 8 hours, we could deploy our artifacts once a day. Sometimes this deployment would be delayed two or three days if those tests stayed in a failed state. The Tested Platform Matrix build would take an entire week and might not pass within that week. For most, if not all development teams, those feedback times are unacceptable.

What are we trying to run that took 8 hours of serial execution?

  • 12,567 JUnit Tests
  • 3020 Functional Tests
  • 328 QUnit Javascript Tests
  • 639 Selenium UI Web Browser Tests

Staging a Revolution in JIRA’s Testing Cycles

By relying on an 8 hour development testing cycle and a week release testing cycle, we added higher risk in our ability to release JIRA on time. A lot of time was spent figuring out what commit broke what test, and sometimes even what day it happened. We needed help with running our tests in parallel and we needed to do it on a massive scale in order to shorten these long feedback loops. Bamboo stages allow us to run more than one build “job” within a stage and more than one stage in a build plan. The JIRA development team has done work to break the test suite into smaller parts, called batches, to launch each of these batches as a job within a Stage. Our functional testing Stage, for example, now includes 24 batches, or “jobs” that run simultaneously. Of course, we need somewhere to execute all of these jobs in parallel. This is where another important capability of Bamboo helps us out: the ability to run multiple remote agents on different types of hardware.

But the build pipeline is not done yet. Once our CI Build passes and “goes green”, what if we wanted to…

1. deploy these tested artifacts to Maven?
2. deploy our tested JIRA distribution to QA environments for manual testing?
3. build other JIRA Plugins so they can consume and validate against these tested artifacts?
4. run more tests, specifically our Tested Platforms Matrix?

We’d like to ensure that we can build and promote through these phases in this diagram going from the center left in development to release validation on the outside of the circles. The two primary loops are signified in yellow for development testing in our CI build and blue for release testing in the Tested Platforms Matrix and other important consumers of JIRA.

How Stages Help Us Reach Our Goals

Like many other development teams, the JIRA team wants to do all of these great tasks and accomplish these goals. The focus is to have our development CI loop as the producer bridge to launch into other outer loop CI activities. However, we can’t act until we can aggregate all of the jobs that run our functional tests into one result, to decide if we want to proceed further into greatness. This is another key feature in Bamboo Stages.

Previously there were 15 functional test build plans all running on their own after a dependent parent build triggered from commit detections. However, it was impossible to launch other build plans against the results of these 15 builds.

Because of this, we would not be able to deploy our tested Maven artifacts from those results. Now, with Stages, our CI build plan can not only have these same 15 jobs but we also get the ability to deploy the artifacts from that commit revision to Maven if and only if all of the functional test jobs pass. Additionally, Bamboo Stages also makes adding Jobs to each Stage an easy process. This is important because we plan on growing our functional test batches from our current set of 20 to 40 with a goal of having each job execute in less time.

Next we modified our other child build plans—that needed to consume JIRA artifacts—to depend upon the result of the CI build plan. For example, the final Stage in the CI plan is to deploy our JIRA artifacts to Maven only if all the previous Stages’ jobs passed, then build and validate important plugins like the JIRA Fisheye, JIRA Workflow and Activity Streams plugins. The final step was to build against the JIRA artifacts that were just deployed in the parent CI plan only if all functional tests passed. All this was done with the use of Bamboo Stages and additionally, another child plan of our CI plan, then deploy the JIRA Standalone distribution to a clean test remote environment for QA manual testing. The best news of all is that the CI build now executes all functional and unit tests on average in 40 minutes and we have ambitions to make this a 10 to 20 minute build in its entirety.

Our Tested Platforms Matrix is the second set of longer running tests that will drive us closer to release readiness. We run exactly the same functional tests in this set of builds. However, the difference here is we run against each distribution that we allow customers to download against most of our supported environments.

What does this mean?

We have:

  • 4 distributions – Standalone, War, Source, and a Windows Installer.
  • 3020 functional tests that were previously defined
  • MySQL, Postgres, MS SQL Server, and Oracle databases that, except SQL Server, run on two operating system platforms, Linux and Windows.

Adding up all of the cells in the Tested Platforms Matrix shows us that we execute 24,160 functional tests.

These 24,160 functional tests are executed against the same Maven artifacts that were deployed from any previously passing CI Build. We can use staging again to help organize the number of jobs launched against the matrix and also aggregate results.

Because of our ability to deploy JIRA artifacts through our development CI build, we can be assured our Tested Platforms Matrix builds run at least once a day. By using Stages, the execution of the matrix can help us complete the entire set of build plans in 5 hours. This allows us to have faster assurance from the Tested Platforms Matrix that we are ready promote what we created from a passing build in CI to make these same artifacts be release candidate artifacts for customer consumption.

Stages for the Win!

Bamboo Stages have been the ultimate win in condensing the amount of time to execute our functional tests in JIRA. It has given us the ability to not only help us launch our functional tests in parallel but also to importantly aggregate the results of those tests into a single pass fail result. This result has enabled us to then, with confidence, deploy and publish our JIRA artifacts for further testing in validation in JIRA and other products that depend on JIRA.

There are other elements to what I have shown you. We use Bamboo’s artifact passing to build once and re-use many times so we can pass the JIRA artifacts from one Stage to the next. This means that all of the functional test batches I described don’t have to checkout JIRA source and compile it. They can re-use the passed artifacts and give us the purity of testing against the exact same artifacts that were built in the compile Stage. When we started using this feature were able to trim even more time from both our CI build and our Tested Platforms Matrix –nearly 45 minutes!

Bryce is a Lead Build Engineer at Decree in Sydney Australia. As a build engineer he is a quality gateway between development and QA/Release and provides services and processes to ensure that quality. He has been working in the software industry since 1996 and specializes in SVN, Mercurial, Bamboo, Maven2, Ant, Ivy, Java, Unix, Linux, Shell Scripting, Continuous Integration and Puppet. To read more from Bryce, visit his blog here:

To keep up-to-date with the latest features released into Bamboo, as well as get tips for tricking out your CI pipeline, check out the Bamboo blog at

You can also follow Atlassian Dev Tools on Twitter: @AtlDevTools. Happy building, everyone!

Bryce Johnson
Bryce is a Lead Build Engineer at Decree in Sydney Australia. As a build engineer he is a quality gateway between development and QA/Release and provides services and processes to ensure that quality. He has been working in the software industry since 1996 and specializes in SVN, Mercurial, Bamboo, Maven2, Ant, Ivy, Java, Unix, Linux, Shell Scripting, Continuous Integration and Puppet.

The Related Post

  LogiGear_Magazine_June 2016_Testing in the New World of DevOps  
…On what you need to know before making the transition to EaaS 1. What are the main differences between cloud-based environments and cloud infrastructure? An environment is a collection of infrastructure elements working in conjunction to enable an application stack to work. For example, a simple 3-tier application, with a web front-end component, a business logic ...
    Eric Minick is internationally recognized as a leading authority on continuous delivery and DevOps. Eric joined IBM four years ago with the acquisition of UrbanCode where he had worked as a developer, technical seller, and evangelist for a decade. Today, he has responsibility for leading the product management team overseeing continuous delivery solutions ...
From the culture shift, to differences in Agile, Dave Farley and Michael Hackett discuss the nitty gritty of Testing in DevOps. For this issue of LogiGear Magazine, our very own Michael Hackett sat down with one of the godfathers of Continuous Delivery, David Farley. In this exclusive interview, David discusses how test teams and automation ...
LogiGear Magazine June Testing in Continuous Delivery Issue 2017
Over the years we’ve provided an extensive number of articles, videos, and infographics that provide a wealth of knowledge about Continuous Delivery.
LogiGear Magazine June Issue 2018: TESTING in DEVOPS
Introduction Everything changes. It’s the only constant. The landscape of software testing is undergoing a fast and dramatic change driven by societal, business and technology changes that have placed software everywhere. Computing is ubiquitous. There is hardly anything today that doesn’t contain a CPU or information transmission capability, with software to drive it. From smart toasters ...
How Halliburton leveraged outsourcing to achieve their goals. Organizations are focusing on speed, both in Continuous Integration and rapid deployment as a competitive advantage. Many software development organizations can significantly shorten development cycles by implementing one or a combination of Agile practices, continuous integration & deployment methods, and feature branches. While these frameworks and techniques ...
It’s no secret that the cloud is growing at an exponential rate. By 2016, two-thirds of the world’s server workloads will exist in the cloud. But according to Cisco’s 2012 Cloud Index, less than half of server workloads currently run in the cloud. Closing the gap between current capabilities and future requirements is a mission-critical ...
LogiGear Magazine June Issue 2020: Transform Your SDLC With Continuous Testing
By Jez Humble and David Farley Continuous Delivery from Jez Humble and David Farley is an important contribution to the field of software development. It takes continuous integration to the logical conclusion and covers how to set up a continuous integration system, delving into everything from check-in to delivery to production. It doesn’t state you ...

Leave a Reply

Your email address will not be published.

Stay in the loop with the lastest
software testing news