Do you want to speed up your automated tests by a factor of 10 and deploy your application continuously? In this article we share how the JIRA development team at Atlassian has accomplished this using Stages in Bamboo.
Stages have allowed the JIRA Development team to take a week’s worth of testing and condense it to one day.
Additionally, we have taken functional testing time from 8 hours to 40 minutes and use it as an “inner loop” continuous integration feedback build.
There are two primary feedback loops that the JIRA development team utilizes:
- Generic “CI” Build – developers get feedback on their code changes quickly.
- Tested Platform Matrix – feedback about how our release is validated against different types of distributions and the different environments they can run on, which include different operating systems and database configurations.
What Did We Do Before Bamboo Stages?
Running our tests in serial, the CI build would take 8 hours to run our functional test suite, which consisted of around 3000 tests. However, we only did this to deploy the JIRA artifacts to Maven because it is important to run all of our tests prior to deployment. This is key for JIRA artifact consumers to make sure they can pin point true integration failures instead of tracking down failures within JIRA. In addition to this build, in order to get faster feedback, we had 15 different build plans to run our functional tests but had no way to act upon those results. Finally, we also ran a unit test build as a fast CI loop that ran in 15 minutes, but only provided unit level testing. We had two builds for functional testing, one for validating deploying artifacts to Maven and the other to give us a faster feedback cycle with those tests. This meant we were potentially testing the same changeset twice and burdening our build hardware. For the Maven deployment functional test build that took 8 hours, we could deploy our artifacts once a day. Sometimes this deployment would be delayed two or three days if those tests stayed in a failed state. The Tested Platform Matrix build would take an entire week and might not pass within that week. For most, if not all development teams, those feedback times are unacceptable.
What are we trying to run that took 8 hours of serial execution?
- 12,567 JUnit Tests
- 3020 Functional Tests
- 328 QUnit Javascript Tests
- 639 Selenium UI Web Browser Tests
Staging a Revolution in JIRA’s Testing Cycles
By relying on an 8 hour development testing cycle and a week release testing cycle, we added higher risk in our ability to release JIRA on time. A lot of time was spent figuring out what commit broke what test, and sometimes even what day it happened. We needed help with running our tests in parallel and we needed to do it on a massive scale in order to shorten these long feedback loops. Bamboo stages allow us to run more than one build “job” within a stage and more than one stage in a build plan. The JIRA development team has done work to break the test suite into smaller parts, called batches, to launch each of these batches as a job within a Stage. Our functional testing Stage, for example, now includes 24 batches, or “jobs” that run simultaneously. Of course, we need somewhere to execute all of these jobs in parallel. This is where another important capability of Bamboo helps us out: the ability to run multiple remote agents on different types of hardware.
But the build pipeline is not done yet. Once our CI Build passes and “goes green”, what if we wanted to…
1. deploy these tested artifacts to Maven?
2. deploy our tested JIRA distribution to QA environments for manual testing?
3. build other JIRA Plugins so they can consume and validate against these tested artifacts?
4. run more tests, specifically our Tested Platforms Matrix?
We’d like to ensure that we can build and promote through these phases in this diagram going from the center left in development to release validation on the outside of the circles. The two primary loops are signified in yellow for development testing in our CI build and blue for release testing in the Tested Platforms Matrix and other important consumers of JIRA.
How Stages Help Us Reach Our Goals
Like many other development teams, the JIRA team wants to do all of these great tasks and accomplish these goals. The focus is to have our development CI loop as the producer bridge to launch into other outer loop CI activities. However, we can’t act until we can aggregate all of the jobs that run our functional tests into one result, to decide if we want to proceed further into greatness. This is another key feature in Bamboo Stages.
Previously there were 15 functional test build plans all running on their own after a dependent parent build triggered from commit detections. However, it was impossible to launch other build plans against the results of these 15 builds.
Because of this, we would not be able to deploy our tested Maven artifacts from those results. Now, with Stages, our CI build plan can not only have these same 15 jobs but we also get the ability to deploy the artifacts from that commit revision to Maven if and only if all of the functional test jobs pass. Additionally, Bamboo Stages also makes adding Jobs to each Stage an easy process. This is important because we plan on growing our functional test batches from our current set of 20 to 40 with a goal of having each job execute in less time.
Next we modified our other child build plans—that needed to consume JIRA artifacts—to depend upon the result of the CI build plan. For example, the final Stage in the CI plan is to deploy our JIRA artifacts to Maven only if all the previous Stages’ jobs passed, then build and validate important plugins like the JIRA Fisheye, JIRA Workflow and Activity Streams plugins. The final step was to build against the JIRA artifacts that were just deployed in the parent CI plan only if all functional tests passed. All this was done with the use of Bamboo Stages and additionally, another child plan of our CI plan, then deploy the JIRA Standalone distribution to a clean test remote environment for QA manual testing. The best news of all is that the CI build now executes all functional and unit tests on average in 40 minutes and we have ambitions to make this a 10 to 20 minute build in its entirety.
Our Tested Platforms Matrix is the second set of longer running tests that will drive us closer to release readiness. We run exactly the same functional tests in this set of builds. However, the difference here is we run against each distribution that we allow customers to download against most of our supported environments.
What does this mean?
We have:
- 4 distributions – Standalone, War, Source, and a Windows Installer.
- 3020 functional tests that were previously defined
- MySQL, Postgres, MS SQL Server, and Oracle databases that, except SQL Server, run on two operating system platforms, Linux and Windows.
Adding up all of the cells in the Tested Platforms Matrix shows us that we execute 24,160 functional tests.
These 24,160 functional tests are executed against the same Maven artifacts that were deployed from any previously passing CI Build. We can use staging again to help organize the number of jobs launched against the matrix and also aggregate results.
Because of our ability to deploy JIRA artifacts through our development CI build, we can be assured our Tested Platforms Matrix builds run at least once a day. By using Stages, the execution of the matrix can help us complete the entire set of build plans in 5 hours. This allows us to have faster assurance from the Tested Platforms Matrix that we are ready promote what we created from a passing build in CI to make these same artifacts be release candidate artifacts for customer consumption.
Stages for the Win!
Bamboo Stages have been the ultimate win in condensing the amount of time to execute our functional tests in JIRA. It has given us the ability to not only help us launch our functional tests in parallel but also to importantly aggregate the results of those tests into a single pass fail result. This result has enabled us to then, with confidence, deploy and publish our JIRA artifacts for further testing in validation in JIRA and other products that depend on JIRA.
There are other elements to what I have shown you. We use Bamboo’s artifact passing to build once and re-use many times so we can pass the JIRA artifacts from one Stage to the next. This means that all of the functional test batches I described don’t have to checkout JIRA source and compile it. They can re-use the passed artifacts and give us the purity of testing against the exact same artifacts that were built in the compile Stage. When we started using this feature were able to trim even more time from both our CI build and our Tested Platforms Matrix –nearly 45 minutes!
BryceTo keep up-to-date with the latest features released into Bamboo, as well as get tips for tricking out your CI pipeline, check out the Bamboo blog at http://blogs.atlassian.com/blog-cat/bamboo/.
You can also follow Atlassian Dev Tools on Twitter: @AtlDevTools. Happy building, everyone!
