How Halliburton leveraged outsourcing to achieve their goals.
Organizations are focusing on speed, both in Continuous Integration and rapid deployment as a competitive advantage. Many software development organizations can significantly shorten development cycles by implementing one or a combination of Agile practices, continuous integration & deployment methods, and feature branches. While these frameworks and techniques shorten development cycles, test teams can quickly fall behind in keeping up with the velocity of product releases. Getting assistance from the outside can be crucial to the success of the entire development organization.
This case study details how Halliburton leveraged LogiGear’s Professional Services team to develop a comprehensive test automation strategy for their Agile teams, and leverage test automaton and application lifecycle management (ALM) tools across the organization. The approach and strategic implementation resulted in a significant reduction in test cycles over prior releases, increased capacity of the test team, and minimized costs using outsourced test automation and Professional Services resources.
Background
Halliburton develops software applications that are extremely complex, with millions of lines of source code, N-Tier architecture and message queuing. The complex nature of the software requires extensive testing and significant resources. When the Production Enhancement (PE) software technology organization adopted continuous integration practices, it became apparent that the current approach to testing needed to change radically to meet deliver expectations.
Cheronda Bright took over the PE testing group at Halliburton shortly after the development team implemented feature branches. The main product team worked on functional enhancements in the development branch while the user experience team made updates to the user interface in a separate feature branch. Features from each branch were merged frequently, which significantly increased the testing workload.
In the past, after every two-week development sprint the software was handed over to the test team that executed all testing manually. New tests were created and run along with any necessary system integration regression testing. Test results were sent to the product management and development teams in order to prioritize fixes to software bugs. After the bugs were addressed the software was sent back to the team for verification testing. When the product management and development teams were satisfied with the results, the software was placed into production.
Testing Challenges
Cheronda knew from previous experience that the only way for testing to keep pace with a continuous integration model was to automate as much of the testing as possible. However, the solution it would not be as simple as just automating tests. The vision was to implement a solution that would result in a truly integrated development environment.
Even though the teams had adopted many Agile practices, collaboration was still a challenge due to separate process management platforms in use. Microsoft Team Foundation Server (TFS) was used by the development team for source control, and user story and bug management by scrum masters and product managers. The test team used Hewlett Packard’s Quality Center (QC) for test management, execution and status reporting. A significant issue for the test team was not having a reliable way to stay updated on what changes were made to the software other than direct communication with the developers. This often resulted in testers creating and running tests that duplicated previously done unit testing.
To begin, Cheronda asked both her testing team and the development team for their input on what they saw as essential to make the project a success. The test team was struggling to keep up with the additional testing required, and the product management team wanted to significantly shorten release cycles while increasing scope. So it was no surprise that reducing testing time for the 3,000 test cases was the top requirement for both teams, along with maintaining quality. The development team also wanted to see testing of impacted code changes executed automatically after each build. And both teams wanted better visibility and reporting for all aspects of the project.
Solution Selection
The first step was to determine if it was possible to implement a common platform for all teams. Ultimately it was determined the Visual Studio/TFS/MTM (Microsoft Test Manager) Integrated Development Environment (IDE) had the potential to improve collaboration across the teams. Using traceability would allow testers to select specific test suites and create test configurations based on the project plan in TFS, and give the entire scrum team access to the project plans and testing strategy in order to create any number of detailed test status reports.
The next step was to identify a viable test automation solution. Cheronda was familiar with keyword testing and made it a requirement that any tool considered would need to support a keyword test method. After evaluating a number of tool options the team settled on TestArchitect from LogiGear. The tool supports Action Based Testing (ABT), an advanced keyword method, and the testing framework provided plug-in integration with both TFS and MTM. Besides TFS/MTM integration, a major factor in the selection was that LogiGear’s Professional Service team was also able to provide implementation support and training for the testing team.
Implementation
Test automation added a level of complexity and required a new way of approaching testing for the PE test team. Going forward, there would be a manual test team and an automation test team working together in close collaboration with the development team.
Cheronda worked with LogiGear Professional Services to develop the project plan and direct various parts of the project through the initial ramp-up. The challenges that had to be addressed included:
- Implementing test automation without impacting project schedules
- Analyzing manual tests in Quality Center
- Training the test team on Action Based Testing and the TestArchitect automation tool
- Training all testers on MTM
- Migrating manual tests from QC to MTM Auto mating manual tests and validating each one before moving them into production
- Setting up tests to run unattended by auto-deploying the execution as apart of the build definition and Continuous Integration processes
- Creating detailed status reports that included images and log files on failure
- Defining the workflow between the manual and automation teams
To begin the project, two Halliburton managers were assigned to be automation leads. These individuals worked with the LogiGear team to identify tests that would be the most beneficial to automate. From 3,000 test cases, 600 were selected for automation. The initial test design and automation work of creating tests using the ABT method was outsourced to LogiGear’s automation engineers. Leveraging LogiGear’s automation expertise allowed the PE test team to continue ongoing testing in parallel with the automation ramp-up.
LogiGear’s Automation Engineers started by creating a library of custom actions. This initial work of populating the action library minimized the programming required for future test creation. The action creation was so that test engineers could begin developing the initial test cases in parallel with the creation of the action library.
While the automation team worked on test creation, LogiGear’s project manager worked closely with the PE configuration management team to integrate TestArchitect with TFS and MTM. As the automated tests were completed, they were checked-in into TFS. A beta test was performed by creating a test plan and then selecting and running tests from MTM on a single machine. After verifying the success of the execution from the Microsoft IDE, the team configured seventeen virtual machines with various setups in order to limit the time to run a full suite of tests to no more than 5 hours.
Next, the team focused on migrating manual tests from QC to TFS. The Halliburton project manager asked LogiGear to investigate the potential of automating the migration. Within a short time, LogiGear’s software development developed a program to automate the migration of tests from QC to MTM.
In parallel with the test automation effort LogiGear trainers provided hands-on instruction in test design and the correct approach for building complex tests required for the PE toolkit. At times the instructors worked alongside individual testers to develop the ideal test solution.
Outcome
LogiGear automation experts worked with the Halliburton team to create a test automation infrastructure consisting of the TestArchitect automaton tool, Microsoft’s TFS and MTM. The PE test team was also trained on the ABT test method that was implemented for both manual and automated testing.
After automating 600 tests, the regression suite that had previously taken eight weeks to complete was run in five hours with TestArchitect using a combination of seventeen virtual and physical machines. The migration to TFS/MTM provided managers and all team members with a much greater level of visibility across the entire project. The work improved testing and team collaboration and met the primary objective of delivering very short-cycle releases by facilitating testing for Continuous Integration deployment, multiple branches and variations.
From the beginning of the project the plan was for test automation to be completely handed off to the PE test team when the integration and transition was complete. Ultimately the decision was made to outsource the automation engineering and execution of automated tests to LogiGear. This approach resulted in much more efficient use of the domain expertise of Halliburton’s testers while leveraging the efficiency and low-cost of LogiGear Automation Engineering staff.
Today Halliburton and LogiGear work collaboratively, constantly fine-tuning the testing workflows, while strategizing across an integrated manual and automated team. The PE team designs and creates test cases in-house, and relies on remote LogiGear Automation Engineers to assist with day-to-day automation programming needs. In cases where actions are required for custom controls LogiGear Automation Engineers review the test requirements and discuss any concerns or suggestions before creating the automation and adding the actions to the shared library.
LogiGear’s test team works one sprint behind software development and provides release notes with every new build delivered. The Halliburton team certifies every test delivered by LogiGear before they are moved to production. This verification step is completed within five hours by utilizing up to 70 virtual machines.
Summary
Halliburton’s PE team embarked on a project to make Continuous Integration and rapid delivery possible. By leveraging test automation and testing expertise and talent from outside their organization they were able to achieve their objectives and implement a sustainable testing solution that allows rapid, high quality testing. The partnership between Halliburton and LogiGear Professional Services proved to be a winning combination.
