Continuous Testing and Continuous Monitoring
What is the goal of Continuous Integration?
Is it to enable Continuous Delivery of the code developers’ produce out to users? Yes, eventually. But first and foremost it is to enable ongoing test and verification of the code. It is to validate that the code produced and integrated with that from other developers and with other components of the application, functions and performs as designed. Once the application has been deployed to a production system, it is also a goal to monitor that application to ensure that it functions and performs as designed in a production environment, as it is being used by end-users.
This brings us to the two practices of DevOps that are enabled by Continuous Integration and Continuous Delivery.
Namely:
- Continuous Testing
- Continuous Monitoring
Continuous Integration and Delivery are both (almost) meaningless without Continuous Test. Not having motoring and hence, not knowing how the application is performing in production makes the whole process of DevOps moot. What good is having an streamlined Continuous Delivery process if the only way you find out about your applications’ functionality or performance being below par is via a ticket opened by a disgruntled user?
Let’s break this down.
Continuous Integration and Continuous Delivery facilitate Continuous Testing
Individual developers work to create code – to fix defects, add new features, enhance features, or to make the code perform faster – could be one of several tasks (work items) they could be working on. When done, they deliver their code and integrate it with the work done by other developers on their team, and with unchanged code their team owns. They would along the way run Unit Tests on their own code.
Once the integration is done, they would do Unit Tests on the integrated code. They may run other tests such as White box Security tests, code performance tests, etc. This work would then be delivered to the common Integration area of the team of teams – integrating the work of all the teams working on the project and all the code components that make up the service, application or system being developed.
This is the essence of the process of Continuous Integration. What makes this process continuous is where the individual developers’ code is integrated with that of their team as and when they check-in the code and it gets delivered for Integration. (For more detail, read Part 2 of this Understanding DevOps series). The important point to note here is the goal of the Continuous Integration process – to validate that the code integrates at all levels without error and that all tests run by developers run without error. Continuous Testing starts right with the developers.
After validating that the complete application (or service or system) is built without error, the application is delivered to the QA area. This delivering of code from the Dev or development environment to the QA environment is the first major step of Continuous Delivery.
There is Continuous Delivery happening as the developers’ deliver their code to their teams’ integration space and to the projects integration space, but this is limited to being within the Dev space. There is no new environment to target. When delivering to QA, we are speaking of a complete transition from one environment to another. QA would have its own environment on which to run its suites of functional and performance tests. DevOps principles preach that this environment be production-like.
In addition QA would potentially also need new data sets for each run of the suites of tests it runs. This may be even one or more times every day as Continuous Integration leads to Continuous Delivery at a steady stream. This means that the Continuous Delivery process would not only require the processes to transition the code from Dev to QA, but to also ‘refresh’ or provision new instances of QA’s ‘Production-like’ environment, complete with the right configurations, and associated test data to run the tests against.
This makes Continuous Delivery a more complex process than just ftp-ing code over. (I will discuss this in more detail in a later post of Infrastructure automation and Infrastructure as code). The key point to note is that the goal of Continuous Delivery is to get the code ready for test. To get the application to the right environment – continuously, so that it can be tested – continuously.
If one extends the process described above to delivering the service, application or system to a staging and eventually a production environment, the process and goal remains the same. The Ops team wants to run their own set of smoke tests, acceptance tests and system stability tests before they deliver the application to the ‘must-stay-up-at-all-costs’ production environment. That is done using the Staging environment. This is a production-like environment that needs to be provisioned just like the QA Environment. It needs to have the necessary scripts and test data for acceptance and performance tests that Ops would run. Only when this last phase of Continuous Testing is complete would the application be delivered to Production. Continuous Delivery processes would perform these tasks too – provision staging and production environments and deliver the application.
Continuous Monitoring
In Production, the Ops team manages and ensures that the application is performing as desired and the environment is stable via Continuous Monitoring. While the Ops teams have their own tools to monitor their environments and systems, DevOps principles suggest that they also monitor the applications. They need to ensure that the applications are performing at optimal levels – down to levels lower than system monitoring tools would allow. This requires that Ops teams use tools that can monitor application performance and issues. It may also require that they work with Dev to build self-monitoring or analytics gathering capabilities right into the applications being built. This would allow for true end-to-end monitoring – continuously.
The Four Continuous Processes of DevOps:
It would not be much of a stretch to say that the practice of DevOps is really up of the implementation of these four basic processes:
- Continuous Integration
- Continuous Delivery
- Continuous Testing
- Continuous Monitoring
Of these, the end goal is really Continuous Testing and Continuous Monitoring. Continuous Integration and Continuous Delivery get the code in the right state and in the right environment to enable these two processes. Testing and Monitoring are what will validate that you have built the right application, which functions and performs as designed. In future posts, I will go into more detail of the implementation of these four processes and the associated practices that enable them.
