Eric Minick brilliantly demonstrates how automation works in Continuous Delivery using this real-world “case study”.
As a product manager at IBM working on our UrbanCode products, I think a lot about Continuous Delivery, how to do it and what it does for teams. Continuous Delivery techniques enable teams to deliver software faster and with less risk. The ideal in continuous delivery is for a developer to push a code change, and have that change rapidly built and tested. Minutes later, the code should either be rejected as having a rejection, or be available to be released into production. Needless to say, most teams are still working towards that ideal.
The Build, Deploy and Release tools that I work on form the skeleton of a continuous delivery pipeline by automating many of the activities that transform code into software, and deploy that software into test and production environments. The “muscle” that attaches to that “skeleton” is automated testing. To determine fitness for production rapidly, teams must be able to get code into test environments quickly and correctly, then execute their tests equally quickly.
Continuous Delivery is a team sport. It requires collaboration between development, testers and operations.
A recent experience with Amazon Prime impacted how I thought about Continuous Delivery. I had noticed a light bulb at home went out and we were out of spares. Instead of adding a trip to the store to my to-do list, my first instinct was to order a light bulb off Amazon with two day shipping. That would be a little crazy if I didn’t have Amazon Prime.
Prime has an annual fee, after which any purchase I make for the year has free two-day shipping. So instead of a $10 pack of light bulbs that I pay $5 to have shipped in a week, I just pay $10. The goods show up before I would bother to go to the store, for less effort, and without paying for gas.
So, what does this have to do with automated software deployments? When you make it cheap and easy to do something, you change behavior in radical ways.
1. Frequency Increases
One of the ways Prime is a winner for Amazon, is that it encourages me to buy more frequently. It is easy and without shipping price penalties I feel like I’m getting a bargain. I also buy tons more music off iTunes than I ever did from music stores.
Likewise, automation makes deployments cheap enough that you do more deployments. We’ve seen customers increase their number of deployments to test environments by an order of magnitude once automated.
For some clients, this has resulted in manual testers being overwhelmed by the flow of change coming towards them. More typically though, more frequent delivery reduces the frequency of bugs being reported which have already been detected and fixed in a more recent version.
Small Things Count: There’s a surprising difference between having a web page where someone can easily request a deployment and automatically deploying to test as the result of a new build or a nightly schedule. On event/schedule gets has a bigger impact over self-service. In Amazon terms, this is “subscribe and save“. New parents, take note, this was made for diapers. Developers, this helps turn CI into CD.
2. It’s About Effort, Not Speed
If I’d needed drain cleaner, speed might have been paramount in deciding how I shopped. Likewise, for some deployment teams, automation’s main benefit is shorter outage windows across development, test, and production environments.
But for most of us, the attractive part of shopping online is just how easy it is. Online, I do my part of the purchasing in 5 minutes, rather than a half hour in the real world. Most teams who are implementing automation are doing so because the deployment guys are stretched beyond capacity and need time back. Shorter outages are a side benefit.
3. Batch Sizes Shrink
I used to avoid buying just one at a time when shopping online. I would wait until I had a few things to get and save on shipping by ordering them together. Looking at my orders before and after purchasing Prime, I’ve seen my orders get smaller and smaller.
The same thing happens with deployment automation. When deployments are a big, scary ordeal teams set up release weekends and release trains where numerous projects go out in a big effort. With automation comes confidence, comfort and eventually more independence project to project. When something is ready to ship, it goes out. It doesn’t wait for a bunch of unrelated other projects to make a release weekend worthwhile.
Similarly, in test environments, because the frequency of deployment has increased the number of changes represented by each update is smaller. This means that when something breaks, the culprit is much clearer. More effort is expended fixing problems and less is spent assigning blame.
This is a nice little side effect. You see faster time to market, and those practicing Lean are thrilled. They see less “inventory” waste, and smaller batch sizes tend to appeal to both the Lean and Agile camps for a host of reasons.
4. Free Returns
Our family now buys many of our clothes through Amazon Prime and shoes from Zappos (an Amazon acquisition). In both cases, the ability to order, try on, and return any rejects with free shipping both ways has been a huge factor. My wife no longer goes to six stores to try things on and can simply order with confidence that if she’s wrong about the fit, reconsidering is easy.
This is the real key to deployment rollback automation. Because rollbacks allow us to reconsider more easily, they enable us to be more aggressive in releasing (even to just test environments). Partial (canary) production deployments build on this pattern.
5. Do Things the “Right” Way
All these little benefits add up to one key thing for Amazon— we do more of our shopping there than we otherwise would have. We buy more stuff how they want us to— on their website.
If you are trying to wrangle people and get them to deploy following the proper approvals, and deployment plans, learn this lesson. Make it easier for people to deploy following protocol than to work around the system. Push button deployment automation is so much easier than manual deployments, that any process you build into that automation system has much higher odds of being followed than something enforced by sternly worded memos.
People just want to get their jobs done. They want to write code. They want to test. They want to deliver value to the customer. While there will always be defenders of the status quo, if you provide an easier way to get things done, most people will follow that path. If the easy path includes all the right checks and controls, the checks and controls will be followed.
*This is me talking about a service I consume and like. Amazon hasn’t endorsed this, and all their trademarks belong to them. Their services, pricing, etc can change. And seriously, think about the gas, jet fuel, and people’s time you waste when you order a light bulb two day shipping. At least get an eight-pack or one of those nifty new LED bulbs. On the other hand, don’t have any guilt about deploying more.
