Moving Toward CI / CD

“The most powerful tool we have as developers is automation.”

CI/CD represents a set of software development practices and processes that allow for rapid development with the help of automated builds, testing, and deployment.

What is CI/CD?

Continuous Integration

Continuous Integration is the process of developers continually merging (integrating) code changes back into the main branch of the repository. One benefit of this process is that it helps avoid large merges at the end of a sprint or delivery cycle. Those giant pushes can often result in a plethora of merge conflicts, errors, and other integration woes.

By merging often, you not only avoid some of the code issues you would face later on, but you also get a chance to test the integration of new features through an automated build/test cycle. Continuous Integration is crucial if you want to be a team that fails fast and quickly reacts to bugs/issues.

Continuous Delivery

Continuous Delivery is a set of practices that ensures you can deploy a proven set of artifacts to any environment at any time. This is possible through the use of automated builds and tests, correct artifact handling and versioning, and deployment tools such as Bamboo or Jenkins.

Continuous Deployment

Continuous Deployment takes Continuous Delivery one step further: it is the process of actually deploying artifacts to a site every time changes are made and pass testing. This is not feasible for a lot of software teams, but is something that they should strive for. Even if you’re not actually deploying every time a change is made, making sure you can is a huge step forward.

Why should we care?

Many teams don’t employ CI/CD practices, believing that manual testing/deployment works “well enough” for them. While that may be true, CI/CD can save time, money, and headaches if implemented correctly.

CI/CD takes most of the human element out of builds and deployments, allowing our engineers to focus on the real problem: engineering. Not only that, but it also gives teams more confidence that what they’re deploying actually works. This helps keep regressions out of production and bugs out of the team’s backlog.

Our experience with CI/CD

Here in Oklahoma City, we are just getting started down the CI/CD journey. A year and a half ago, we had absolutely no automation for our new suite of products. Any time we needed new services deployed, one of our engineers had to manually compile the code and deploy it to our Kubernetes cluster. This had a few problems associated with it:

• Only one person could deploy changes. If they weren’t around or was too busy, too bad. This not only slowed down development, but also our manual QA process.
• It was extremely error-prone. If our deployer had local code changes or any other variance in his environment, the entire build could fail, or some weird issues could get into the deployed services.
• There were no automated tests. The only way to verify a deployment was successful was to manually click around the site and then get QA involved to test.
• There was no visibility. We had no way to know what was deployed to the site at any given time, relying completely on this one engineer to tell us what was going on.
• We had no saved artifacts. If we wanted to promote a build, guess what: we had to completely re-build it! Not only was this time consuming, but it also meant those particular artifacts had not actually been tested.

Since then, we have made great strides in our own processes. We now use Atlassian’s Bamboo as an automated build system connected to our code repositories in Bitbucket. On every merge to the main branch, Bamboo compiles the changes, runs unit tests, and then deploys the artifacts to an isolated testing environment to run them against a gauntlet of automated tests. After all the tests pass, those same exact artifacts are then deployed to a QA environment for some more manual tests. We also have the ability to then one-click deploy those same artifacts to a handful of supported environments.

How it has helped

I’m sure you can tell just from what I said above that these new processes have helped a ton. However, I’d like to enumerate some of the specific benefits we’ve seen.

1. Nobody has to babysit deployments. Since Bamboo handles them all automatically after a merge, we know that all of our changes will automatically get where they need to go, without having to watch over (or perform) the actual deployment process.

2. Our test suite is always run. No more “I forgot to run the tests before committing.” Code changes with failing tests will always be caught before they make it into our important environments.

3. We know what we’re deploying. If we want to deploy a particular build to a new environment, we can rest assured that we’re pushing the exact artifacts that we tested against, not just new artifacts built from the same code.

4. Visibility. If a build fails, the whole team is notified on Slack. This allows us to react to failures quickly and as a team, rather than only one person being responsible.

5. Simpler process. As we on-board new developers, we don’t have to worry about them not knowing how to deploy their changes immediately. Because everything is automated, they can be confident that their code changes will get where they need to go.

6. Fast feedback loop. If something in our unit/integration tests fails, we know about it quickly. This allows our developers to fix problems as soon as they crop up, rather than letting them sit for much longer.

Baby steps

I’d like to reiterate that we didn’t get to this place overnight; it’s been a lot of trial and error, a lot of tiny steps forward. I think that’s the most important thing about CI/CD: you’re never really finished with it. We’re already thinking up more improvements we can make to our pipeline, more optimizations we can take advantage of.

If your team is facing any of the problems I outlined above, or you think you might benefit from a more CI/CD oriented approach, I’d encourage you to try it! Think of one thing you can change about your process to get you one step closer to CI/CD. If you take enough of those baby steps, you never know how far you might get!