Getting intimate with CI/CD and CI/CD Pipeline

Introduction:

In my previous article Why CI/CD? How was a life without CI/CD & how is it now? I talked about the Why of CI/CD. If you read that, got the concept and wanna hear more about CI/CD then In this article we are going to ding more into the concepts of CI/CD.

So,

What is CI/CD?

CI/CD is simply a method to frequently deliver apps to customers by introducing automation into the stages of app development. The main concepts attributed to CI/CD are continuous integration, continuous delivery, and continuous deployment.

CI/CD is a solution to the problems integrating new code can cause for development and operations teams (AKA "integration hell").

Specifically, CI/CD introduces ongoing automation and continuous monitoring throughout the lifecycle of apps, from integration and testing phases to delivery and deployment.

So, there are three terms:

• Continuous Integration
• Continuous Delivery
• Continuous Deployment

Let's look into them one by one

Continuous Integration

Continuous Integration (CI) is a development practice that requires developers to integrate code into a shared repository several times a day. Each check-in is then verified by an automated build, allowing teams to detect problems early.

When you have a team of developers, each of whom is responsible for a separate feature, you need to integrate the different features before youre ready for a release. By integrating so frequently, your team can surface errors earlier. And when those are caught, the amount of backtracking needed to find the cause is also much reduced. Therefore, your team can resolve the integration errors much faster.

Continuous Integration doesnt get rid of bugs, but it does make them dramatically easier to find and remove
Martin Fowler, Chief Scientist, ThoughtWorks

How to Practice Continuous Integration

If you want to practice CI, the steps roughly go like this:

1. Developers check out code from the repository to work on it locally. Ideally, they create a new branch for the feature they want to implement.
2. When their feature branch is ready, they run tests locally in their development environments.
3. Once all tests pass, they push the commits to the single source repository.
4. Whenever there are changes on the repository, a CI server checks out the changes and performs a build and test. A build and test is when the CI server builds the entire system on the developers feature branch and runs all the unit and integration tests.
5. The CI server notifies the team of the integration result. There should generally be four outcomes: failed build, successful build, failed tests, successful tests.
6. If theres a failure, the team fixes the issue ASAP.
7. When the feature branch is merged to the main branch, they repeat steps 26.
8. They continue to develop and repeat the steps 26 whenever theres new code to be checked in to the repository.

There are minor variations of the steps, depending on the tools you chose and the processes you agree upon within the team. **The main principles of CI** are that you:

1. Check in code in frequently.
2. Automate the build and test portion.
3. Always test the code locally before checking it in.
4. Never merge any failed branches to the main branch.
5. Return its status back to successful if youre the developer who causes the failed build or test.
6. Make it your top priority to do so once the fail happens.

Continuous Deployment (CD)

Continuous Deployment is closely related to Continuous Integration and refers to the release into production of software that passes the automated tests.

So why do you need to care about continuous deployment as part of your development process? Well, when there are releases, there will be deployment steps. These deployment steps tend to repeat for each release. Instead of performing the deployment manually for each release, why not have the deployment steps be executed automatically? Of course, ideally, this code has been built and tested successfully by the CI server too.

Essentially, it is the practice of releasing every good build to users

SO, Whats the Difference Between Continuous Deployment and Continuous Delivery? Aren't they same thing?

You may be confused by the fact that theres another term CD can stand for: continuous delivery. So whats the difference between the two?

According to Jez Humble, continuous deployment is about automating the release of a good build to the production environment. In fact, Humble thinks it might be more accurate to call it continuous release. On the other hand, continuous delivery is about ensuring that every good build is potentially ready for production release. At the very least, its sent to the user acceptance test (UAT) environment. Your business team can then decide when a successful build in UAT can be deployed to production and they can do so at the push of a button.

Sometimes its unwise to have every build be an actual release. Such is the case with embedded software. So we need a slightly different definition for builds that potentially can be releases but need not be automatically deployedhence the existence of continuous delivery.

To keep it simple, heres a diagram.

When youve successfully implemented continuous deployment, it implies that continuous delivery is achieved as well. But the converse isnt true.

The term continuous deployment came before the term continuous delivery. However, do note that in most enterprise setups, the business side of the company prefers to manually trigger the deployment. So, chances are your organization will actually be implementing continuous delivery, which is a good enough solution.

Now let's move onto the concept of CI/CD Pipeline

Now that we have understood the concepts of CI and CD, its time we get more into the weeds of what a CI/CD pipeline is.

A CI/CD pipeline can be easily understood as the process pathway through which we can deliver a single unit of production-ready software. Your team will choose which services theyll use to build this; theres no single canonical implementation of a CI/CD pipeline.
The simplest version of the pipeline can best be described by this activity chain:

Each of these steps in the activity chain can be handled by a single service or piece of software. Alternatively, you can split the tasks into several different tools.

CI/CD Pipeline Phases

While each stage of a CI/CD pipeline may be performed manually, the actual benefit of CI/CD pipelines is achieved via automated processes. A CI/CD pipeline is made up of discrete subgroups of operations which are organized into pipeline phases.

Following are some common pipeline phases:
Source
The source contains the initial source code repository and any change in the code triggers CI/CD tools to automatically apply modifications. User-initiated processes, programmed timelines, and the outcomes of other pipelines are other typical triggers. The process begins with a Push-request, written manually, which causes the development of Webhooks to run the pipeline.

Build
At this stage complies an app with its source code repository. But, in fact, the build stage is not necessarily used for such languages as Python, JavaScript, and Ruby. Inability to pass the build phase indicates a basic project malfunction, which should be addressed as soon as possible.

Test
The implementation of automated tests to check the validity of code and the functionality of the program is included in the test phase. This stages automation facilitates the quality assurance for the running code. Testing eliminates potentially repeatable issues from making their way to the customers. Besides, here happens the Merge stage, which is responsible for the finalization of the assessment, needed before the release to the source code repository.

Deploy
This is the last step of the CI/CD pipeline, and its done if the previous testing and merging go without conflicts and errors. Deployment means your code goes to the servers, like Azure or AWS to make it functional.

Example of a CI/CD Pipeline Toolchain

To better illustrate a pipeline, Ill conjure up a fictional example involving an enterprise web software written in Java. Suppose I want to start from scratch and I want to keep things easy. Id make the following choices:

Source code control: Ill host the code on GitHub (hosted service) as a private repository. This way, I can use its integrations with major software and services to establish triggers whenever code commits are checked in.

Continuous integration: Ill get CircleCI and connect it with GitHub integrations. This allows every code commit to notify CircleCI via webhooks. When code changes notify CircleCI, it will then pull the code from the GitHub repository and proceed to build and run the tests. Any failures or successes can be sent by email or Slack notification.

Deploy to UAT: Suppose my UAT is running on a server in AWS ECS. Ill configure CircleCI to deploy automatically to the AWS UAT server when the build and tests are successful.

Deploy to Production: For deployment to production, Ill reuse the integration steps for deploying to UAT.
With sufficient configuration at the start and by employing three major services, Ive designed a good enough CI/CD pipeline.

CI/CD pipeline principles and best practices

There are no ironclad rules that define the best ways to create CI/CD pipelines. However, there are certain guidelines to follow while building these pipelines.

• If the build is broken, fix it immediately. Teams should drop any work if the changes introduced break the pipeline.
• Shift left in testing. If the team has a large number of tests, first run fundamental and faster tests such as code quality.
• Use a consistent environment. An application deployed to a handcrafted environment is most likely to fail when it hits the production environment.
• Bake in code quality checks. You can easily integrate many open source tools into the CI/CD pipeline to provide comprehensible documentation for developers.
• Gauge your CI/CD pipeline's speed. How long does the build sit in queue before it's picked up by an agent? How long does it take to provision a new pipeline for an application? As a general rule of thumb, if it takes longer than grabbing a coffee to build, test and deploy the changes, that's a signal that the pipeline is not providing fast feedback.
• Document everything. Describe how the pipeline functions. If something fails and there is documentation around how the pipeline works developers can attempt to fix the issues by reading the pipeline documentation and issuing pull requests. This enhances collaboration across teams as well.

Overall considerations to build a CI/CD pipeline

CI/CD pipelines aim to streamline software development and delivery, but real-world implementations can widely differ from the theoretical concepts. Organizations have specific problems to tackle, resources to draw upon and technology decisions to weigh.

Consider investments in time and resources to manage the infrastructure that supports CI/CD pipelines. On-premises repos and version control such as Git, along with build servers such as Jenkins, require a lot of effort to patch and maintain.

If a team chooses to run the CI/CD pipelines with hosted providers, such as GitHub and Azure DevOps, there are additional considerations. A CI/CD pipeline in the cloud typically deploys the application as a hosted workload on that cloud's platform and will require the team to assign underlying infrastructure resources (IaaS, PaaS or SaaS).

There are also many security questions to answer with CI/CD in the cloud. How do you authenticate users and grant them appropriate access to resources? How does the provider store credentials or service connection strings to access outside resources? How do CI/CD pipelines with a cloud provider access internal resources if required? Many providers allow organizations to run the hosted agent in their internal network, but this requires outbound connections to allow egress traffic, which may require a security review and exception.

There is no one-size-fits-all answer to implementing a CI/CD pipeline. However, there are common attributes and CI/CD best practices that can help you design, implement and continuously improve your software delivery process. Start with very basic pipelines and then continuously gather feedback and improve them over time.

Now you can tell others that:

Connect with me:

LinkedIn
Instagram
Twitter

References:

https://incora.software/insights/what-is-the-role-of-ci-cd-pipeline-in-software-development/75
https://www.redhat.com/en/topics/devops/what-is-ci-cd
https://www.plutora.com/blog/understanding-ci-cd-pipeline
https://www.techtarget.com/searchsoftwarequality/CI-CD-pipelines-explained-Everything-you-need-to-know
https://www.techtarget.com/searchitoperations/tip/How-to-build-a-CI-CD-pipeline-with-examples