How to manage rollouts and rollbacks using ArgoCD

Learn how to manage canary, blue-green deployments and rollbacks using Argo Rollouts. These methods can greatly reduce downtime during deployments.
Damaso Sanoja is passionate about helping others make data-driven decisions to achieve their goals.

How to manage rollouts and rollbacks using ArgoCD

Learn how to manage canary, blue-green deployments and rollbacks using Argo Rollouts. These methods can greatly reduce downtime during deployments.

Imagine you’re implementing a critical backend update through your typical continuous integration, continuous deployment (CI/CD) pipeline and the new version of the application unexpectedly crashes. This would disrupt the entire user base. This inconvenience underscores the critical role of strategic deployment patterns. Utilizing methods like canary or blue-green releases can greatly reduce downtime, ensuring reliable, smooth updates for uninterrupted online engagements. However, manually implementing these strategies is out of the question since, far from helping, they would add a layer of complexity regarding traffic routing and application management.

Argo Rollouts is a progressive delivery tool that uses the Kubernetes controller and custom resource definition (CRD) constructs to enhance the Kubernetes built-in deployment capabilities. This provides more control during the application deployment process and allows you to implement blue-green and canary deployments easily. Moreover, if necessary, you can use Argo Rollouts to roll back to a previous version, ensuring minimal downtime and improving the reliability of your application deployments.

This tutorial explains canary and blue-green deployments, compares their differences, and shows how to implement both strategies using Argo Rollouts.

Before Getting Started

While you can read this article without completing the tutorial steps, following along will give you firsthand experience of the capabilities of Argo Rollouts. Before starting, you need the following:

Install the controller using these commands:

kubectl create namespace argo-rollouts
kubectl apply -n argo-rollouts -f https://github.com/argoproj/argo-rollouts/releases/latest/download/install.yaml

Install the plugin on macOS with this command:

curl -LO https://github.com/argoproj/argo-rollouts/releases/latest/download/kubectl-argo-rollouts-darwin-amd64

Make sure that you replace darwin with linux if you have a Linux machine.

Lastly, clone the tutorial repository to an appropriate location:

git clone https://github.com/argoproj/rollouts-demo.git

With these requirements out of the way, it’s time to explore canary and blue-green deployment strategies.

Canary Releases and Blue-Green Deployments Explained

Canary and blue-green deployments are two strategies used in Argo Rollouts to manage the progressive delivery of applications. While both methods aim to minimize downtime and enhance user experience, their approach differs.

Canary deployments start by releasing new versions to a few users. If things go well, everyone else gets the update. This approach lets you test new features safely and pull back if needed without impacting all users.

During canary deployments with Argo Rollouts, two key aspects are managed: splitting traffic between the current (stable) and new (canary) versions and keeping a mix of replicas of both versions active. There are three main phases in a standard canary deployment process:

  1. Argo initially directs all user traffic to the current production version with all replicas running it. For instance, if you have five replicas, all five are serving the stable version.
  2. Argo starts the canary deployment based on your setup when you introduce a new version. If you choose to route 20 percent of traffic to the new version, then one out of five replicas will run this new version, and the canaryService will manage user traffic to it. You can pause the rollout at this stage to evaluate the new version’s performance.
  3. Argo gradually replaces old replicas with the new version in set increments, say from 20 percent to 100 percent, if the new version performs well and you opt to proceed. During this process, Argo ensures the total number of replicas remains constant, like keeping five replicas active.

Blue-green deployments use two identical environments: one hosts the stable version (blue) and the other the new version (green). This method is perfect for updates with no downtime as the new version matches the live environment, letting you switch all users to it at once.

Meanwhile, blue-green deployments with Argo Rollouts involves the following process:

  1. Argo directs all traffic to the stable version with activeService. If you have three replicas, all three serve the production version.
  2. Argo, upon updating the app, sets up a new (preview) environment with previewService, deploying three replicas of the new version. You typically pause here to check if the new version aligns well with production.
  3. You promote the rollout after verifying everything works as expected, which then leads Argo to switch activeService to the new version and ends the use of the old version’s replicas.

The following diagram shows you an overview of both strategies and how they carry out the update process:

canary vs blue-green deployment

After this high-level overview of how Argo Rollouts performs updates, it’s time to move on to how you can implement each strategy.

Implementing Canary Deployments with Argo Rollouts

For simplicity, you’ll use the same example for this canary deployment; that is, an application with five replicas is updated in 20 percent increments with intermediate pauses for performance monitoring. During the initial stage, Argo routes 20 percent of users to the new version, which means that one out of five replicas will run the new version, while the remaining four replicas will continue with the stable version. At that point, the process stops, waiting for you to confirm that it’s time to start promoting the rest of the users to the new version.

Setting Up a Canary Release

To set up a canary deployment, you create Ingress controllers to control the routing traffic and corresponding services. You will find these files in rollouts-demo/examples/base/canary. Additionally, you must create a file where you set up Argo Rollouts; this example uses the name canary-rollout.yaml.

You can find the file in rollouts-demo/examples/canary/canary-rollout.yamlcanary-rollout.yaml uses the Rollout resource to set up the canary deployment:

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: canary-demo
spec:
  replicas: 5
  revisionHistoryLimit: 1
  selector:
    matchLabels:
      app: canary-demo
  template:
    metadata:
      labels:
        app: canary-demo
    spec:
      containers:
      - name: canary-demo
        image: argoproj/rollouts-demo:blue
        imagePullPolicy: Always
        ports:
        - name: http
          containerPort: 8080
          protocol: TCP
        resources:
          requests:
            memory: 32Mi
            cpu: 5m
  strategy:
    canary:
      canaryService: canary-demo-preview
      steps:
      - setWeight: 20
      - pause: {}
      - setWeight: 40
      - pause: {duration: 10}
      - setWeight: 60
      - pause: {duration: 10}
      - setWeight: 80
      - pause: {duration: 10}

Let’s break down the code:

  • kind: Rollout identifies the resource as a Rollout.
  • name: canary-demo names the Rollout canary-demo.
  • revisionHistoryLimit: 1 restricts the stored revisions for the Rollout to one.
  • image: argoproj/rollouts-demo:blue sets the initial image for the canary-demo as argoproj/rollouts-demo:blue.
  • strategy specifies the deployment strategy to use, such as canary or blue-green, determining how updates are introduced to your environment.
  • canary defines this rollout as a canary deployment, allowing incremental updates with a subset of users before full rollout.
  • steps outlines a sequence of actions (steps) to be executed during the rollout process. In this example, eight steps were defined, but you can adjust the number of steps to your requirements.
  • setWeight adjusts the percentage of traffic directed to the new version (canary) during an update, facilitating gradual exposure to users. In this example, the initial step redirects 20 percent of the traffic to the new version and then halts. After promotion, the process resumes in 20 percent increments until the update is completed.
  • pause: {duration: TIME} temporarily halts the rollout process for a specified duration, providing a window to validate stability before proceeding. Not specifying the duration means pausing indefinitely until manually resumed (promotion). In this example, the first pause is indefinite, while the duration of the rest is ten seconds each. You could set each pause to indefinite or any combination that works for you.

This simple rollout works to explain the canary strategy in a tutorial; for a comprehensive list of all the fields available in the Rollout specification, you can check the docs.

Rolling Out a New Version Using a Canary Release

To implement the rollout, begin by navigating to the root directory of the demo repository:

cd /rollouts-demo/

Create a new namespace for the rollout, which is named canary in this example:

kubectl create ns canary

Run kustomize to deploy the rollout in your Kubernetes cluster:

kustomize build examples/canary | kubectl -n canary apply -f -

To verify that everything is working as expected, run the following command to list all rollouts in the canary namespace:

kubectl argo rollouts list rollouts -n canary

The output should be similar to the following:

NAME         STRATEGY   STATUS        STEP  SET-WEIGHT  READY  DESIRED  UP-TO-DATE  AVAILABLE
canary-demo  Canary     Healthy       8/8   100         5/5    5        5           5  

Now, you know the rollout is active, but you lack visibility into the ongoing processes. With Argo, you can monitor the progressive deployment using either the CLI or your browser. For a comprehensive view, the rest of the tutorial features split-screen screenshots, displaying the CLI output on the left and the Argo Rollouts Dashboard on the right.

If you want to achieve a similar result, you need to have at least two terminal windows open simultaneously, both at the rollouts-demo root directory.

In the first one, run the command that executes Argo’s dashboard UI:

kubectl argo rollouts dashboard

Leave the terminal open and, using your browser, go to this address: http://localhost:3100/rollouts

Then from the second terminal, run the following command:

kubectl argo rollouts get rollout canary-demo -n canary --watch

The command uses the Argo Rollouts get command with the --watch flag to provide a live view of the canary-demo rollout in the canary namespace.

The following image shows both outputs:

incremental

You are now at the starting point of the diagram shown in the last section. There are five replicas running the argoproj/rollouts-demo:blue image, which is currently the stable version.

To start the canary deployment, open a third terminal and use the command argo rollouts set image. This command sets the image argoproj/rollouts-demo:yellow as the new version you want to deploy:

kubectl argo rollouts set image canary-demo -n canary "*=argoproj/rollouts-demo:yellow"

After a few seconds, you should see the following:

update is paused

As specified in canary-rollout.yaml, 20 percent of users are using the new version (canary), and the rest are using the version that is still considered stable. The deployment is indefinitely halted as instructed.

To continue, you can either use the Promote button from the UI or run the command promote from the CLI:

kubectl argo rollouts promote canary-demo -n canary

You will see the execution of the rest of the steps with their respective ten-second pauses. Once the promotion of the canary version is finished, you will see an output similar to this:

fully deployed

Rolling Back a Canary Release

In this first run, everything went well. But what would happen if it didn’t?

One of the advantages of canary deployments is that you can roll back the process if the new version is not yet ready for production. To simulate such a scenario, run the following command:

kubectl argo rollouts set image canary-demo -n canary "*=argoproj/rollouts-demo:red"

It is the same command that you used for the first update; the difference is that it now points to the rollouts-demo:red image, which you can consider as another app version.

The output should now be like this:

process paused

The canary release is paused; however, instead of promoting the changes, roll back to the latest stable version. To do this, click the Abort button in the UI or use the command abort, which reverts all steps:

kubectl argo rollouts abort canary-demo -n canary 

Here’s what you will see once the rollback is completed:

rollback to previous image

Note that although all replicas are healthy and are running the previous version, the status is degraded. This is because the update process was unsuccessful, and Argo is waiting for your next action: either insist on another update or set the current version as stable. Do the latter with this command:

kubectl argo rollouts set image canary-demo -n canary "*=argoproj/rollouts-demo:yellow"

As expected, everything now looks good, and the system is ready for the next update:

healthy again

Implementing Blue-Green Deployments with Argo Rollouts

Similar to the approach used for the canary deployment, you’ll use the scenario from the diagram, starting with three replicas of the application running the stable version. As before, the progressive deployment unfolds in three stages: setting up the deployment, preparing the preview environment, and executing the update.

Setting Up a Blue-Green Deployment

This time, you will find all the required files in the rollouts-demo/examples/blue-green location:

  • bluegreen-ingress.yaml defines an Ingress resource named bluegreen-demo with annotations for Nginx settings.
  • bluegreen-preview-ingress.yaml defines an Ingress resource named bluegreen-demo-preview.
  • bluegreen-preview-service.yaml creates the corresponding bluegreen-demo-preview service.
  • bluegreen-service.yaml creates the corresponding bluegreen-demo service.
  • bluegreen-rollout.yaml defines the blue-green rollout.

As before, let’s focus on the file that matters, bluegreen-rollout.yaml:

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: bluegreen-demo
  labels:
    app: bluegreen-demo
spec:
  replicas: 3
  revisionHistoryLimit: 1
  selector:
    matchLabels:
      app: bluegreen-demo
  template:
    metadata:
      labels:
        app: bluegreen-demo
    spec:
      containers:
      - name: bluegreen-demo
        image: argoproj/rollouts-demo:blue
        imagePullPolicy: Always
        ports:
        - name: http
          containerPort: 8080
          protocol: TCP
        resources:
          requests:
            memory: 32Mi
            cpu: 5m
  strategy:
    blueGreen:
      autoPromotionEnabled: false
      activeService: bluegreen-demo
      previewService: bluegreen-demo-preview

The structure of the rollout is very similar to that of the canary deployment, but instead of steps, there are three fields that define the process:

  • autoPromotionEnabled: false disables automatic promotion of the new ReplicaSet to the active service. In this example, the process is indefinitely paused, just as it was done with the canary deployment. However, you can use a combination of autoPromotionEnabled: true and autoPromotionSeconds to automatically promote the rollout after a stipulated time.
  • activeService specifies the active service for the blue-green deployment.
  • previewService sets the preview service for the blue-green deployment.

Rolling Out a New Version Using a Blue-Green Deployment

Implementing the blue-green strategy involves similar steps to implementing canary releases.

Still in the rollouts-demo directory, create a new namespace for the rollout. Here, it’s called blue-green:

kubectl create ns blue-green

Run kustomize to deploy the rollout in your Kubernetes cluster:

kustomize build examples/blue-green | kubectl -n blue-green apply -f -

Check the rollout:

kubectl argo rollouts list rollouts -n blue-green

The output is similar to this:

NAME            STRATEGY   STATUS        STEP  SET-WEIGHT  READY  DESIRED  UP-TO-DATE  AVAILABLE
bluegreen-demo  BlueGreen  Healthy        -     -           3/3    3        3           3

To monitor the process, run the command:

kubectl argo rollouts get rollout bluegreen-demo -n blue-green --watch

Here’s what you see if you monitor the rollout:

stable versions

You are at the starting point, and all users use the rollouts-demo:blue image.

Begin an update using this command:

kubectl argo rollouts set image bluegreen-demo -n blue-green "*=argoproj/rollouts-demo:yellow"

Here’s the output:

6 replicas

The process is paused; take a moment to review what has happened. When you started the update, Argo created the preview environment. There are now six replicas, but only three are serving users with the current stable version. The other three replicas are waiting for your next action.

Promote the new version (preview) to stable using:

kubectl argo rollouts promote bluegreen-demo -n blue-green
3 of 3 replicas

As you can see, the update proceeded without any problems.

Rolling Back a Blue-Green Deployment

If you need to roll back to the previous version, you just have to start the process again. To do this, simply run the following:

kubectl argo rollouts set image bluegreen-demo -n blue-green "*=argoproj/rollouts-demo:blue"

Now, you have to promote the image, and you will return to the previous version:

process starts over

Overall, once you define your rollout and know the main commands, managing any strategy in Argo Rollouts is simple.

Canary vs. Blue-Green Strategies

In this tutorial, you learned how to implement both canary and blue-green strategies, so deciding on the optimal approach requires you to find a balance between your specific requirements and constraints and what each strategy has to offer you.

The choice between canary and blue-green implementations in Argo Rollouts depends on several factors, including app traits, available infrastructure, budget, downtime tolerance, and user experience goals. The key takeaway is that no strategy is flawless. Canary deployments mitigate risks by rolling out changes gradually but often require comprehensive monitoring. Blue-green deployments offer instant rollback and minimal downtime but are twice as resource-intensive, impacting costs.

aviator releases

Aviator.co | Blog

Subscribe

Be the first to know once we publish a new blog post

Join our Discord

Learn best practices from modern engineering teams

Get a free 30-min consultation with the Aviator team to improve developer experience across your organization.

Powered by WordPress