Controlled deployments allow teams to minimize disruptions and ensure that new releases improve an application and don’t introduce problems. They allow you to gradually expose new features to users, and give you room to monitor and validate performance, before a full-scale rollout.

This article will walk you through the essentials of rollouts and rollbacks. It will introduce Flux CD and explain how to implement both rollouts and rollbacks with Flux CD — covering setup and configuration, and including an example of a canary deployment strategy.

It will also explain how to integrate Flagger with Flux CD, to further enhance deployment strategies, especially for progressive delivery.

Prerequisites

Before starting, make sure you have the following:

• A Kubernetes cluster.
• kubectl installed and configured to interact with your cluster.
• A Git repository for storing Kubernetes manifests.

Setting up Flux CD

First, you’ll install Flux CLI:

curl -s https://fluxcd.io/install.sh | sudo bash

Flux needs your GitHub credentials to log in and perform some actions on your repository. Export your GitHub personal access token and username:

export GITHUB_TOKEN=<your-token> #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
export GITHUB_USER=<your-username>

The flux bootstrap GitHub command sets up the Flux controllers on a Kubernetes cluster and sets them to synchronize the cluster’s state with a Git repository. It also uploads the Flux manifests to the Git repository and sets up Flux CD to automatically update itself based on changes in the Git repository.

To do this, run the following command:

echo $GITHUB_TOKEN | flux bootstrap github \
  --owner=$GITHUB_USER \
  --repository=<github-repo-name> \
  --branch=main \
  --path=./flux-cluster \
  --personal
  --private=false

The command above does the following:

• Creates a Git repository, <github-repo-name> (in this example, roll-flux), on your GitHub account.
• Adds Flux component manifests to the repository.
• Deploys Flux components to your Kubernetes cluster. You can run kubectl get all -n flux-system to check out the components.
• Configures Flux components to track the path /flux-cluster in the repository.
• Uses the --private=false flag to create a public repository.

Now clone the repository you created (in my example, roll-flux) to your local machine:

git clone git@github.com:$GITHUB_USER/roll-flux.git
cd roll-flux

Now run the following to create a GitRepository manifest pointing to the https://github.com/khabdrick/roll-flux/ main branch. 

flux create source git nginx \
  --url=https://github.com/khabdrick/roll-flux \
  --branch=main \
  --interval=2m \
  --export > ./flux-cluster/rollflux-source.yaml

In the command above:

• A Git repository named nginx is created.
• The source-controller checks the Git repository every two minutes, as indicated by the --interval flag.
• It clones the main branch of the https://github.com/khabdrick/roll-flux/ repository.
• When the current Git repository revision differs from the latest fetched revision, a new artifact is archived.

After the command is run, you should have the corresponding file rollflux-source.yaml.

Now deploy the roll-flux application using GitOps. Configure Flux CD to build and apply the kustomize directory located in the roll-flux repository. 

Use the following flux create command to create a Kustomization that applies the roll-flux manifest files:

flux create kustomization nginx \
  --target-namespace=default \
  --source=quotes \
  --path="./kustomize" \
  --prune=true \
  --wait=true \
  --interval=10m \
  --retry-interval=2m \
  --export > ./flux-cluster/nginx-kustomization.yaml

Add the manifest to your Git repository:

git add -A
git commit -m "Add Kustomize settings"
git push origin main

Understanding rollouts

A rollout is the process of deploying a new version of an application to your Kubernetes cluster. Flux CD uses kustomize or helm controllers to manage rollouts.

Let’s say we have a simple Nginx deployment. We’ll create a file for our deployment and the kustomization.yaml file in the ./kustomize directory. Take note of the file path in the repo.

The deployment file we have in the repository contains a Kubernetes Service and a Deployment for an Nginx application. The Service of type LoadBalancer named nginx directs incoming TCP traffic on port 80 to the pods labeled app: nginx on the same port, facilitating external access to the Nginx application. The Deployment named nginx specifies running three replicas of the Nginx container with version 1.19.2. 

Performing a rollout

To perform a rollout, update the image version in your Kustomize configuration and push the changes to your Git repository:

# Kustomize/deployment.yaml
...
   spec:
     containers:
     - name: nginx
       image: nginx:1.19.3
       ports:
       - containerPort: 80

Commit and push the changes:

bash git add base/deployment.yaml
git commit -m "Update nginx to 1.19.3"
git push origin main

Flux will automatically sync the changes and deploy your application. You can check by running the following command:

kubectl -n default get deployments,services

Understanding rollbacks

A rollback is the process of reverting to a previous version of an application when a new deployment fails or causes issues. Flux CD makes rollbacks straightforward by allowing you to undo changes in your Git repository.

If the new version of Nginx (1.19.3) causes issues, you can easily roll back to the previous version (1.19.2) by undoing the change, or reverting to the previous version, in your Git repository:

git revert <commit-hash-of-update>
git push origin main

Flux CD will automatically detect the rollback in your Git repository and apply the previous version (1.19.2) to your Kubernetes cluster.

Advanced rollout strategies with Flux CD

Flux CD supports advanced rollout strategies through integration with Flagger. Flagger provides automated canary deployments, A/B testing, and blue/green deployments.

In this section, we will use Flagger to create a canary deployment for the Nginx application and gradually shift traffic to the new version while monitoring metrics. If the new version meets the specified thresholds, the canary becomes the primary deployment.

Canary deployment with Flagger

First, install Flagger with the following command:

kubectl apply -f https://raw.githubusercontent.com/fluxcd/flagger/main/artifacts/flagger/crd.yaml

Create a canary resource:

apiVersion: flagger.app/v1beta1
kind: Canary
metadata:
 name: nginx
spec:
 targetRef:
   apiVersion: apps/v1
   kind: Deployment
   name: nginx

 service:
   port: 80

 analysis:
   interval: 1m
   threshold: 10
   metrics:
     - name: request-success-rate
       threshold: 99
       interval: 1m

   webhooks:
     - name: load-test
       type: pre-rollout
       url: http://flagger-loadtester.test/
       timeout: 30s

Push to GitHub:

git add kutomize/deployment.yaml 
git commit -m "added flagger to Kustomize"
git push origin main

This YAML configuration defines a Flagger canary resource for the nginx deployment in the default namespace. It specifies that Flagger should monitor the nginx deployment (with API version apps/v1) and expose it on port 80.

The canary analysis will run every minute (interval: 1m) for a total of 10 iterations (iterations: 10) and allow a maximum of 10 failed checks (threshold: 10). Each iteration checks whether the request-success-rate metric meets or exceeds the 99% success rate (threshold: 99) over a one-minute interval (interval: 1m). Additionally, a pre-rollout webhook (type: pre-rollout) is configured to run a load test via the specified URL (http://flagger-loadtester.test/) with a timeout of 30 seconds (timeout: 30s). If the canary deployment fails to meet the specified metrics or the webhook test, Flagger will rollback to the previous stable version.

Canary deployment analysis with Flagger

To trigger the canary analysis, you can update the deployment (Kustomize/deployment.yaml).

In your Kustomize/deployment.yaml, update the Nginx image version to a new tag — for example,  nginx:1.19.4 — and push to GitHub:

...
    spec:
      containers:
      - name: nginx
        image: nginx:1.19.4
        ports:
        - containerPort: 80

How does Flagger work?

1. Monitoring the deployment: Flagger continuously monitors the target deployment specified in the targetRef of the canary resource. In our example, this is the nginx deployment.
2. Detecting new versions: When a new version of the application is pushed to the container registry and the deployment manifest is updated (for instance, when a new image tag is applied), Flagger detects this change. This is typically done by changing the image field in the deployment specification.
3. Initiating canary analysis: Once a change is detected, Flagger starts the canary analysis process. It creates a canary version of the deployment and routes a small percentage of traffic to it while monitoring key performance indicators (KPIs) such as success rate and request duration.

Rollback with Flagger

Let’s assume the new version of the Nginx deployment (nginx:1.19.4) is failing to meet the success rate criteria:

1. Flagger detects that the success rate for nginx:1.19.4 has fallen to 95%.
2. The threshold is set to 99%, so Flagger initiates a rollback.
3. Flagger reverts the deployment to the previous stable version .
4. The traffic is shifted back to the stable version, ensuring the stability of the application.

Conclusion

Flux CD simplifies the process of rollouts and rollbacks in Kubernetes clusters by leveraging GitOps principles. By continuously monitoring your Git repository, Flux CD ensures that your cluster state matches the desired state defined in your manifests.

By following the steps outlined in this article, you can confidently manage application updates and rollbacks.

To deepen the knowledge you’ve gained from this tutorial, explore advanced concepts such as integrating Flagger with other monitoring tools like Prometheus for more granular metrics.