In this guide, you’ll learn how to set up both automated (manual) and automatic rollbacks using GitHub Actions and Heroku. Manual rollbacks will be triggered via the GitHub Actions UI, while automatic rollbacks will be triggered by alerts from your monitoring service. This setup ensures your application can quickly revert to a stable state if there are issues with a new deployment.

Prerequisites

• A GitHub repository with your code.
• An existing Heroku application.
• Your Heroku API key.
• GitHub Secrets set up with:
  • HEROKU_API_KEY
  • HEROKU_APP_NAME

Automated Rollbacks

Create a GitHub Actions workflow to handle manual rollbacks. Create a file named .github/workflows/rollback.yml with the following content:

name: Automated Rollback Heroku Deployment

on:
  workflow_dispatch:

jobs:
  rollback:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout code
        uses: actions/checkout@v2

      - name: Set up Python
        uses: actions/setup-python@v2
        with:
          python-version: '3.x'

      - name: Install Heroku CLI
        run: curl https://cli-assets.heroku.com/install.sh | sh

      - name: Install jq
        run: sudo apt-get install jq

      - name: Fetch Previous Heroku Release
        id: fetch_release
        env:
          HEROKU_API_KEY: ${{ secrets.HEROKU_API_KEY }}
          HEROKU_APP_NAME: ${{ secrets.HEROKU_APP_NAME }}
        run: |
          releases=$(heroku releases --json --app ${{ secrets.HEROKU_APP_NAME }} | jq -r '.[] | select(.description | contains("Deploy")) | .version')
          previous_release=$(echo $releases | awk '{print $(NF-1)}')
          echo "::set-output name=previous_release::v$previous_release"

      - name: Rollback Heroku Release
        env:
          HEROKU_API_KEY: ${{ secrets.HEROKU_API_KEY }}
          HEROKU_APP_NAME: ${{ secrets.HEROKU_APP_NAME }}
        run: |
          heroku releases:rollback ${{ steps.fetch_release.outputs.previous_release }} --app ${{ secrets.HEROKU_APP_NAME }}

This workflow fetches the list of releases and determines the second-to-last release version using the Heroku CLI, jq and awk commands. The rollback is then performed using this version. You can manually input the version you want by adjusting the workflow. Once the changes are pushed to GitHub, you can trigger the workflow from the GitHub Actions tab.

Automatic Rollbacks

For automatic rollbacks, use a monitoring service like Papertrail, Loggly, or New Relic to send alerts to a webhook. This webhook triggers a GitHub Actions workflow to perform the rollback. Create a file named .github/workflows/automatic_rollback.yml with the following content:

name: Automatic Rollback Heroku Deployment

on:
  repository_dispatch:
    types: [rollback]

jobs:
  rollback:
    runs-on: ubuntu-latest

    steps:
      - name: Rollback Heroku Release
        env:
          HEROKU_API_KEY: ${{ secrets.HEROKU_API_KEY }}
          HEROKU_APP_NAME: ${{ secrets.HEROKU_APP_NAME }}
        run: |
          heroku releases:rollback ${{ github.event.client_payload.release }} --app ${{ secrets.HEROKU_APP_NAME }}

Set Up Monitoring and Alerts

First, set up Heroku to send logs to Papertrail. Go to your Heroku dashboard, select your application, and enter the Papertrail URL in the “Overview” tab.

Note: If you don’t see Papertrail you need to first install and provision Papertrail on the app you are working with.

Create a Papertrail Alert

Now, navigate to the log search page and search for logs related to your Heroku application by entering the name of your app on Heroku. Once you have your search query set up to find the relevant logs, save the search by clicking the “Save Search” button.

Configure the Alert

Now you can create an alert on this saved search. In the section we will use a webhook POST request to a FASTAPI backend that we will build in the next section.

Webhook Configuration

Create a webhook endpoint that triggers the GitHub repository dispatch event. Here’s an example using FastAPI:

from fastapi import FastAPI, Request, HTTPException
import requests

app = FastAPI()

GITHUB_TOKEN = "your_github_token"  # Replace with your GitHub personal access token
OWNER = "your_github_username"  # Replace with your GitHub username
REPO = "your_repository"  # Replace with your GitHub repository
EVENT_TYPE = "rollback"
HEROKU_API_KEY = "your_heroku_api_key"  # Replace with your Heroku API key
HEROKU_APP_NAME = "your_heroku_app_name"  # Replace with your Heroku app name

@app.post("/webhook")
async def webhook(request: Request):
    data = await request.json()

    try:
        # Example logic to parse and verify the alert data
        alert_name = data.get('alert_name')
        alert_severity = data.get('severity')
        error_rate = data.get('error_rate')

        # Example condition: Trigger rollback if the error rate exceeds 5% and severity is critical
        if alert_name == "High Error Rate" and alert_severity == "critical" and error_rate > 5:
            # Fetch the list of Heroku releases
            releases_response = requests.get(
                f"https://api.heroku.com/apps/{HEROKU_APP_NAME}/releases",
                headers={
                    "Authorization": f"Bearer {HEROKU_API_KEY}",
                    "Accept": "application/vnd.heroku+json; version=3"
                }
            )
            
            if releases_response.status_code != 200:
                raise HTTPException(status_code=releases_response.status_code, detail="Failed to fetch Heroku releases")
            
            releases = releases_response.json()
            if len(releases) < 2:
                raise HTTPException(status_code=400, detail="Not enough releases to perform rollback")
            
            # Determine the version before the current version
            previous_release = releases[-2]['version']
            
            # Trigger GitHub Action for rollback
            response = requests.post(
                f"https://api.github.com/repos/{OWNER}/{REPO}/dispatches",
                headers={
                    "Authorization": f"Bearer {GITHUB_TOKEN}",
                    "Accept": "application/vnd.github.v3+json",
                },
                json={
                    "event_type": EVENT_TYPE,
                    "client_payload": {"release": f"v{previous_release}"},
                },
            )
            
            if response.status_code == 204:
                return {"status": "success", "message": "Rollback triggered successfully"}
            else:
                return {"status": "error", "message": response.json()}
        else:
            return {"status": "ignored", "message": "Alert conditions not met for rollback"}
    except KeyError as e:
        raise HTTPException(status_code=400, detail=f"Missing expected field: {e}")
    except Exception as e:
        raise HTTPException(status_code=500, detail=str(e))

This FastAPI application listens for incoming webhook requests on /webhook endpoint and verifies the alert conditions. In this case, if the alert is named “High Error Rate“, has a severity level of “critical“, and the error rate exceeds 5%. If these conditions are met, it proceeds to initiate a rollback process. 

Note: The alert conditions may vary based on the data you are receiving from the monitoring service.

When the alert conditions are met, the application fetches the list of releases from Heroku using the Heroku API. If there are sufficient releases, the code determines the version before the current release, and sends a request to GitHub’s repository dispatch API to trigger a GitHub Action with an event type “rollback”, passing the previous release version as a payload. This triggers a predefined workflow in the GitHub repository to perform the rollback.

The application handles potential errors and exceptions that may occur during the process.

You can deploy this FastAPI webhook to a cloud service like Heroku, AWS Lambda function, Google’s cloud run function or any other hosting provider that supports Python applications, we used Heroku for this guide. 

Note: If you are using Heroku, you should set up a separate Heroku app for this to avoid any failure when the main app deployment fails, causing the webhook to also fail.

Returning to the alert setup in the previous section, select “Webhook” as the alert destination and enter the URL of your FastAPI webhook endpoint. For example, mine is https://rollback-trigger-b89aab9d5465.herokuapp.com/webhook.

To test, simulate a bad alert with:

curl -X POST https://your-fastapi-app.herokuapp.com/webhook -H "Content-Type: application/json" -d '{
    "alert_name": "High Error Rate",
    "severity": "critical",
    "error_rate": 6
}'

Check the GitHub Actions tab to confirm the rollback was triggered.

Conclusion

With this guide, you can manage both automated (one-click manual) and automatic rollbacks using GitHub Actions and Heroku. You can extend this setup by securing sensitive data and tailoring it to your specific use case.