Trunk-Based Development in Microservices
Trunk-based development (TBD) is an effective approach for version control, known for reliable, quick deployments with short lived branching. This approach encourages continuous integration and reduces merge conflicts by focusing on a single primary branch where developers commit frequently.
TBD is especially beneficial for microservices, as teams manage multiple services that must collaborate and be updated independently. The continuous integration of changes makes TBD ideal for environments requiring swift updates while maintaining stability.
In contrast to Trunk-based development, architectures like feature branching and Gitflow are more complex to set up and work with.
This blog will explore:
- How trunk-based development compares to feature branching and Gitflow, and how it addresses their limitations
- Core principles of TBD, and its benefits over other approaches.
- A hands-on example which will include setting up a repository, committing to the main branch, building a simple microservice, writing unit tests, and integrating CI/CD pipelines.
- Additionally, we’ll also look at using
- Short-lived feature branches
- Conflict resolution
- Rollback techniques
Understanding trunk-based development
Trunk-based development centers around making frequent, modest commits to the main branch. Developers work directly on the main branch instead of creating separate feature branches, enabling continuous integration. This approach maintains project stability and allows early problem detection.
In comparison, Feature branching makes distinct branches for every feature, which can remain isolated for weeks. Although this keeps things structured, it frequently results in bigger, more complex mergers and increases conflicts. Additionally, insufficient testing on isolated branches may delay bug detection.
Similarly, Gitflow uses multiple branches for development, features, and releases. Although it offers organizational structure, it can slow development with strict branch rules, creating bottlenecks and delays. This can be challenging in fast-paced environments where quick updates are needed.
With TBD, developers integrate changes into the main branch as soon as they’re ready, keeping changes small and testing them continuously. This reduces merge conflicts and maintains stability, enabling faster releases. TBD particularly suits fast-paced environments where quick, regular updates across multiple services are essential. By avoiding long-lived branches, TBD allows for smoother integration and quicker deployments.
Trunk-based development for microservices
Trunk-based development focuses on small, short-lived branches that merge into a single main branch. Integrating incremental changes reduces conflict by catching issues early, leading to less rework. This approach excels in microservices environments where frequent service interactions make updates potentially disruptive.
TBD streamlines update coordination across services as all teams work against a single branch. Teams discover incompatibilities immediately when changes occur, rather than during the end of a sprint or branch merges. Continuous integration ensures every commit is tested, enabling early bug detection.
For example, a retail company uses microservices for inventory, payments, and customer accounts. A payments team implementing a new payment method can integrate changes progressively. Every change is tested automatically against dependent services like inventory updates and customer purchase history. Through small, continuous updates, TBD enables smooth feature deployment while maintaining service synchronization.
Hands-on using trunk-based development
Let’s walk through the process of setting up a simple “Hello World” microservice while following the principles of Trunk-Based Development. This hands-on example will demonstrate how to commit changes frequently, integrate continuously, and use automated testing, all while maintaining a stable and efficient codebase.
Creating the repository
The first step is to create a new Git repository with a single main branch. Let’s say that our project directory is called TBD-Test, use the following command to create a folder and initialize Git in the repository:
mkdir TBD-Test
git init
We will also be adding a remote link to the upstream repository, which is housed on GitHub.
git remote add <link-to-upstream-repository>
For more details on GitHub and remote repository, check out this documentation by GitHub.
In TBD, developers work in short-lived feature branches that are quickly reviewed and merged into the main branch. Development prioritizes maintaining a stable, current main branch. Limiting branch lifetimes simplifies codebase stability and accelerates change integration.
Instead of direct commits to the main branch, branch protections and restrictions ensure changes undergo necessary reviews. Pull requests serve as the standard method for merging changes, enabling thorough review, automated checks, and maintaining codebase stability before merging.
If you are not familiar with Git, consider their quick tutorial on working with branches.
Project structure
Once the repository is set up, define a simple project structure to ensure organized building, testing, and deployment. We’ll work with two microservices: a “Hello World” API and a “Bye World” API.
Once the repository is set up, it’s time to define a simple structure for your project. This keeps everything organized and ensures that the application can be easily built, tested, and deployed. Here we will be working on two microservices, one will be called the “Hello World” API and the other will be called the “Bye World” API.
Our Project structure consists of:
- hello-api/app/ folder – Contains the core application code which is a simple flask program which will respond with “Hello World” upon request.
- bye-api/app/ folder – Contains the core application code which is a simple flask program which will respond with “Bye” upon request.
- hello-api/test/ folder – Responsible for unit and integration tests for your hello-api service.
- bye-api/test/ folder – Responsible for unit and integration tests for your bye-api service.
- README.md – Basic documentation for setting up and running the service.
So the project contents look like this:
.
├── bye-api
├── hello-api
└── README.md
Frequent commits
Since we are using TBD, each commit is a small, isolated task to smooth integration and catch issues early. So these changes would be committed via a short-lived feature branch called file-structure
, and we create a pull request from this newly created file structure branch to main. For this we will use the GitHub CLI tool:
gh pr create --title "Basic file structure" --body ""
Developing the “Hello World” microservice
We’ll use Flask to build the “Hello World” microservice, but the same principles can also be applied to other frameworks and programming languages. The service will have a simple /hello
endpoint that responds with the message “Hello, World!”.
For this, we will go to the app folder and create two files named __init__.py and routes.py.
cd hello-api
mkdir app
touch __init__.py
touch routes.py
The __init__.py
file is typically used for initializing your Flask app and any configurations, while the logic for specific routes and functionality can go in separate modules.
Contents of __init__.py
from flask import Flask
app = Flask(__name__)
from app import routes
The routes.py
contains the logic for all the routes. Here’s the code to build the “Hello World” service. We will add this to the routes.py file.
Contents of routes.py
from app import app
@app.route('/hello')
def hello_world():
return "Hello, World!"
This is the simplest version of the service and is ready to be tested. Now, we will commit to this change to a new branch hello-feature
:
git checkout -b hello-feature
git add .
git commit -m "Add the /hello endpoint to return Hello World message"
git push origin hello-feature
Just like before, create a new pull request:
gh pr create --title "Add /hello endpoint" --body ""
Automated testing
Testing is a key part of trunk-based development. To maintain stability, we must ensure that every change is automatically tested before merging into the main branch. Let’s add a simple unit test to verify that the /hello endpoint works as expected.
First, navigate to the Root directory of the Hello World microservice and create a test file:
app-test.py
import unittest
from app import app
class TestHelloWorld(unittest.TestCase):
def setUp(self):
self.app = app.test_client()
def test_hello(self):
response = self.app.get('/hello')
self.assertEqual(response.data.decode(), "Hello, World!")
if __name__ == '__main__':
unittest.main()
After this is completes, we will add it to the already stacked list of commits:
git checkout -b hello-test
git add .
git commit -m "Add the test for /hello endpoint"
git push origin hello-test
Create a new pull request:
gh pr create --title "Add a test for /hello endpoint" --body ""
Continuous integration pipeline
Next, we set up a CI pipeline using GitHub Actions that runs these tests automatically every time a commit is made to the main branch. This ensures that the code remains stable, and that any issues are caught early.
For this we will create a .github/workflows
folder in the root directory, and this will add a workflow file. To know more about GitHub Actions and workflow, check out the documentation.
Now, we will navigate to the .github/workflows
folder and create a hello-ci.yaml
file. This file will specify how the tests should run on each commit to the main branch.
hello-ci.yaml
name: Hello CI Pipeline
on:
push:
branches:
- main
pull_request:
branches:
- main
jobs:
test:
runs-on: ubuntu-latest
services:
docker:
image: python:3.9
options: --network host
steps:
- name: Checkout code
uses: actions/checkout@v2
- name: Set up Python
uses: actions/setup-python@v2
with:
python-version: '3.9'
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install flask
pip install -r requirements.txt || true
- name: Run tests
run: |
python -m unittest discover -s hello-api/test
This CI pipeline is triggered on every pull requests targeting the main branch. To maintain code quality and stability, branch protection rules can be configured to ensure that these pull requests cannot be merged until the CI pipeline completes successfully. This guarantees that all automated tests and checks pass before any changes are integrated into the main branch.
In the above example, the pipeline runs on the Ubuntu-latest environment using Python 3.9. It begins by checking out the code, installing dependencies including Flask
and others listed in requirements.txt
, and finally running tests with the unittest
framework to ensure stability and prevent new issues.
Short-lived feature branches
In trunk-based development, frequent merges to the main branch help spot and fix conflicts early, keeping the codebase stable. Aviator’s av CLI manages conflicts effectively with stacked PRs by creating and managing Short-Lived Feature Branches. To setup CLI, read the quick start guide.
What are stacked PRs
Stacked PRs involve creating multiple small, focused pull requests that build on top of each other. Each PR includes a specific change, making it easier to review and test. Instead of working on a large, complex PR, developers can submit manageable updates one at a time, which reduces integration problems. Stacked PRs help keep the main branch stable while allowing incremental changes to be added without causing delays or conflicts.
For each small change, create a new short-lived branch using:
av stack branch <branch_name>
And then synchronize changes across the entire stack using:
av stack sync
This lets you work on each change separately and synchronize changes as you made improvements anywhere in the stack, keeping things simple and reducing the risk of conflicts.
Example
We will now show how to use stacked PRs to improve the trunk-based development workflow. In this, you will create a temporary branch, add specific changes, test them, and then merge the branch back into the main branch once all checks pass.
You can check out the official GitHub page for installation instructions for your specific operating system. After you are done installing the CLI tool, link it to GitHub. In order to interact with GitHub, av uses the GitHub API token. If you have a GitHub CLI installed, av will automatically use the token from the GitHub CLI. It is recommended to install both.
To explain the branching example, we will work on the other microservice: “Bye World”. We will use av CLI to create a new branch:
av stack branch bye-endpoint
We will again go to the app folder and create a two files named __init__.py
and routes.py
.
__init__.py
from flask import Flask
app = Flask(__name__)
from app import routes
routes.py
from app import app
@app.route('/bye')
def bye_world():
return "Bye, World!"
Once the code is ready, commit the change to the local repository.
$ git add .
$ git commit -m "Add /bye endpoint to the microservice"
After this is done, lets open a PR for this change, for this, we will use the av CLI tool.
av pr create
This command creates or updates a PR in the stacked branch you created.
Next, we will add a unit test to ensure the new functionality works as expected. For this we will create another stacked branch from the current bye-endpoint branch. For this you can use the following command:
av stack branch test-bye-endpoint
Then navigate to the Root directory of the Hello World microservice and create a test directory with app-test.py
file.
app-test.py
import unittest
from app import app
class TestByeWorld(unittest.TestCase):
def setUp(self):
self.app = app.test_client()
def test_bye(self):
response = self.app.get('/bye')
self.assertEqual(response.data.decode(), "Bye, World!")
if __name__ == '__main__':
unittest.main()
When the test is ready, commit the change to the local repository.
git add .
git commit -m "Add test for /bye endpoint"
After this is done, let’s open a PR for this new change, for this, we will use the av CLI tool.
av pr create
This will create an additional PR that is stacked on top of the PR created in the previous step. This ensures that each change can be independently reviewed and revised while keeping the entire stack in sync.
Now, we will create a GitHub Action CI for the Bye World microservice as well. For this we will create another stacked branch from the current test-bye-endpoint branch. For this you can use the same av stack branch
command:
av stack branch bye-ci
Next, we will navigate to the .github/workflow folder and create a bye-ci.yaml file for the Bye World Microservice as well.
bye-ci.yaml
name: Bye CI Pipeline
on:
push:
branches:
- main
pull_request:
branches:
- main
jobs:
test:
runs-on: ubuntu-latest
services:
docker:
image: python:3.9
options: --network host
steps:
- name: Checkout code
uses: actions/checkout@v2
- name: Set up Python
uses: actions/setup-python@v2
with:
python-version: '3.9'
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install flask
pip install -r requirements.txt || true
- name: Run tests
run: |
python -m unittest discover -s bye-api/test
And commit the change to the local repository, and open a PR:
git add .
git commit -m "Add CI Pipeline"
av pr create
Now, navigate to your repository on GitHub and open the pull request from the feature branch to the main. This is where the CI pipeline will automatically run tests to verify the new feature’s stability.
Once all tests pass in the CI pipeline, you can review the code and merge the entire stack together or separately into the main branch.
Once these PRs are merged, you can easily delete the temporary branch locally and remotely using the sync command:
av stack sync --rebase-to-trunk
Syncing and keeping branches up-to-date
As you work on multiple stacks, you can use:
av stack sync --all
This will update and rebase all the branches across all your stacks regularly. This command ensures that the commits in any part of the stack are propagated across the entire stack, ensuring that each branch is up to date with the latest code and avoiding surprises later on.
Visualizing and organizing stacked PRs:
To see an overview of all your branches and their order, use:
av stack tree
This gives you a clear view of how branches relate to each other, helping you manage and merge them in the right order. If you need to adjust the order of branches or commits, you can use:
av stack reorder
Branch protection
Branch protection rules in GitHub ensure that only validated changes are merged into the protected branches. With conditional CI pipelines, you should focus on requiring specific checks to pass before allowing merges.
Running separate CIs for different microservices
To run separate CI workflows for different microservices within the same monorepo, we can modify our GitHub Actions workflow files to use path filters. This ensures that CI runs only when relevant files are changed.
First, let’s update the workflow files:
.github/workflows/hello-ci.yaml
name: Hello CI Pipeline
on:
push:
branches:
- main
paths:
- 'hello-api/**'
pull_request:
branches:
- main
paths:
- 'hello-api/**'
jobs:
test:
runs-on: ubuntu-latest
# Add status check name for branch protection
name: Hello API Tests
steps:
# Previous steps remain the same
# Add explicit status check
- name: Report status
if: always()
run: |
if [ ${{ job.status }} == 'success' ]; then
exit 0
else
exit 1
fi
.github/workflows/bye-ci.yaml
name: Bye CI Pipeline
on:
push:
branches:
- main
paths:
- 'bye-api/**'
pull_request:
branches:
- main
paths:
- 'bye-api/**'
jobs:
test:
runs-on: ubuntu-latest
# Add status check name for branch protection
name: Bye API Tests
steps:
# Previous steps remain the same
# Add explicit status check
- name: Report status
if: always()
run: |
if [ ${{ job.status }} == 'success' ]; then
exit 0
else
exit 1
fi
This ensures that a status check is reported as success when the file is not modified for the related service. This status check can then be used in the branch protection rules.
Branch protection rules with conditional CIs
Now, let’s set up branch protection rules that work with our conditional CI setup:
- Go to your repository settings.
- Navigate to “Branches” under “Code and automation”.
- Click “Add classic branch protection rules” under “Branch protection rules”.
- For “Branch name pattern,” enter main. Then click on the checkboxes with these rules:
- Require a pull request before merging
- Require status checks to pass before merging.
- Select both the status checks “Bye API Tests” and “Hello API Tests” as required checks
With this, your branch protection will be applied.
Integrating Aviator MergeQueue
By integrating the Aviator’s MergeQueue, there will be an additional layer of safety, making it easy to incorporate new features into the system, which has an orderly way of merging. MergeQueue ensures that any changes that are merged in mainline are validated with the latest code changes.
It combines all queued changes sequentially, executing CI tests over the resultant code base, and later merging such changes to avoid the problems of broken builds associated with conflicting modifications. It enhances the integration of these changes while preserving the state of branches and offers further assurance that only carefully tested modifications are integrated.
.github/workflows/aviator.yaml
version: 1.0.0
merge_rules:
labels:
trigger: mergequeue
skip_line: skip-line
require_all_checks_pass: true
publish_status_check: true
ci_timeout_mins: 20
preconditions:
use_github_mergeability: false
merge_mode:
type: parallel
parallel_mode:
max_parallel_builds: 3
check_mergeability_to_queue: true
override_required_checks: []
require_all_draft_checks_pass: true
merge_strategy:
name: squash
Just like above, create a new stacked branch and push it to your remote repository.
Remember to regularly sync your stacked branches with the main to catch conflicts early:
git fetch origin main
av stack sync --rebase-to-trunk
av stack push
Once the stack is ready to merge, you can post a GitHub comment on the PR at the top of your stack to merge the entire stack:
/aviator stack merge
Aviator MergeQueue start the validation and eventually merges all stacked PRs once validation is complete.
Deployments and rollbacks
Despite frequent commits and testing, merge conflicts and dependency errors can occur. A reliable rollback process is essential for main branch stability. Aviator Releases simplifies rollback management and quick fix deployment.
Aviator Releases provides centralized release management, organizing deployment pipelines and automating version control. Teams can manage rollbacks, cherry-pick changes, and implement strategies like canary releases and blue-green deployments from one platform.
Aviator’s integration with trunk-based development ensures main branch stability and smooth integration. Its tools for quick fixes, structured rollbacks, and advanced testing strategies help teams confidently manage software updates while minimizing risk.
Best practices for TBD in microservices
Short-lived feature branches: Keep branches short to avoid code drift
In trunk-based development, feature branches should be brief and minimal. This prevents significant divergence between main and feature branches. Extended feature branches risk becoming outdated and creating integration issues. Short feature branches allow developers to work on isolated changes and merge quickly, minimizing code drift and maintaining codebase consistency.
Our example demonstrated several main branch commits and two pull requests from feature-stacked branches.
This is what our commit graph looks like, and we can further improve these individual branches if required.
Incremental code reviews: Tips for frequent, small reviews
Regular review of small, incremental changes is preferable to reviewing completed large features. This speeds up the review process and enables early issue detection without development delays. Frequent commits facilitate regular code reviews, keeping changes small and manageable.
As shown in our example, we created two pull requests:
You and your teammates can review these pull requests and make incremental changes. After committing new changes, sync all stacked PR branches with this av command:
av stack sync --all
This command will sync all the branches which you are using for stacked PRs by rebased them with their parent branches and pushing them to the remote repository for you.
Selective testing in pipelines
Continuous integration is a key feature of Trunk-based development, running tests in the CI/CD pipeline after each main branch commit.
The GitHub actions workflow we created automates this pipeline. Tests trigger automatically when code is pushed to main or when pull requests are created. View test results in your GitHub repository’s action tab.
Not every change requires running all tests. Selective testing reduces pipeline time by focusing on relevant tests only. This provides faster feedback and confirms new changes don’t break existing functionality, while avoiding pipeline overload.
Simple reverts
In trunk-based development, simple reverts maintain stability when issues arise despite continuous integration. Quick reversion of problematic changes in the trunk branch enables fast recovery. Automated rollbacks and clear procedures prevent minor issues from disrupting development. Small, isolated commits simplify error identification during reverts.
Ensuring backward compatibility: Strategies to ensure compatibility
In microservices, backward compatibility is essential for service interactions. API and data contract versioning ensures older services can communicate with newer versions. Feature toggles enable incremental deployment, allowing production testing of new features while maintaining compatibility. This approach enables smooth system updates without disrupting existing functionality.
Efficient deployment in microservices
Trunk-based development requires stable deployments with minimal downtime and quick rollback capabilities. Continuous deployment integrates naturally, delivering recent changes to staging or production. Blue-green deployments and canary releases enable gradual deployment, minimizing risk and facilitating rollbacks. Feature toggles allow partial deployment while maintaining system stability.
Immutable infrastructure and monitoring systems support efficient deployments and rapid recovery. Standard APIs and backward compatibility reduce service coupling, enabling smooth system-wide updates.
Conclusion
Trunk-based development excels in fast-paced environments requiring stable, frequent updates. By focusing on small, frequent commits to the main branch, TBD minimizes merge conflicts and maintains codebase stability. This approach eliminates bottlenecks from isolated branches or complex branching strategies, enabling real-time service evolution through continuous updates. In microservices architecture, TBD’s continuous integration helps maintain service alignment and interaction, creating an efficient workflow that promotes code quality and rapid delivery.
Short-lived feature branches and quick fixes enhance TBD’s benefits without adding overhead. Tools like Aviator’s av CLI help manage stacked pull requests, allowing developers to focus on clear, incremental changes that preserve main branch stability. Automated testing and continuous integration verify each commit before merging, enabling early bug detection and resolution. TBD empowers teams to make gradual improvements while avoiding complex merges and long-lived branch risks, making it ideal for dynamic, collaborative projects requiring reliable scaling.
FAQs
What is trunk-based development?
Trunk-based development is a version control management practice where developers merge small, frequent updates to a core “trunk” or main branch. Since it streamlines merging and integration phases, it helps achieve CI/CD and increases software delivery and organizational performance.
How does trunk-based development differ from traditional Git workflows?
Trunk-based development differs from traditional Git workflows like feature branching and GitFlow by focusing on a single main branch for all development. Developers commit changes directly to the main branch in small, frequent updates, as opposed to working in long-lived feature branches. This encourages faster integration and reduces the risk of merge conflicts, ensuring a more stable and quickly evolving codebase.
What are some best practices for managing dependencies in TBD?
In trunk-based development, managing dependencies is crucial to prevent bottlenecks. Best practices include using dependency management tools to lock versions and ensure compatibility, and avoiding tightly coupling services in a way that requires frequent changes across services. Additionally, keeping dependencies up to date with automated pipelines and minimizing direct dependencies between services helps maintain flexibility and speed.
What is the difference between feature-based and trunk-based development?
Feature-based development involves working on separate branches for individual features, often for extended periods before merging them into the main branch. This can lead to integration challenges and conflicts when merging. In contrast, trunk-based development encourages developers to commit small changes directly to the main branch, resulting in continuous integration. This minimizes the risks of code drift and merge conflicts, enabling faster releases and a more stable codebase.
Does Google use trunk-based development?
The development workflows are also mainly Git-based. Two of the most widely used workflows are feature-based and trunk-based development (aka TBD). Software development teams at Netflix, Google, Facebook, and other tech giants use these workflows.