Rebuilding Google’s Engineering Productivity (EngProd)
For years, Google has made significant investments in engineering productivity via EngProd, but it wasn’t until recently that much of the tech industry started to focus on it. The recent rise of remote work and the evolution of AI have pushed this discipline into the spotlight.
As a team of former Googlers, we started looking for tools to fill the gap in the real-world. Since Google builds everything in-house, not everything is easily replicable. That led us to create Aviator — rebuilding Google’s Engineering Productivity (EngProd) on the modern engineering stack to solve collaboration challenges at every stage of the development process, from code reviews to builds, testing, merging, and deployment.
What’s Software Engineering Productivity? The anti-metrics approach
Everyone defines engineering productivity differently.
- In traditional context, it is a discipline to instrument and evaluate the efficiency of engineers to build high quality software.
- With a modern context, we look beyond metrics into elevating the developer experience by introducing modern software engineering practices, and changing culture.
So we are taking the anti-metrics approach. Instead of building the shiny dashboards, we are building automated workflows across the entire SDLC – development, code reviews, builds, tests and deployments. This helps us focus on solving real developer problems instead of just pointing at the problems.
Engineering productivity metrics are still important, Google themselves run the DORA research that suggests a series of key performance indicators (KPIs) to understand the effectiveness of an engineering organization, and engineers have a number of ways to improve DORA metrics. However, metrics are only a compass to identify what’s wrong, not the solution.
Why Engineering Productivity? 5 benefits for the development teams to consider
- It’s a system-level data-driven and AI-supported approach
- Resolving inefficiencies is cheaper and more effective as a long-term strategy
- It helps developers facilitate, improve, and streamline their routine tasks via automated workflows, saving time & effort
- It’s a comprehensive approach covering the entire lifecycle
- It improves the transparency and consistency of the development process at all stages, ensuring overall better and sustainable end results.
The development teams have been evolving their roles and processes to deal with the data overload, as well as the new AI and automation tools. And it’s now more important than ever to have a system and process-focused framework to ensure that this results in some qualitative improvement rather than chaos.
The Google stack challenge
While Google’s tools are built for their massive-scale infrastructure, not every company operates at Google’s level. Furthermore, unlike Google most companies rely on standard external tools such as GitHub, JIRA, Datadog, etc. So the challenge here is not as simple as porting over what Google has built internally, rather we have to craft a flexible framework that is compatible with standard development tools stack.
One interesting challenge with building Google’s Engineering ProductivIty is creating a balance of flexibility with opinionated systems. At Google, everything is opinionated, and that works out well to define standardization across the organization. When we moved from one team to another within Google, it never required retraining on the tooling.
But this cannot be applied for general purpose. There are certain aspects that should be opinionated (e.g. engineers should create smaller changes and have fast review cycles) but some that need flexibility (e.g. not everyone runs a deploy pipeline for every commit). Creating that balance requires understanding where to draw the line.
Balancing flexibility with opinionated workflows for engineering productivity
To overcome the Google stack challenge and create the balance of flexibility with opinionated systems, we’ve created a complete developer experience infrastructure for modern software teams. It’s a suite of 4 engineering productivity tools to scale the developer workflow:
We’ve also made sure that Aviator platform works with any build tool including Bazel, NX, Pants, Turborepo, or Gradle. Check it out and help us with your feedback – it’s free for teams under 15 developers.
Now, let’s dive into the details of how Aviator’s stack works.
Monorepos
One of the foundation principles that we built Aviator’s stack on is Trunk based development in Monorepos. Many of you might be rolling your eyes with the mention of monorepos – many times it’s considered unscalable, unnecessary, or even anti-pattern. However, there are a few things that are still true in most companies today:
- majority of developers contribute primarily in 2-3 active repositories
- Monorepo may not always mean strictly a single repository
- Even when there are multiple separate repositories, there is some logical layer to define dependence of these repos with each other.
Releases ← Rapid
Google’s internal release management system is called Rapid. The objective of Rapid is to deliver scalable, hermetic and reliable releases. Using Rapid, every engineering team can self-serve their own release process. This is the key in achieving faster delivery and elevating the engineering productivity and effectiveness:
- Frequent releases with fewer changes are tied to organizational excellence. DORA research also supports this philosophy.
- To enable frequent releases, more developers need to be comfortable with the deployment process.
- To get comfortable, the deployment workflow needs to be simple, intuitive and reliable.
Aviator Releases
Aviator Releases product follows the same principles to offer federated deployments enabling any engineer (or non-engineer) to manage deployments. It offers continuous delivery using trunk based development.
But unlike Google’s Rapid, Aviator Releases works with a GitHub/GitLab. It also connects with the popular CD looks like GitHub Actions, ArgoCD, Buildkite to provide:
- One click deploys and rollbacks
- Apply hotfixes as cherrypicks to multiple releases together
- Scheduled releases and environment promotions
- Changelog for each microservice and each environment
- Release verification and gatekeeping
MergeQueue ← TAP presubmit
Google’s continuous integration system is called TAP (Test Automation Platform). As a developer when you are ready to merge changes, you run a presubmit that picks your changes, and test them against the latest head. This helps detect any failures before the CL (Google’s version of pull request) gets submitted. This worked at scale because of their distributed graphs, so that TAP only ran the affected tests instead of the entire corpus.
Aviator MergeQueue
MergeQueue is a highly customizable version of TAP presubmit:
- Instead of running a command, a developer requests a pull request (PR) to be queued
- The PR is validated with a codemix of the latest head
- To avoid a global lock, it also verifies the codemix of any unmerged changes ahead of this PR in the queue
- If affected targets are available, the MergeQueue optimizes to validate related changes together
- It reduces the impact from flaky test by using optimistic validation.
AttentionSet
Almost every developer in a large team is not happy with the status quo of code reviews. Although there may not be a perfect solution to solve slow code reviews without dropping the code review quality, some tools can make a big difference in response times.
One such tool from Google is called AttentionSet. It’s part of the Gerrit review tool that tracks whose attention is needed for a review. That means:
- If a code review is sent from the author to the reviewer, the attention is passed to the reviewer
- Likewise, when the reviewer responds back with a review (with approval or comments), the attention is passed back to the author.
- Similarly, if a CI fails or if someone tagged you in the PR, the attention is assigned to you.
Now this Attention assignment acts as the inbox for the developer. Depending on the notification preferences of the developer, they can be notified via email, slack or Chrome Extension.
Aviator AttentionSet design follows a very similar pattern, with Slack notifications and Chrome Extension to keep the developer informed while ensuring not to break the flow-state.
Stacked PRs ← Small CLs
From the very early days, Google had been a big promoter of small changes. With small changes, you get:
- faster feedback cycle, as small changes are easier to review
- the quality of review is much higher
- you get feedback much faster before investing heavily into one direction
- breaking the changes into small self-contained changes means you can continue working on other changes while it’s in review
But to encourage smaller changes, better tooling is needed. Google’s internal source control system (Piper) supports stacking changes natively, making this experience seamless.
Aviator Stacked PRs CLI
Aviator’s open-source stacked PRs CLI provides a similar capability on top of Git. One of the biggest challenges of maintaining a long chain of stacked branches is being able to amend or update those branches. Moreover, with Git’s squash-merge strategy, it’s easy to get into merge conflict as soon as you merge a PR. Aviator’s CLI is meant to handle these rough edges providing a seamless experience to manage 10s or 100s of stacked branches natively in any Git-based review tool.
FlexReview ← GwsQ
Code ownership for code reviews a sensitive but complex topic. Code reviews are a great tool in any organization to expand knowledge sharing and reduce bus factor. But delegation of code reviews can be messy in large organization leading to:
- reviews not getting assigned to the right reviewer
- reviews not getting distributed fairly
- too many reviewers getting assigned
Google’s original code owners model relied on OWNERS file, which turned into GitHub CODEOWNERS. On top of this GwsQ to perform smart operations on code reviews to assign the right reviewers.
Aviator FlexReview
At Aviator, we took a more opinionated approach to improving the code review experience as a part of the overall engineering productivity improvements. Aviator FlexReview is a code review ownership framework that helps teams dynamically understand the domain expertise and assign reviewers. This also accounts for availability, timezone, current workload, size of the code change and author’s expertise when choosing the right reviewers.
The role of AI in Engineering Productivity
In several companies, a large portion of the code is already generated with the help of AI. This brings both opportunities and challenges in the developer experience space. While AI can accelerate code generation and automate routine tasks, it also requires robust tooling to create guardrails and maintain code quality.
- AI assists, not replaces: We are just starting to understand the developer-AI interfaces
- Quality guardrails: AI-generated code needs stricter oversight, we need code quality and code review quality benchmarks to understand the impact of AI tools.
- New developer focus: With AI handling routine tasks, developers will focus more on planning, problem-solving, and collaboration—areas where DevEx is key.
- Smaller, empowered teams: As engineering teams shrink, each developer will take on more responsibility. We will need tools to improve knowledge sharing and provide context.
Standardization
Developer experience cannot be improved simply by introducing new tools. To truly make an impact we need to work on three levers: People, Practices and Tools. As the industry gets more mature, the tools will evolve, but people and practices will remain the same.
Google managed to standardize many of the software delivery practices that helped them pioneer the DevEx space. These are the same five delivery practices that are embodied in the Aviator’s platform:
#1. Monorepo
A monorepo setup helps establish consistent standards across projects by centralizing build configurations, linting rules, and development workflows. This standardization reduces cognitive overhead for developers moving between projects and ensures uniform quality controls. Having all code in one place also simplifies dependency management and cross-project changes.
#2. Trunk-based development
Trunk-based development is a source control branching model where developers collaborate on code in a single branch called ‘trunk’, resisting any long-lived feature branches. This approach promotes continuous integration by ensuring all developers work off the latest code. It reduces merge conflicts and integration issues while enabling continuous delivery of code changes.
#3. Continuous delivery
Continuous delivery is a software development practice where code changes are automatically built, tested, and prepared for release to production. This approach enables teams to deploy code changes more frequently and reliably. By automating the delivery pipeline, teams can maintain a constant flow of updates while ensuring quality and stability. Note that continuous delivery is different from continuous deployment, and it’s more important to have a continuous delivery even if the deployment is triggered manually or at a preset cadence.
#4. Small reviews
Small, focused code reviews are helps maintain high code quality and developer productivity. By breaking changes into smaller, digestible pieces, reviewers can provide more thorough feedback and catch potential issues earlier. This approach also reduces cognitive load on reviewers and speeds up the review process, leading to faster iteration cycles.
#5. Universal ownership
Universal ownership for developer assets promotes well defined shared responsibility and knowledge sharing across the team. When everyone feels ownership over the codebase, it encourages collaboration, reduces knowledge silos, and ensures that any team member can contribute to any part of the project. This approach also helps distribute the review workload more evenly and builds a stronger engineering culture.
What do I need for Engineering Productivity? 5 areas to start with
While not every organization needs to implement Google’s entire toolkit, several key lessons are universally applicable:
- Invest in Tooling: Good tools pay for themselves many times over in improved productivity
- Automate Aggressively: Every manual process is an opportunity for automation
- Measure Everything: Data-driven decisions lead to better outcomes
- Standardize Thoughtfully: Balance standardization with flexibility
- Focus on Developer Experience: Happy developers are productive developers
Looking forward
As the industry continues to evolve, the fundamental principles of Google’s Engineering Productivity model serve as a blueprint for building efficient, scalable, and maintainable software development organizations. The focus on automation, standardization, and measurement provides a framework that organizations of any size can adapt to their needs.
The success of this model has inspired a new generation of tools and platforms, including those being developed by former Google engineers who are now bringing these practices to the broader market.
Interested in trying out the Engineering Productivity model with Aviator? Get a demo or start for free.