AI Hasn't Changed What We're Trying to Achieve
Both the strengths and weaknesses of the SPACE framework are that SPACE is not a set of metrics. SPACE is a set of principles that, at its core, says developer productivity is nuanced, it's multidimensional; you have to look across many different angles. And those dimensions still hold.
The challenge is the metrics that sit underneath them. And this is a place where, as an industry, I feel like we've lost our minds.
AI is a means to an end. The things we are trying to accomplish now with AI are the same things we were trying to accomplish before. We want to produce innovation quickly, easily, and at high quality that leads to good business outcomes. AI is changing the way we get there, but it's not changing what we're trying to achieve.
Because of the hype of AI, because of all the costs associated with AI, it creates this anxiety where teams are throwing away everything they've learned over the last many decades about how to measure our engineering experiences effectively. And it's not even that we're just throwing it away. We're making negative progress.
The number of times over the last couple of years when I've heard very senior executives talking very publicly about measuring lines of code coming from AI — "40% of our code base comes from AI" — what in the world are we doing? We've known for decades that measuring lines of code is a heavily problematic metric.
The Metrics Hierarchy
At the top layer, we should be measuring outcomes—our innovation velocity, our top-level quality, our business outcomes. Those metrics should not change because we're still trying to achieve the same things.
What changes is how we get there. I think of these more as diagnostic measures—data points that give us insights into how our systems are working, how the innovation is flowing. Here we have some brand new metrics like AI adoption and token usage, not the things I would necessarily want to set top-level goals around. If you set top-level goals around token usage, you get tokenmaxxing. But they provide a lot of depth and color into the patterns that lead to those outcomes.
Token Usage Is a Cost Metric
Those costs are real and growing, and we should be measuring them. We should be understanding how to leverage them most efficiently. But just because I'm using a lot of tokens doesn't necessarily mean I am using AI more effectively. It doesn't mean I'm having better outcomes. People thought those things correlated, and that evidence just isn't there.
It says a lot more about how I structure my context and the efficiency of that context than about the usage itself.
In this rush to justify and show that we are using AI and on the leading edge suddenly people realize if we have leaderboards for these things, people will optimize for it.
What Should Engineering Leaders Actually Measure?
The first bit of advice is you can't try to distill things down to a single metric. You aim to measure across many dimensions because that acts as guardrails that make sure you capture more nuance in the engineering process.
You certainly want to measure across a quality dimension. Something like change failure rate, which is in the DORA core four, is still an incredibly valid metric. It was valid before AI and is now valid with AI. We still want low failure rates of the code we're flowing through.
Measuring developer satisfaction is still incredibly important. If AI is causing our developers to find their work less meaningful, more stressful, they're burning out; that is a signal we should be capturing. We might be getting all these gains from AI, but what is the cost of it?
We love measuring PR throughput. I think PR throughput still has a ton of value. It is a garbage measure of individual productivity—it is now an even worse measure of individual productivity with AI—but it is an incredibly valuable measure of system flow.
Can we move code efficiently through the system, or are there lots of friction points and bottlenecks?
Whether that's human code or AI code, we should still be looking at that. Is it code generation? No. Is it code review? Well, now probably it is. Is it validation? Yeah, like that's probably where we have a new bottleneck.
And all of the tried and true things—how quickly do we respond to code review requests, how much time are we spending in meetings, how much bureaucratic overhead are we having? The things that were always important are still important.
Measure the Full Idea-to-Customer Cycle
One example of a metric that wasn't worth the effort before is measuring the entire cycle of an idea—from when we have a first planning construct of an idea until it is in the hands of a customer.
Of course it's what we always should have been measuring. What is our overall innovation velocity? But it's really difficult to do, because you have to stitch together planning artifacts, coding artifacts, build and test artifacts, and deployment artifacts. We could use PRs as a proxy for that. Well, PRs are becoming less of a proxy for that over time.
This is a metric that maybe we always should have been doing, but now we're going to be forced to as we move towards more of an intent-first world away from a PR-first one.
AI's ROI: Mixed Signals
Clearly, AI is increasing PR throughput. But if you just trace that code through to deployment, there was a great recent study by the National Bureau of Economic Research that actually found that very little of the new code generated actually survives all the way to deployment—maybe only 50%.
As we try to draw correlations between our AI usage and our outcome metrics, do we see PR velocity go up? Yes, clearly we do. But do we see change failure rate also improve? There, the signal gets a lot more complicated.
I have not seen any evidence with any sort of statistical significance that our change failure rate is any better, which means we're producing more and more code, but it's certainly not any more stable than it was.
I have a study that I hope to publish soon where we looked across eight different dimensions of code quality—maintainability, robustness, testability, readability, security, and things of that nature. Across all eight dimensions, none of them had a statistically significant positive relationship with AI usage. And in fact, most of them trended slightly negative.
That shouldn't be surprising. Humans are really good at writing code. The question is, what's the ROI of having a human do it versus AI doing it?
Developers Love AI, But It's Complicated
In our just-published study, The SPACE of AI, we asked developers how strongly they agree with the statement that they like their job better because they're using AI. Overwhelmingly, developers say, Yes, I like my job better because I am using AI. Developers overwhelmingly feel more productive when they're using AI. And as we feel more productive, we are happier with our jobs.
But when you go just a little bit under the surface—do you like what it's doing to your code review processes? Do you like what it's doing to quality? Do you even like what it's doing to your work-life balance? Because now we feel all of this pressure to keep up. We're working more and more. It might be leading to a bit of a burnout epidemic.
Yeah, of course I love AI. It makes me feel so effective. But there's so much about it that I hate. And it's driving me crazy.
We need to think about how we are deploying AI beyond just accelerating coding, which is the thing that devs have always liked doing themselves anyway.
Multi-Agent Work Sounds Great; Our Brains Disagree
In that SPACE of AI paper, we included research by the UX researcher Sarah Cesari, where she did observational studies of developers—actually watched developers work with AI. What she found is that even very senior developers who are well-experienced with using AI, when they were put under even just a little bit of time pressure, all reverted from multi-agent workflows to sequential workflows.
Why? Because our brains aren't good at multitasking. That's not how the human brain is wired. We talk about this world where we're all going to be managers of agents. And it turns out, that's really, really hard to do. That's not what we've been trained to do.
We're all going to be managers of agents. Well, turns out not everyone has the skills required to be a good manager.
As our tools make orchestration easier and allow us to visualize what's happening better, that's going to help a lot. Training will help a lot. But there's also this gap between this idealized way we'll work as humans and the way that human brains actually work.
Less Meeting Beat AI By 2x
I recently published a case study in a paper called Engineering Thrive that looked at an organization of 3,000 engineers. We ran an experiment where all we did was clean up low-quality, redundant meetings that people didn't need to be in, and then sequence their meetings so they had bigger gaps of time throughout the day. Just calendar cleanup stuff—to give people bigger uninterrupted blocks of time to do deep focused work, to get into a flow state.
Not only did it create more focus time, but devs took that time and reinvested it in coding. And the gains in PR throughput that they got from that were 2x the gains that we are seeing from AI usage.
Culture Predicts AI Success
Where do we see clusters of high performance, and what are the similarities between these organizations? What I found is that the signals predictive of an organization maximizing their value from AI are not any of the things I would have expected going in. It isn't about the age of their engineering system. It isn't about the size of their engineering workforce. It isn't about their specific segment. It's a culture problem.
Organizations that have a culture around experimenting with new use cases in a safe way—where it's OK to fail—and where leadership strongly advocates for using AI in new and inventive ways. I found that organizations where leadership were strong advocates for AI usage had 7x the daily adoption as organizations that didn't. 7x. That's just a cultural thing.
The most effective way of learning new AI skills is something we call social proof. It's not sitting in a structured training. The most valuable thing is watching a trusted peer, a trusted colleague, solve a real-world problem.
It's not "I'll go watch this video." It's "Could you share your screen and show me how you did this thing?". That requires a culture of mentorship.
The more an individual uses AI, the better at it they get. Of course. But we can actually measure that the more the people around you use AI, the better you get at it. Teams that work effectively as a team—they are separating out from the teams that didn't.
Don't Let AI Kill Collaboration
AI does appear to be improving productivity, not just along dimensions of activity but also allowing us to work more efficiently. It seems to have some evidence of positive impact on satisfaction, although that's nuanced. The majority of developers feel it's allowing them to deliver more business value.
The one wrinkle is communication and collaboration. That is the one dimension of SPACE where we did not see most developers say that AI is helping. And maybe not coincidentally, it's also the most under-measured dimension of SPACE.
Individuals who can't collaborate effectively with their peers are 10% less productive. We need to think about how we measure the things that allow us to detect pain points in our collaboration—measuring the effectiveness of our meetings, our code review processes, how long I have to wait for a peer to pick up my code review.
My fear is that code reviews go away because we have AI just do all of our code reviews. But code reviews are about so much more than just finding defects. It's about transferring knowledge, getting transparency on what my peers are working on, and finding new ways of doing things. As we automate some of that stuff away with AI, how are we replacing that incredibly important team-building aspect?
What's Next: Agent Experience
My next big research area is around something we're calling agent experience. There's all of this research into what are the factors that allow developers to do their best work, but not what are the factors that have to be true to allow agents to do their best work.
If agents can't do their best work, that puts a lot more pressure and dissatisfaction on the developer.
The human side of maximizing human-AI collaboration—we should approach tackling developer experience problems. On the agent side, we should tackle agent experience problems. Do agents have clear context? Are they working towards clear goals? Is the information they've been given well-scoped, or do you feed in the entire code base every time?
How do we define agent experience, how do we maximize it, and ultimately, what does that do to developer experience?