Engineering Is a Black Box
At a LeadDev directors' roundtable, one of the leaders described how we come up with our estimates as "hand-wavy stuff." I think about engineering confidence as not having that hand-wavy stuff. Many people depend on expert judgment, and I think expert judgment is there to validate the delivery evidence you have to build confidence.
I hear all the time that engineering is a black box. I typically don't even say it — people just come out and say it to me, and I'm stunned because they're absolutely right. We have all these metrics in the world that we track everything on. But when the business wants to know when we can deliver something, it's all based on belief.
AI is producing more than we could ever imagine before, but we're still defining confidence in the same old way we've been doing it for decades. And it's causing a lot of problems.
AI Is Giving Us False Confidence
I think AI is giving us some false confidence because we can do more. If we have more output, of course we can do more work. At that same roundtable, another leader said their executive team came and said, "We want to see three times more productivity." I asked the fundamental question: what was their underlying reason for asking for that? What is engineering not doing currently that it should be doing?
Regardless of how we measure team performance — that's a whole other conversation — are we giving confidence to executive leaders who know nothing about engineering metrics that we're going to deliver? It's still a lot based on opinion and belief.
A lot of the engineers I've worked with feel really good about the delivery and what they say they can do. Then the crunch comes, and they're pulling their hair out: we've got to make this date, we've got to get it done. That's where you start getting burnout. And AI has compressed that timeline significantly.
Belief-Based vs. Evidence-Based Confidence
At a LeadDev dinner, I was challenging people on confidence. One leader said he brings his principal engineers to the board to give them confidence. I kept asking, what is that confidence based on? He said they delivered everything they were supposed to deliver in the quarter. That's good, but what was it based on? I kept digging, and it turned out it was based on belief.
That team had been together for a very long time and understood the codebase inside and out, so they had good judgment. Judgment is valuable, but it's still based on belief. It wasn't based on any delivery evidence.
Engineering confidence meets expectations when you deliver consistently. But there's a journey there, because surprises almost always happen. Each of those deliveries was a different length of time, a different size. So how do you know with confidence that you can deliver within the time you say, or when the business says it needs it?
The person next to me said there's no way you can predict two or four weeks out with confidence on big programs or big features. I challenged that. I think you probably can — it's about how you go about doing it. And that's where you have to teach it, because no one understands it. Once people have that "aha" moment, you get into some fascinating things you can do with it. But it all depends on the leadership you have, because I've been in situations where it gets very uncomfortable once you have that level of transparency and accountability for leadership.
Overloading Teams Signals Low Confidence
Pushing the team provides focus on the outcome you want. But when OKRs are set so that completing 70% is the target rather than 100%, I'd first ask who is making those OKRs. From my lens, that's someone saying: I don't have confidence in engineering delivery, so I'm just going to overload them and hopefully get what I want out of them.
When you have confidence in engineering, it becomes less of a risk center and more of a value center.
I think engineering creates the value for the business 90% of the time, because we're literally delivering the new functionality customers are going to consume. But everywhere we look, we're getting laid off because we're seen as a cost center. Confidence is low, so we say, "We're going to overload you," and hope you make 70% of it. Wouldn't it be nice to have confidence that they can make 90, 95, or even 100% of it?
I don't come in and prescribe a way of working to teams. Typically you have quarterly planning, and I'll implement what I do in that planning to get the confidence. I also look at team performance and line them up, because I want to make sure we're able to deliver what we say we can deliver. My preference is to understand where we are going and what the outcome is, and plan for that. Why do we need to stop what we're doing every quarter to plan for where we already know we need to go? I haven't gotten my way yet because the industry is very much quarterly planning, PI planning, sprints. And that's totally fine — I let teams go to town with that. As long as we're tracking towards the outcome, that's the goal.
A Visibility Problem
The friction between leadership pushing for more with AI and engineers trying to keep up is, from my perspective, a visibility problem. Many business leaders, and this is by far not all of them, don't understand engineering at all. You give them DORA metrics, and they ask, "What is this?
But if you can provide something that shows them where engineering is and bring them along on the journey of delivery, they can start understanding the challenges we're having, and that sets expectations appropriately.
Right now, the expectation is that we can do a lot more work, which is true. But there's friction along the whole SDLC that we have to solve to get the outputs the organization wants us to have.
What If We Slowed Down?
At the roundtable, one leader said his team was delivering features faster than ever, and the feedback he was getting was that customers weren't able to take all the features — the business was telling them they were going too fast. Everybody else was nodding their heads. I asked, "If the feedback you're getting is that you're going too fast, why don't you slow down?" Literally everybody at the table laughed out loud. I said, no, I'm serious. Why don't you slow down?
If you have this abundant capacity, what can you do with it? There's a big gap with junior developers. Why not make your junior developers write code by hand and support it in production so they build up those skills before they move into AI programming, where you get the benefit?
Then you also get the benefit of them understanding how to actually write code and support it in a production environment.
Think about how you learn, by making mistakes. If AI writes a bunch of code for you that you don't really understand, and you don't have time for all the pull requests, it's going to go out, and you'll have problems. And that compounds.
Give People Time Back
I was in Atlanta with my team, and on the drive back from dinner I asked, "What if we gave people time back?"
The first company that figures this out and has, say, a 30-hour work week instead of a 40-hour work week is going to have talent banging down the door. Retention rates are going to skyrocket.
And with AI helping us do more, you should still see your profits increase. In the age of abundance, there's another option: give people time back.
Another leader at that roundtable said the cognitive overload of his engineers is increasing. They start something, the agent is working, so they start something else, and work in process just keeps going up. But nobody's training engineers how to be managers. So they're doing more and more, and as a result they're getting more and more burnt out.
What Is Your Confidence Vote Based On?
In SAFe, you go to PI planning and say what you can do in the next 12 weeks. Then you go around the room and take a confidence vote — everybody raises their hand for low, medium, or high confidence. What I challenge is: what is that vote based on? What kind of confidence are you really voting on? If it's belief-based confidence, you can run into a huge problem when the business starts planning around it.
At one organization, we had to bring in a global consulting firm to build an application for a very busy part of the year for the business because there was no confidence our engineering organization could deliver it in time. They said, "Yes, we can totally do this, no problems at all,” with no evidence whatsoever. But they were the experts, so the business and other departments started planning around that commitment.
A few weeks later, my product partner asked how confident they were in hitting the date, because it was clear they were struggling. There was a lot of talk about still being confident, and no evidence that they were on target. The date came, and it didn't hit. It didn't even make it to production. Because the business planned around confidence that was based on belief, other departments had to scramble to get things in place for their busy season.
That's why confidence is so important for engineering. Without it, you run the risk of impacting huge parts of the organization.
Ask for Delivery Evidence
Ask what evidence you need to feel confident the delivery is going to make it, and then follow it up. We all have feedback loops in place. Let's report on them, so we can see what's actually going on and have options earlier to make better decisions.
What I've seen is that a surprise always happens. Here are all these unknown unknowns, or we were too optimistic about our estimation. That's all based on belief, not evidence.
When that consulting firm came in, I kept asking what evidence they had that they'd deliver. It was a whole bunch of narrative. What I'm looking for is delivery evidence: show me your delivery so I can have confidence you're going to be on time and on budget.
Forward visibility is really important, and not a lot of people know how to do it. We have all these activity metrics in engineering, but they're always looking at the past. How do you look forward to give confidence that you're going to hit those dates? That's the challenge. I have some charts on my website that can seed that conversation.
Visibility Beats Speed
At one global distributor I was at, we had 144 features to deliver, and we didn't find out until six months after we should have started. So we had six months to deliver it. At the first status meeting, everyone was in the room, the VP of engineering, my manager, peers, product. The VP of engineering looked at me right out of the gate and said, Jay, how's your team doing? I said, well, let me show you.
I pulled up one of the charts and showed them exactly where my team was. It showed they had about twice as much work as they could realistically deliver in the time we had left. That right there showed the constraint.
That visibility really beats any type of speed you can have. Because then you can say, what are we going to do about it? And that's where the game changes.
When you have confidence and visibility into what engineering is doing, it changes the conversation. Without confidence, it's chaos.