More Isn't the Reward

a hundred thousand flaky tests taught me what Jevons Paradox can mean for leaders

Quite a few years back I built a testing framework and scaffolding that made it stupid easy for anyone in our org to write an automated test and get it running in our distribution system. A template would create the scaffolding, and a simple submission template would get it into a system that could distribute the test to any number of devices. I was proud of it…and within a month we went from a few thousand automated tests to a few hundred thousand.

I thought I’d built more confidence, but I quickly learned that more isn’t always better. What I’d actually built was more failure. Flaky tests multiplied right along with everything else, and the team started spending hours every day chasing down which failures were real and which ones were noise. Instead of getting safer or building more confidence, we just got busier.

That’s Jevons paradox, and I didn’t know the name for it until last Friday, when Brent and I spent an hour on our podcast talking about it in the context of AI and hardware. The short version: in 1865, economist William Stanley Jevons noticed that as steam engines got more efficient at burning coal, total coal consumption went up, not down. Cheaper power didn’t mean less use. It meant more machines, more mills, more everything, because the cost per unit of output dropped and demand rushed in to fill the gap.

We talked about how this shows up with GPUs and AI compute. Every leap in efficiency doesn’t shrink the industry’s appetite for chips, it grows it, because suddenly things that were too expensive to try become worth trying. And after my conversation with Brent, I started seeing the same pattern in a lot more places around me.

Coal, Chips, and Your CI Pipeline

The scaffolding story of mine isn’t rare. Delivery and quality work has its own version of Jevons paradox built into it that’s been around for years. We build faster test suites, better CI/CD pipelines, more automated coverage, all so we can ship with more confidence and less risk. And a lot of times it actually works. But other times, the org discovers that deployment is cheaper, and instead of using that as a safety margin, it starts shipping more often before they’re actually ready to deliver quality at higher velocity. More releases, extra branches in flight, and too many feature flags stacked on top of each other lead to a much larger surface area of things that can go wrong. This is how I indirectly created thousands of flaky tests.

I’ve watched teams treat many of these improvements like a scoreboard. Test coverage goes up, so risk must be going down. But coverage measures what you decided to test, not what changed underneath you while you were busy testing it.

Your Calendar Ate Your Time

This pattern exists in a lot of places. We build teams we trust and we make delegation simple. We hand off decisions, and we stop being the bottleneck on things that don’t need us. We expect hours of freed up time, and instead, the freed-up time gets absorbed almost instantly, by more meetings, more Slack conversations, and more “quick syncs” that exist because other work expands to fill that space. This example probably overlaps with Parkinson’s Law, but I think it fits Jevons Paradox as well.

This is a lot like the coal mines. Lower the cost of a thing, and people don’t do less of it. They do more, right up until the new cost feels expensive again. Trust doesn’t create space, it creates capacity, and capacity gets spent.

The Perils of Onboarding

There’s another subtler parallel in how organizations scale. Good onboarding, strong documentation, and clear systems make it cheaper to bring someone new up to speed. This is a good thing a lot of the time. Too many times, however, I’ve seen that cheap onboarding is usually the thing that gives leadership the confidence to grow headcount faster, because the marginal cost of a new hire just dropped. The efficiency fuels the expansion.

I think this is why “we’ll fix it once we have a better process” so rarely delivers the calm people expect. The process isn’t broken. It’s working exactly as Jevons would predict. It’s just not doing what you assumed efficiency would do, which was to give you less to manage.

What Do You Actually Do With This?

I don’t think the answer is to stop building efficient systems, testing pipelines, or trust with your team. The answer is to stop assuming efficiency is the same thing as margin. Margin is a choice you make after the efficiency shows up, not a side effect you get for free.

This means that when your pipelines get faster, or when delegation frees up your calendar, you get to decide whether that speed becomes more releases or more stability, or whether time “saved” becomes more meetings or actual thinking time. Jevons paradox isn’t a trap you’re doomed to fall into. It’s a trap you fall into by default, unless you pay attention to what happens with the system when the cost drops.

Efficiency will always tempt you to do more. As leaders, or honestly anyone who pays attention to quality, deciding on purpose what “more” is worth doing.

---

If you’d like to talk more about this, my calendar is open.