James Shore’s argument lands in one line: an AI coding agent that writes code faster but leaves you more to maintain has made you slower, not faster. We reviewed that essay against our own AI development platform — reading our pipeline, our standards and our generated output — and agreed. The change we adopted is a maintenance-burden metric with lifetime denominators.
Why “time to working code” is the wrong headline number
Every agent demo is measured the same way: how long from prompt to something that runs. It is the easiest thing to film and the least useful thing to know. Writing is the cheap half of software. The expensive half is everything after the merge — the next developer reading it, the dependency that goes end-of-life, the abstraction that made sense for one screen and now has eleven callers. Shore’s title is the entire thesis, and it is a better thesis than most of what trended alongside it on Hacker News that week. An agent that triples output while doubling what you carry has not made you faster. It has moved the invoice. We build client software on an AI development platform, so this is not abstract for us: every project our pipeline generates is a project somebody has to live with afterwards. If our agents inflate that, we are selling velocity and delivering debt.
Does an AI coding agent actually reduce maintenance costs?
Not automatically, and not measurably at present. Public agent evaluations score whether a generated patch resolves a reported issue — that is a resolution rate, not a maintenance number. Nothing on those leaderboards tells you what an accepted patch costs to own in month six, and no independent benchmark for maintainability of agent-written code exists to cite. That absence is the finding, and it argues Shore’s point better than any table would. When we compared his argument against our pipeline, the uncomfortable part was not that our agents produce unmaintainable code. It was that we had no number that would have told us either way. Cost per task, token spend, duration, success rate, cycle time — all reported. Bugs filed against work we had already delivered sat in the database, populated and honest, and no metric surfaced them. “The agent felt fine” is not a measurement.
The metric we adopted, and the two better-looking versions we rejected
The metric is deliberately boring: bugs filed divided by tasks completed, across the project’s entire life. The interesting work was rejecting the two versions that looked stronger on paper. The first was an internal task quality-rating table — richer, more granular, exactly what you would design starting fresh. It has a single manual writer, so it is nearly empty, so a metric built on it would return a comfortable null for almost every project and read as health. The second was our default reporting window, a rolling thirty days. Maintenance burden does not respect a rolling window. A project that shipped hard for a year and went quiet last month scores a perfect zero on a thirty-day view, and that zero is a lie about code somebody still maintains. Lifetime denominators, or nothing.
bug_rate = bugs_filed / tasks_completed # lifetime, not a rolling window
# null when tasks_completed == 0
The null matters as much as the ratio. A project with nothing completed yet has an undefined bug rate, not a zero one, and a dashboard that renders 0.0 there has invented a perfect score for a project that has done nothing. Three rules came out of this that transfer to any stack:
- Build the metric on the table that is actually populated, not the one with the nicer schema. An elegant, empty source produces confident nonsense.
- Match the window to the phenomenon. Debt accumulates over a project’s life, so the denominator has to cover a project’s life.
- Never fabricate a zero. Absence of data and absence of bugs look identical on a chart and mean opposite things.
The gap the review actually found: nobody upgrades the dependencies
Reading our own CI generator was the part I did not enjoy. Every project our pipeline provisions gets a generated workflow — build, tests, lint, the usual gates — and not one line of dependency maintenance. No Dependabot configuration. No scheduled audit job. We generate applications with real dependency trees and hand them over with no mechanism that will ever tell the owner a transitive package went critical. That is a maintenance cost we manufacture at provisioning time and then never report. The fix is small and was already on our roadmap, which is why the review narrowed to it instead of inventing something larger: emit a Dependabot config and an audit step in the generated workflow. Cheap, dull, and it removes a class of debt from every future project rather than paying it down one project at a time.
One interaction has to be checked before that lands, and it is exactly what I would get wrong if I moved fast. Our platform can auto-merge pull requests once the gates go green. An agent that opens dependency bumps and merges its own dependency bumps is not automation — it is an unattended supply-chain path. Dependency PRs need a different gate from feature PRs. I would rather ship the audit job alone than ship both and discover the interaction in production.
Where we already agreed with him: maintainability-first standards
Not everything in the review went against us. Shore’s implicit prescription — bias the machine toward code that is cheap to keep — is something our pipeline already does structurally, and it is the piece I would copy first if I were building this elsewhere. Our agents write against an injected standards library that is maintainability-first by construction: the simplest implementation that satisfies the requirement, not the most impressive one. What makes it hold up over time is that the library scores itself. Each standard tracks how often it was injected and how the resulting work turned out, and the ranking uses a Wilson lower-bound confidence interval rather than a raw success rate. That choice matters more than it sounds. A rule that succeeded once out of once shows 100% and would otherwise outrank a rule that succeeded ninety-nine times out of a hundred.
Raw success rates reward small samples, and a standards library ranked on them slowly fills with lucky noise. Correcting for sample size is the difference between a library that improves and one that drifts. It is the same discipline as the lifetime denominator: pick the statistic that survives contact with sparse data.
FAQ
What counts as a maintenance cost for AI-generated code?
Everything after the merge: bugs filed against delivered work, dependency upgrades nobody automated, time the next developer spends understanding an abstraction, and rework when a shortcut becomes load-bearing. None of it appears in time-to-working-code. That is precisely why an agent can look fast and still leave you worse off overall.
Why measure bug rate over a project’s lifetime rather than the last 30 days?
A rolling window rewards silence. A project that shipped heavily for a year and paused last month reports zero bugs on a 30-day view, which says nothing about the code somebody still owns. Debt accrues across a project’s life, so the denominator — tasks completed — has to cover that same life to mean anything.
Should an AI coding agent auto-merge dependency updates?
Not on the same gate as feature work. An agent that both opens and merges its own dependency bumps creates an unattended path from an upstream package straight into production. Generate the update PRs and the audit job, then require a human or a distinct, stricter gate before those particular merges land.
Is there a benchmark for AI coding agent maintainability?
Not one we can point to. Public agent evaluations measure whether a patch resolves an issue, which is a resolution rate, not a maintenance number. Until something measures cost of ownership months later, the honest move is to instrument your own projects — bug rate over completed work is a crude start that beats no number at all.
Get a free recommendation from a senior engineer.