AI coding seats didn't fix delivery. METR found experienced developers were 19% slower with AI — yet still believed they were faster. Here's why the tool isn't the intervention, the workflow is.In early 2025, a research group called METR ran a study that most engineering leaders would rather not read. They took sixteen experienced open-source developers, gave them real tasks drawn from their own repositories, and randomly decided for each task whether the developer was allowed to use AI tools or not.The developers were fluent with those tools. They expected the assistance to speed them up by about a quarter. The measured result went the other way: tasks completed with AI took roughly nineteen percent longer than tasks completed without it.The finding that should keep leaders awake is not the slowdown itself but the perception gap around it. Even after finishing, the developers still believed AI had made them about twenty percent faster.Hold that number next to the one everybody quotes. A widely cited controlled experiment from GitHub found developers completing a specific task — writing an HTTP server in JavaScript — about fifty-six percent faster with Copilot than without.Both studies are real. Both are competently run. The gap between them is not a contradiction to be resolved by picking a favorite. It is the whole subject of this article. The greenfield task rewarded generation because the human held little context that the model could not supply. The expert working in a codebase they already carried in their head got slowed down, because the binding constraint was never typing speed. It was judgment, and the tool added a layer of review and correction on top of judgment the developer already had.That is the hidden variable behind almost every disappointing AI rollout. The tool is not the intervention. The workflow around it is.
Buying a tool is not changing a process
Most organizations adopt AI the way they adopt any other developer tool. Procurement approves seats, a champion runs a lunch-and-learn, dashboards start counting accepted suggestions, and everyone waits for the delivery graph to bend.It usually does not bend, and the reason is structural. An assistant dropped into an unchanged process speeds up one local step while leaving every downstream step exactly as constrained as before.The clearest public evidence for this comes from Google's DORA program, which has been measuring software delivery performance across thousands of organizations for years. In its 2024 report, a twenty-five percent increase in AI adoption was associated with an estimated one and a half percent decrease in delivery throughput and a seven percent decreasein delivery stability.Read that slowly. More AI correlated with slightly worse delivery, not better. This is correlational data from a survey, not proof that AI harms delivery, but the mechanism the researchers proposed is easy to recognize on any team: faster code generation produced larger change sets, and larger change sets are harder to review, harder to test, and riskier to ship.The 2025 edition sharpened the picture rather than reversing it. Throughput's relationship with AI adoption turned positive as teams learned to work with the tools, but instability kept climbing, and the report's own summary of the dynamic is worth quoting directly:"Friction doesn't vanish so much as move: It shifts from manual grind to deciding and verifying."That sentence is the argument of this piece in one line. When you make one part of the pipeline cheaper, work does not disappear. It piles up wherever your capacity is weakest.You can watch this happen at the level of a single ticket. A vague requirement handed to a capable model does not fail loudly. It produces a plausible, well-structured, confidently wrong implementation that passes a shallow read. A generated test suite does not refuse an ambiguous spec — it encodes whatever behavior the model guessed at, then guards that behavior against future change, so the wrong answer becomes load-bearing.Speed at the keyboard turns into latency in the review queue, defects that surface a week later, and a quiet tax on the senior engineers who now spend their afternoons reconstructing intent from code they did not write. Local acceleration, global drag.
The coordination layer agents actually need
If the problem is the workflow, the fix is a coordination layer that most teams have never built, because unaided humans carry it implicitly. When a senior engineer picks up a task, they already know the acceptance criteria, the parts of the system that are fragile, the conventions the team enforces, and who to ask when something is unclear.A model has none of that unless you make it explicit and machine-readable. Doing so is now a discipline with a name — context engineering — and it is less glamorous and more decisive than prompt wording.A workable coordination layer has a few concrete parts:A specification states what a change should do and, just as importantly, how anyone will know it worked: the acceptance criteria in checkable terms.Repository-level guidance — a committed instructions file — tells any agent the local conventions, the directories it may touch, and the commands that must pass before work is considered done.Reusable skills package a task the team performs often, a migration pattern or a release checklist, so behavior is consistent instead of reinvented per prompt.Tool access is granted deliberately: an agent that can read the issue tracker, run the test suite, and open a draft pull request is useful, and one with unscoped write access to production data is a liability.This is where a standard like the Model Context Protocol (MCP) earns attention. Anthropic published MCP as an open standard in late 2024 to solve a boring but real problem: every AI-to-system connection used to be a bespoke integration, which meant access was inconsistent and impossible to govern. A shared protocol lets you connect an agent to your issue tracker, your documentation, or your observability stack through one auditable interface. The point is not the protocol as technology. It is that access becomes something you can see, scope, and revoke, which is the precondition for trusting an agent with anything that matters.Specifications carry a caveat that the enthusiastic often skip. A precise spec removes ambiguity about implementation. It does not tell you whether the feature is worth building, which trade-off serves the customer, or when to say no. Spec-driven development moves product judgment upstream and makes it explicit. It does not replace it. An organization that treats a written spec as a substitute for deciding what is worth doing has automated the easy half of the problem and abandoned the hard half.
Where AI earns its place across the lifecycle
The useful question is not whether to use AI in the SDLC but where, bounded by what, checked by whom, and measured how. Walking the lifecycle stage by stage makes the pattern concrete, and the pattern repeats: give the agent a bounded task, keep the consequential decision with a human, and track an outcome rather than an activity.
Planning and requirements
The bottleneck here is rarely writing tickets. It is ambiguity that survives into implementation, where it is expensive to remove. A model is genuinely useful as an adversarial reader of intent. Point it at a draft epic and a slice of the existing system and ask it to surface undefined terms, missing error cases, contradictory acceptance criteria, and edge conditions nobody named.One team I would describe as typical used this to turn "users can export their data" into a set of pointed questions: which formats, what row ceiling, which fields for a deleted account, what happens when the export outlives the session. Those questions were always latent. The agent just made them cheap to ask before code existed.What cannot be delegated is the decision the questions provoke. Whether deleted-account exports are even permitted is a legal and product call, not a modeling one. Keep the agent in the role of interrogator and the human in the role of decider.Outcome to measure: the share of stories that reach a sprint review without a requirements-driven reversal, and the reduction in rework traced to unclear scope — not tickets refined per week.
Architecture and design
The constraint at design time is that consequences are delayed and expensive. A poor choice about data ownership or a service boundary is cheap to make and brutal to undo. AI helps most as a fast generator of options and a stress-tester of proposals. Ask it to lay out three approaches to a new integration with the trade-offs each imposes on latency, cost, and operability, or to argue against a design you already favor by naming its failure modes under load.This widens the search before a human commits. The commitment stays human, and it stays human for a reason a model cannot supply: accountability for a decision the organization will live inside for years.Security belongs in this stage rather than at the end, which is exactly what the NIST Secure Software Development Framework insists on when it puts "prepare the organization" and threat consideration ahead of production. Permissions, data boundaries, and trust zones are architecture, not a checklist someone runs the day before launch.Outcome to measure: design flaws and security concerns caught at review versus in production, and the time it takes to produce a decision record complete enough for the next engineer to understand why.
Implementation
Coding is the stage everyone pictures, and the one where the METR result should temper expectations. Generation is real leverage where the human lacks context: scaffolding a new service, writing the tenth variant of a familiar CRUD endpoint, translating a well-specified function into a language the author knows less well, or drafting the boilerplate around a clear interface. It is weaker, and sometimes negative, where a fluent engineer already holds the mental model, because the agent's output must still be read, doubted, and reconciled against knowledge the human had all along.The bounded task is a small, specified change with tests, produced as a draft for review, never an unattended commit to a shared branch. The human decision is the review itself, and it has to be a real one.Stack Overflow's 2025 developer survey found the most common frustration with AI tools — cited by nearly half of respondents — was output that is "almost right, but not quite," the category that costs more to debug than to write. Trust in AI accuracy in that same survey fell to under a third.The implication for implementation is direct: treat generated code as an unfamiliar contributor's pull request, not as your own work with autocomplete.Outcome to measure: review cycle time and the escaped-defect rate on AI-assisted changes compared with a baseline of similar work — not lines produced or suggestions accepted.
Testing and quality assurance
Testing is where AI's failure mode is most seductive, because a generated test suite looks like diligence. The real bottleneck is coverage of the behavior that actually matters, especially the unhappy paths, and a model left to itself will happily write many tests that assert the code does what it currently does. That is tautology dressed as verification.Used well, an agent expands a human-specified oracle: given the intended behavior and the edge cases a person named, it drafts the cases, proposes malformed and boundary inputs a tired author would skip, and flags untested branches. It is good at breadth and bad at deciding what correctness means.Defining correctness is the human's job and cannot move. Someone has to state the invariant the tests defend, or the suite will defend an accident. A useful, bounded application is regression triage: point the agent at a failing run and ask it to localize the likely cause and summarize the diff that introduced it, leaving the fix to a person.Outcome to measure: escaped defects and change failure rate — not test count, which is an activity metric that rises fastest exactly when a suite is least meaningful.
Deployment, operations, and maintenance
In operations the bottleneck is time under pressure. During an incident, the scarce resource is a human's attention while a system misbehaves. Agents help by compressing the boring, high-latency parts of the work: assembling a timeline from logs and traces, correlating a spike with the most recent deploys, drafting the first version of an incident summary, or proposing a rollback plan for a human to approve. In steady state they are useful for the maintenance nobody schedules, such as flagging dependency updates with known advisories or drafting the changelog from merged work.The decision that cannot be delegated is the consequential action itself, and this is where autonomy has to be proportional to reversibility and risk. Letting an agent open a pull request that bumps a library is low-stakes and easily undone. Letting one execute a production rollback, rotate a credential, or run a destructive migration without a human in the loop trades a small time saving for a standing risk that will eventually be triggered.Attach human approval to the irreversible and consequential, and stop attaching it to the trivial, because approval fatigue is its own failure mode.Outcome to measure: time to restore service, change failure rate, and deployment frequency — measured against a real baseline rather than a hopeful anecdote.
A composite: cutting review latency without cutting corners
The following scenario is a composite drawn from common patterns, not a single named company, and it is labeled as such deliberately.A mid-sized product team had already bought AI coding seats and, predictably, seen no improvement in delivery. Their own metrics told the story: median time from pull request opened to merged had grown from about a day to nearly three, and their change failure rate had crept up. Generation was faster and everything after it was slower. This is the DORA dynamic reproduced in one team.Instead of buying more or banning the tools, they redesigned a single workflow: the path a change takes from ticket to merge. Three changes did the work:Required a short spec with checkable acceptance criteria before any AI-assisted implementation began, which cut the volume of confidently-wrong first drafts.Added a committed repository instructions file and gave the coding agent scoped tool access through a single reviewed interface, so it could run the test suite and open a draft pull request but could not touch anything else.Put an AI reviewer in front of the human reviewer, tasked narrowly with catching mechanical problems, missing tests, and convention violations, so that human reviewers spent their attention on design and intent rather than on lint.The controls were the point. The agent never merged. Every change carried a spec a human had approved and tests a human had judged sufficient.Over a quarter, comparing similar classes of work rather than cherry-picked wins, median review time came back down below a day and the change failure rate returned to its prior level while throughput rose. Notice what moved: not the amount of code generated, which had always been high, but the latency and risk in the steps around it. They did not make the model better. They made the workflow able to absorb what the model produced.
What each role has to learn
An operating model changes jobs, and the changes are specific rather than a vague call for everyone to become a prompt engineer. Nobody needs to become a prompt engineer. Everyone needs to get better at stating intent that can be checked and at reviewing evidence instead of vibes.Product manager — the new capability is writing testable intent. Acceptance criteria that a human could argue about are now criteria an agent will implement literally, which raises the cost of vagueness and rewards precision about what "done" means.Engineer — the shift is from author to editor and specifier. The valuable skills become decomposing a problem into bounded, verifiable units, reading generated code with informed suspicion, and knowing when the context they hold makes the tool a net cost.QA engineer — the job moves up a level, from writing cases to defining the oracle of correctness and auditing whether a generated suite tests behavior that matters or merely restates the implementation.Engineering manager and architect — the new literacy is governing autonomy and reading system-level signals. That means deciding which decisions require a human, resisting the pull toward more automation than the team can supervise, and treating security and data boundaries as design inputs.Across all of these roles, one meta-skill recurs and deserves naming as an insight in its own right: the ability to notice and escalate uncertainty. The METR perception gap — developers feeling faster while measurably slowing down — means individual intuition about AI's value is not trustworthy. The people who thrive are the ones who distrust their own sense of speed and ask for the number.
From scattered assistants to a system you can operate
A single reviewed workflow is a starting point, not the destination. Over time the separate assistants — one drafting code, one triaging incidents, one reviewing changes — start to look like components of a system. Turning them into one that you can actually operate, rather than a pile of clever automations, depends on a handful of properties that have nothing to do with model quality.Shared context and permissions come first: agents need consistent, scoped access to the same sources of truth, and that access has to be grantable and revocable, which is precisely the governance problem a standard like MCP exists to make tractable.Audit trails come next, because when an agent takes an action you must be able to reconstruct what it saw, what it did, and why, both to debug and to satisfy anyone who asks how a change reached production.Observability has to extend to the agents themselves, so that their cost, latency, and error rate are monitored like any other production dependency rather than assumed to be free and correct.Failure recovery has to be designed in, because a system with autonomous actors needs a clear answer to the question of what happens, and who is paged, when one of them does the wrong thing confidently.Here is the warning, and it is the one most worth heeding. The temptation once a few agents work is to wire them into an elaborate multi-agent choreography — handoffs and sub-agents and planners — before the single-agent basics are reliable. Premature autonomy is expensive in a specific way: it multiplies the surface where errors compound and buries the audit trail under indirection, so that when something breaks you cannot tell which agent, on what context, made the call.Autonomy should be earned incrementally, proportional to how reversible its actions are and how well you can see what it did. A system you cannot observe is not an advanced operating model. It is an outage you have not scheduled yet.
Measuring the thing that actually matters
None of this can be steered without the right measurements, and the most common mistake is measuring the tool instead of the delivery. Activity metrics are the seductive ones because vendors report them and they always look good: licenses deployed, active users, suggestions accepted, lines of code generated.Every one of these can rise while delivery gets worse, and the METR study is the reason to take that seriously. If accepted suggestions had been the yardstick in that experiment, the tool would have scored a triumph in the exact scenario where it made experts slower. Activity is not outcome, and self-reported productivity is, on the available evidence, an actively misleading signal.Outcome metrics are the ones tied to the business. The DORA set is the well-validated place to start: lead time for change, deployment frequency, change failure rate, and time to restore service, ideally supplemented by review cycle time, escaped defects, rework rate, and some honest read on developer cognitive load.The discipline that makes these numbers mean anything is comparison. You need a baseline captured before the change, and you need to compare similar classes of work, because a quarter that shipped mostly greenfield features against one that fought legacy migrations will differ for reasons that have nothing to do with AI. A single before-and-after figure does not establish causation, and anyone who presents one as proof is selling something.The right posture is closer to running an experiment than announcing a result, and the honest conclusion is sometimes that a workflow which shone on one repository or one task type did not generalize to another.
A way to start that survives contact with production
The practical path is narrow on purpose.Pick one workflow where the pain is measurable and the blast radius is contained — the review-to-merge path from the case above is a good candidate — and instrument it before you touch it, so you have a baseline that is not a memory. State the outcome you expect in advance and the safety boundary you will not cross: what the agent may do, what stays human, what can never happen unattended. Run it as a time-boxed experiment on real work, not a demo.Then do the part most teams skip, which is to inspect the failures honestly — the confidently-wrong drafts, the tests that guarded the wrong behavior, the reviews that rubber-stamped a mistake — and treat those as the data. Scale only after the outcome metrics, not the activity metrics, hold up against the baseline across comparable work.The through-line of every section here is the same, and it is worth stating plainly rather than dressing up. AI does not improve delivery by writing code faster. It improves delivery when a team rebuilds the flow of intent, context, verification, and ownership around it so that the extra output can be absorbed without breaking. The seat is the easy purchase. The operating model is the work, and it is the part no vendor can sell you.This is how we think about shipping software at VRB Tech. We build, modernize, and advise as an AI-first engineering and consulting partner, and we care more about what reaches production than about what demos well. If you are weighing how to fold AI into your own delivery without loosening your grip on quality, we are happy to have a real conversation about it, with no sales pitch and no pressure. vrbtech.co