Oleg
Chursin
WorkNotesPhotography
Almost right: the plausible-but-wrong failure mode of generated code, and how big it is

Why agent patches pass the visible checks and still do the wrong thing, what three studies measured about the size of that gap, and how to catch it before you merge.

Contents · 11 sections

The agent pull request that costs a reviewer an afternoon is rarely the one that fails to build. CI catches that in minutes. The expensive one is green: it compiles, the new test passes, the description says the issue is fixed, and the diff reads like something a competent colleague would write. It fixes the example in the issue, and only that example: later in this note there's a one-line patch that parses "1,299.50" correctly and reads "9,946,342.75" as 9946.

The usual response is "write better tests", and it's true as far as it goes. But it skips the question a reviewer or tech lead actually needs answered: when an agent's patch passes the checks you can see, how often is it still wrong, and wrong in what way? Without a number, you can't decide how much a green run is worth, and you can't tell whether your review process is calibrated to the risk or to a feeling.

This note sizes that gap with the measurements that exist. A survey says how often developers run into it. A peer-reviewed study of SWE-bench Verified says how many "solved" patches behave differently from the human fix, and why. Two METR studies say how far "passes the maintainer's tests" sits from "would be merged". Then I'll show a small demo where two patches pass the same suite and a differential test tells them apart, and finish with what I'd change in a review process on the strength of all this. It builds on the first note in this series, which splits an agent into model, harness, tools and context; here the subject is the output, and the gap between what it looks like and what it does.

"Almost right" is the AI problem developers report most often

The name for this failure mode comes from a survey answer. The 2025 Stack Overflow Developer Survey's AI section asked which problems or frustrations people had encountered with AI tools, select all that apply. The most common answer, chosen by 66% of the 31,476 people who answered, was "AI solutions that are almost right, but not quite". Second, at 45.2%, was "Debugging AI-generated code is more time-consuming". The survey's own write-up connects the two, saying the first often leads to the second.

The same page shows how that experience translates into trust. Asked how much they trust the accuracy of AI output in their workflow, 3.1% of respondents said they highly trust it and 29.6% somewhat trust it, about 33% in total. On the other side, 26.1% somewhat distrust it and 19.6% highly distrust it, about 46%. That's from 33,244 responses, and it sits next to 84% of respondents using or planning to use AI tools, per the same page. People use the tools and don't believe the output, which is a coherent position only if they check it.

The 2026 survey changed the question, which makes it hard to say whether things moved. The 2026 AI page reports that coding assistants and agents are now the dominant use case (66%), and the full 2026 AI data asks about trust with different answer options: 48.0% of 14,304 respondents chose "I trust it when I can easily verify the output", and 6.6% trust it "for many tasks, including important work decisions". As of 18 October 2026, the 2026 AI data has no frustrations question at all, so the 2025 "almost right" figure is still the latest direct measure I could find.

Two things are worth keeping from the survey, and one thing to leave behind. Keep the fact that this is the most reported pain, ahead of slow debugging, opaque code and skill loss. Keep the 2026 framing too, because "trust when I can verify" is a good description of the only safe policy. Leave behind any idea that 66% tells you how often generated code is wrong. It's the share of respondents who have encountered the problem, not a defect rate. For a rate you need studies that look at the patches themselves.

Plausible-but-wrong comes in four shapes

Before counting, it helps to be precise about what is being counted, because "almost right" covers several different defects that need different checks.

The standards bodies give the general frame. NIST AI 600-1, the Generative AI Profile (July 2024), defines confabulation as "the production of confidently stated but erroneous or false content", and its section 2.2 widens that to outputs that "diverge from the prompts or other input" or contradict earlier statements in the same context. The same section makes a point that matters for code review: generated explanations can make wrong answers more convincing, because a model may lay out logical steps for an answer that is itself incorrect. An agent's PR description is exactly that kind of explanation.

OWASP files the problem under misinformation. LLM09:2025 Misinformation pairs hallucination with overreliance, and lists "unsafe code generation" (suggesting insecure or non-existent libraries) among its examples. The list has since been renumbered: the 2026 edition, published 4 August 2026, moves it to LLM07:2026 Misinformation, and the new text is closer to what coding agents do. Its examples include "Incorrect or Fabricated Code and Dependencies" and "Incorrect State Inference in Workflows", where the model concludes that a condition has been met when it hasn't, and one of its scenarios is titled "Fabricated Task Completion". It also routes the neighboring problems elsewhere: the execution and handling of unsafe generated code goes to LLM10:2026 Improper Output Handling, and the registration of hallucinated package names as an attack to LLM04:2026 Supply Chain. Package hallucination is its own subject, and this track gives it a separate note (AIE-48).

Those frames are about language models in general. For agent patches, the empirical work points to four engineering categories, and I'll use them for the rest of this note. The first three line up with the patterns the SWE-bench study below uses when it compares a generated patch with the human one; the fourth comes from METR's review rubric.

Under-implementation. The patch leaves out a change the intended fix makes: it handles the case in the issue and misses a sibling case the issue implies, or fixes the reported input and leaves the same bug one call away. The SWE-bench study calls this an absent semantic change.

Over-implementation. The patch changes behavior the task didn't ask to change: an extra guard, a "while I'm here" rewrite, a looser parser. It passes the new test and quietly alters what existing callers get. The study's term is a supplementary semantic change.

Divergent-but-passing. The patch makes the right change in a different way from the intended fix, and the difference only shows on inputs no test exercises. Neither the reviewer nor the tests have a reason to look. In the study, these are divergent implementations of the same change.

Quality failures. The logic is right, but the change isn't finished: missing tests, missing docs, lint failures, code that is hard to maintain. This is the category a maintainer rejects without needing to find a bug.

The first three are behavioral and the fourth isn't, which is why the next two sections measure different things. The SWE-bench study is about behavior. The METR studies are mostly about the fourth category, with some behavior mixed in.

On SWE-bench Verified, roughly one "solved" patch in nine is wrong

The most careful measurement of plausible-but-wrong patches I know of is Wang, Pradel and Liu's "Are 'Solved Issues' in SWE-bench Really Solved Correctly?". It was posted to arXiv in March 2025, revised to v2 on 9 September 2025, and, per its Crossref record, published in the ICSE '26 proceedings (the 48th IEEE/ACM International Conference on Software Engineering), pages 169-181, with an issue date of 12 April 2026. So it's peer-reviewed, not a preprint.

The setup matters for reading the numbers. The authors took the patches that three issue-solving tools (CodeStory, LearnByInteract and OpenHands with CodeAct v2.1) produced for SWE-bench Verified, the 500-task subset that, per the paper, 93 developers hired by OpenAI filtered for tasks with overly specific or unrelated tests. They kept the plausible patches: the ones that pass SWE-bench's own validation. That's 877 patches. Then they asked how many of those are actually correct, in three steps.

Step one: run the tests SWE-bench skips

SWE-bench judges a patch with the tests associated with the issue: at least one fail-to-pass test that shows the bug is fixed and, where available, pass-to-pass tests that guard against regressions. In practice, the paper explains, both SWE-bench and SWE-bench Verified run only the test files that the task's test patch modified. Tests elsewhere in the repository that touch the same code don't run. When the authors ran every available developer test, 7.8% of the plausible patches failed on average (7.2% to 8.4% across the three tools), which cut the tools' reported resolution rates by 4.5 points on average. Their example is a Django session-decoding fix that passes the PR's tests and breaks a separate test of the legacy decoder.

That first finding has nothing to do with clever analysis. It's a validation gap: the check the benchmark runs is smaller than the check the repository already has. I suspect many CI setups for agent work have the same gap, wherever "run the affected tests" is the gate.

Step two: compare behavior with the human fix

The core of the paper is a technique the authors call PatchDiff, released with a replication package. For each plausible patch it finds functions that sit between the developer tests and the code the patch changed, then prompts a model (GPT-4o-mini in the main study) to write tests that pass under one patch and fail under the other. A test that does that is a differentiating test: it proves the two patches behave differently on some input.

PatchDiff found differentiating tests for 260 of the 877 plausible patches, 29.6%. That's a lower bound on divergence, since a test generator that fails to find a difference proves nothing. The paper's own comparison supports that reading: on a 300-patch sample, swapping the generator model raised the detection rate from 28.0% with GPT-4o-mini to 39.0% with DeepSeek-V3 and 47.7% with Qwen3. Conventional test generators did far worse on the subset they could run at all. Pynguin and CoverUp produced no differentiating tests for 133 patches, and an issue-reproduction approach (LIBRO, re-implemented) found them for 5.9%.

Step three: decide which differences are bugs

A behavioral difference isn't automatically a defect. The human fix isn't the only correct fix, and some inputs have undefined behavior. So the authors sampled 77 of the 260 divergent patches and examined each by hand, against the issue, the repository, the developer tests and the differentiating tests.

Two results from that sample matter most. First, the shape of the differences. 46.8% came from similar but divergent implementations of the same change (my divergent-but-passing), and 27.3% from the generated patch changing more behavior than the human one (my over-implementation). Only 5.2% were cases where the generated patch left out a change the human made (my under-implementation). The authors note the asymmetry themselves: the generated patches tended to add, not omit. A further 20.8% had no alignment with the human fix at all.

Second, the verdict. 22 of the 77 (28.6%) were certainly incorrect: 11 regressions (the issue fixed, something else broken), 6 partial fixes, 3 with irrelevant behavior changes and 2 with plain implementation errors. Only 4 (5.2%) were certainly correct. The remaining 51, two thirds of the sample, couldn't be decided, because the issue didn't specify the behavior that differed. I'll come back to that 66.2%, because it's the most important number in the paper.

Half of the 22 incorrect patches (11) would not have been caught by running all the developer tests. And 8 of the 22 traced back to faulty supplementary changes: the extra behavior the agent added on its own initiative.

Fig. 1 puts the three steps together. Extrapolating the sample to all 260 divergent patches, and adding the 23 patches that failed the full suite in step one but weren't flagged by PatchDiff, the authors estimate that 11.0% of plausible patches are incorrect, roughly one in nine.

There's one discrepancy to flag before you quote this. The arXiv abstract page says the combined weaknesses inflate resolution rates by "6.2 absolute percent points". The v2 PDF on the same record says 6.4 in its abstract, in the RQ4 results and in the introduction, and that PDF is already typeset with the ICSE '26 reference format and DOI, which suggests it's the accepted text. I couldn't confirm that against ACM's copy, which returned 403 to my requests, so I can't say for certain which figure the final version carries. I use "about 6.4 points" because it's what the full text derives, and the difference doesn't change any conclusion.

Be careful with the 11.0% in the other direction too. It assumes the incorrect patches are spread evenly among the divergent ones the authors didn't sample, and it counts only certainly incorrect patches. The 51 undecidable ones aren't in it. The true rate of patches a maintainer would consider wrong could be higher, and nothing in the study bounds it from above.

Passing the maintainer's tests is not the same as mergeable

The SWE-bench study asks whether a patch behaves correctly. METR asked a blunter question: would anyone merge it?

In its August 2025 research update, METR took 18 real issues from two large open-source repositories that were part of its developer productivity study: 15 from stdlib-js and 3 from hypothesis. The issues had taken the original maintainers between 20 minutes and four hours, 1.3 hours on average. An agent built on Claude 3.7 Sonnet (in METR's Inspect ReAct scaffold) attempted each one without seeing the maintainer's tests, and METR scored the result two ways.

By the maintainer-written tests, the agent succeeded 38% of the time (±19%, 95% confidence interval). By manual review of 15 of the PRs, for whether each could be merged without substantial extra work, the success rate was zero. None of the 15 was mergeable as-is.

METR also estimated how long each PR would take a human to fix. The average was 42 minutes. For the PRs that had passed all the maintainer's tests, it was 26 minutes, with a minimum of 15. Against a 1.3-hour original, that remaining work is about a third of the job, which is METR's own framing. METR is explicit that these are rough, high-variance judgments made by comparing the agent PR with the human one, not by the original authors.

Fig. 2 shows the gap and the fix times side by side.

The failure breakdown says what the gap is made of. METR scored each reviewed PR on five failure modes. Among the 4 reviewed PRs that passed the tests, 100% had inadequate test coverage, 75% had missing or incorrect documentation, 75% had linting, formatting or typing issues, and 50% had other code-quality problems. And 25%, one of the four, still had incorrect core functionality according to the human reviewers, despite passing every maintainer test. Every one of the 15 reviewed PRs had at least three of the five problems, and 20% had all five.

So in METR's sample the dominant gap is the fourth category, quality, with a measurable slice of the behavioral kind on top. The sample is tiny, and METR says so. Four test-passing PRs can't give you a rate. They can tell you the gap is real on a well-maintained codebase, and what it's made of.

The 2026 follow-up, with real maintainers

METR came back to the question in a research note dated 10 March 2026. METR labels its notes as less reviewed than its research posts, so weigh it accordingly, but the design is stronger than the 2025 study in the ways that matter. Four active maintainers from three SWE-bench Verified repositories (two from scikit-learn, one each from Sphinx and pytest) reviewed 296 AI-generated PRs covering 95 issues, blinded to whether a human or an agent wrote them. The patches came from Claude 3.5 Sonnet, Claude 3.7 Sonnet, Claude 4 Opus, Claude 4.5 Sonnet and GPT-5.

Maintainer review is noisy, so METR also resubmitted 47 of the original human patches, which had really been merged. The maintainers merged only about 68% of those. METR normalizes every score against that golden baseline, and after that adjustment the maintainer merge rate came out about 24.2 percentage points (standard error 2.7) below the automated grader's pass rate. The summary puts it plainly: roughly half of test-passing patches from agents of mid-2024 to late 2025 would not be merged into main.

Two details of the review setup matter. Maintainers were told to ignore testing requirements, which METR itself calls generous to the agents, so on a team that requires tests I'd expect the gap to be wider. And they had no CI, so no linting or test runs to lean on; METR told them instead that the patches had passed an automated grader built from tests. That cuts both ways: a maintainer can't see a lint failure, but has been told the code works. The patches are also small: METR says the human SWE-bench Verified fixes in its sample change about 17 lines on average, against more than 500 in the 2025 study.

The note also tried to measure whether the gap is closing. It found the maintainer merge rate improving about 9.6 points per year more slowly than the grader score, but that estimate has a standard error of 5.5, is significant only at the 10% level, and changes depending on which models are included. METR calls it weak evidence. I'd treat it as no evidence either way.

The gap is structural: tests are a weak oracle and agents grade themselves kindly

I don't read these numbers as a bug in a particular model. They follow from how agents decide they're done, and the sources point to several mechanisms.

A passing suite is a weak oracle. The SWE-bench paper starts from the observation that test suites are "rarely exhaustive", so a patch can pass and still miss what the developers expected. Its own data shows how weak: half of the certainly-incorrect patches passed every developer test in the repository, not only the ones SWE-bench ran. METR's 2025 update draws the matching conclusion for benchmarks: algorithmic scoring "may overestimate" real-world performance.

Issues under-specify behavior. This is the 66.2% from the SWE-bench study. For two thirds of the divergent patches the authors examined, nothing in the issue, the repository or the human fix said which behavior was right. The agent made a reasonable choice, the human made a different one, and the tests didn't care. An agent asked to fix a bug has to fill those gaps somehow, and it fills them with plausible defaults. The output looks right because it's the kind of code that usually is right.

The agent's own stopping signal is "looks done". Anthropic's best-practices guide for Claude Code says it directly: "Claude stops when the work looks done", and without a check it can run, looking done is the only signal available. The same guide names this failure the "trust-then-verify gap": the agent "produces a plausible-looking implementation that doesn't handle edge cases". The fix it recommends is verification: tests, scripts, screenshots, and "If you can't verify it, don't ship it."

Self-evaluation is lenient. In Harness design for long-running application development (March 2026), Anthropic's Prithvi Rajasekaran reports that agents asked to evaluate their own work tend to respond by "confidently praising the work", even when a human sees it as mediocre. The effect is strongest for subjective quality like design, but the post says that even on verifiable tasks agents sometimes show poor judgment. Its account of building a separate evaluator is instructive: out of the box, the QA agent found real issues and then talked itself into approving the work anyway, and it tested superficially rather than probing edge cases. Separating the judge from the worker helped, because tuning a standalone evaluator to be skeptical was "far more tractable" than making a generator critical of its own output. That's vendor engineering writing about its own product, so it's evidence of what one team observed, not a measured rate.

Training may reward what's checkable. METR's 2025 update offers a hypothesis rather than a finding: if model training increasingly relies on rewards that are computed automatically, agents may get better at whatever automatic checks measure faster than at what they don't. Its 2026 note's per-model breakdown fits the picture loosely (the gains it describes between Claude 4 Opus and Claude 4.5 Sonnet are mostly in code quality), but that is a description of five models, not a test of the hypothesis. I'd hold it as a reason to keep measuring, not as an explanation.

Put those together and you get the mechanism. The agent optimizes toward the signals in its loop. Tests cover a fraction of the intended behavior, the issue covers less, and the agent's own judgment leans generous. Whatever falls outside all three gets filled with something plausible. Fluency is what makes the result hard to catch.

Reviewers miss it for predictable reasons

If generated code is often almost right, review should catch the rest. There are documented reasons it often doesn't.

The risk frameworks name the mechanism. NIST AI 600-1 describes automation bias, "excessive deference to automated systems", in its section on human-AI configuration, and notes that people may come to over-rely on these systems or unjustifiably rate their output above other sources. OWASP's LLM09:2025 defines overreliance as users placing excessive trust in generated content and failing to verify it. The 2026 entry, LLM07:2026, adds that humans and systems tend to treat fluent, well-structured output as authoritative, and that in agentic systems this overreliance "is frequently embedded in system design". A CI gate that turns green on the PR's own tests is a small example of that.

My reading is that agent PRs make this worse in a specific way, though I know of no study that measures it directly. A human contributor's mistakes often leave traces: inconsistent naming, a half-finished refactor, a missing case that matches a gap in their understanding. A generated patch is uniformly fluent, so the surface doesn't point you at the wrong line. The SWE-bench examples show what that looks like. In the paper's motivating case, a SymPy fix for an evaluate(False) crash made the error go away by skipping the imaginary-coordinate check whenever evaluation is off, so invalid points could now be created. The human fix corrected the check itself. Both diffs are a couple of lines, and both read as reasonable.

The OpenSSF's security-focused guide for AI code assistants (dated 1 August 2025) sets the right default: "Assume AI-written code can have bugs or vulnerabilities, because it often does." It also says that AI-generated code isn't a shortcut around code review, testing, static analysis or documentation. That's the attitude. The difficulty is applying it to every PR without burning out.

The opposite failure: a reviewer that always finds something

The tempting fix is to add an AI reviewer and tell it to be thorough. Anthropic's best-practices guide warns about the result: a reviewer prompted to find gaps "will usually report some, even when the work is sound", and chasing every finding leads to over-engineering: extra abstraction, defensive code and tests for cases that can't happen. Its advice is to tell the reviewer to flag only gaps that affect correctness or the stated requirements. The same page recommends a fresh session for review, since a reviewer with clean context "won't be biased toward code it just wrote".

That gives you two ways to fail. Under-review lets plausible-but-wrong code through. Over-review buries the real problem under invented ones and pushes the agent toward the over-implementation the SWE-bench study already found in 27.3% of divergent patches. Whether a fresh-context reviewer actually catches more is an empirical question this track takes up separately (AIE-32). For this note the point is narrower: a reviewer, human or model, needs evidence about behavior, not more opinions about the diff.

Two patches, one green suite: a differential test separates them

Here's the failure in miniature, small enough to run. A pricing module has a parsePrice function, and an issue says it rejects prices with thousands separators: "1,299.50" comes back as NaN. This is the code on main:

original.mjs
// main, before the fix: only plain numbers parse.
export function parsePrice(input) {
  const value = Number(input.trim());
  return Number.isFinite(value) ? value : NaN;
}

Two agents propose fixes. Each changes one line, both look fine, and both make the issue's example work. Save them next to original.mjs:

patch-a.mjs
export function parsePrice(input) {
  const value = Number(input.trim().replaceAll(',', ''));
  return Number.isFinite(value) ? value : NaN;
}
patch-b.mjs
export function parsePrice(input) {
  const value = parseFloat(input.trim().replace(',', ''));
  return Number.isFinite(value) ? value : NaN;
}

The repository's tests, plus the one the PR adds for the issue, run against whichever patch the IMPL environment variable points at:

price.test.mjs
import { test } from 'node:test';
import assert from 'node:assert/strict';

const { parsePrice } = await import(process.env.IMPL);

test('parses plain prices', () => {
  assert.equal(parsePrice('12.50'), 12.5);
  assert.equal(parsePrice(' 7 '), 7);
});

test('rejects non-numbers', () => {
  assert.ok(Number.isNaN(parsePrice('abc')));
});

// Added by the PR that closes the issue.
test('accepts a thousands separator', () => {
  assert.equal(parsePrice('1,299.50'), 1299.5);
});

Both patches pass all three tests. Reading the two files, a reviewer has no strong reason to prefer either. Patch B is wrong twice, though, and each mistake matches a category from the SWE-bench study.

First, String.prototype.replace with a string pattern replaces only the first match, so "1,299,000.00" keeps its second comma. That's a divergent implementation of the right change, the 46.8% category. Second, swapping Number for parseFloat changes behavior nobody asked to change. MDN's parseFloat page shows it parsing "4.567abcdefgh" as 4.567, while Number coercion turns any string that isn't a complete number literal into NaN. So "12abc", which main rejected, now parses as 12. That's a supplementary change, the 27.3% category, and it's the kind the paper links to a third of its certainly-incorrect patches.

A differential test catches both by asking two questions the example tests don't. Does every price the app can format round-trip through the parser? And on the inputs the issue didn't touch, does the patch still agree with main?

price.diff.test.mjs
import { test } from 'node:test';
import assert from 'node:assert/strict';

const { parsePrice } = await import(process.env.IMPL);
const main = await import('./original.mjs');

// Seeded generator, so a failure reproduces exactly.
function random(seed) {
  return () => {
    seed = (seed * 1664525 + 1013904223) % 2 ** 32;
    return seed / 2 ** 32;
  };
}
const enUS = new Intl.NumberFormat('en-US', { minimumFractionDigits: 2 });

test('round-trips any formatted price', () => {
  const next = random(42);
  for (let i = 0; i < 500; i++) {
    const digits = 2 + Math.floor(next() * 9);
    const price = Math.floor(next() * 10 ** digits) / 100;
    const text = enUS.format(price);
    assert.equal(parsePrice(text), price, text);
  }
});

test('matches main on input without commas', () => {
  const inputs = ['12', '0.5', '1e3', '12abc', '3.4.5', '', '0x10'];
  for (const text of inputs) {
    assert.deepEqual(parsePrice(text), main.parsePrice(text), text);
  }
});

Run both files against each patch with Node's built-in test runner:

IMPL=./patch-a.mjs node --test price.test.mjs price.diff.test.mjs
IMPL=./patch-b.mjs node --test price.test.mjs price.diff.test.mjs

I ran this with Node 24.18.0. Patch A passes all five tests. Patch B passes the three visible tests and fails both differential ones: the round-trip fails on "9,946,342.75", which it parses as 9946, and the comparison with main fails on "12abc", which comes back as 12 where main returned NaN. Fig. 3 shows the sequence.

A failed assertion throws and ends the test, so "12abc" is only the first difference. Patch B also turns "3.4.5" into 3.4 and "0x10" into 0, where main gave NaN and 16. And it changes "" from 0 to NaN: Number coercion turns an empty string into 0, so main was reading a blank field as a price of 0. That's the point of comparing with main. It flags every change in behavior, including a change that happens to fix a latent bug, and a person decides which ones were intended.

Two caveats about the demo. Patch A isn't proven correct, only not shown wrong: it also accepts "1,2,3" as 123, and whether it should is a specification question no test here answers. That's the SWE-bench study's 66.2% bucket in one line. And the second differential test needs a reference to compare against. Here it's main, which works for any change that should leave existing behavior alone. For the behavior the issue does change, you need a property, like the round-trip, or a second implementation.

In a real codebase I'd use a property-based testing library rather than a hand-rolled generator. fast-check is a property-based testing library for JavaScript and TypeScript (version 4.10.2 on npm as of 18 October 2026); it generates inputs, runs the property over many cases and shrinks a failure to the smallest input that reproduces it. The method matters more than the tool, and this track has a separate note on differential testing of agent patches (AIE-19) that goes further.

Pass rate, divergence and mergeability answer different questions

Three studies, three numbers for roughly the same kind of patch: an estimated 11.0% of passing patches certainly wrong, about half of passing patches not mergeable, and zero of fifteen mergeable as-is. They don't contradict each other. They measure different things, and the one you should track depends on the decision you're making.

Measure What it counts Source Use it when you're deciding
Test pass rate Patches the visible checks accept The benchmark or your CI Whether the agent can do the core task at all
Behavioral divergence Patches that act differently from a reference on some input PatchDiff, differential tests How much of a green run is unverified behavior
Certainly incorrect Divergent patches a human judges wrong Manual review of divergences Whether green runs are hiding defects
Mergeable as-is Patches a maintainer would accept without changes Maintainer review How much human time an agent PR really saves

Each has a cost. Pass rate is cheap and automatic, and it's the number every agent will look best on. Divergence needs a reference and a test generator; it's automatable, and it over-reports, since some differences are fine. "Certainly incorrect" needs a person with context, and the SWE-bench study shows how often even an expert can't decide. Mergeability needs your actual reviewers, and it's noisy: the maintainers in METR's 2026 study merged only about 68% of patches that had already been merged once.

The easy mistake in arguments about agents is quoting one of these numbers as if it answered the question of another. A vendor's SWE-bench score answers the first row. Your reviewers' experience answers the last. The gap between them, by METR's 2026 estimate about 24 points on small, human-filtered issues, is the cost of plausible-but-wrong output, paid in review time.

A green run should start the review of an agent PR, not end it

Here's what I'd do, given this evidence, if I owned a team's process for agent PRs.

Treat the PR's own tests as necessary, not sufficient. A green run on the tests the agent wrote or touched tells you the agent can satisfy its own understanding of the task. The SWE-bench study shows that running the full suite, not only the affected tests, catches 7.8% of plausible patches on its own. If your CI runs "affected tests" for speed, run everything before merge on agent PRs.

Require evidence of behavior on the changed paths. Ask for, or generate, at least one test that compares the patched function with main on inputs the issue didn't touch, plus a property for the behavior it did change. The differential test file in the demo above is about 30 lines for a small function. Without that, the reviewer is checking fluency.

Read every line the issue didn't ask for. Supplementary changes were 27.3% of the sampled divergent patches and accounted for 8 of the 22 certainly-incorrect ones in the SWE-bench sample. When an agent "also" hardens a parser or adds a guard, that's the hunk to read slowly. A good rule for the agent's instructions is the one in Anthropic's review prompt example: check that "nothing outside the task's scope changed".

Send ambiguity back as a question. Two thirds of divergent patches in the SWE-bench sample couldn't be judged because the issue didn't specify the behavior. When a reviewer finds a divergence they can't call, that's a specification gap, and the fix is to settle the behavior and add a test, not to guess. Specs and interview-first workflows are their own topic (AIE-16).

Keep a human merge bar, and calibrate any AI reviewer to correctness. Tell a review agent to report only issues that affect correctness or the stated requirements, as the Claude Code guide suggests, and keep a person accountable for merge. This track's pillar on reviewing changes no human wrote (AIE-30) covers what that person should look for.

Count your own "almost right" rate. None of the published numbers describes your codebase, your agents or your issue quality. Start counting two things: agent PRs that pass CI but need rework before merge, and the rework time. Use METR's five failure modes (core functionality, test coverage, documentation, lint and types, other code quality) as the rubric, so your numbers are comparable to theirs. A spreadsheet is enough. After a month you'll know whether your gap looks like METR's or like something else, and every change you make to instructions, tools or review can be judged against it.

None of this is exotic. It's the engineering hygiene the OpenSSF guide says AI code doesn't get to skip, aimed at the specific ways generated code goes wrong.

What we don't know

Current-model rates. Both quantitative studies use agents from 2024 and 2025: three issue-solving tools from early 2025 in the SWE-bench study, and models up to Claude 4.5 Sonnet and GPT-5 in METR's 2026 note. As of 18 October 2026 I found no measurement of plausible-but-wrong rates for the frontier agents released in 2026. What would settle it is the PatchDiff method, or METR's maintainer-review design, rerun on current agents.

How much of the gap is elicitation. METR's 2026 note says plainly that it doesn't claim a capability limit: the agents got one attempt, no feedback and a single harness, and better prompting might close much of the quality gap. Nobody has published the same tasks run with and without a stronger harness, review feedback or project instructions.

The size of the undecidable bucket in real work. The 66.2% of divergent patches the SWE-bench authors couldn't judge reflects SWE-bench Verified issues, which had already been through human screening. Whether real tickets are better or worse specified, and so whether the share is larger or smaller, is untested.

Which inflation figure the final paper carries. The arXiv abstract says 6.2 points and the v2 full text, typeset with the ICSE '26 DOI, says 6.4. ACM's copy wasn't accessible to me, so I couldn't check the published version directly. The difference is small, but if you quote the number, quote the version.

Small samples behind the mergeability numbers. METR's 2025 manual review covers 15 PRs from two repositories, only 4 of which passed the tests, and its ±19% interval applies to the test-pass rate, not to mergeability. The 2026 note is larger (296 PRs, 95 issues) but covers three Python repositories and small patches, about 17 changed lines on average for the human fixes, and is published as a less-reviewed research note.

The trend. Whether the gap between test-passing and mergeable is narrowing is open. METR's estimate of a 9.6-point-per-year slower improvement for maintainer merges has a standard error of 5.5 and shifts depending on which models are included.

Whether reviewers really catch fewer defects in generated code. The argument above, that uniform fluency hides where a generated patch is wrong, is my reading of the mechanism, backed by the risk frameworks' warnings about automation bias. I found no controlled study that seeds the same defects into human-written and agent-written PRs and compares reviewer detection rates. That experiment would settle it.

What developers mean by "almost right". The 66% figure counts people who have encountered the problem in 2025. The 2026 survey dropped the question, so there's no trend, and no survey measures how often a given respondent's AI output is almost right. A repeated survey item, or better, telemetry on rework of accepted suggestions, would help.

References

  1. AI - 2025 Stack Overflow Developer Survey. Stack Overflow. Accessed .
  2. AI - 2026 Stack Overflow Developer Survey. Stack Overflow. Accessed .
  3. AI data - 2026 Stack Overflow Developer Survey. Stack Overflow. Accessed .
  4. Are "Solved Issues" in SWE-bench Really Solved Correctly? An Empirical Study (arXiv v2). You Wang, Michael Pradel, Zhongxin Liu (arXiv). Accessed .
  5. Crossref metadata for DOI 10.1145/3744916.3764576 (ICSE 2026 proceedings). Crossref. Accessed .
  6. PatchDiff: a technique for differential patch testing (replication package). ZJU-CTAG (GitHub). Accessed .
  7. Research Update: Algorithmic vs. Holistic Evaluation. METR. Accessed .
  8. Research note: Many SWE-bench-Passing PRs Would Not Be Merged into Main. METR. Accessed .
  9. Harness design for long-running application development. Anthropic. Accessed .
  10. Best practices for Claude Code. Anthropic. Accessed .
  11. NIST AI 600-1: Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile. NIST. Accessed .
  12. LLM09:2025 Misinformation. OWASP GenAI Security Project. Accessed .
  13. OWASP Top 10 for LLM Applications 2026 - release overview. OWASP GenAI Security Project (GitHub). Accessed .
  14. LLM07:2026 Misinformation (canonical source). OWASP GenAI Security Project (GitHub). Accessed .
  15. Security-Focused Guide for AI Code Assistant Instructions. OpenSSF Best Practices and AI/ML Working Groups. Accessed .
  16. parseFloat(). MDN Web Docs. Accessed .
  17. Number - Number coercion. MDN Web Docs. Accessed .
  18. Test runner (node:test). Node.js. Accessed .
  19. fast-check - Introduction. fast-check. Accessed .
  20. fast-check package metadata. npm. Accessed .

Found this useful? Pass it on.

Set in Archivo on a twelve-column grid. Built with Lit web components and rendered to static HTML.

© 2026 Oleg Chursin
Brooklyn, New York