When a coding agent hands you a bad diff, the first explanation everyone reaches for is "the model got worse". It's a satisfying hypothesis because it puts the problem somewhere you can't touch. It's also, often enough, wrong. The best-documented Claude Code regression of 2026 turned out to be three changes in the product around the model, and Anthropic's own postmortem says plainly that "the API was not impacted".

The trouble is that "the agent" is a bundle. There's a model, a harness that runs the model in a loop, a set of tools the model acts through, the context that fills up as it works, and the machine the whole thing runs on. Change any of them and the output changes. If you can't say which one moved, you'll fix the wrong one, and you won't be able to tell whether your fix helped.

This piece takes the bundle apart. It defines each part precisely, collects the best public evidence I could find for how much each one moves output quality (with the caveat, stated up front, that almost all of it is published by the vendor whose product it describes), and ends with a diagnostic order you can run the next time a session goes sideways. It's the first part of a series; later pieces work each lever in depth.

An agent is a model running inside a harness

Anthropic's documentation is unusually explicit about the split. How Claude Code works describes an "agentic loop" with three phases, "gather context, take action, and verify results", and says the loop is powered by two components: models that reason and tools that act. Claude Code itself is "the layer around the model that provides the tools and manages the context the model sees", and the page names that layer: "This surrounding layer is what the term agentic harness refers to."

The eval side uses the same vocabulary. Anthropic's Demystifying evals for AI agents (January 2026) defines an agent harness, "or scaffold", as "the system that enables a model to act as an agent", and then draws the consequence: "When we evaluate 'an agent,' we're evaluating the harness and the model working together." The same guide separately defines an evaluation harness, the infrastructure that runs tasks, records transcripts and grades outcomes. Keep those two apart; we'll need both.

That one sentence about evaluating the pair is the thesis of this whole piece. A benchmark score belongs to a model and a harness, on a particular machine. So does a bad session. Nobody has ever observed "the model" writing code by itself in your repository. What you observed was a system, and systems have more than one knob.

Fig. 1 shows the loop with each lever attached where it acts. The model decides what to do next. The harness decides what the model sees and how hard it thinks. The tools decide what an action can be and what comes back from it. The context is everything that has accumulated in the window. The environment is the container, the CPU and memory, the network and the clock. It isn't part of the loop at all, which is exactly why it's easy to miss.

Five levers, defined so you can tell them apart

The definitions matter, because the boundaries are where blame gets misassigned. Here's how I draw them, with the source that justifies each line.

The model: weights plus how much it is allowed to think

The model is the weights, plus the test-time compute you grant it. In Claude's current lineup the main knob for that compute is effort. Claude Code's model configuration page says effort levels "control adaptive reasoning, which lets the model decide whether and how much to think on each step", and adds a line that trips people up: "The effort scale is calibrated per model, so the same level name does not represent the same underlying value across models."

So medium on one model and medium on another are not the same setting. The same page gives a concrete case. Opus 5.5 defaults to medium, one level below Opus 5's default of high, and "in Anthropic's testing, Opus 5.5 at medium matches or exceeds Opus 5 at high on coding and knowledge-work evaluations". That is a vendor claim about its own models; I'm citing it for what the knob means, not as proof of the result. If you migrate models and carry your effort setting over, you've changed two variables, not one.

The harness: everything between your prompt and the API call

The harness is the system prompt, the default effort it sends, its context management, its permission mode and how it retries failed API calls (headless runs report each retry as an api_retry event). How Claude Code works describes the context part: as the window fills, Claude Code "clears older tool outputs first, then summarizes the conversation if needed", and instructions from early in the session "may be lost". The postmortem below adds one more item to the list: which of the model's own earlier reasoning gets sent back. The important property is that harness defaults can differ from API defaults for the same model.

As of 2026-10-18, the API's Models overview lists the default effort on the Claude API as high for Claude Sonnet 5.5 and Fable 5.1, and medium for Opus 5.5 and Haiku 5.5. Claude Code's model configuration gives its own resolution order: an explicit choice wins, then your settings, then the model's default, which is "high on every model that supports effort, except that Opus 5.5, Sonnet 5.5, and Haiku 5.5 default to medium, Opus 4.7 defaults to xhigh". Put those side by side and Sonnet 5.5 runs at high if you call the API without setting effort, and at medium if you open Claude Code without setting it. Same weights, same name, different system.

That's not a criticism of either default. It's the reason "I tried the same model in the API and it was fine" proves less than it sounds like.

The tools: an interface the model reads

Tools are the definitions, descriptions and result shapes the model acts through. In the Model Context Protocol's tools specification (revision 2026-07-28), a tool is a protocol object returned by tools/list, with a name, a human-readable description, an inputSchema and an optional outputSchema. The model never sees your implementation. It sees the description and whatever text the tool returns, and it decides what to do next from those alone.

The harness also shapes tool output on the way back. Claude Code's MCP documentation says it "displays a warning when MCP tool output exceeds 10,000 tokens and limits output to 25,000 tokens by default", configurable with MAX_MCP_OUTPUT_TOKENS. A successful result over that limit, or a successful text result over 50,000 characters, is saved to a file, and the model gets a message naming the path instead of the content. Error results get different treatment: error text longer than about 11,000 characters keeps only its first and last 5,000 characters. Those are tool-shaped decisions made by the harness, which is a good example of why the levers aren't independent.

The context: a budget that degrades as it fills

Context is everything in the window: system instructions, project instruction files, files the agent read, tool results and the conversation so far. Anthropic's context engineering write-up (September 2025) calls it "a critical but finite resource" and describes an "attention budget" that every new token depletes. Claude Code's best-practices guide opens with the same constraint: "Claude's context window fills up fast, and performance degrades as it fills."

The environment: the fifth lever

The environment is the runtime the agent works in: CPU and memory limits, installed packages, network access, time limits, even the time of day the run happened. Anthropic's product docs describe the agent as models, tools and the harness that manages context; the machine isn't on that list. The evidence below makes the case that it belongs there. For an agent that installs dependencies and runs test suites, the machine is part of the problem it's solving.

The April 2026 regression came from the harness

The postmortem is the best public case study we have of a quality drop that users blamed on the model and that turned out to be something else. It's worth reading in full; here's the part that matters for diagnosis. Anthropic traced the reports to "three separate changes that affected Claude Code, the Claude Agent SDK, and Claude Cowork", and confirmed "that our API and inference layer were unaffected".

Change 1: a lower default effort

On March 4, Claude Code's default reasoning effort went from high to medium, to cut very long latencies in high mode. Anthropic's own evals had shown medium "achieved slightly lower intelligence with significantly less latency for the majority of tasks". Users disagreed with the trade, and the change was reverted on April 7. It affected Sonnet 4.6 and Opus 4.6. The postmortem notes that after in-product notices and an inline selector, "most users retained the medium effort default", which tells you how much weight a default carries.

Change 2: a context-management bug

On March 26, Claude Code shipped a change meant to clear old thinking once, for sessions idle for more than an hour, using the clear_thinking_20251015 API header with keep:1. The implementation cleared it on every turn for the rest of the session instead. In the postmortem's words, Claude "would continue executing, but increasingly without memory of why it had chosen to do what it was doing", which "surfaced as the forgetfulness, repetition, and odd tool choices people reported". Because each request dropped blocks, each also missed the prompt cache, which Anthropic believes explains separate reports of usage limits draining faster. It was fixed on April 10 in v2.1.101, and it affected Sonnet 4.6 and Opus 4.6.

Change 3: one line in the system prompt

On April 16, alongside Opus 4.7, a system prompt instruction was added to reduce verbosity: "Length limits: keep text between tool calls to ≤25 words. Keep final responses to ≤100 words unless the task requires more detail." It had passed multiple weeks of internal testing with no regressions on the evaluations Anthropic ran at the time. The postmortem says that "in combination with other prompt changes, it hurt coding quality". During the investigation, Anthropic ran ablations (removing lines from the system prompt one at a time) against a broader set of evaluations, and one of them "showed a 3% drop for both Opus 4.6 and 4.7". It was reverted on April 20, and the postmortem says all three issues were resolved as of that date, in v2.1.116.

Fig. 2 puts the three windows on one calendar. The overlap is the lesson. The postmortem says it directly: "Because each change affected a different slice of traffic on a different schedule, the aggregate effect looked like broad, inconsistent degradation." The effort change hit everyone on two models who kept the default. The thinking bug hit only sessions that had crossed the idle threshold. The prompt line hit three models for four days. Any single user saw some mix of these, and the mix varied by model, by session age and by date.

Two details make the case sharper. First, Anthropic's internal evals did not reproduce the problem at first: reports were "challenging to distinguish from normal variation in user feedback", and "neither our internal usage nor evals initially reproduced the issues identified". Second, the thinking bug "made it past multiple human and automated code reviews, as well as unit tests, end-to-end tests, automated verification, and dogfooding". If a vendor with full access to its own harness needed more than a week to find the root cause, a team guessing from outside needs a better method than intuition.

There's one more thing I checked, because it matters for detection. I read the changelog entries for 2.1.101 (April 10) and 2.1.116 (April 20). Neither mentions the thinking fix or the system prompt revert. Release notes are not a complete record of harness behavior. If you want to know what changed between two runs, the version number is necessary, and it isn't sufficient.

The environment can move a score by six points

The second piece of evidence is about the lever nobody names. In Quantifying infrastructure noise in agentic coding evals (February 2026), Anthropic ran Terminal-Bench 2.0 under six resource configurations, from strict enforcement of each task's specified CPU and RAM to completely uncapped. "Everything else stayed constant: same Claude model, same harness, same task set."

The headline: "the gap between the most- and least-resourced setups on Terminal-Bench 2.0 was 6 percentage points (p < 0.01)". The mechanism is more interesting than the number, and it comes in two regimes.

• Up to about 3x the specified resources, extra headroom fixes the infrastructure, not the task. Infrastructure error rates fell from 5.8% at strict enforcement to 2.1% at 3x (p < 0.001), while success rates moved "within the margins of noise (p=0.40)". Most tasks that crashed at 1x "would have failed regardless".
• Above 3x, extra headroom changes what's being measured. Between 3x and uncapped, infrastructure errors fell another 1.6 points while success rose almost 4. Generous limits let the agent try approaches that only work with generous allocations, "such as pulling in large dependencies, spawning expensive subprocesses, and running memory-intensive test suites".

The post's example is a task that needs Bayesian network fitting. Some models start by installing pandas, networkx and scikit-learn. With room, that works. Under tight limits "the pod runs out of memory during installation, before the agent writes a single line of solution code". A leaner strategy exists and some models default to it. So the resource limit decides which model's habits happen to succeed. That's a property of the environment and the model together, not of either alone.

The finding wasn't a one-model quirk: the post says the core result replicated across different Anthropic models, with a consistent direction and varying magnitude. The effect was smaller on SWE-bench, where tasks are less resource-hungry: scores rose monotonically with RAM but were only 1.54 points higher at 5x than 1x, across 227 problems with 10 samples each.

The post also puts the six points in proportion. Across the moderate range of configurations the spread was "just below 2 percentage points", and "naive binomial confidence intervals already span 1-2 percentage points"; the infrastructure effects "stack on top of that, not within it". So sampling noise and machine noise add up, and both are the size of the gaps people argue about on leaderboards. The post adds an observation it explicitly didn't quantify: "pass rates fluctuate with time of day, likely because API latency varies with traffic patterns and incidents". Its recommendation for readers of leaderboards is that differences "below 3 percentage points deserve skepticism until the eval configuration is documented and matched".

Anthropic's eval guide has a neater example of the evaluation setup distorting a result. Opus 4.5 "initially scored 42% on CORE-Bench" until a researcher found rigid grading (it penalized "96.12" when expecting "96.124991…"), ambiguous specs and stochastic tasks; after fixes and "a less constrained scaffold", the score went to 95%, per Demystifying evals for AI agents. Nothing about the model changed between those two numbers.

For your own team the implication is direct. If your agent runs in CI on a shared runner, its quality depends on what else that runner is doing. Two runs of the same task on different days are not a controlled comparison unless you've controlled the machine.

Tools are an interface the model has to read

The tools lever has the weakest public evidence and the most practical guidance. Start with the claim, and be precise about its status.

Anthropic's Writing effective tools for agents (September 2025) says: "Claude Sonnet 3.5 achieved state-of-the-art performance on the SWE-bench Verified evaluation after we made precise refinements to tool descriptions, dramatically reducing error rates and improving task completion." I couldn't find a published ablation behind that sentence: no before-and-after numbers, no breakdown by change. It's a vendor's account of its own result. I treat it as a strong hypothesis that's cheap to test, not as a measured effect size.

The mechanisms in the same post are concrete enough to act on:

• Identifiers the model can reason about. The post found that "merely resolving arbitrary alphanumeric UUIDs to more semantically meaningful and interpretable language (or even a 0-indexed ID scheme) significantly improves Claude's precision in retrieval tasks by reducing hallucinations". It recommends returning fields like name and file_type rather than uuid and mime_type.
• Tools that match natural subdivisions of the task. Fewer, more purposeful tools mean fewer descriptions in context and less multi-step work the model has to chain correctly.
• Truncation and pagination with sensible defaults. "For Claude Code, we restrict tool responses to 25,000 tokens by default." The current MCP docs confirm the 25,000-token default for MCP tool output, as covered above.
• Error messages written for the model. The post recommends that validation errors "clearly communicate specific and actionable improvements, rather than opaque error codes or tracebacks". The MCP specification encodes the same idea at protocol level: tool execution errors are returned with isError: true and "contain actionable feedback that language models can use to self-correct", and clients "SHOULD provide tool execution errors to language models to enable self-correction". Given the Claude Code truncation rule above, put the instruction the model needs in the first few thousand characters of a long error, not in the middle.

Here's what that looks like on the wire, as the result of a tools/call response. In the 2026-07-28 revision every result carries a resultType, which is "complete" for a finished call, error or not, per the protocol's base specification. The first result is the kind an agent can't recover from; the second tells it what to do next.

{
  "resultType": "complete",
  "content": [{ "type": "text", "text": "Error 422" }],
  "isError": true
}

{
  "resultType": "complete",
  "content": [{
    "type": "text",
    "text": "No order matches orderId 'A-1043'. orderId is the order number shown to the customer, e.g. 'A-10043'. Call search_orders with the customer's email to list their order numbers."
  }],
  "isError": true
}

The older Building effective agents (December 2024) frames all this as the agent-computer interface, or ACI, and asks you to "plan to invest just as much effort in creating good agent-computer interfaces (ACI)" as in human-computer interfaces. Its best anecdote is from Anthropic's own SWE-bench agent: the model made mistakes with relative file paths after moving out of the root directory, so they changed the tool to require absolute paths, "and we found that the model used this method flawlessly". The post says they "spent more time optimizing our tools than the overall prompt". It's nearly two years old, so read it as the origin of the idea rather than current product documentation.

Context is a budget, and more of it is not better

The context lever has the best non-vendor evidence in this piece, and it cuts against the most common practice.

The context engineering post gives the mechanism. Needle-in-a-haystack studies "uncovered the concept of context rot: as the number of tokens in the context window increases, the model's ability to accurately recall information from that context decreases", and the post attributes this to architecture: transformers let every token attend to every other, which "results in n² pairwise relationships for n tokens", stretched thinner as length grows. Its prescription is to find "the smallest possible set of high-signal tokens that maximize the likelihood of some desired outcome". The next note in this series, on context rot, looks at the peer-reviewed long-context results behind this.

Now the counterweight. Gloaguen and colleagues, in Evaluating AGENTS.md (an arXiv preprint, v3 revised 2026-09-29), tested repository context files in two settings: SWE-bench tasks with LLM-generated context files, and a new set of issues from repositories that contain developer-committed ones. Their abstract reports that "providing context files does not generally improve task success rates, while increasing inference cost by over 20% on average", across different LLMs and coding agents and for both kinds of file. It also says what did work: instructions in the files "are well followed by coding agents", and the files are "useful for specifying non-standard coding practices", while "repository overviews, although popular and recommended by model providers, are not helpful".

That's a preprint and it should be read as one. But it's the closest thing we have to a controlled experiment on the context lever, and it says the default move (write a long file that describes the repository) costs tokens and buys little. A dedicated note in the context engineering series goes through the study in detail.

Vendor guidance has moved the same way. Claude Code's best-practices guide says to keep CLAUDE.md concise and to ask of each line, "Would removing this cause Claude to make mistakes?", and warns that "bloated CLAUDE.md files cause Claude to ignore your actual instructions". The OpenSSF guide for AI code assistant instructions (August 2025) gives the same advice for security rules: they "should be kept concise, specific, and actionable", to influence the assistant "without overwhelming it". The Agent Skills specification builds the budget into the format with progressive disclosure: about 100 tokens of metadata per skill loaded at startup, a body under 5,000 tokens recommended, loaded only when the skill activates, and a main file under 500 lines. And per How Claude Code works, MCP tool definitions "are deferred by default and loaded on demand via tool search, so only tool names and server instructions consume context until Claude uses a specific tool".

The context lever also interacts with the harness lever in a way the postmortem makes vivid. The thinking bug was a context bug: the harness was removing the model's own reasoning from its context. Nothing in your CLAUDE.md could have compensated for that, and nothing about the weights caused it.

Every harness component encodes an assumption that expires

The most useful idea I found on how the levers interact comes from Anthropic's Harness design for long-running application development (March 2026). The author built a three-agent harness (a planner that expands a one-line prompt into a spec, a generator that builds it, and an evaluator that drives the running app through Playwright and files bugs) and compared it against a single agent on the same prompt, using Opus 4.5.

The cost table is stark. The solo run took 20 minutes and cost $9. The full harness took 6 hours and cost $200. The author's account of the outputs is qualitative but specific: the solo run's game maker didn't have a working play mode, and the harness run's did. That's a single example judged by the person who built the harness, so it shows what the trade looks like, not how often it pays.

Two observations from the same post matter more than the numbers.

The first is about self-evaluation. "When asked to evaluate work they've produced, agents tend to respond by confidently praising the work", the post says, even when a human can see the quality is mediocre. Out of the box, the author writes, "Claude is a poor QA agent"; in early runs it found real issues "then talk[ed] itself into deciding they weren't a big deal". A separate evaluator tuned to be skeptical helped. That's a structural fix for a model weakness, delivered through the harness.

The second is the principle I'd put on the wall: "every component in a harness encodes an assumption about what the model can't do on its own, and those assumptions are worth stress testing, both because they may be incorrect, and because they can quickly go stale as models improve." When Opus 4.6 arrived, the author removed the sprint construct entirely and moved the evaluator to a single pass at the end of the run. The build the post shows for that simpler harness, a browser-based digital audio workstation rather than the game maker, took about 4 hours and $124. Different prompt, so don't read that as a saving against the $200; read it as a harness that got simpler because the model needed less help. The evaluator, the post concludes, "is not a fixed yes-or-no decision. It is worth the cost when the task sits beyond what the current model does reliably solo."

Read alongside the postmortem, this explains a pattern you'll see on any team that's used agents for a while. Rules, wrappers and workarounds accumulate to compensate for some model's habits. The model changes. The workarounds stay, and now they're the thing degrading output. The April verbosity line is a close cousin: an instruction written for one new model's tendency (Opus 4.7's verbosity), shipped to older models too, and it cost quality on both. That's why the postmortem says Anthropic has since added guidance "to ensure model-specific changes are gated to the specific model they're targeting".

Diagnose in a fixed order: verification, pins, then one lever at a time

Here's the procedure I'd run when output quality drops. The order matters more than any single step, because each step makes the next one interpretable.

1. Check that the agent can verify its own work

Before blaming anything, ask whether the agent had a check it could run. Claude Code's best-practices guide makes this its first recommendation: "Claude stops when the work looks done. Without a check it can run, 'looks done' is the only signal available, and you become the verification loop." The check is "anything that returns a signal Claude can read in the conversation: a test suite, a build exit code, a linter, a script that diffs output against a fixture". The same guide names the failure you're probably seeing: "Claude produces a plausible-looking implementation that doesn't handle edge cases."

If there was no runnable check, you haven't found a model problem or a harness problem yet. You've found a missing guardrail, and fixing it will do more than any other change. The OpenSSF guide makes the same point from the security side: "AI-generated code isn't a shortcut around engineering processes such as code reviews, testing, static analysis, documentation, and version control discipline." The guardrails series later in this track starts here.

2. Pin and record the system, then compare against an earlier pinned run

You can't attribute a change you didn't record. For every run you want to compare, record the harness version, the exact model ID, the effort level and the environment. A few traps, all from Claude Code's own docs:

• Model aliases move. The model configuration page says aliases "point to the recommended version for your provider and update over time", and that to pin you use the full model name, such as claude-opus-5-5. The API's Models overview says every Claude model ID "is a pinned snapshot, including the dateless IDs used from the 4.6 generation on".
• The harness updates itself. The setup guide says native installations update automatically in the background and documents a stable channel, "typically about one week old, skipping releases with major regressions". On 2026-10-18 the npm registry listed 2.1.302 (published 2026-10-16) as latest and 2.1.292 as stable. DISABLE_AUTOUPDATER=1 turns background updates off, per the environment variables reference.
• An environment variable beats the flag. The same reference says CLAUDE_CODE_EFFORT_LEVEL takes precedence over the --effort flag and the /effort command. If it's set in someone's shell profile, your --effort flag silently does nothing.
• Local configuration leaks in. Per Run Claude Code programmatically, without --bare a claude -p run discovers and loads hooks, skills, MCP servers, auto memory and CLAUDE.md "as an interactive session would". --bare skips that discovery, which is what you want for a controlled run. It also never reads OAuth credentials or the keychain, so the run needs ANTHROPIC_API_KEY (or another provider's credentials) in the environment.

This isn't only engineering hygiene. NIST's Generative AI Profile (AI 600-1, July 2024), action GV-1.6-003, lists "Underlying foundation models, versions of underlying models, and access modes" among the items to consider in a GAI system inventory. If your organization keeps such an inventory, the harness version belongs in it next to the model.

Here's a trial runner that pins all of it. Each trial clones the repository fresh into a temporary directory (the eval guide says each trial should start "from a clean environment", because shared state between runs can cause correlated failures), installs the locked dependencies, runs the agent once, then runs your test suite as the grader and prints one JSON line (append it to a .jsonl file per experiment). It assumes a Node project with a lockfile and an npm test script, and it needs Node 22 or later for the npm package, per the setup guide, plus git and jq. A run that fails inside Claude Code, for example by hitting --max-turns, exits non-zero, per the headless docs; the script records that exit code instead of dropping the trial, because a trial that silently disappears biases the pass rate upward. I haven't run this against a real task set for this article, so treat it as a method, not a result.

#!/usr/bin/env bash
# Usage: run-trial.sh <repo-url> <base-commit> <task-file> <condition>
set -euo pipefail

repo="$1"; base="$2"; condition="$4"
prompt="$(cat "$3")"               # read before cd, so a relative path works
cc_version="2.1.302"
model="claude-opus-5-5"            # full ID, not the "opus" alias
effort="${EFFORT:-medium}"

unset CLAUDE_CODE_EFFORT_LEVEL     # it would override --effort
export DISABLE_AUTOUPDATER=1

dir="$(mktemp -d)"
git clone --quiet "$repo" "$dir"
git -C "$dir" checkout --quiet "$base"
cd "$dir"
npm ci --silent                    # same dependencies in every trial

# A failed run (for example, --max-turns reached) exits non-zero.
# Record it as a trial instead of letting set -e drop it.
code=0
result="$(npx --yes "@anthropic-ai/claude-code@${cc_version}" \
  --bare -p "$prompt" \
  --model "$model" --effort "$effort" \
  --permission-mode acceptEdits \
  --allowedTools "Bash(npm test *)" \
  --max-turns 40 --no-session-persistence \
  --output-format json)" || code=$?
cost="$(jq '.total_cost_usd // null' <<<"$result" 2>/dev/null || true)"

# The grader is the project's own tests, not the agent's opinion.
if npm test --silent >/dev/null 2>&1; then pass=true; else pass=false; fi

jq -nc --arg c "$condition" --arg v "$cc_version" --arg m "$model" \
  --arg e "$effort" --argjson pass "$pass" --argjson code "$code" \
  --argjson cost "${cost:-null}" \
  '{condition:$c, cc:$v, model:$m, effort:$e, pass:$pass,
    exit:$code, cost:$cost}'

Every flag there is documented in the CLI reference. The total_cost_usd field comes from the JSON output format, which the headless docs say includes it "and a per-model cost breakdown". They also say both figures "are client-side estimates and can differ from your actual bill", so use them to compare conditions, not to reconcile invoices. Because --bare skips CLAUDE.md discovery, a condition that tests an instruction needs to pass it explicitly, for example with --append-system-prompt-file. That puts it in the system prompt rather than in a project file, which is a different channel from what your team uses day to day; say so when you report the result.

3. Change one lever, run enough trials, and read the interval

A good shape for a team experiment is four conditions on one fixed task: a baseline, effort lowered one level, a deliberately vague rule added to the instructions, and one tool description degraded. Ten trials each is the number people reach for. The trap is reading ten trials as if they were a hundred.

Agent runs are non-deterministic; the eval guide says so directly and defines each attempt as a trial for exactly this reason. With a binary pass/fail per trial, a binomial confidence interval tells you how much a pass rate could move by chance. The NIST/SEMATECH e-Handbook of Statistical Methods recommends the Wilson interval over the textbook normal approximation: its worth "does not strongly depend upon the value of n and/or p", and its lower limit can't go negative. The code is the handbook's formula:

// 95% Wilson score interval for k passes out of n trials.
export function wilson(k, n, z = 1.96) {
  const p = k / n;
  const denom = 1 + (z * z) / n;
  const centre = (p + (z * z) / (2 * n)) / denom;
  const half =
    (z * Math.sqrt((p * (1 - p)) / n + (z * z) / (4 * n * n))) / denom;
  return [centre - half, centre + half];
}

console.log(wilson(7, 10).map((x) => x.toFixed(2))); // [ '0.40', '0.89' ]
console.log(wilson(5, 10).map((x) => x.toFixed(2))); // [ '0.24', '0.76' ]

Those are hypothetical counts, not measurements, but the arithmetic is the point. A baseline that passes 7 of 10 and a changed condition that passes 5 of 10 look like a 20-point drop. Their intervals, 0.40 to 0.89 and 0.24 to 0.76, overlap almost entirely. At 100 trials each, 70 and 50 give 0.60 to 0.78 and 0.40 to 0.60, and the intervals no longer overlap.

Overlap is a picture, not a test, and it's a conservative one. To compare two conditions, test the difference. The handbook's comparison of two proportions uses a z statistic for large samples and Fisher's exact test for small ones. Applied to the hypothetical counts, Fisher's exact test on 7 of 10 against 5 of 10 gives a two-sided p of about 0.65: nothing. The z-test on 70 of 100 against 50 of 100 gives z ≈ 2.89, p ≈ 0.004: a clear difference, even though the intervals nearly touch. Now shrink the effect to something like the postmortem's 3% drop. Even 700 of 1,000 against 670 of 1,000 gives z ≈ 1.44, p ≈ 0.15. A thousand trials per condition, and a three-point effect still doesn't show.

Fig. 3 animates that comparison. The practical rule I take from it: when a lever changes the pass rate by less than your interval width, report "no detectable change at n trials", not "no change". Record token cost alongside pass rate, since several of these levers (effort, context files, evaluator agents) mostly move cost.

Pass rate also isn't the only number that matters. The eval guide separates pass@k, the chance of at least one success in k trials, from pass^k, the chance that all k succeed. Its example: at a 75% per-trial success rate, three passes in a row happen about 42% of the time. If your team reruns a failed session until it works, pass@k is what they experience. If a change has to be right the first time, pass^k is the one to watch, and it falls fast.

4. Start simple and add components only when they pay

The last step is a design rule rather than a diagnostic one. Building effective agents recommends "finding the simplest solution possible, and only increasing complexity when needed", and notes that agentic systems "often trade latency and cost for better task performance". The 2026 harness-design post is that principle applied: build the elaborate harness, then strip pieces out as the model makes them unnecessary. When you add an instruction, a tool, a subagent or an evaluator, write down which assumption about the model it encodes. That note is what tells you, six months later, whether it can go.

Every lever has a price

None of these levers is free, and the sources are candid about the costs.

Effort costs latency and tokens. The postmortem's whole first change was a latency trade, and Claude Code's model configuration says each level "trades token spend against capability" and that max "may show diminishing returns and is prone to overthinking, so test before adopting it broadly". In Spending your effort, a September 2026 post by Thariq Shihipar on the claude.dev blog that Claude Code's model configuration page links to, the author reports that effort changed "how much verification and edgecase testing Claude did and how much of its own judgement it used", and that on one underspecified build task with Opus 5.5 the runs took 1.5 minutes at low and 67 minutes at max. That's one person's informal tests, not a controlled study, but it's a useful reminder that higher effort also means the agent makes more decisions on your behalf.

Context costs attention, and money. Every instruction, tool description and file read spends the budget the context engineering post describes, and the AGENTS.md preprint puts a number on one common case: over 20% more inference cost on average, without a general improvement in success.

Extra agents are the expensive end. In the harness-design post, the planner-generator-evaluator harness cost $200 against $9 for a solo run of the same prompt. The post's own conclusion is that the evaluator pays only when the task sits beyond what the model does reliably alone.

Environment headroom changes what you're measuring. More resources reduce spurious failures up to a point, and past it, per the infrastructure-noise study, they reward agents that brute-force with heavy tools. Pick the limits that match production, not the ones that make the score look best.

There's one boundary I'm drawing on purpose. Permission modes and sandboxing are harness components, and they affect quality: an agent that can't run the test suite can't verify. But what an agent should be allowed to do, how to contain it and when a human must approve an action are security questions, and this site's Security track treats them separately, in its note on excessive agency and sandboxing. Here they appear only as variables you need to hold fixed when you compare runs.

If I had to compress this into a recommendation for a tech lead who owns a team's agent setup: make verification the first thing you invest in, pin and log the full system for every run you care about, and treat every instruction or harness tweak as an experiment with a hypothesis and a stated sample size. "The model got worse" is a hypothesis too, and it's the last one you should test, because it's the one you can do the least about.

What we don't know

Nobody has published a controlled study that varies all five levers on one task set. The evidence above is a patchwork: one postmortem, one resource study, one harness case study, one preprint on context files. Each varies one lever. What would settle it is an independent team running a fixed task set across model, harness version, effort, tool descriptions, context files and resource limits in a factorial design, with intervals.

Almost all of the evidence is from the vendor, about its own product. The postmortem, the noise study, the harness post and the tool-writing post are all Anthropic engineering writing about Claude. They're unusually forthcoming, and I've cited them for what happened and what Anthropic measured. They aren't independent replication. I couldn't find an independent study that isolates harness changes from model changes for any commercial coding agent.

The postmortem's 3% figure comes from an eval whose contents aren't published. We know it was one of "a broader set of evaluations" and that it showed the drop on Opus 4.6 and 4.7. We don't know the task mix, the trial count or the interval, so we can't say how large the effect was relative to noise. We don't even know whether "3%" means three percentage points or three percent of the score; the post doesn't say.

The tool-description claim has no published ablation. Anthropic attributes Claude 3.5 Sonnet's SWE-bench Verified result partly to tool-description refinements. Without the before-and-after numbers, the size of the tools lever is unknown. A public ablation on any agent would help.

Release notes don't record all harness changes. The 2.1.101 and 2.1.116 changelog entries don't mention the fixes the postmortem attributes to those versions. I couldn't confirm whether system prompt changes are ever listed. Until vendors publish prompt diffs or a behavior changelog, the version number is the best handle you have, and it's an incomplete one.

The time-of-day effect is unquantified. Anthropic observed pass rates varying with time of day and said it had "not formally quantified this effect". Anyone running agents in CI would want to know how big it is.

The context-files result is a preprint. The AGENTS.md study is on arXiv, revised to v3 on 2026-09-29, and hasn't been peer-reviewed as far as I could tell. Its conclusion is specific (overviews don't help; non-standard practices do), and it's the best evidence on that lever, but it should be read as provisional.

Some of the guidance is old. Building effective agents dates from December 2024. Its "simplest solution" principle reappears in the March 2026 harness-design post, but I couldn't find a recent Anthropic page that restates the ACI recommendations as current guidance. I've cited it for the idea and the anecdote, not as current product documentation.