We had a clean result. Our full agent configuration (the instructions, skills and hooks we have piled up) made the agent worse at real work. Replaying merged pull requests, the stripped-down setups matched the merged answer four times out of four. Our full setup managed two.

It even had a mechanism. We read the transcripts. In one failed cell, the agent with our config compared its work against origin/master, decided the change had already been merged, and reverted its own work. It finished with an empty diff. Our config tells it to check origin/master. Case closed: too much instruction, the agent overthinks.

It was wrong. The agent had read the answer key.

How a replay leaks

The experiment (part 1 has the setup) replayed real pull requests: start the agent at the commit before the PR, give it the task, compare what it produces with what actually merged.

We started each replay in a git worktree. A worktree is a second checkout of the same repository, and it shares the repository’s object store and refs. So from inside a replay at an old commit, the agent could still see everything after it: origin/master, the merge commit, and the landing branch that held the exact answer. Nothing hid them. A plain git log would not show them, but git log --all would, and git show would open any of them.

We went back and classified every counted replay cell by whether any command it ran touched a commit from the future. Eight of twelve did, spread across all three setups. The “four of four versus two of four” comparison was void.

And the cell with the mechanism, the agent that reverted its own work? It looked at origin/master, saw the real merged PR sitting there, and correctly concluded the work was already done. It was the most rational agent in the experiment. We had built it a world where the future was on disk.

Clever Hans was the horse who could do arithmetic until someone noticed he was reading his questioner’s face. The horse was not cheating. The setup was answering for him.

The fix

Each replay now gets a clone that holds only the parent commit and its ancestors:

git branch -f isolate <parent>
git clone --no-local --single-branch --no-tags --branch isolate <repo> <dest>
git -C <dest> remote remove origin
git -C <dest> reflog expire --expire=now --all

No remote, no tags, one branch, no reflog. We checked it: the merge commit is unreachable, and git log --all ends at the parent. The --no-local matters: by default, git clones a local path by hard-linking the source’s whole object store, future commits included.

What changed when we reran

With the leak closed, the full setup no longer lost on correctness: it matched the merged answer in 4 of 6 cells, the same as Anthropic’s default setup. What survived was cost: the same answers at roughly 1.6 to 1.7 times the price, and a hook of ours that blocked the repository’s own validator in every full-setup cell. The verdict went from “our config makes the agent wrong” to “our config makes the agent expensive.” Those lead to different decisions, and we had very nearly acted on the first one.

The same leak had voided the first version of another task in the same week. It was caught by the same audit and rerun the same way.

What did not need rerunning: the question-and-answer half of the experiment, because that repository’s history holds no answers to our questions. Knowing why a result is unaffected is part of the audit, not a footnote to it.

The pattern

It is not the first time an audit of our own instrument changed a result. Our control group was broken and it cost us 4.2 points; a citation judge resolved paths wrong and scored real citations as fabricated. Each time the bad number came with a plausible story. Each time the story was what made it dangerous.

So the rule we follow: before comparing the arms, audit what each arm could see. For replays of version-controlled work, that means asking whether the answer is reachable from the starting state, because if it is, some agent will find it, and it will be right to.

We wrote the technique up as a public skill, agent-eval-replay-history-leak.

Part 2 of a series on what our process experiments found. Part 1: the agent’s tests agreed with the agent. Next: say what done looks like.