tl;dr — The “full repository dump” baseline in our benchmark packed files until one didn’t fit, skipped it, and kept going. That’s not a budget, it’s a size filter. It quietly admitted 69 of 97 files and dropped the largest files, among them the three classes the architecture question asked about. The model correctly reported them “absent from the provided files,” and we scored that as the baseline’s ceiling. Fixing the packer: 6.60 → 10.80 out of 12. Every cross-strategy comparison we’d published was anchored to a control that was wrong by 4.2 points.
Fourteen lines of ordinary code
for relative in paths:
body = open(os.path.join(repo_root, relative)).read()
block = "===== FILE: {0} =====\n{1}\n".format(relative, body)
if used + len(block) > budget:
continue # <-- this
chunks.append(block)
used += len(block)continue, not break. When a file doesn’t fit the remaining budget, skip it and try the next one. It reads like politeness — pack as much as possible — and it passes review, because every individual line is correct.
What it actually implements is: prefer small files. Once the budget gets tight, every large file gets skipped and every small one still slides in. The bias grows as the budget fills, and it is invisible from the outside, because the output is a perfectly well-formed source dump.
At a 600,000-character budget over jsoup, it admitted 69 of 97 files. The ones it dropped were the largest: Parser.java, Tokeniser.java, TreeBuilder.java, HtmlTreeBuilder.java, HtmlTreeBuilderState.java, TokeniserState.java.
The question we then asked it
How is the parser subsystem structured? Describe the roles of the tokeniser, the tree builder, and the parser state machine.
Every class in that question was in the set the packer had silently dropped. Seven small files from parser/ were present — ParseError.java, ParseSettings.java, TokenData.java — so the dump looked like it covered the parser package.
The model answered honestly: those classes are “absent from the provided files.”
It was right. We scored it 4/12 and recorded it as what a full-context dump can achieve.
The number
| score /12 | |
|---|---|
| baseline, skip-and-continue packer | 6.60 |
| baseline, fixed | 10.80 |
Our control was understated by 4.2 points out of 12, and everything else was measured against it. Every “this compressed mode reaches N% of full-context quality” claim in the first writeup was computed against a denominator that was wrong in the flattering direction — making every compression strategy look better than it was.
The second-order damage is worse than the first. A wrong treatment arm is one wrong row. A wrong control is every row.
Why nothing caught it
There was no error. No exception, no warning, no truncation notice. stop_reason was end_turn. The cost was normal. The answer was fluent, correctly formatted, and internally consistent.
And critically: the answer was true. The model wasn’t hallucinating or hedging — it accurately described the context it had been given. The bug was one layer up, in the gap between what we thought we handed it and what we actually did.
That gap is invisible to every check that examines the output.
What we changed
Two things, and the second matters more than the first.
Stop at the budget instead of skipping past it:
if used + len(block) > budget:
breakTruncating at a prefix is still lossy — but it’s lossy in a way that’s ordered and legible rather than correlated with file size.
Make the artifact declare its own incompleteness:
Repository source dump. TRUNCATED: the first 69 of 97 matching files in path
order, cut off by a 600000-character budget. Files after 'parser/TokenData.java'
are absent from this dump but do exist in the repository.Now the model knows the difference between “this class doesn’t exist” and “this class wasn’t given to me” — and so does anyone reading the transcript. Then we raised the budget so nothing truncates at all, and checked the result by hand: 97 of 97 files included, with Parser.java and Tokeniser.java present. The 97 is jsoup at d24b16d9, which the harness pins; the repository is at 96 today, so a rerun on a later checkout counts differently.
The general version
Every one of us has written continue where break belonged. That’s not the lesson. The lesson is about which bug you can afford to have there.
In production code, a size-biased packer is a mild performance quirk. In a benchmark’s control group, it’s a systematic error multiplied across every comparison you publish — and it presents as a result, which means it gets written up rather than investigated.
So: audit the control first, and audit it hardest. Not “does it run” but “does it contain what I claim it contains.” For a full-context baseline that is a three-line assertion. It was not in this harness when the bug bit; it is now, behind a baseline_require_full flag so a deliberately budgeted dump can still label itself instead of failing. Each line catches a different failure: the count catches a truncated dump, and the Parser.java line catches globs that matched nothing, where the count is 0 of 0 and passes:
assert included == len(paths), f"{included} of {len(paths)}"
assert any(p.endswith("/Parser.java") for p in paths[:included])Ten seconds to write. It would have saved this entire post.
Next in this series: I published a finding about RAG. It was a finding about my config.
Harness, raw records, and full method: voitta-rag/benchmark/.
