tl;dr — A colleague asked us to benchmark four context-compression tools: “should cost <$20 for a controlled eval.” It cost $80.15. Answering the questions was $28.25; grading the answers was $51.90. Grading cost more than the work because grading meant an agent opening the repository and checking whether every cited file and line number actually exists. That check is the entire reason we learned anything — the cheapest, most token-efficient tool in the lineup fabricated 29 of its 31 citations, and no amount of measuring token ratios would have caught it. The $20 version of this benchmark exists. It recommends the wrong tool.

---

The ask

May 14th, in Slack, after someone posted Repomix and someone else posted llm-tldr:

If one of you get time can you run and eval on the same codebase for the same task and let me know if firstly these actually improve the output and secondly which one is better should cost <$20 for a controlled eval

That is a good ask. It’s specific, it’s scoped, it names a budget, and the budget is a reasonable guess. Four tools, one codebase, a handful of questions — twenty dollars of API calls sounds about right.

Then the plan we wrote in response was 30 questions × 5 modes = 150 judged runs, gated behind an interview session to draft the question set. And it sat unrun for ten weeks, because that is not a thing you do on a Tuesday.

What finally unblocked it was cutting it to five questions and one repository. What made it expensive was the part we didn’t cut.

The bill

	cost	share
answering (70 cells, Claude Sonnet 5)	$28.25	35%
judging (70 cells, Claude Opus 5 + repo tools)	$51.90	65%
total	$80.15

Roughly 4x the target, and the overrun is almost entirely one line item: verification costs more than the thing being verified.

Why grading is expensive

The naive way to score a benchmark like this is to ask a model whether the answer looks good. That’s one API call per cell, it’s cheap, and it measures fluency.

We required every factual claim to carry a file:line citation, and then gave the judge read_file, grep, and glob over the actual repository with instructions to resolve each one. Does Tokeniser.java:135 exist? Does it say what the answer says it says?

That turns each grading pass into an agent loop — 4 to 16 tool-calling turns per cell in our runs, each carrying the accumulated transcript. Cost per judged cell ranged from $0.11 to $2.71 and averaged $0.74.

We are paying for the difference between plausible and true, and that difference is priced like the labour it is.

What the expensive part bought

One number, which paid for the whole exercise:

	score /12	tokens in	verified citations	fabricated
llm-tldr	2.40	5,179	2	29
full source dump	10.80	404,878	65	11

llm-tldr cut input tokens 44x. On a token-savings benchmark it wins outright. On a plausibility-scored benchmark it does fine — the answers are well-structured, confident, and specific.

It fabricated 29 of 31 source citations, with zero correct citations on three of the five questions.

A $20 benchmark does not find this. It reports a 97% token reduction, notes that answer quality “held up reasonably,” and recommends the tool. Then someone adopts it, and the cost moves from your API bill to your code review — where it’s paid in engineer-hours by people who don’t know the citations are unreliable.

There’s a second thing it bought, which is that the same instrument caught our own mistakes. Two of our four harness bugs — a control group biased against large files, a filter that silently swapped the retrieval corpus — produced plausible numbers rather than errors, and were only visible because the citation column disagreed with the score column. We published one of them as a finding before we caught it.

The honest limits

Since this post is about what the money bought, it should be equally clear about what it didn’t.

• Five questions. Enough to catch a large effect, not enough to rank close ones. The 10.40–10.80 cluster — full dump, Repomix, prose-compressed dump — is a tie as far as this data can tell. Believe the big gaps, not the ordering within a point.
• One repository, deliberately unfamiliar. The “we already know this codebase” case, where structural indexes should do best, isn’t measured at all.
• Single judge, single pass, no inter-rater check. We verified the judge’s citation resolutions spot-wise, not systematically.
• Agentic token counts are cumulative across a tool loop, and aren’t directly comparable to a one-shot mode’s single request.

Any of these could be bought down. All of them cost more money.

The actual lesson about eval budgets

The instinct behind “<$20 for a controlled eval” is right: don’t gold-plate the measurement, get a number, move on. We agree enough that shrinking the plan is the only reason this ever ran.

But there’s a specific thing you cannot cut, and it’s the thing that’s expensive: the eval has to check something the model cannot fake. Fluency is free to measure and free to fake. Token counts are free to measure and don’t tell you whether the answer is true. A resolvable citation is neither.

So the trade isn’t “cheap eval vs. thorough eval.” It’s:

• $20 buys you a token-ratio comparison and a confident recommendation, which in our case was the wrong tool.
• $80 buys you the knowledge that the recommendation was wrong, plus the mechanism, plus — because the same instrument turned on our own harness — the discovery that two of our published findings were artifacts.

Sixty-three dollars is cheap for finding out you were about to be wrong in public.

And the thing about the $20 version is that it doesn’t feel wrong. It produces a table, the table has numbers, the numbers are reproducible. That’s the trap: a benchmark that measures the wrong thing doesn’t come back empty. It comes back confident.

---

Next and last in this series: keep, change, kill — what we’re doing differently. Earlier: why nobody needed to fit the codebase in the window, 44x fewer tokens and every citation was fake, our control group was broken, I published a finding about RAG that was a finding about my config, and two axes of compression.

Harness, raw records, and full method: voitta-rag/benchmark/. Answering on Claude Sonnet 5, judging on Claude Opus 5, both at effort high.