tl;dr — Six posts of findings are worth nothing if nothing changes. So: we’re killing context-packing as the default for code questions, because handing the model tools beat every packing strategy on quality and cost. We’re changing what counts as a valid context representation — anything that strips line numbers is disqualified from citation-requiring work, which puts a hard requirement on our own retrieval product. And we’re keeping two things that turned out to be load-bearing: writing down the caveat you don’t have time to test, and shipping the smallest runnable cut instead of the correct plan. The correct plan sat unrun for ten weeks. The small cut found four bugs in a week, two of them ours. Publication found a fifth.
The boundary matters: this is five questions against one pinned Java repository, judged by one model in a single pass. That is enough to change our default workflow. It is not a universal ranking of context strategies, and we’ve tried to keep the claims below inside it.
The series, in order: nobody needed to fit the codebase in the window · 44x fewer tokens, and every citation was fake · our control group was broken · a finding about RAG that was a finding about my config · two axes of compression · what it costs to know · keep, change, kill (this post).
Everything, in one table
| strategy | score /12 | tokens in | $/question | verified cites | fabricated |
|---|---|---|---|---|---|
| agentic exploration | 11.40 | 288,342 | 0.6387 | 142 | 2 |
| llm-tldr → agentic | 11.00 | 288,436 | 0.6378 | 124 | 0 |
| full source dump | 10.80 | 404,878 | 0.8552 | 65 | 11 |
| Repomix | 10.40 | 406,277 | 0.9374 | 69 | 17 |
| prose-compressed dump | 10.40 | 363,613 | 0.7777 | 86 | 1 |
llm-tldr (extract) | 6.60 | 66,895 | 0.1520 | 83 | 1 |
| RAG retrieval | 5.00 | 4,237 | 0.0353 | 31 | 15 |
llm-tldr (semantic search) | 2.40 | 5,179 | 0.0236 | 2 | 29 |
The RAG row is confounded and a clean rerun is still blocked: the adapter handed the model index-prefixed paths (jsoup/src/…) that the judge resolved against the checkout root, so citations that were real scored as unresolved. Its fabrication count is an upper bound and its score moves with it (voitta-rag#57, #58). No other row is affected.
Everything below is a decision taken from that table. The arguments are in the six posts; this is what we do about them.
Kill: “get the codebase into the window” as the default
Start with the thing we’re stopping. The most useful result was the one we added almost as an afterthought: hand the model read_file, grep and glob, inject nothing, and it beat the full dump on quality while using fewer tokens and less money.
Every tool we benchmarked is an answer to how do I fit the codebase into the window. On this workload that’s the wrong question, and the tools inherit the wrongness.
So tool access is the default for code Q&A, and packing is the exception — reserved for when something rules out the agentic loop: no tool-calling surface, a hard latency ceiling, or per-query economics that can’t absorb 7–16 round trips.
We’re deliberately not generalising past code. Agentic exploration wins here because source is navigable: greppable identifiers, imports that point somewhere, filenames that mean something. Undifferentiated prose has none of that, and retrieval should do better there. The claim is about codebases.
Change: citability is a hard requirement
The benchmark’s whole discriminating power came from one rule — every claim needs a file:line, and the judge resolves it against real source. That rule caught what no quality score caught: the most token-efficient arm fabricated 29 of its 31 citations.
The mechanism is structural, not incidental. llm-tldr‘s semantic search reports "line": 1 for every unit; its extract subcommand carries real line_number fields and the same tool goes from 2 verified citations to 83. voitta-rag’s chunks carry a chunk_index and no lines at all. A representation that omits locations doesn’t degrade gracefully — the model still has to satisfy the citation requirement, so it invents a plausible number. You get confident, specific wrongness: the most expensive failure mode there is, because it’s the one that survives review.
- Line spans are a product requirement for our retrieval layer, filed as voitta-rag#52. Without them the component can’t be used where claims must be verifiable.
- We evaluate integrations, not tools. “llm-tldr scores 2.4” was never a true sentence — the same tool scores 6.6 through a different subcommand. Adoption decisions name the adapter.
- Citation resolution is standard in our evals, not a special feature of this one.
Change: what our own retrieval product is for
This one stings. Retrieval landed at the bottom, and we tested the obvious excuse — corpus mismatch — by indexing exactly the benchmark’s file set. It scored marginally worse — and then a fresh run with the adapter’s path bug fixed reversed even that, landing the two configurations 0.40 apart in the other direction. At five questions that is not a result, it is noise with a sign. The corpus excuse is untested, not refuted, and we are not going to claim otherwise in either direction. That is enough to stop treating retrieval as our default for code questions. It is not enough to declare its quality ceiling, and we’re not going to.
What it does settle is an earlier portfolio audit, which argued that voitta-rag owns the commodity half and hasn’t built its differentiator: retrieval is rentable, and the broker — routing across representations — is the actual asset, still unbuilt. The benchmark gives that direction evidence, though the retrieval rerun remains outstanding. The differentiator has two parts:
- Citability. Line spans, verifiable claims. Table stakes, and the thing missing.
- Routing. The right strategy varies by question class, and a layer that picks is worth more than a layer that retrieves.
Keep: the caveat you don’t have time to test
When we first published llm-tldr at 2.40, we wrote one sentence we couldn’t support with data: this measures one adapter, not the tool’s ceiling.
That sentence cost nothing and was the highest-value thing in the post. It’s why someone went back, tried extract, found the score nearly triples and the fabrications drop from 29 to 1 — surfacing the best quality-per-token arm in the benchmark, previously invisible.
Keep writing the falsifiable caveat. When you know the shape of what would overturn your result, say so in the artifact. It’s the cheapest insurance against publishing a wrong conclusion permanently, and it converts a dead end into a queued experiment.
Keep: ship the smallest runnable cut
The full plan was 30 questions × 5 modes = 150 judged runs, gated behind an interview session to draft the question set. Correct, thorough, and it sat unrun for ten weeks.
The version that ran was five questions and one repository. It produced the headline, two corrections to our own published claims, a pile of harness bugs and a filed product requirement — in about a week.
The lesson isn’t “small is better.” It’s that the spec was the blocker and its thoroughness was the reason. A plan that requires a meeting to start doesn’t start. Every axis we cut turned out to be a config entry plus one function once the harness existed.
What this changes on Tuesday
| when the question is… | reach for | why |
|---|---|---|
| high-stakes code Q&A | agentic exploration (read_file/grep/glob) | best answers, most verified citations, cheaper than packing |
| high-volume, low-stakes (triage, classification) | llm-tldr structural index | 61% of the quality at a sixth of the cost |
| code retrieval as the proposed default | wait for line spans | still bottom-tier after a clean re-run, and #52 blocks citation-requiring work |
| prose corpora | retrieval | the navigability argument for agentic exploration doesn’t hold |
How we run evals, as commitments
Not doctrine — the things we got wrong, written as what we’ll do next time:
- We will audit the control first and hardest. Everything is measured against it, so an error there multiplies across every row. Ours was understated by 4.2 points and invalidated a whole writeup. Two lines of assertion would have caught it.
- We will print what each arm actually received before theorising about why it lost. The better our explanation for a surprising result, the more suspicious we should be — a good mechanism is exactly what stops you checking the inputs.
- We will assert non-empty output per cell. A full bill with an empty answer is a config bug, not a model result. We hit it twice.
- We will measure the rendered answer, not
tokens_out. On a thinking model, thinking is billed as output and swamps everything: one arm shrank its visible answer 21% whiletokens_outtripled. - We will instrument cost per cell from the first run. Verification cost more than the work it verified ($50.74 against $28.28), and we only know that because we added the counter partway through.
What we’re running next
Graphify gets its own evaluation, not a row in this one. The obvious next move is to build a knowledge graph up front and work from it downstream. This benchmark can’t test that fairly: all five questions are “find/trace/plan against this specific code,” which is deterministic-structure territory, not sensemaking. Running it here would measure it on someone else’s home turf and confirm a foregone conclusion — the exact failure this series spent six posts documenting. It needs a relational question class built for it, with kill criteria written before the run.
The rerun is done, and its figures are in the table above.
Then breadth, in this order: repetitions before we interpret close scores, a second repository and language, and a prose corpus before we make any claim wider than code.
And a fixed cadence. New tools arrive faster than a benchmark can absorb them — three of the ten modes here weren’t in the plan when the plan was written. A benchmark that chases every entrant never has a still target and never converges; it just accrues arms. So the next pass is a retrospective: same instrument, whatever exists then, re-checking whether these conclusions still hold.
The thing under all of it
Every finding in this series is the same idea at a different altitude: a measurement you cannot audit is not a measurement.
The citation check is that applied to the model’s output. The control audit and the scope dump are it applied to our own harness — which failed the standard twice and published both failures as findings before we caught them. The tokens_out trap is it applied to the metric itself.
We drafted this conclusion saying four harness bugs. During publication week we found a fifth — the index-prefix mismatch above — so we corrected the count before publishing. The thesis demonstrated itself before the series was finished.
Five defects across the harness and its integrations, then, each capable of producing a wrong-but-plausible number rather than an error. The only reason we found any of them is that we’d built one check the numbers had to agree with. Without it, this series would have been a confident, well-formatted, reproducible recommendation for the wrong tool.
That’s the actual deliverable. Not the ranking — the instrument. The ranking is already going stale; the instrument is what makes the next retrospective cheap.
Harness, raw records, and full method: voitta-rag/benchmark/. Everything in this series is reproducible from the committed JSONL.
Correction, 2026-09-25. The RAG row in the table above has been corrected, and the paragraph on our own retrieval product rewritten. This post first published 5.20 for that arm and said the corpus-mismatch excuse was dead. A fresh run with the adapter’s path bug fixed gives 5.00, and reverses the corpus comparison by the same margin it originally ran — so that excuse is untested rather than refuted, and the sentence claiming otherwise is gone. The re-run and the fix are in voitta-rag#57 and #58.
Context benchmark series — part 7 of 7: ← Previous · the full index is at the top of this post.
