tl;dr — Six posts of findings are worth nothing if nothing changes. So: we’re killing context-packing as the default for code questions, because handing the model tools beat every packing strategy on quality and cost. We’re changing what counts as a valid context representation — anything that strips line numbers is disqualified from citation-requiring work, which puts a hard requirement on our own retrieval product. And we’re keeping two things that turned out to be load-bearing: writing down the caveat you don’t have time to test, and shipping the smallest runnable cut instead of the correct plan. The correct plan sat unrun for ten weeks. The small cut found four bugs in a week, two of them ours. Publication found a fifth.

The boundary matters: this is five questions against one pinned Java repository, judged by one model in a single pass. That is enough to change our default workflow. It is not a universal ranking of context strategies, and we’ve tried to keep the claims below inside it.

The series, in order: nobody needed to fit the codebase in the window · 44x fewer tokens, and every citation was fake · our control group was broken · a finding about RAG that was a finding about my config · two axes of compression · what it costs to know · keep, change, kill (this post).

Everything, in one table

strategy	score /12	tokens in	$/question	verified cites	fabricated
agentic exploration	11.40	288,342	0.6387	142	2
llm-tldr → agentic	11.00	288,436	0.6378	124	0
full source dump	10.80	404,878	0.8552	65	11
Repomix	10.40	406,277	0.9374	69	17
prose-compressed dump	10.40	363,613	0.7777	86	1
llm-tldr (extract)	6.60	66,895	0.1520	83	1
RAG retrieval (rerun pending)	5.20	4,337	0.0289	26	24
llm-tldr (semantic search)	2.40	5,179	0.0236	2	29

The RAG row is confounded and a clean rerun is still blocked: the adapter handed the model index-prefixed paths (jsoup/src/…) that the judge resolved against the checkout root, so citations that were real scored as unresolved. Its fabrication count is an upper bound and its score moves with it (voitta-rag#57, #58). No other row is affected.

Everything below is a decision taken from that table. The arguments are in the six posts; this is what we do about them.

Kill: “get the codebase into the window” as the default

Start with the thing we’re stopping. The most useful result was the one we added almost as an afterthought: hand the model read_file, grep and glob, inject nothing, and it beat the full dump on quality while using fewer tokens and less money.

Every tool we benchmarked is an answer to how do I fit the codebase into the window. On this workload that’s the wrong question, and the tools inherit the wrongness.

So tool access is the default for code Q&A, and packing is the exception — reserved for when something rules out the agentic loop: no tool-calling surface, a hard latency ceiling, or per-query economics that can’t absorb 7–16 round trips.

We’re deliberately not generalising past code. Agentic exploration wins here because source is navigable: greppable identifiers, imports that point somewhere, filenames that mean something. Undifferentiated prose has none of that, and retrieval should do better there. The claim is about codebases.

Change: citability is a hard requirement

The benchmark’s whole discriminating power came from one rule — every claim needs a file:line, and the judge resolves it against real source. That rule caught what no quality score caught: the most token-efficient arm fabricated 29 of its 31 citations.

The mechanism is structural, not incidental. llm-tldr‘s semantic search reports "line": 1 for every unit; its extract subcommand carries real line_number fields and the same tool goes from 2 verified citations to 83. voitta-rag’s chunks carry a chunk_index and no lines at all. A representation that omits locations doesn’t degrade gracefully — the model still has to satisfy the citation requirement, so it invents a plausible number. You get confident, specific wrongness: the most expensive failure mode there is, because it’s the one that survives review.

• Line spans are a product requirement for our retrieval layer, filed as voitta-rag#52. Without them the component can’t be used where claims must be verifiable.
• We evaluate integrations, not tools. “llm-tldr scores 2.4” was never a true sentence — the same tool scores 6.6 through a different subcommand. Adoption decisions name the adapter.
• Citation resolution is standard in our evals, not a special feature of this one.

Change: what our own retrieval product is for

This one stings. Retrieval landed at the bottom, and we tested the obvious excuse — corpus mismatch — by indexing exactly the benchmark’s file set. It scored marginally worse, so that excuse is dead. But the citation column driving much of the gap is the confounded one, and the rerun is still blocked. That is enough to stop treating retrieval as our default for code questions. It is not enough to declare its quality ceiling, and we’re not going to.

What it does settle is an earlier portfolio audit, which argued that voitta-rag owns the commodity half and hasn’t built its differentiator: retrieval is rentable, and the broker — routing across representations — is the actual asset, still unbuilt. The benchmark gives that direction evidence, though the retrieval rerun remains outstanding. The differentiator has two parts:

• Citability. Line spans, verifiable claims. Table stakes, and the thing missing.
• Routing. The right strategy varies by question class, and a layer that picks is worth more than a layer that retrieves.

Keep: the caveat you don’t have time to test

When we first published llm-tldr at 2.40, we wrote one sentence we couldn’t support with data: this measures one adapter, not the tool’s ceiling.

That sentence cost nothing and was the highest-value thing in the post. It’s why someone went back, tried extract, found the score nearly triples and the fabrications drop from 29 to 1 — surfacing the best quality-per-token arm in the benchmark, previously invisible.

Keep writing the falsifiable caveat. When you know the shape of what would overturn your result, say so in the artifact. It’s the cheapest insurance against publishing a wrong conclusion permanently, and it converts a dead end into a queued experiment.

Keep: ship the smallest runnable cut

The full plan was 30 questions × 5 modes = 150 judged runs, gated behind an interview session to draft the question set. Correct, thorough, and it sat unrun for ten weeks.

The version that ran was five questions and one repository. It produced the headline, two corrections to our own published claims, a pile of harness bugs and a filed product requirement — in about a week.

The lesson isn’t “small is better.” It’s that the spec was the blocker and its thoroughness was the reason. A plan that requires a meeting to start doesn’t start. Every axis we cut turned out to be a config entry plus one function once the harness existed.

What this changes on Tuesday

when the question is…	reach for	why
high-stakes code Q&A	agentic exploration (read_file/grep/glob)	best answers, most verified citations, cheaper than packing
high-volume, low-stakes (triage, classification)	llm-tldr structural index	61% of the quality at a sixth of the cost
code retrieval as the proposed default	wait for line spans and the clean rerun	the current result is confounded, and #52 blocks citation-requiring work
prose corpora	retrieval	the navigability argument for agentic exploration doesn’t hold

How we run evals, as commitments

Not doctrine — the things we got wrong, written as what we’ll do next time:

• We will audit the control first and hardest. Everything is measured against it, so an error there multiplies across every row. Ours was understated by 4.2 points and invalidated a whole writeup. Two lines of assertion would have caught it.
• We will print what each arm actually received before theorising about why it lost. The better our explanation for a surprising result, the more suspicious we should be — a good mechanism is exactly what stops you checking the inputs.
• We will assert non-empty output per cell. A full bill with an empty answer is a config bug, not a model result. We hit it twice.
• We will measure the rendered answer, not tokens_out. On a thinking model, thinking is billed as output and swamps everything: one arm shrank its visible answer 21% while tokens_out tripled.
• We will instrument cost per cell from the first run. Verification cost more than the work it verified, and we only know that because we added the counter partway through.

What we’re running next

Graphify gets its own evaluation, not a row in this one. The obvious next move is to build a knowledge graph up front and work from it downstream. This benchmark can’t test that fairly: all five questions are “find/trace/plan against this specific code,” which is deterministic-structure territory, not sensemaking. Running it here would measure it on someone else’s home turf and confirm a foregone conclusion — the exact failure this series spent six posts documenting. It needs a relational question class built for it, with kill criteria written before the run.

The rerun. One clean pass with every known harness defect fixed, with the earlier figures marked superseded rather than quietly replaced.

Then breadth, in this order: repetitions before we interpret close scores, a second repository and language, and a prose corpus before we make any claim wider than code.

And a fixed cadence. New tools arrive faster than a benchmark can absorb them — three of the ten modes here weren’t in the plan when the plan was written. A benchmark that chases every entrant never has a still target and never converges; it just accrues arms. So the next pass is a retrospective: same instrument, whatever exists then, re-checking whether these conclusions still hold.

The thing under all of it

Every finding in this series is the same idea at a different altitude: a measurement you cannot audit is not a measurement.

The citation check is that applied to the model’s output. The control audit and the scope dump are it applied to our own harness — which failed the standard twice and published both failures as findings before we caught them. The tokens_out trap is it applied to the metric itself.

We drafted this conclusion saying four harness bugs. During publication week we found a fifth — the index-prefix mismatch above — so we corrected the count before publishing. The thesis demonstrated itself before the series was finished.

Five defects across the harness and its integrations, then, each capable of producing a wrong-but-plausible number rather than an error. The only reason we found any of them is that we’d built one check the numbers had to agree with. Without it, this series would have been a confident, well-formatted, reproducible recommendation for the wrong tool.

That’s the actual deliverable. Not the ranking — the instrument. The ranking is already going stale; the instrument is what makes the next retrospective cheap.

---

Harness, raw records, and full method: voitta-rag/benchmark/. Everything in this series is reproducible from the committed JSONL.

Context benchmark series — part 7 of 7: ← Previous · the full index is at the top of this post.