tl;dr — Six posts of findings are worth nothing if nothing changes. So: we’re killing context-packing as the default strategy for code questions, because handing the model tools beat every packing strategy on quality and cost. We’re changing what we consider a valid context representation — anything that strips line numbers is now disqualified from citation-requiring work, which puts a hard requirement on our own retrieval product. And we’re keeping two things that turned out to be load-bearing: writing down the caveat you don’t have time to test, and shipping the smallest runnable cut instead of the correct plan. The correct plan sat unrun for ten weeks. The small cut found four bugs in a week, two of them ours.

---

This is the last post in a series about benchmarking the context layer — the headline result, the citation failure, the broken control, the filter that swapped a corpus, the two axes, and what it cost. Findings are the easy part. This is what we’re changing.

Kill: “get the codebase into the window” as the default

The single most useful result was the one we added almost as an afterthought. Agentic exploration — hand the model read_file, grep, glob, inject nothing — scored 11.40/12 on 288K cumulative tokens at $0.64/question, against the full dump’s 10.80 on 405K at $0.86. Better answers, fewer tokens, less money.

Every tool we benchmarked is an answer to how do I fit the codebase into the window. On this workload that’s the wrong question, and the tools inherit the wrongness. So:

For code Q&A, tool access is now the default and packing is the exception. We stop reaching for a context-building strategy first and reach for it only when something rules out the agentic loop — no tool-calling surface, a hard latency ceiling, or per-query economics that can’t absorb 7-16 round trips.

We’re deliberately not generalising this past code. Agentic exploration wins here because source is navigable: greppable identifiers, imports that point somewhere, filenames that mean something. A corpus of undifferentiated prose has none of that, and retrieval should do better there. The claim is about codebases, not about context in general.

Change: citability becomes a hard requirement, not a nice-to-have

The benchmark’s whole discriminating power came from one rule — every claim needs a file:line, and the judge resolves it against real source. That rule caught the thing no quality score caught: the most token-efficient arm fabricated 29 of 31 citations.

And the mechanism turned out to be structural, not incidental. Two different tools, two different vendors, one identical failure:

	what it returns	verified	fabricated
llm-tldr semantic search	"line": 1 for every unit	2	29
llm-tldr extract	real line_number fields	83	1
voitta-rag chunks	chunk_index, no lines	26	24

A representation that omits locations doesn’t degrade gracefully. The model still has to satisfy the citation requirement, so it invents a plausible number, and you get confident specific wrongness — the most expensive failure mode there is, because it’s the one that survives review.

So the changes:

• Line spans are a product requirement for our retrieval layer. Filed as voitta-rag#52. Not a backlog nice-to-have; without it the component cannot be used in any workflow that requires verifiable claims, which is most of them.
• We evaluate integrations, not tools. “llm-tldr scores 2.4” was never a true sentence — the same tool scores 6.6 through a different subcommand. Any adoption decision names the adapter.
• Citation resolution is standard in our evals, not a special feature of this one. It’s the only check we ran that distinguished compressed-and-correct from compressed-and-confabulating.

Change: what our own retrieval product is for

This one stings, and it sharpens a conclusion from an earlier portfolio audit.

Plain retrieval scored 5.20, the bottom tier — with the caveat that a path-prefix bug in the adapter cost that arm citation credit it had earned, so read the gap, not the decimal. We then tested the obvious excuse — corpus mismatch — by building an index of exactly the benchmark’s file set. It scored 4.80. Marginally worse. The excuse was wrong, and so was the replacement hypothesis we reached for next.

The prior audit’s finding was that voitta-rag owns commodity and hasn’t built its differentiator — the retrieval half is rentable, the broker (routing across representations) is the actual asset and is unbuilt. The benchmark converts that from an opinion into a measurement: retrieval quality is not a place we can win. The bottom of the table is not a position to defend.

What the data says the differentiator has to be:

1. Citability — line spans, verifiable claims (voitta-rag#52). Table stakes, and the thing missing.
2. Routing — the benchmark is a per-query argument for the broker. The right strategy varies by question class: agentic exploration wins overall, llm-tldr-structural is the best cost-per-point at $0.15/question, retrieval is cheapest per call and worst per answer. A layer that picks is worth more than a layer that retrieves.

That’s not a pivot, it’s the audit’s conclusion with numbers behind it.

Keep: the caveat you don’t have time to test

When we first published llm-tldr at 2.40, we wrote one sentence we couldn’t support with data: this measures one adapter, not the tool’s ceiling.

That sentence cost nothing and was the highest-value thing in the post. It’s the reason someone went back, tried extract, and found the score nearly triples and the fabricated citations drop from 29 to 1 — which also surfaced the best quality-per-token arm in the entire benchmark, previously invisible.

Keep writing the falsifiable caveat. When you know the shape of what would overturn your result, say so in the artifact. It’s the cheapest possible insurance against publishing a wrong conclusion permanently, and it converts a dead end into a queued experiment.

Keep: ship the smallest runnable cut

The full plan was 30 questions × 5 modes = 150 judged runs, gated behind an interview session to draft the question set. Correct, thorough, and it sat unrun for ten weeks.

The version that ran was five questions and one repository. It produced the headline result, two corrections to our own published claims, four harness bugs, and a filed product requirement — in about a week.

The lesson isn’t “small is better.” It’s that the spec was the blocker, and the spec’s thoroughness was the reason. A plan that requires a meeting to start doesn’t start. Every axis we cut turned out to be a config entry plus one function once the harness existed; we added five more modes afterwards without touching the runner.

Build the thing that produces a number this week. Expand along axes, not by rewriting.

Change: the daily workflow, and which tool comes out first

The findings above are product decisions. Here’s the part that changes what we actually type on a Tuesday.

Default move for a code question: give the agent the repo and tools. Not “build context, then ask.” This is the biggest habit change, and it mostly means deleting a step we used to think was diligence.

Reach for the structural index when cost dominates, not when quality does. llm-tldr-structural at $0.15/question for 6.60/12 is the right tool for high-volume, low-stakes passes — triage, first-pass classification, anything running over thousands of items where 61% of the quality at a sixth of the cost is the correct trade. It is the wrong tool for the question you’re going to act on.

Stop reaching for retrieval on code. Bottom tier (5.20), and roughly as many fabricated citations as verified — an upper bound on the fabrications, since a prefix artifact scored some real ones as unresolved. That’s not a tuning problem we should keep poking at; it’s a “wait for line spans” problem (voitta-rag#52). Retrieval stays in the toolkit for prose corpora, where the navigability argument that makes agentic exploration win doesn’t apply.

Match the tool to the question class, because the per-class table is not flat. Agentic exploration led everywhere, but the gaps varied a lot: on path-tracing it scored 11 against llm-tldr-structural’s 3, while on change-planning the spread was 11 to 9. Where the compressed arm is close, the cost difference is worth taking.

What we’re deliberately not answering: should graphify run first?

There’s an obvious next move — build a knowledge graph up front and let everything downstream work from it. We’re not answering it here, and the reason we’re not is the last thing this series has to say.

Graphify is untested. It’s in none of the tables above, and this benchmark can’t test it fairly: all five questions are “find/trace/plan against this specific code,” which is deterministic-structure territory, not sensemaking territory. Running graphify against them and publishing a number would measure it on someone else’s home turf and confirm a foregone conclusion — the exact failure this series spent six posts documenting.

The alternative was to hold the series until we’d drafted a relational question class and run the arm. We didn’t, and that’s a deliberate call rather than a shortcut.

We’ve argued elsewhere that agent-era delivery looks like eventual consistency: independent work converging on a shared truth, where the scarce job is no longer execution but owning the merge — “designing the seams, resolving the conflicts, deciding what converges and what gets discarded.”

This is a merge decision. The findings above have converged: they’re stable, they’re reproducible from committed records, and they change what we do on Monday. The graphify arm hasn’t converged — it doesn’t have a question class yet, let alone a number. Holding seven finished posts hostage to an eighth unfinished one isn’t rigour, it’s the failure mode that already cost this project ten weeks: a plan too complete to start.

So the merge is: ship what converged, discard the rest from this cut. Graphify becomes the next series, not a footnote in this one — which also means it gets a question class designed for what it’s actually good at, rather than being bolted onto five questions written for other tools.

And the honest constraint underneath: new tools arrive faster than any benchmark can absorb them. Three of the ten modes here didn’t exist in our plan when the plan was written. A benchmark that chases every new entrant never has a still target and therefore never converges — it just accrues arms. The discipline isn’t running every tool; it’s picking a still-enough target, converging, publishing, and revisiting on a cadence.

Which makes the next one a retrospective rather than a rerun: same instrument, new entrants, and a fresh look at whether the conclusions above still hold. The instrument is the durable asset. The ranking has a shelf life, and that’s fine — it’s supposed to.

Change: how we run evals

Concretely, the checklist that came out of getting this wrong twice:

• Audit the control first and hardest. Everything is measured against it, so an error there multiplies across every row. Ours was understated by 4.2 points and it invalidated a whole writeup. Two lines of assertion would have caught it.
• Print what each arm actually received before theorising about why it lost. The better your explanation for a surprising result, the more suspicious you should be — a good mechanism is exactly what stops you checking the inputs.
• Assert non-empty output per cell. A full bill with an empty answer is a config bug, not a model result. We hit it twice at different thresholds.
• Measure the rendered answer, not tokens_out. On a thinking model, thinking is billed as output and swamps everything: one arm shrank its visible answer 21% while tokens_out tripled.
• Instrument cost per cell from the start. Verification cost more than the work it verified ($51.90 vs $28.25) and we only know that because we added the counter partway through.

The thing under all of it

Every finding in this series is the same idea at a different altitude: a measurement you cannot audit is not a measurement.

The citation check is that applied to the model’s output. The control audit and the scope dump are it applied to our own harness — which failed the standard twice, and published both failures as findings before we caught them. The tokens_out trap is it applied to the metric itself.

The uncomfortable version: our harness produced four wrong-but-plausible numbers, and the only reason we found them is that we’d built one check the numbers had to agree with. Without it, this series would have been a confident, well-formatted, reproducible recommendation for the wrong tool.

That’s the actual deliverable. Not the ranking — the instrument. The ranking is already going stale; the instrument is what makes the next retrospective cheap.

---

Harness, raw records, and full method: voitta-rag/benchmark/. Everything in this series is reproducible from the committed JSONL.