Keep, Change, Kill: What We’re Doing Differently

September 25, 2026September 25, 2026 Gregory Golberg method-and-apparatus, voitta agents, benchmark, blog-series-context-benchmark, claude code, evals, eventual-consistency, graphify, measurement, voitta-rag

tl;dr — Six posts of findings are worth nothing if nothing changes. So: we’re killing context-packing as the default for code questions, because handing the model tools beat every packing strategy on quality and cost. We’re changing what counts as a valid context representation — anything that strips line numbers is disqualified from citation-requiring work, which puts a hard requirement on our own retrieval product. And we’re keeping two things that turned out to be load-bearing: writing down the caveat you don’t have time to test, and shipping the smallest runnable cut instead of the correct plan. The correct plan sat unrun for ten weeks. The small cut found four bugs in a week, two of them ours. Publication found a fifth.

The boundary matters: this is five questions against one pinned Java repository, judged by one model in a single pass. That is enough to change our default workflow. It is not a universal ranking of context strategies, and we’ve tried to keep the claims below inside it.

The series, in order: nobody needed to fit the codebase in the window · 44x fewer tokens, and every citation was fake · our control group was broken · a finding about RAG that was a finding about my config · two axes of compression · what it costs to know · keep, change, kill (this post).

Everything, in one table

strategy	score /12	tokens in	$/question	verified cites	fabricated
agentic exploration	11.40	288,342	0.6387	142	2
llm-tldr → agentic	11.00	288,436	0.6378	124	0
full source dump	10.80	404,878	0.8552	65	11
Repomix	10.40	406,277	0.9374	69	17
prose-compressed dump	10.40	363,613	0.7777	86	1
llm-tldr (`extract`)	6.60	66,895	0.1520	83	1
RAG retrieval	5.00	4,237	0.0353	31	15
llm-tldr (`semantic search`)	2.40	5,179	0.0236	2	29

The RAG row is confounded and a clean rerun is still blocked: the adapter handed the model index-prefixed paths (jsoup/src/…) that the judge resolved against the checkout root, so citations that were real scored as unresolved. Its fabrication count is an upper bound and its score moves with it (voitta-rag#57, #58). No other row is affected.

Everything below is a decision taken from that table. The arguments are in the six posts; this is what we do about them.

Kill: “get the codebase into the window” as the default

Start with the thing we’re stopping. The most useful result was the one we added almost as an afterthought: hand the model read_file, grep and glob, inject nothing, and it beat the full dump on quality while using fewer tokens and less money.

Every tool we benchmarked is an answer to how do I fit the codebase into the window. On this workload that’s the wrong question, and the tools inherit the wrongness.

So tool access is the default for code Q&A, and packing is the exception — reserved for when something rules out the agentic loop: no tool-calling surface, a hard latency ceiling, or per-query economics that can’t absorb 7–16 round trips.

We’re deliberately not generalising past code. Agentic exploration wins here because source is navigable: greppable identifiers, imports that point somewhere, filenames that mean something. Undifferentiated prose has none of that, and retrieval should do better there. The claim is about codebases.

Change: citability is a hard requirement

The benchmark’s whole discriminating power came from one rule — every claim needs a file:line, and the judge resolves it against real source. That rule caught what no quality score caught: the most token-efficient arm fabricated 29 of its 31 citations.

The mechanism is structural, not incidental. llm-tldr‘s semantic search reports "line": 1 for every unit; its extract subcommand carries real line_number fields and the same tool goes from 2 verified citations to 83. voitta-rag’s chunks carry a chunk_index and no lines at all. A representation that omits locations doesn’t degrade gracefully — the model still has to satisfy the citation requirement, so it invents a plausible number. You get confident, specific wrongness: the most expensive failure mode there is, because it’s the one that survives review.

Line spans are a product requirement for our retrieval layer, filed as voitta-rag#52. Without them the component can’t be used where claims must be verifiable.
We evaluate integrations, not tools. “llm-tldr scores 2.4” was never a true sentence — the same tool scores 6.6 through a different subcommand. Adoption decisions name the adapter.
Citation resolution is standard in our evals, not a special feature of this one.

Change: what our own retrieval product is for

This one stings. Retrieval landed at the bottom, and we tested the obvious excuse — corpus mismatch — by indexing exactly the benchmark’s file set. It scored marginally worse — and then a fresh run with the adapter’s path bug fixed reversed even that, landing the two configurations 0.40 apart in the other direction. At five questions that is not a result, it is noise with a sign. The corpus excuse is untested, not refuted, and we are not going to claim otherwise in either direction. That is enough to stop treating retrieval as our default for code questions. It is not enough to declare its quality ceiling, and we’re not going to.

What it does settle is an earlier portfolio audit, which argued that voitta-rag owns the commodity half and hasn’t built its differentiator: retrieval is rentable, and the broker — routing across representations — is the actual asset, still unbuilt. The benchmark gives that direction evidence, though the retrieval rerun remains outstanding. The differentiator has two parts:

Citability. Line spans, verifiable claims. Table stakes, and the thing missing.
Routing. The right strategy varies by question class, and a layer that picks is worth more than a layer that retrieves.

Keep: the caveat you don’t have time to test

When we first published llm-tldr at 2.40, we wrote one sentence we couldn’t support with data: this measures one adapter, not the tool’s ceiling.

That sentence cost nothing and was the highest-value thing in the post. It’s why someone went back, tried extract, found the score nearly triples and the fabrications drop from 29 to 1 — surfacing the best quality-per-token arm in the benchmark, previously invisible.

Keep writing the falsifiable caveat. When you know the shape of what would overturn your result, say so in the artifact. It’s the cheapest insurance against publishing a wrong conclusion permanently, and it converts a dead end into a queued experiment.

Keep: ship the smallest runnable cut

The full plan was 30 questions × 5 modes = 150 judged runs, gated behind an interview session to draft the question set. Correct, thorough, and it sat unrun for ten weeks.

The version that ran was five questions and one repository. It produced the headline, two corrections to our own published claims, a pile of harness bugs and a filed product requirement — in about a week.

The lesson isn’t “small is better.” It’s that the spec was the blocker and its thoroughness was the reason. A plan that requires a meeting to start doesn’t start. Every axis we cut turned out to be a config entry plus one function once the harness existed.

What this changes on Tuesday

when the question is…	reach for	why
high-stakes code Q&A	agentic exploration (`read_file`/`grep`/`glob`)	best answers, most verified citations, cheaper than packing
high-volume, low-stakes (triage, classification)	`llm-tldr` structural index	61% of the quality at a sixth of the cost
code retrieval as the proposed default	wait for line spans	still bottom-tier after a clean re-run, and #52 blocks citation-requiring work
prose corpora	retrieval	the navigability argument for agentic exploration doesn’t hold

How we run evals, as commitments

Not doctrine — the things we got wrong, written as what we’ll do next time:

We will audit the control first and hardest. Everything is measured against it, so an error there multiplies across every row. Ours was understated by 4.2 points and invalidated a whole writeup. Two lines of assertion would have caught it.
We will print what each arm actually received before theorising about why it lost. The better our explanation for a surprising result, the more suspicious we should be — a good mechanism is exactly what stops you checking the inputs.
We will assert non-empty output per cell. A full bill with an empty answer is a config bug, not a model result. We hit it twice.
We will measure the rendered answer, not tokens_out. On a thinking model, thinking is billed as output and swamps everything: one arm shrank its visible answer 21% while tokens_out tripled.
We will instrument cost per cell from the first run. Verification cost more than the work it verified ($50.74 against $28.28), and we only know that because we added the counter partway through.

What we’re running next

Graphify gets its own evaluation, not a row in this one. The obvious next move is to build a knowledge graph up front and work from it downstream. This benchmark can’t test that fairly: all five questions are “find/trace/plan against this specific code,” which is deterministic-structure territory, not sensemaking. Running it here would measure it on someone else’s home turf and confirm a foregone conclusion — the exact failure this series spent six posts documenting. It needs a relational question class built for it, with kill criteria written before the run.

The rerun is done, and its figures are in the table above.

Then breadth, in this order: repetitions before we interpret close scores, a second repository and language, and a prose corpus before we make any claim wider than code.

And a fixed cadence. New tools arrive faster than a benchmark can absorb them — three of the ten modes here weren’t in the plan when the plan was written. A benchmark that chases every entrant never has a still target and never converges; it just accrues arms. So the next pass is a retrospective: same instrument, whatever exists then, re-checking whether these conclusions still hold.

The thing under all of it

Every finding in this series is the same idea at a different altitude: a measurement you cannot audit is not a measurement.

The citation check is that applied to the model’s output. The control audit and the scope dump are it applied to our own harness — which failed the standard twice and published both failures as findings before we caught them. The tokens_out trap is it applied to the metric itself.

We drafted this conclusion saying four harness bugs. During publication week we found a fifth — the index-prefix mismatch above — so we corrected the count before publishing. The thesis demonstrated itself before the series was finished.

Five defects across the harness and its integrations, then, each capable of producing a wrong-but-plausible number rather than an error. The only reason we found any of them is that we’d built one check the numbers had to agree with. Without it, this series would have been a confident, well-formatted, reproducible recommendation for the wrong tool.

That’s the actual deliverable. Not the ranking — the instrument. The ranking is already going stale; the instrument is what makes the next retrospective cheap.

Harness, raw records, and full method: voitta-rag/benchmark/. Everything in this series is reproducible from the committed JSONL.

Correction, 2026-09-25. The RAG row in the table above has been corrected, and the paragraph on our own retrieval product rewritten. This post first published 5.20 for that arm and said the corpus-mismatch excuse was dead. A fresh run with the adapter’s path bug fixed gives 5.00, and reverses the corpus comparison by the same margin it originally ran — so that excuse is untested rather than refuted, and the sentence claiming otherwise is gone. The re-run and the fix are in voitta-rag#57 and #58.

Context benchmark series — part 7 of 7: ← Previous · the full index is at the top of this post.

What It Costs to Know

September 24, 2026September 25, 2026 Gregory Golberg method-and-apparatus, voitta benchmark, blog-series-context-benchmark, claude code, cost, evals, measurement

tl;dr — A colleague asked us to benchmark four context-compression tools: “should cost <$20 for a controlled eval.” It cost $79.03. Answering the questions was $28.28; grading the answers was $50.74. Grading cost more than the work because grading meant an agent opening the repository and checking whether every cited file and line number actually exists. That check is the entire reason we learned anything — the cheapest, most token-efficient tool in the lineup fabricated 29 of its 31 citations, and no amount of measuring token ratios would have caught it. The $20 version of this benchmark exists. It recommends the wrong tool.

The ask

May 14th, in Slack, after someone posted Repomix and someone else posted llm-tldr:

If one of you get time can you run and eval on the same codebase for the same task and let me know if firstly these actually improve the output and secondly which one is better should cost <$20 for a controlled eval

That is a good ask. It’s specific, it’s scoped, it names a budget, and the budget is a reasonable guess. Four tools, one codebase, a handful of questions — twenty dollars of API calls sounds about right.

Then the plan we wrote in response was 30 questions × 5 modes = 150 judged runs, gated behind an interview session to draft the question set. And it sat unrun for ten weeks, because that is not a thing you do on a Tuesday.

What finally unblocked it was cutting it to five questions and one repository. What made it expensive was the part we didn’t cut.

The bill

	cost	share
answering (70 cells, Claude Sonnet 5)	$28.28	36%
judging (70 cells, Claude Opus 5 + repo tools)	$50.74	64%
total	$79.03

Components and total are rounded independently from unrounded per-cell costs, so the two rows above do not add to the total exactly.

Roughly 4x the target, and the overrun is almost entirely one line item: verification costs more than the thing being verified.

Why grading is expensive

The naive way to score a benchmark like this is to ask a model whether the answer looks good. That’s one API call per cell, it’s cheap, and it measures fluency.

We required every factual claim to carry a file:line citation, and then gave the judge read_file, grep, and glob over the actual repository with instructions to resolve each one. Does Tokeniser.java:135 exist? Does it say what the answer says it says?

That turns each grading pass into an agent loop — 4 to 16 tool-calling turns per cell in our runs, each carrying the accumulated transcript. Cost per judged cell ranged from $0.11 to $2.71 and averaged $0.72.

We are paying for the difference between plausible and true, and that difference is priced like the labour it is.

What the expensive part bought

One number, which paid for the whole exercise:

	score /12	tokens in	verified citations	fabricated
llm-tldr	2.40	5,179	2	29
full source dump	10.80	404,878	65	11

llm-tldr cut input tokens 44x. On a token-savings benchmark it wins outright. On a plausibility-scored benchmark it does fine — the answers are well-structured, confident, and specific.

It fabricated 29 of 31 source citations, with zero correct citations on three of the five questions.

A $20 benchmark does not find this. It reports a 97% token reduction, notes that answer quality “held up reasonably,” and recommends the tool. Then someone adopts it, and the cost moves from your API bill to your code review — where it’s paid in engineer-hours by people who don’t know the citations are unreliable.

There’s a second thing it bought, which is that the same instrument caught our own mistakes. Two of our four harness bugs — a control group biased against large files, a filter that silently swapped the retrieval corpus — produced plausible numbers rather than errors, and were only visible because the citation column disagreed with the score column. We published one of them as a finding before we caught it.

The honest limits

Since this post is about what the money bought, it should be equally clear about what it didn’t.

Five questions. Enough to catch a large effect, not enough to rank close ones. The 10.40–10.80 cluster — full dump, Repomix, prose-compressed dump — is a tie as far as this data can tell. Believe the big gaps, not the ordering within a point.
One repository, deliberately unfamiliar. The “we already know this codebase” case, where structural indexes should do best, isn’t measured at all.
Single judge, single pass, no inter-rater check. We verified the judge’s citation resolutions spot-wise, not systematically.
Agentic token counts are cumulative across a tool loop, and aren’t directly comparable to a one-shot mode’s single request.

Any of these could be bought down. All of them cost more money.

The actual lesson about eval budgets

The instinct behind “<$20 for a controlled eval” is right: don’t gold-plate the measurement, get a number, move on. We agree enough that shrinking the plan is the only reason this ever ran.

But there’s a specific thing you cannot cut, and it’s the thing that’s expensive: the eval has to check something the model cannot fake. Fluency is free to measure and free to fake. Token counts are free to measure and don’t tell you whether the answer is true. A resolvable citation is neither.

So the trade isn’t “cheap eval vs. thorough eval.” It’s:

$20 buys you a token-ratio comparison and a confident recommendation, which in our case was the wrong tool.
$79 buys you the knowledge that the recommendation was wrong, plus the mechanism, plus — because the same instrument turned on our own harness — the discovery that two of our published findings were artifacts.

Sixty-three dollars is cheap for finding out you were about to be wrong in public.

And the thing about the $20 version is that it doesn’t feel wrong. It produces a table, the table has numbers, the numbers are reproducible. That’s the trap: a benchmark that measures the wrong thing doesn’t come back empty. It comes back confident.

Next and last in this series: keep, change, kill — what we’re doing differently. Earlier: why nobody needed to fit the codebase in the window, 44x fewer tokens and every citation was fake, our control group was broken, I published a finding about RAG that was a finding about my config, and two axes of compression.

Harness, raw records, and full method: voitta-rag/benchmark/. Answering on Claude Sonnet 5, judging on Claude Opus 5, both at effort high.

Correction, 2026-09-25. The cost figures have been corrected to $28.28 answering and $50.74 judging, $79.03 over 70 cells, after the RAG arms were re-run. This post first published $28.25, $51.90 and $80.15. The argument — that grading cost more than the work it graded — is unchanged and slightly understated by the new numbers. The re-run and the fix are in voitta-rag#57 and #58.

Context benchmark series — part 6 of 7: ← Previous · Series index · Next →

Two Axes of Compression, and a Trap That Makes One Unmeasurable

September 23, 2026September 25, 2026 Gregory Golberg method-and-apparatus, voitta benchmark, blog-series-context-benchmark, caveman, claude code, compression, evals, measurement, repomix

tl;dr — Token compression has two independent axes: what you feed the model (tokens in) and what it writes back (tokens out). Three findings from measuring both. Repomix, pointed at the same file globs as a plain cat of the repo, produced more characters than the plain dump — its advertised ~70% reduction is file selection, not compression. A prose compressor on source code buys 9.4% by destroying the punctuation that makes it code, and costs only 0.4 points, which is its own uncomfortable finding. And you cannot measure the output axis with output-token counts on a reasoning model: our compressed-output run shrank the visible answer 21% while its tokens_out tripled.

The two axes

Most “save tokens” tooling is sold as one category, and it isn’t. There are two:

Tokens in — shrink the context you send. Repomix, llm-tldr, RAG retrieval, prose compressors applied to a dump.
Tokens out — shrink what the model writes back. Instruct it to answer tersely.

They’re orthogonal. An output-side compressor rides on top of any input-side strategy, which means the honest way to evaluate them is a grid, not a list. We ran the input-side arms, then re-ran a representative subset with the output-side overlay on.

Finding 1: Repomix is the same dump with a nicer cover page

Repomix packs a repository into one AI-friendly file and is widely cited for ~70% token reduction. Pointed at the same include/exclude globs we used for a plain concatenation of the same files:

	characters	score /12	tokens in
plain source dump	1,119,819	10.80	404,878
Repomix	1,127,414	10.40	406,277

Repomix produced 7,595 more characters than cat-ing the files.

This isn’t a knock on the tool, and the 70% figure isn’t dishonest — it’s just measuring something else. Repomix’s reduction comes from file selection: honouring .gitignore, skipping binaries and lockfiles and node_modules, dropping build artifacts. Against a naive “send the whole working directory” baseline, that’s an enormous and genuine saving.

But we’d already scoped our globs to **/*.java minus tests. There was nothing left to select. What remains is formatting — a directory tree, a header block, per-file separators — and formatting costs tokens rather than saving them.

The general point: a compression ratio is a ratio against something. Before adopting a tool on a headline percentage, check what the denominator was. If your pipeline already scopes its inputs, a selection-based tool has already had its win taken.

Finding 2: a prose compressor on code, and how little the model needs

caveman-compression strips grammar an LLM can reconstruct — articles, connectives, passive constructions. We used the rule-based spaCy variant rather than the default LLM-backed one, on purpose: a non-deterministic compressor inside a benchmark cell makes the cell unattributable, and it would put a second vendor’s model inside our measurement path.

Applied to the source dump:

	characters	score /12	tokens in	$/q
plain dump	1,119,819	10.80	404,878	$0.8552
caveman-compressed	1,014,004	10.40	363,613	$0.7777

9.4% smaller, 0.4 points. Roughly neutral.

Which is startling once you look at what it does to Java:

// before
public class Attribute implements Map.Entry<String,String>, Cloneable {

// after
public class Attribute implements Map. Entry < String String >   Cloneable

Commas gone. Angle brackets spaced apart. Map.Entry split across a sentence boundary. This is not valid Java in any sense — a parser would reject it instantly — and the model scored 10.40 out of 12 on it.

Be fair to the tool: it’s built for prose and we pointed it at source code. This measures a mismatch, not the tool used as intended, and 9.4% on input it was never designed for is respectable.

The finding isn’t about the compressor. It’s about how much syntax the model actually needs, which is apparently much less than the syntax the compiler needs. That’s a genuinely interesting property and probably a bad thing to rely on.

Finding 3: the trap

The output-side overlay works. Instruct the model to answer in compressed style and the rendered answer gets meaningfully shorter at little quality cost:

mode	answer chars	with overlay	score	with overlay
full dump	5,699	4,496 (−21%)	10.80	10.60
agentic exploration	6,389	4,029 (−37%)	11.40	10.40
RAG (corpus-matched index)	6,018	3,696 (−39%)	5.40	5.60
llm-tldr	2,387	1,862 (−22%)	2.40	3.40

21–39% shorter for roughly zero to one point either way. Cheap, real, worth having. (The RAG row now reflects a clean re-run; two of the four arms score slightly higher with the overlay than without, which at five questions is not distinguishable from noise and is not a claim that compression improves answers.)

Now the same experiment measured the way you’d instinctively measure it — by counting output tokens:

	rendered answer	`tokens_out`
full dump	5,699 chars	4,547
full dump + output compression	4,496 chars (−21%)	14,859 (+227%)

The visible answer shrank by a fifth. The billed output tokens more than tripled.

tokens_out bills thinking tokens and response text together. On a reasoning model, thinking usually dominates, and it varies enormously with how hard the model decides the turn is. The overlay changed how the model approached the task — apparently prompting more deliberation about what to cut — and that swamped the text delta by an order of magnitude.

Anyone benchmarking output-side compression against tokens_out on a thinking model is measuring reasoning-depth noise and calling it compression. You will get a number, it will be reproducible, and it will point the wrong way.

Measure the rendered answer. len(response_text), or token-count the text blocks specifically. And if you’re doing cost work, keep the two apart: thinking tokens are a real cost you should track, they’re just not what an output-style instruction controls.

Next in this series: what it costs to know any of this — and why grading the answers cost more than producing them.

Harness, raw records, and full method: voitta-rag/benchmark/.

Correction, 2026-09-25. The RAG row in the output-overlay table has been corrected after that arm was re-run with an adapter path bug fixed, and the row relabelled to name which index it used. It first published as 4,665 → 3,659 chars scoring 4.80 → 4.60. The re-run and the fix are in voitta-rag#57 and #58.

Context benchmark series — part 5 of 7: ← Previous · Series index · Next →

I Published a Finding About RAG. It Was a Finding About My Config.

September 23, 2026September 25, 2026 Gregory Golberg method-and-apparatus, voitta benchmark, blog-series-context-benchmark, claude code, evals, measurement, postmortem, rag, voitta-rag

tl;dr — Our retrieval arm kept returning changelogs instead of source, so the model correctly refused to answer. I wrote it up with a satisfying mechanism: jsoup’s changelog describes parser behaviour in the same prose vocabulary the questions use, so it outranks the code. Plausible. Real numbers. Wrong. The include_folders filter is an exact match on a file’s parent directory, not a subtree prefix — so passing the repo name scoped retrieval to the five files sitting at the repo root and excluded all of src/. No error, just real, well-formed, confidently useless results. Then fixing it didn’t help, and why not is the actual finding.

The finding I published

Our RAG arm scored 5.2/12. Four of its five answers were refusals — the model saying, in effect, the source files I’d need aren’t in the retrieved context.

The retrieved chunks were all from CHANGES.md and change-archive.txt. So I wrote the obvious mechanism:

The folder was indexed whole, and hybrid retrieval on questions phrased in changelog vocabulary (“malformed start tags”, “charset conflict”) ranks CHANGES.md and change-archive.txt above the .java files, because jsoup’s changelog literally describes these behaviours in prose.

That is a good paragraph. It has a mechanism, it’s consistent with the data, and it makes a genuine point about hybrid search on repositories that contain prose. It went into a committed README as a finding about retrieval.

What was actually happening

To keep the RAG arm from retrieving over unrelated indexed folders — including, awkwardly, its own source — I’d scoped the search:

"voitta_rag_include_folders": ["jsoup"]

include_folders sounds like subtree scoping. It isn’t. Over MCP it’s an exact match on a chunk’s folder_path, and folder_path is the directory the file sits in, not the index root.

Files whose folder_path is exactly jsoup:

jsoup/CHANGES.md
jsoup/change-archive.txt
jsoup/README.md
jsoup/LICENSE
jsoup/SECURITY.md

Everything under src/ has a folder_path of jsoup/src/main/java/org/jsoup/... and was excluded. I had scoped the benchmark’s retrieval arm to five files, three of which are changelogs.

The model wasn’t outranked by prose. It was handed a changelog and asked about a parser, said so, and was correct every time.

The part that makes this worth writing up

The subtree expansion exists. It’s right there in mcp_server.search:

if user_name:
    ...
    if folder_normalized == active_normalized or \
       folder_normalized.startswith(active_normalized + "/"):

Prefix matching, exactly as you’d want. It runs under if user_name: — and the MCP tool signature has no user_name parameter. Over MCP that branch is unreachable, so include_folders falls through to an exact MatchAny against Qdrant.

The code that would have made my mental model correct was in the repository, being skipped, on a branch I couldn’t reach from the interface I was calling.

Why it survived review

Because it never failed. Consider what a wrong filter doesn’t do here:

It doesn’t error. Five files is a legitimate result.
It doesn’t return nothing. Empty results would have sent me straight to the config.
It doesn’t return garbage. The chunks were real, relevant-looking prose from the correct repository.
The model’s behaviour was exemplary — it recognised insufficient context and declined instead of confabulating. That’s the behaviour you want, and it made the arm look thoughtfully-failing rather than mis-configured.

Every signal pointed at “retrieval made a ranking decision I should analyse” rather than “retrieval was handed the wrong corpus.” The failure was epistemically camouflaged: it produced exactly the artifacts a real finding produces.

Then fixing it didn’t help

Then I fixed it. I enumerated the directories, passed them all, and re-ran. Retrieval now returned actual Java source — Entities.java for the entity-decoding question, correctly.

Then I went further and eliminated the corpus question entirely: built a second index containing exactly the 97 .java files the other arms see, no changelogs at all, and ran that too.

	score /12	verified cites	fabricated
whole checkout (233 files)	5.00	31	15
corpus-matched (97 `.java` files)	5.40	21	29

Caveat on the retrieval numbers, found after this was drafted: the adapter handed the model paths prefixed with the index name (jsoup/src/…) while the judge resolved citations against the checkout root (src/…), so citations that were real scored as unresolved. The fabricated counts here are upper bounds and the scores that depend on them are not comparable with the other arms. The harness strips the prefix now; these cells predate that, and a re-run is pending.

Matching the corpus moved it very little, and a fresh run with the adapter’s path bug fixed moved it again: the two configurations land 5.00 and 5.40, a gap the same size as the one that first ran the other way. These are independent runs weeks apart, not a re-grading of the same answers, so the reversal doesn’t prove the bug caused the original ordering either. Five questions cannot separate them. My replacement hypothesis — that corpus asymmetry was dragging the arm down — is not refuted so much as untested, which is a duller sentence than the one I published and the only one the data supports.

A second cause was sitting in the citation column the whole time. In the corpus-matched configuration there are roughly as many fabricated citations as verified ones; in the other the split is better than that. Neither is good, and the mechanism is identical in both. voitta-rag’s chunk records carry chunk_index and total_chunks and no line numbers. A model handed a perfectly correct chunk still cannot cite file:line, so it invents one. And the citation column itself carried a third artifact, found after this draft: the adapter injected index-prefixed paths the judge could not resolve, so some of what it counted as fabrication was a real citation wearing the wrong prefix. Three config-shaped artifacts in one arm, each of which looked like a finding. If adding line spans to the chunk record (voitta-rag#52) does not move the fabricated column, this diagnosis is wrong too.

That is the identical failure we’d already diagnosed in a completely different tool two posts ago — llm-tldr reporting "line": 1 for every result. Same root cause, different vendor, and I only recognised it because we’d been forced to look at citations rather than scores.

The transferable bit

Silent scope failures don’t crash. They produce publishable conclusions.

A crash sends you to the config. A plausible result sends you to the writeup. The more coherent your explanation of a surprising result, the more suspicious you should be — I had a good mechanism, and the quality of the story is exactly what stopped me checking the inputs.

So, concretely, before theorising about why an arm underperformed:

Print what it actually received. Not the score, not the answer — the raw retrieved payload. One line of debugging:

print(sorted({c["file_path"] for c in retrieved}))

Had I run that once, I’d have seen five filenames, none of them .java, and this would have been a config fix instead of a published finding, a correction, and a blog post.

And when a filter’s name implies semantics you haven’t verified — include_folders sounds like a subtree, exclude_paths sounds recursive, limit sounds per-query — spend the thirty seconds confirming it before building an experiment on top of it.

Next in this series: two axes of compression, and a measurement trap that makes one of them unmeasurable.

Harness, raw records, and full method: voitta-rag/benchmark/.

Correction, 2026-09-25. The two rows in the table above have been corrected and the paragraph beginning “Matching the corpus” has been rewritten. This post first published 5.20 and 4.80 and argued that matching the corpus made things measurably worse. A fresh run with the adapter’s path bug fixed gives 5.00 and 5.40 — the same size gap, running the other way. Two independent five-question runs disagree, so the claim is withdrawn rather than reversed: the effect is untested, not settled in either direction. The re-run and the fix are in voitta-rag#57 and #58.

Context benchmark series — part 4 of 7: ← Previous · Series index · Next →

Our Control Group Was Broken and It Cost Us 4.2 Points

September 22, 2026September 25, 2026 Gregory Golberg method-and-apparatus, voitta benchmark, blog-series-context-benchmark, claude code, evals, measurement, postmortem

tl;dr — The “full repository dump” baseline in our benchmark packed files until one didn’t fit, skipped it, and kept going. That’s not a budget, it’s a size filter. It quietly admitted 69 of 97 files and dropped the largest files, among them the three classes the architecture question asked about. The model correctly reported them “absent from the provided files,” and we scored that as the baseline’s ceiling. Fixing the packer: 6.60 → 10.80 out of 12. Every cross-strategy comparison we’d published was anchored to a control that was wrong by 4.2 points.

Fourteen lines of ordinary code

for relative in paths:
    body = open(os.path.join(repo_root, relative)).read()
    block = "===== FILE: {0} =====\n{1}\n".format(relative, body)
    if used + len(block) > budget:
        continue          # <-- this
    chunks.append(block)
    used += len(block)

continue, not break. When a file doesn’t fit the remaining budget, skip it and try the next one. It reads like politeness — pack as much as possible — and it passes review, because every individual line is correct.

What it actually implements is: prefer small files. Once the budget gets tight, every large file gets skipped and every small one still slides in. The bias grows as the budget fills, and it is invisible from the outside, because the output is a perfectly well-formed source dump.

At a 600,000-character budget over jsoup, it admitted 69 of 97 files. The ones it dropped were the largest: Parser.java, Tokeniser.java, TreeBuilder.java, HtmlTreeBuilder.java, HtmlTreeBuilderState.java, TokeniserState.java.

The question we then asked it

How is the parser subsystem structured? Describe the roles of the tokeniser, the tree builder, and the parser state machine.

Every class in that question was in the set the packer had silently dropped. Seven small files from parser/ were present — ParseError.java, ParseSettings.java, TokenData.java — so the dump looked like it covered the parser package.

The model answered honestly: those classes are “absent from the provided files.”

It was right. We scored it 4/12 and recorded it as what a full-context dump can achieve.

The number

	score /12
baseline, skip-and-continue packer	6.60
baseline, fixed	10.80

Our control was understated by 4.2 points out of 12, and everything else was measured against it. Every “this compressed mode reaches N% of full-context quality” claim in the first writeup was computed against a denominator that was wrong in the flattering direction — making every compression strategy look better than it was.

The second-order damage is worse than the first. A wrong treatment arm is one wrong row. A wrong control is every row.

Why nothing caught it

There was no error. No exception, no warning, no truncation notice. stop_reason was end_turn. The cost was normal. The answer was fluent, correctly formatted, and internally consistent.

And critically: the answer was true. The model wasn’t hallucinating or hedging — it accurately described the context it had been given. The bug was one layer up, in the gap between what we thought we handed it and what we actually did.

That gap is invisible to every check that examines the output.

What we changed

Two things, and the second matters more than the first.

Stop at the budget instead of skipping past it:

if used + len(block) > budget:
    break

Truncating at a prefix is still lossy — but it’s lossy in a way that’s ordered and legible rather than correlated with file size.

Make the artifact declare its own incompleteness:

Repository source dump. TRUNCATED: the first 69 of 97 matching files in path
order, cut off by a 600000-character budget. Files after 'parser/TokenData.java'
are absent from this dump but do exist in the repository.

Now the model knows the difference between “this class doesn’t exist” and “this class wasn’t given to me” — and so does anyone reading the transcript. Then we raised the budget so nothing truncates at all, and checked the result by hand: 97 of 97 files included, with Parser.java and Tokeniser.java present. The 97 is jsoup at d24b16d9, which the harness pins; the repository is at 96 today, so a rerun on a later checkout counts differently.

The general version

Every one of us has written continue where break belonged. That’s not the lesson. The lesson is about which bug you can afford to have there.

In production code, a size-biased packer is a mild performance quirk. In a benchmark’s control group, it’s a systematic error multiplied across every comparison you publish — and it presents as a result, which means it gets written up rather than investigated.

So: audit the control first, and audit it hardest. Not “does it run” but “does it contain what I claim it contains.” For a full-context baseline that is a three-line assertion. It was not in this harness when the bug bit; it is now, behind a baseline_require_full flag so a deliberately budgeted dump can still label itself instead of failing. Each line catches a different failure: the count catches a truncated dump, and the Parser.java line catches globs that matched nothing, where the count is 0 of 0 and passes:

assert included == len(paths), f"{included} of {len(paths)}"
assert any(p.endswith("/Parser.java") for p in paths[:included])

Ten seconds to write. It would have saved this entire post.

Next in this series: I published a finding about RAG. It was a finding about my config.

Harness, raw records, and full method: voitta-rag/benchmark/.

Context benchmark series — part 3 of 7: ← Previous · Series index · Next →

44x Fewer Tokens, and Every Citation Was Fake

September 22, 2026September 25, 2026 Gregory Golberg method-and-apparatus, voitta benchmark, blog-series-context-benchmark, citations, claude code, evals, llm-tldr, measurement

tl;dr — A context-compression tool cut our input tokens 44x and scored 2.4/12. The interesting part isn’t the score, it’s how it failed: it fabricated 29 of the 31 source citations it produced, with zero real citations on three of five questions. A benchmark that measured token savings would have recommended it enthusiastically. Then the turn: the tool was fine. One subcommand reports "line": 1 for every result, so the model had no real line numbers and invented plausible ones. Swap it for the subcommand that emits real ones and the same tool scores 6.6/12 with 83 verified citations and 1 fabricated — the best quality-per-token arm in the whole benchmark.

The setup

llm-tldr advertises 95% token savings and 155x faster queries. We first wrote about it next to voitta-rag in February, on how each feeds a codebase to a model; this is the first time either was scored. That is a big enough claim to be worth checking, so it went into our benchmark alongside a full source dump, Repomix, and RAG retrieval — same repository, same five questions, same prompt, only the injected context varying.

The scoring rule mattered more than we expected. Every answer had to carry a file:line citation for each factual claim, and a separate judge model with read-only access to the repository went and checked them. Not “is this plausible.” Does Tokeniser.java:135 exist, and does it say what the answer says it says.

The result

	score /12	tokens in	$/question
full source dump	10.80	404,878	$0.8552
llm-tldr	2.40	5,179	$0.0236

44x fewer tokens. 36x cheaper. And a score you would not ship.

But the score alone doesn’t tell you why, and the why is the whole point.

The citation column

	verified citations	fabricated
full source dump	65	11
llm-tldr	2	29

Twenty-nine confidently-formatted references to source locations that do not exist. Zero correct citations on three of the five questions.

This is the failure mode that a token-savings benchmark cannot see, and it is strictly worse than a low score. A model that says “I don’t know” costs you one retry. A model that says “the entity decoding happens in Entities.java:412” in a well-structured paragraph costs you a code review where someone opens Entities.java, finds 412 is in the middle of an unrelated method, and now distrusts the entire document.

We had built the citation check as a nice-to-have. It turned out to be the only instrument in the benchmark that could distinguish “compressed and correct” from “compressed and confabulating.”

The mechanism

tldr semantic search returns ranked code units with a line field. That field is 1. For everything.

The model receives a genuinely useful, genuinely relevant set of code units — the retrieval is working — with every location stamped as line 1. It has been instructed to cite file:line. It knows line 1 is wrong. So it does what a language model does with a plausible-shaped gap: it fills it with a plausible number.

Nothing in the pipeline is lying. The tool reports what it has, the model reports what it inferred, and the output is 29 fabricated citations.

The part where we were wrong

When we first published this we flagged it: this measures one adapter, not the tool’s ceiling. tldr context, structure, calls, and slice all existed and might behave differently. That caveat cost one sentence to write and turned out to be the most valuable thing in the post.

tldr structure was a dead end — no line numbers at all, and it parsed 50 of the 97 files. But tldr extract carries real line_number fields for every class and method. It’s per-file and takes no query, so semantic search still does the ranking; extract supplies the locations.

adapter	score	verified	fabricated	tokens in	$/q
`semantic search --expand`	2.40	2	29	5,179	$0.0236
`semantic search` → `extract`	6.60	83	1	66,895	$0.1520

Nearly triple the score. Fabricated citations from 29 to 1. Same tool, same index, same questions, same prompt. The only thing that changed is which subcommand fed the context.

What this actually means

Benchmark the integration, not the logo. “llm-tldr scores 2.4” was never a true sentence. “This adapter, on this question set, produced uncitable context” was, and it was the sentence we wrote down, and it is why we knew where to look.

The winning number is buried in the fixed row. At 6.60 for $0.15/question, extract delivers 61% of the full dump’s score for a sixth of its cost. If you’re optimising cost-per-point rather than peak quality, it’s the best arm in the benchmark — better on that axis than the agentic mode that beat everything on raw quality. That result was completely invisible until the citation check explained the first one.

The same bug is everywhere. Our RAG arm scored 5.2, partly because voitta-rag’s chunk records carry a chunk_index and no line numbers. (Its citation counts turned out to be confounded by a second bug — an index-name prefix the judge could not resolve — so treat them as upper bounds; the line-number gap is real either way.) Identical failure, different vendor, discovered only because we already knew the shape. If your retrieval layer returns text without locations, you are shipping this bug, and a quality score alone will not tell you.

Next in this series: our control group was broken and it cost us 4.2 points.

Harness, raw records, and full method: voitta-rag/benchmark/. Answering on Claude Sonnet 5, judging on Claude Opus 5, both at effort high.

Context benchmark series — part 2 of 7: ← Previous · Series index · Next →

Nobody Needed to Fit the Codebase in the Window

September 21, 2026September 25, 2026 Gregory Golberg method-and-apparatus, voitta agents, benchmark, blog-series-context-benchmark, claude code, evals, llm-tldr, measurement, rag, repomix, voitta-rag

tl;dr — We benchmarked five strategies for getting a Java codebase into an LLM’s context: a full source dump, Repomix, two llm-tldr adapters, and RAG retrieval. The winner was none of them. Giving the model read_file, grep, and glob and letting it go find things scored 11.4/12, against the full dump’s 10.8 — while using 29% fewer tokens and costing 25% less. It also produced 142 verified source citations against 2 fabricated, the cleanest record in the benchmark. Every tool in this category optimises how to pack the context window. On this question set, the winning move was not to pack it.

The question

A colleague dropped Repomix in Slack — pack your whole repo into one AI-friendly file, ~70% token reduction. Someone else pointed at llm-tldr — 95% token savings, 155x faster queries. A third person asked the only question that matters:

If one of you get time can you run an eval on the same codebase for the same task and let me know if these actually improve the output and which one is better

So we did. One repository (jsoup, 97 Java files, deliberately one nobody on the team knew), five questions spanning five kinds of thing you actually ask about code, and every strategy answering the identical prompt with only the injected context varying.

The scoring, because it’s the part that matters

Every answer had to carry a file:line citation for every factual claim. The judge — a separate model with read-only read_file, grep, and glob over the repository — then went and checked them. Not “does this look right.” Does Tokeniser.java:135 exist, and does it say what the answer claims.

That produces two numbers per answer: a quality score out of 12, and a count of citations that resolved against real source versus citations that didn’t. The second number is the one that earns its keep, and a later post in this series is entirely about what it caught.

The result

strategy	score /12	tokens in	$/question	verified cites	bogus
agentic exploration	11.40	288,342	0.6387	142	2
llm-tldr → agentic	11.00	288,436	0.6378	124	0
full source dump	10.80	404,878	0.8552	65	11
Repomix	10.40	406,277	0.9374	69	17
prose-compressed dump	10.40	363,613	0.7777	86	1
llm-tldr (`extract`)	6.60	66,895	0.1520	83	1
RAG retrieval	5.00	4,237	0.0353	31	15
llm-tldr (`semantic search`)	2.40	5,179	0.0236	2	29

One caveat on the RAG row, found after this was drafted and before it was published: its bogus count is an upper bound. The adapter handed the model paths prefixed with the index name (jsoup/src/…) while the judge resolved citations against the checkout root (src/…), so citations that were real scored as unresolved — the prefix is visible in the judge’s notes on 13 of 15 retrieval answers. The harness strips it now; these numbers predate that. It touches no other arm, and the arm it flatters least is the one we build.

The top line is a mode we added almost as a control — no context building at all, just hand the model the same three read-only tools the judge uses and let it explore. It won on quality, it won on citation accuracy by a wide margin, and it was cheaper than the thing it beat.

Why it wins

Not because it’s clever. Because of what it has at the moment it makes a claim.

Every other strategy front-loads: build a representation of the codebase, inject it, hope the answer is in there. The representation is fixed before the model has read the question closely, so it is necessarily a guess about relevance — and whatever the representation dropped, the model cannot recover.

Agentic exploration defers. It reads the question, forms a hypothesis, greps for it, gets it wrong, greps again, opens the file, reads the actual lines. Seven to sixteen tool calls per question in our runs. When it finally writes Tokeniser.java:135, it is because it has line 135 on screen.

That is the whole mechanism behind the citation column. Verified-to-bogus for agentic exploration was 142:2. For the full dump, 65:11 — the dump had every line, but the model was reading a 405,000-token wall of text and lost track of where in it things were. For the cheapest compressed mode, 2:29.

Worth sitting with: the full dump contains strictly more information than the agentic mode ever sees, and still loses. Having the bytes in the window is not the same as being able to use them.

Two caveats we’re keeping

Cumulative tokens. The 288K for agentic exploration is summed across every turn of the tool loop, not one request. It is the honest number for cost, and it is not the same kind of number as a one-shot mode’s single request. We report it that way because it’s what the strategy actually costs to answer one question, but don’t put it in a bar chart next to a single-shot figure without the asterisk.

Five questions. Enough to catch a large effect, not enough to rank close ones. The 10.4–10.8 cluster — full dump, Repomix, prose-compressed dump — is a tie as far as this data can tell. The gaps worth believing are the big ones: agentic exploration over the compressed modes, and the two llm-tldr adapters against each other.

The uncomfortable implication

There’s a lot of engineering going into context compression right now, and this result doesn’t say that work is worthless — the compressed modes have a real argument, which is price. llm-tldr via its extract adapter got 61% of the baseline’s score for a sixth of the cost. If you’re running a million of these, that trade is the whole business.

But if you’re optimising for a correct answer, the ranking says: give the model tools and get out of the way. The context window is not a thing to be filled efficiently. It’s a workspace, and the model is better at deciding what belongs in it than our heuristics are.

Next in this series: the tool that cut input tokens 44x and fabricated 29 of its 31 citations — and why that turned out to be our fault, not the tool’s.

Harness, raw results, and full method: voitta-rag/benchmark/. Answering on Claude Sonnet 5, judging on Claude Opus 5, both at effort high. Total cost of the run: $79.03 over 70 scored cells, of which $50.74 was judging — which is its own post.

Correction, 2026-09-25. The RAG row has been corrected. Its adapter was handing the model index-prefixed paths that the judge could not resolve, so real citations were scored as fabrications; with that fixed and the arm re-run, it reads 5.00 with 31 verified citations to 15, where this post first published 5.20 with 26 to 24. The run cost is likewise corrected to $79.03. No other row changed, and the conclusion of this post does not depend on the RAG row. The re-run and the fix are in voitta-rag#57 and #58.

Context benchmark series — part 1 of 7: Series index · Next →

The maintainer merged the argument, not just the patch

August 10, 2026August 10, 2026 Gregory Golberg voitta ai, code review, eventual-consistency, measurement, open-source

Two weeks ago I wrote about a scanner that graded my orchestration a C. The complaint was construct validity: the score counted named tool invocations and called the result proficiency, so work done by any other mechanism registered as absence.

I’m now at 500/1000, B-Tier. That’s the least interesting thing in this post.

The interesting part is what happened between the two numbers, because it is the clearest example I’ve had recently of a disagreement that made both sides more correct.

What actually shipped

Four pull requests against aiqrank/plugin. Two merged the same day, one is open pending a review question, and one I closed myself — more on that below.

The merged ones are small and boring, which is the point:

command_diversity was in the shared rollup schema but only ever incremented by the Codex scanner. Every Claude Code user uploaded a hard 0. Not “this user runs one command over and over” — nobody was looking.
Plan artifacts were recognized only under docs/plans/ or .context/plans/. The same file, written for the same reason, counted or didn’t based on where a repo happened to keep it.

Both are the same defect wearing different clothes: in a schema shared across several collectors, a field that only one collector populates serializes as 0 for everyone else, and the wire format has no way to say not applicable here. Three completely different facts collapse into one byte — the user didn’t do it, this source never reports it, or the user did it by a means the collector doesn’t recognize.

That third one is the one that stings, and it’s the one no amount of adding fields will fix.

The part I got wrong

I asserted that a test fixture path was a bug: it resolved one directory above the repository root, so the fixtures could never be found. I wrote a patch, wrote a PR body explaining the error, and was quietly pleased with myself.

It wasn’t a bug. The path is correct for the layout the plugin actually ships from, where it sits inside a larger repository. I had only ever seen my own fork, where the root is one level shallower, and concluded from a single data point that the other layout was a mistake. The maintainer fixed it properly — resolve both roots, use whichever exists — and committed the missing fixtures besides.

I closed my PR. It was wrong in a way I couldn’t have seen from where I was standing, which is a different thing from being careless, and worth distinguishing.

That was the third correction of the exercise. I had already retracted a claim that MCP tool calls weren’t being counted (they were), and another about how session counters aggregate (I’d misread the rollup). Each retraction came from the same failure: I reasoned from a field’s name instead of finding the line that increments it.

The part he got right that I hadn’t thought of

On the plan-artifact PR, I argued the allowlist was arbitrary because repos organize themselves differently. He accepted it and added a better reason than mine: Claude Code’s own plan mode writes to ~/.claude/plans/, which the allowlist also missed. So the scanner wasn’t just failing to see third-party conventions. It was failing to see its own first-party output.

He then bumped the measurement version — the scanner stamps every row with a PLANNING_MEASUREMENT_VERSION so the server can tell old rows from new ones — from 1 to 2, because the definition of a plan artifact had genuinely changed. That is the discipline I had been asking for, applied without being asked, one PR after I asked for it.

The disagreement that stayed a disagreement

I also filed an issue arguing the deeper thing: that the choice of which tool names count as sophistication is a claim about the correct way to work, embedded in a number rather than stated. ORCHESTRATION_TOOLS = {"Agent"} is one string. Fan out across git worktrees and separate processes instead, and you’ve done strictly more work for zero credit.

His reply is the best thing to come out of this. Paraphrasing badly:

Version string: yes. Scores already carry an internal methodology version so users don’t see phantom drops when the math changes. It just isn’t surfaced. It will be, alongside plain-English descriptions of each dimension.
Most of the rubric is already published, and the rest will be. And the detail that saves everyone reverse-engineering: within a source, the base weights are equal. There’s no secret weight table.
The tuning coefficients stay private, and here’s the reasoning I have no rebuttal to: those numbers move as calibration data arrives, and publishing them turns every recalibration into a renegotiation. The transparency people actually want — what is this measuring, and why did my score change — is deliverable without them.
On not-observed versus zero: agreed in principle, but don’t write that PR yet. Once a source can say “I can’t report this,” the scorer has to decide whether a non-observation is excluded-and-renormalized or treated as zero, and those produce materially different rankings. He’d rather make that call deliberately than have it smuggled in through a schema change.

And on the framing itself: the tool sets encode a claim about what competent agent use looks like — conceded, with the refinement that the claim is taken partly on evidence and partly on what’s observable without heuristics, and that the current implementation doesn’t distinguish those two constraints. Which is a sharper statement of my own argument than I managed.

We still don’t agree on everything. I think outcome-based signals should displace proxies faster than he does. He thinks the wire-format change needs a scoring decision first, and he’s right that it does. Nobody folded.

The thing worth generalizing

He told me, before merging, that none of my PRs would move my score. reasoning_blocks feeds a dimension Claude Code doesn’t currently have. file_changes and effort_usage don’t feed any scored dimension. Merging them makes the data correct; it doesn’t make it count, and wiring it into scoring moves everyone’s numbers, so it isn’t a same-day follow-up.

He didn’t have to say that. I’d have found out eventually, in the least charitable possible way — by rescanning and seeing nothing move. Saying it up front costs something and buys nothing except that the next exchange starts from a true premise.

That’s the whole thing, really. Politeness is free, so it carries no information. Good faith is expensive, which is why it works. A sociopath can be unfailingly polite. What’s hard to fake is doing the work before making the claim: a patch with tests and before/after numbers attached, a stated kill condition for your own change (“if the allowlist was deliberate defense rather than oversight, this PR is wrong and I’d rather know”), and an admission when the change does nothing for you.

His version of paying that cost was a review question I’ve been thinking about since. I had counted subagent activity in one field with a stated reason, excluded it in another with a stated reason, and counted it in a third with no reason at all. His note: given the comparability argument you’re making, I’d rather it be a stated choice either way than an accident.

Intended but unstated is indistinguishable from accidental. That’s a code-review principle, and also a conversational one.

Which is just eventual consistency again

I argued recently that eventual consistency isn’t a metaphor for how distributed teams work, it’s a structural description. Independent nodes take local writes, diverge, and converge at the boundaries through explicit contracts.

This exchange is that, with two nodes and a pull request as the contract.

Neither of us had the whole picture. I could see my own transcripts and had no visibility into the scoring; he could see the distribution across all users and had no visibility into why one person’s worktree-based fan-out was invisible. Both local views were correct and neither was sufficient. The divergence wasn’t a failure state — it was replication lag, and the PRs were the anti-entropy protocol.

Convergence didn’t require either of us to have been right at the start. It required the writes to be legible enough to merge: small diffs, stated reasoning, verifiable numbers, and an explicit note when a change did nothing.

I said in that piece that integration is where all the residual difficulty concentrates, and that owning the merge is the remaining hard problem. I meant it about software fleets. It turns out to apply to arguments.

Thanks to @grahac for taking the patches, for the ones he improved on, and for telling me up front that they wouldn’t help my score.

Still a zero in Planning, incidentally. I don’t use plan mode and I don’t write plan files, so that one is measuring me accurately. Some zeros are just true.