Keep, Change, Kill: What We’re Doing Differently

September 25, 2026September 25, 2026 Gregory Golberg method-and-apparatus, voitta agents, benchmark, blog-series-context-benchmark, claude code, evals, eventual-consistency, graphify, measurement, voitta-rag

tl;dr — Six posts of findings are worth nothing if nothing changes. So: we’re killing context-packing as the default for code questions, because handing the model tools beat every packing strategy on quality and cost. We’re changing what counts as a valid context representation — anything that strips line numbers is disqualified from citation-requiring work, which puts a hard requirement on our own retrieval product. And we’re keeping two things that turned out to be load-bearing: writing down the caveat you don’t have time to test, and shipping the smallest runnable cut instead of the correct plan. The correct plan sat unrun for ten weeks. The small cut found four bugs in a week, two of them ours. Publication found a fifth.

The boundary matters: this is five questions against one pinned Java repository, judged by one model in a single pass. That is enough to change our default workflow. It is not a universal ranking of context strategies, and we’ve tried to keep the claims below inside it.

The series, in order: nobody needed to fit the codebase in the window · 44x fewer tokens, and every citation was fake · our control group was broken · a finding about RAG that was a finding about my config · two axes of compression · what it costs to know · keep, change, kill (this post).

Everything, in one table

strategy	score /12	tokens in	$/question	verified cites	fabricated
agentic exploration	11.40	288,342	0.6387	142	2
llm-tldr → agentic	11.00	288,436	0.6378	124	0
full source dump	10.80	404,878	0.8552	65	11
Repomix	10.40	406,277	0.9374	69	17
prose-compressed dump	10.40	363,613	0.7777	86	1
llm-tldr (`extract`)	6.60	66,895	0.1520	83	1
RAG retrieval	5.00	4,237	0.0353	31	15
llm-tldr (`semantic search`)	2.40	5,179	0.0236	2	29

The RAG row is confounded and a clean rerun is still blocked: the adapter handed the model index-prefixed paths (jsoup/src/…) that the judge resolved against the checkout root, so citations that were real scored as unresolved. Its fabrication count is an upper bound and its score moves with it (voitta-rag#57, #58). No other row is affected.

Everything below is a decision taken from that table. The arguments are in the six posts; this is what we do about them.

Kill: “get the codebase into the window” as the default

Start with the thing we’re stopping. The most useful result was the one we added almost as an afterthought: hand the model read_file, grep and glob, inject nothing, and it beat the full dump on quality while using fewer tokens and less money.

Every tool we benchmarked is an answer to how do I fit the codebase into the window. On this workload that’s the wrong question, and the tools inherit the wrongness.

So tool access is the default for code Q&A, and packing is the exception — reserved for when something rules out the agentic loop: no tool-calling surface, a hard latency ceiling, or per-query economics that can’t absorb 7–16 round trips.

We’re deliberately not generalising past code. Agentic exploration wins here because source is navigable: greppable identifiers, imports that point somewhere, filenames that mean something. Undifferentiated prose has none of that, and retrieval should do better there. The claim is about codebases.

Change: citability is a hard requirement

The benchmark’s whole discriminating power came from one rule — every claim needs a file:line, and the judge resolves it against real source. That rule caught what no quality score caught: the most token-efficient arm fabricated 29 of its 31 citations.

The mechanism is structural, not incidental. llm-tldr‘s semantic search reports "line": 1 for every unit; its extract subcommand carries real line_number fields and the same tool goes from 2 verified citations to 83. voitta-rag’s chunks carry a chunk_index and no lines at all. A representation that omits locations doesn’t degrade gracefully — the model still has to satisfy the citation requirement, so it invents a plausible number. You get confident, specific wrongness: the most expensive failure mode there is, because it’s the one that survives review.

Line spans are a product requirement for our retrieval layer, filed as voitta-rag#52. Without them the component can’t be used where claims must be verifiable.
We evaluate integrations, not tools. “llm-tldr scores 2.4” was never a true sentence — the same tool scores 6.6 through a different subcommand. Adoption decisions name the adapter.
Citation resolution is standard in our evals, not a special feature of this one.

Change: what our own retrieval product is for

This one stings. Retrieval landed at the bottom, and we tested the obvious excuse — corpus mismatch — by indexing exactly the benchmark’s file set. It scored marginally worse — and then a fresh run with the adapter’s path bug fixed reversed even that, landing the two configurations 0.40 apart in the other direction. At five questions that is not a result, it is noise with a sign. The corpus excuse is untested, not refuted, and we are not going to claim otherwise in either direction. That is enough to stop treating retrieval as our default for code questions. It is not enough to declare its quality ceiling, and we’re not going to.

What it does settle is an earlier portfolio audit, which argued that voitta-rag owns the commodity half and hasn’t built its differentiator: retrieval is rentable, and the broker — routing across representations — is the actual asset, still unbuilt. The benchmark gives that direction evidence, though the retrieval rerun remains outstanding. The differentiator has two parts:

Citability. Line spans, verifiable claims. Table stakes, and the thing missing.
Routing. The right strategy varies by question class, and a layer that picks is worth more than a layer that retrieves.

Keep: the caveat you don’t have time to test

When we first published llm-tldr at 2.40, we wrote one sentence we couldn’t support with data: this measures one adapter, not the tool’s ceiling.

That sentence cost nothing and was the highest-value thing in the post. It’s why someone went back, tried extract, found the score nearly triples and the fabrications drop from 29 to 1 — surfacing the best quality-per-token arm in the benchmark, previously invisible.

Keep writing the falsifiable caveat. When you know the shape of what would overturn your result, say so in the artifact. It’s the cheapest insurance against publishing a wrong conclusion permanently, and it converts a dead end into a queued experiment.

Keep: ship the smallest runnable cut

The full plan was 30 questions × 5 modes = 150 judged runs, gated behind an interview session to draft the question set. Correct, thorough, and it sat unrun for ten weeks.

The version that ran was five questions and one repository. It produced the headline, two corrections to our own published claims, a pile of harness bugs and a filed product requirement — in about a week.

The lesson isn’t “small is better.” It’s that the spec was the blocker and its thoroughness was the reason. A plan that requires a meeting to start doesn’t start. Every axis we cut turned out to be a config entry plus one function once the harness existed.

What this changes on Tuesday

when the question is…	reach for	why
high-stakes code Q&A	agentic exploration (`read_file`/`grep`/`glob`)	best answers, most verified citations, cheaper than packing
high-volume, low-stakes (triage, classification)	`llm-tldr` structural index	61% of the quality at a sixth of the cost
code retrieval as the proposed default	wait for line spans	still bottom-tier after a clean re-run, and #52 blocks citation-requiring work
prose corpora	retrieval	the navigability argument for agentic exploration doesn’t hold

How we run evals, as commitments

Not doctrine — the things we got wrong, written as what we’ll do next time:

We will audit the control first and hardest. Everything is measured against it, so an error there multiplies across every row. Ours was understated by 4.2 points and invalidated a whole writeup. Two lines of assertion would have caught it.
We will print what each arm actually received before theorising about why it lost. The better our explanation for a surprising result, the more suspicious we should be — a good mechanism is exactly what stops you checking the inputs.
We will assert non-empty output per cell. A full bill with an empty answer is a config bug, not a model result. We hit it twice.
We will measure the rendered answer, not tokens_out. On a thinking model, thinking is billed as output and swamps everything: one arm shrank its visible answer 21% while tokens_out tripled.
We will instrument cost per cell from the first run. Verification cost more than the work it verified ($50.74 against $28.28), and we only know that because we added the counter partway through.

What we’re running next

Graphify gets its own evaluation, not a row in this one. The obvious next move is to build a knowledge graph up front and work from it downstream. This benchmark can’t test that fairly: all five questions are “find/trace/plan against this specific code,” which is deterministic-structure territory, not sensemaking. Running it here would measure it on someone else’s home turf and confirm a foregone conclusion — the exact failure this series spent six posts documenting. It needs a relational question class built for it, with kill criteria written before the run.

The rerun is done, and its figures are in the table above.

Then breadth, in this order: repetitions before we interpret close scores, a second repository and language, and a prose corpus before we make any claim wider than code.

And a fixed cadence. New tools arrive faster than a benchmark can absorb them — three of the ten modes here weren’t in the plan when the plan was written. A benchmark that chases every entrant never has a still target and never converges; it just accrues arms. So the next pass is a retrospective: same instrument, whatever exists then, re-checking whether these conclusions still hold.

The thing under all of it

Every finding in this series is the same idea at a different altitude: a measurement you cannot audit is not a measurement.

The citation check is that applied to the model’s output. The control audit and the scope dump are it applied to our own harness — which failed the standard twice and published both failures as findings before we caught them. The tokens_out trap is it applied to the metric itself.

We drafted this conclusion saying four harness bugs. During publication week we found a fifth — the index-prefix mismatch above — so we corrected the count before publishing. The thesis demonstrated itself before the series was finished.

Five defects across the harness and its integrations, then, each capable of producing a wrong-but-plausible number rather than an error. The only reason we found any of them is that we’d built one check the numbers had to agree with. Without it, this series would have been a confident, well-formatted, reproducible recommendation for the wrong tool.

That’s the actual deliverable. Not the ranking — the instrument. The ranking is already going stale; the instrument is what makes the next retrospective cheap.

Harness, raw records, and full method: voitta-rag/benchmark/. Everything in this series is reproducible from the committed JSONL.

Correction, 2026-09-25. The RAG row in the table above has been corrected, and the paragraph on our own retrieval product rewritten. This post first published 5.20 for that arm and said the corpus-mismatch excuse was dead. A fresh run with the adapter’s path bug fixed gives 5.00, and reverses the corpus comparison by the same margin it originally ran — so that excuse is untested rather than refuted, and the sentence claiming otherwise is gone. The re-run and the fix are in voitta-rag#57 and #58.

Context benchmark series — part 7 of 7: ← Previous · the full index is at the top of this post.

What It Costs to Know

September 24, 2026September 25, 2026 Gregory Golberg method-and-apparatus, voitta benchmark, blog-series-context-benchmark, claude code, cost, evals, measurement

tl;dr — A colleague asked us to benchmark four context-compression tools: “should cost <$20 for a controlled eval.” It cost $79.03. Answering the questions was $28.28; grading the answers was $50.74. Grading cost more than the work because grading meant an agent opening the repository and checking whether every cited file and line number actually exists. That check is the entire reason we learned anything — the cheapest, most token-efficient tool in the lineup fabricated 29 of its 31 citations, and no amount of measuring token ratios would have caught it. The $20 version of this benchmark exists. It recommends the wrong tool.

The ask

May 14th, in Slack, after someone posted Repomix and someone else posted llm-tldr:

If one of you get time can you run and eval on the same codebase for the same task and let me know if firstly these actually improve the output and secondly which one is better should cost <$20 for a controlled eval

That is a good ask. It’s specific, it’s scoped, it names a budget, and the budget is a reasonable guess. Four tools, one codebase, a handful of questions — twenty dollars of API calls sounds about right.

Then the plan we wrote in response was 30 questions × 5 modes = 150 judged runs, gated behind an interview session to draft the question set. And it sat unrun for ten weeks, because that is not a thing you do on a Tuesday.

What finally unblocked it was cutting it to five questions and one repository. What made it expensive was the part we didn’t cut.

The bill

	cost	share
answering (70 cells, Claude Sonnet 5)	$28.28	36%
judging (70 cells, Claude Opus 5 + repo tools)	$50.74	64%
total	$79.03

Components and total are rounded independently from unrounded per-cell costs, so the two rows above do not add to the total exactly.

Roughly 4x the target, and the overrun is almost entirely one line item: verification costs more than the thing being verified.

Why grading is expensive

The naive way to score a benchmark like this is to ask a model whether the answer looks good. That’s one API call per cell, it’s cheap, and it measures fluency.

We required every factual claim to carry a file:line citation, and then gave the judge read_file, grep, and glob over the actual repository with instructions to resolve each one. Does Tokeniser.java:135 exist? Does it say what the answer says it says?

That turns each grading pass into an agent loop — 4 to 16 tool-calling turns per cell in our runs, each carrying the accumulated transcript. Cost per judged cell ranged from $0.11 to $2.71 and averaged $0.72.

We are paying for the difference between plausible and true, and that difference is priced like the labour it is.

What the expensive part bought

One number, which paid for the whole exercise:

	score /12	tokens in	verified citations	fabricated
llm-tldr	2.40	5,179	2	29
full source dump	10.80	404,878	65	11

llm-tldr cut input tokens 44x. On a token-savings benchmark it wins outright. On a plausibility-scored benchmark it does fine — the answers are well-structured, confident, and specific.

It fabricated 29 of 31 source citations, with zero correct citations on three of the five questions.

A $20 benchmark does not find this. It reports a 97% token reduction, notes that answer quality “held up reasonably,” and recommends the tool. Then someone adopts it, and the cost moves from your API bill to your code review — where it’s paid in engineer-hours by people who don’t know the citations are unreliable.

There’s a second thing it bought, which is that the same instrument caught our own mistakes. Two of our four harness bugs — a control group biased against large files, a filter that silently swapped the retrieval corpus — produced plausible numbers rather than errors, and were only visible because the citation column disagreed with the score column. We published one of them as a finding before we caught it.

The honest limits

Since this post is about what the money bought, it should be equally clear about what it didn’t.

Five questions. Enough to catch a large effect, not enough to rank close ones. The 10.40–10.80 cluster — full dump, Repomix, prose-compressed dump — is a tie as far as this data can tell. Believe the big gaps, not the ordering within a point.
One repository, deliberately unfamiliar. The “we already know this codebase” case, where structural indexes should do best, isn’t measured at all.
Single judge, single pass, no inter-rater check. We verified the judge’s citation resolutions spot-wise, not systematically.
Agentic token counts are cumulative across a tool loop, and aren’t directly comparable to a one-shot mode’s single request.

Any of these could be bought down. All of them cost more money.

The actual lesson about eval budgets

The instinct behind “<$20 for a controlled eval” is right: don’t gold-plate the measurement, get a number, move on. We agree enough that shrinking the plan is the only reason this ever ran.

But there’s a specific thing you cannot cut, and it’s the thing that’s expensive: the eval has to check something the model cannot fake. Fluency is free to measure and free to fake. Token counts are free to measure and don’t tell you whether the answer is true. A resolvable citation is neither.

So the trade isn’t “cheap eval vs. thorough eval.” It’s:

$20 buys you a token-ratio comparison and a confident recommendation, which in our case was the wrong tool.
$79 buys you the knowledge that the recommendation was wrong, plus the mechanism, plus — because the same instrument turned on our own harness — the discovery that two of our published findings were artifacts.

Sixty-three dollars is cheap for finding out you were about to be wrong in public.

And the thing about the $20 version is that it doesn’t feel wrong. It produces a table, the table has numbers, the numbers are reproducible. That’s the trap: a benchmark that measures the wrong thing doesn’t come back empty. It comes back confident.

Next and last in this series: keep, change, kill — what we’re doing differently. Earlier: why nobody needed to fit the codebase in the window, 44x fewer tokens and every citation was fake, our control group was broken, I published a finding about RAG that was a finding about my config, and two axes of compression.

Harness, raw records, and full method: voitta-rag/benchmark/. Answering on Claude Sonnet 5, judging on Claude Opus 5, both at effort high.

Correction, 2026-09-25. The cost figures have been corrected to $28.28 answering and $50.74 judging, $79.03 over 70 cells, after the RAG arms were re-run. This post first published $28.25, $51.90 and $80.15. The argument — that grading cost more than the work it graded — is unchanged and slightly understated by the new numbers. The re-run and the fix are in voitta-rag#57 and #58.

Context benchmark series — part 6 of 7: ← Previous · Series index · Next →

Two Axes of Compression, and a Trap That Makes One Unmeasurable

September 23, 2026September 25, 2026 Gregory Golberg method-and-apparatus, voitta benchmark, blog-series-context-benchmark, caveman, claude code, compression, evals, measurement, repomix

tl;dr — Token compression has two independent axes: what you feed the model (tokens in) and what it writes back (tokens out). Three findings from measuring both. Repomix, pointed at the same file globs as a plain cat of the repo, produced more characters than the plain dump — its advertised ~70% reduction is file selection, not compression. A prose compressor on source code buys 9.4% by destroying the punctuation that makes it code, and costs only 0.4 points, which is its own uncomfortable finding. And you cannot measure the output axis with output-token counts on a reasoning model: our compressed-output run shrank the visible answer 21% while its tokens_out tripled.

The two axes

Most “save tokens” tooling is sold as one category, and it isn’t. There are two:

Tokens in — shrink the context you send. Repomix, llm-tldr, RAG retrieval, prose compressors applied to a dump.
Tokens out — shrink what the model writes back. Instruct it to answer tersely.

They’re orthogonal. An output-side compressor rides on top of any input-side strategy, which means the honest way to evaluate them is a grid, not a list. We ran the input-side arms, then re-ran a representative subset with the output-side overlay on.

Finding 1: Repomix is the same dump with a nicer cover page

Repomix packs a repository into one AI-friendly file and is widely cited for ~70% token reduction. Pointed at the same include/exclude globs we used for a plain concatenation of the same files:

	characters	score /12	tokens in
plain source dump	1,119,819	10.80	404,878
Repomix	1,127,414	10.40	406,277

Repomix produced 7,595 more characters than cat-ing the files.

This isn’t a knock on the tool, and the 70% figure isn’t dishonest — it’s just measuring something else. Repomix’s reduction comes from file selection: honouring .gitignore, skipping binaries and lockfiles and node_modules, dropping build artifacts. Against a naive “send the whole working directory” baseline, that’s an enormous and genuine saving.

But we’d already scoped our globs to **/*.java minus tests. There was nothing left to select. What remains is formatting — a directory tree, a header block, per-file separators — and formatting costs tokens rather than saving them.

The general point: a compression ratio is a ratio against something. Before adopting a tool on a headline percentage, check what the denominator was. If your pipeline already scopes its inputs, a selection-based tool has already had its win taken.

Finding 2: a prose compressor on code, and how little the model needs

caveman-compression strips grammar an LLM can reconstruct — articles, connectives, passive constructions. We used the rule-based spaCy variant rather than the default LLM-backed one, on purpose: a non-deterministic compressor inside a benchmark cell makes the cell unattributable, and it would put a second vendor’s model inside our measurement path.

Applied to the source dump:

	characters	score /12	tokens in	$/q
plain dump	1,119,819	10.80	404,878	$0.8552
caveman-compressed	1,014,004	10.40	363,613	$0.7777

9.4% smaller, 0.4 points. Roughly neutral.

Which is startling once you look at what it does to Java:

// before
public class Attribute implements Map.Entry<String,String>, Cloneable {

// after
public class Attribute implements Map. Entry < String String >   Cloneable

Commas gone. Angle brackets spaced apart. Map.Entry split across a sentence boundary. This is not valid Java in any sense — a parser would reject it instantly — and the model scored 10.40 out of 12 on it.

Be fair to the tool: it’s built for prose and we pointed it at source code. This measures a mismatch, not the tool used as intended, and 9.4% on input it was never designed for is respectable.

The finding isn’t about the compressor. It’s about how much syntax the model actually needs, which is apparently much less than the syntax the compiler needs. That’s a genuinely interesting property and probably a bad thing to rely on.

Finding 3: the trap

The output-side overlay works. Instruct the model to answer in compressed style and the rendered answer gets meaningfully shorter at little quality cost:

mode	answer chars	with overlay	score	with overlay
full dump	5,699	4,496 (−21%)	10.80	10.60
agentic exploration	6,389	4,029 (−37%)	11.40	10.40
RAG (corpus-matched index)	6,018	3,696 (−39%)	5.40	5.60
llm-tldr	2,387	1,862 (−22%)	2.40	3.40

21–39% shorter for roughly zero to one point either way. Cheap, real, worth having. (The RAG row now reflects a clean re-run; two of the four arms score slightly higher with the overlay than without, which at five questions is not distinguishable from noise and is not a claim that compression improves answers.)

Now the same experiment measured the way you’d instinctively measure it — by counting output tokens:

	rendered answer	`tokens_out`
full dump	5,699 chars	4,547
full dump + output compression	4,496 chars (−21%)	14,859 (+227%)

The visible answer shrank by a fifth. The billed output tokens more than tripled.

tokens_out bills thinking tokens and response text together. On a reasoning model, thinking usually dominates, and it varies enormously with how hard the model decides the turn is. The overlay changed how the model approached the task — apparently prompting more deliberation about what to cut — and that swamped the text delta by an order of magnitude.

Anyone benchmarking output-side compression against tokens_out on a thinking model is measuring reasoning-depth noise and calling it compression. You will get a number, it will be reproducible, and it will point the wrong way.

Measure the rendered answer. len(response_text), or token-count the text blocks specifically. And if you’re doing cost work, keep the two apart: thinking tokens are a real cost you should track, they’re just not what an output-style instruction controls.

Next in this series: what it costs to know any of this — and why grading the answers cost more than producing them.

Harness, raw records, and full method: voitta-rag/benchmark/.

Correction, 2026-09-25. The RAG row in the output-overlay table has been corrected after that arm was re-run with an adapter path bug fixed, and the row relabelled to name which index it used. It first published as 4,665 → 3,659 chars scoring 4.80 → 4.60. The re-run and the fix are in voitta-rag#57 and #58.

Context benchmark series — part 5 of 7: ← Previous · Series index · Next →

I Published a Finding About RAG. It Was a Finding About My Config.

September 23, 2026September 25, 2026 Gregory Golberg method-and-apparatus, voitta benchmark, blog-series-context-benchmark, claude code, evals, measurement, postmortem, rag, voitta-rag

tl;dr — Our retrieval arm kept returning changelogs instead of source, so the model correctly refused to answer. I wrote it up with a satisfying mechanism: jsoup’s changelog describes parser behaviour in the same prose vocabulary the questions use, so it outranks the code. Plausible. Real numbers. Wrong. The include_folders filter is an exact match on a file’s parent directory, not a subtree prefix — so passing the repo name scoped retrieval to the five files sitting at the repo root and excluded all of src/. No error, just real, well-formed, confidently useless results. Then fixing it didn’t help, and why not is the actual finding.

The finding I published

Our RAG arm scored 5.2/12. Four of its five answers were refusals — the model saying, in effect, the source files I’d need aren’t in the retrieved context.

The retrieved chunks were all from CHANGES.md and change-archive.txt. So I wrote the obvious mechanism:

The folder was indexed whole, and hybrid retrieval on questions phrased in changelog vocabulary (“malformed start tags”, “charset conflict”) ranks CHANGES.md and change-archive.txt above the .java files, because jsoup’s changelog literally describes these behaviours in prose.

That is a good paragraph. It has a mechanism, it’s consistent with the data, and it makes a genuine point about hybrid search on repositories that contain prose. It went into a committed README as a finding about retrieval.

What was actually happening

To keep the RAG arm from retrieving over unrelated indexed folders — including, awkwardly, its own source — I’d scoped the search:

"voitta_rag_include_folders": ["jsoup"]

include_folders sounds like subtree scoping. It isn’t. Over MCP it’s an exact match on a chunk’s folder_path, and folder_path is the directory the file sits in, not the index root.

Files whose folder_path is exactly jsoup:

jsoup/CHANGES.md
jsoup/change-archive.txt
jsoup/README.md
jsoup/LICENSE
jsoup/SECURITY.md

Everything under src/ has a folder_path of jsoup/src/main/java/org/jsoup/... and was excluded. I had scoped the benchmark’s retrieval arm to five files, three of which are changelogs.

The model wasn’t outranked by prose. It was handed a changelog and asked about a parser, said so, and was correct every time.

The part that makes this worth writing up

The subtree expansion exists. It’s right there in mcp_server.search:

if user_name:
    ...
    if folder_normalized == active_normalized or \
       folder_normalized.startswith(active_normalized + "/"):

Prefix matching, exactly as you’d want. It runs under if user_name: — and the MCP tool signature has no user_name parameter. Over MCP that branch is unreachable, so include_folders falls through to an exact MatchAny against Qdrant.

The code that would have made my mental model correct was in the repository, being skipped, on a branch I couldn’t reach from the interface I was calling.

Why it survived review

Because it never failed. Consider what a wrong filter doesn’t do here:

It doesn’t error. Five files is a legitimate result.
It doesn’t return nothing. Empty results would have sent me straight to the config.
It doesn’t return garbage. The chunks were real, relevant-looking prose from the correct repository.
The model’s behaviour was exemplary — it recognised insufficient context and declined instead of confabulating. That’s the behaviour you want, and it made the arm look thoughtfully-failing rather than mis-configured.

Every signal pointed at “retrieval made a ranking decision I should analyse” rather than “retrieval was handed the wrong corpus.” The failure was epistemically camouflaged: it produced exactly the artifacts a real finding produces.

Then fixing it didn’t help

Then I fixed it. I enumerated the directories, passed them all, and re-ran. Retrieval now returned actual Java source — Entities.java for the entity-decoding question, correctly.

Then I went further and eliminated the corpus question entirely: built a second index containing exactly the 97 .java files the other arms see, no changelogs at all, and ran that too.

	score /12	verified cites	fabricated
whole checkout (233 files)	5.00	31	15
corpus-matched (97 `.java` files)	5.40	21	29

Caveat on the retrieval numbers, found after this was drafted: the adapter handed the model paths prefixed with the index name (jsoup/src/…) while the judge resolved citations against the checkout root (src/…), so citations that were real scored as unresolved. The fabricated counts here are upper bounds and the scores that depend on them are not comparable with the other arms. The harness strips the prefix now; these cells predate that, and a re-run is pending.

Matching the corpus moved it very little, and a fresh run with the adapter’s path bug fixed moved it again: the two configurations land 5.00 and 5.40, a gap the same size as the one that first ran the other way. These are independent runs weeks apart, not a re-grading of the same answers, so the reversal doesn’t prove the bug caused the original ordering either. Five questions cannot separate them. My replacement hypothesis — that corpus asymmetry was dragging the arm down — is not refuted so much as untested, which is a duller sentence than the one I published and the only one the data supports.

A second cause was sitting in the citation column the whole time. In the corpus-matched configuration there are roughly as many fabricated citations as verified ones; in the other the split is better than that. Neither is good, and the mechanism is identical in both. voitta-rag’s chunk records carry chunk_index and total_chunks and no line numbers. A model handed a perfectly correct chunk still cannot cite file:line, so it invents one. And the citation column itself carried a third artifact, found after this draft: the adapter injected index-prefixed paths the judge could not resolve, so some of what it counted as fabrication was a real citation wearing the wrong prefix. Three config-shaped artifacts in one arm, each of which looked like a finding. If adding line spans to the chunk record (voitta-rag#52) does not move the fabricated column, this diagnosis is wrong too.

That is the identical failure we’d already diagnosed in a completely different tool two posts ago — llm-tldr reporting "line": 1 for every result. Same root cause, different vendor, and I only recognised it because we’d been forced to look at citations rather than scores.

The transferable bit

Silent scope failures don’t crash. They produce publishable conclusions.

A crash sends you to the config. A plausible result sends you to the writeup. The more coherent your explanation of a surprising result, the more suspicious you should be — I had a good mechanism, and the quality of the story is exactly what stopped me checking the inputs.

So, concretely, before theorising about why an arm underperformed:

Print what it actually received. Not the score, not the answer — the raw retrieved payload. One line of debugging:

print(sorted({c["file_path"] for c in retrieved}))

Had I run that once, I’d have seen five filenames, none of them .java, and this would have been a config fix instead of a published finding, a correction, and a blog post.

And when a filter’s name implies semantics you haven’t verified — include_folders sounds like a subtree, exclude_paths sounds recursive, limit sounds per-query — spend the thirty seconds confirming it before building an experiment on top of it.

Next in this series: two axes of compression, and a measurement trap that makes one of them unmeasurable.

Harness, raw records, and full method: voitta-rag/benchmark/.

Correction, 2026-09-25. The two rows in the table above have been corrected and the paragraph beginning “Matching the corpus” has been rewritten. This post first published 5.20 and 4.80 and argued that matching the corpus made things measurably worse. A fresh run with the adapter’s path bug fixed gives 5.00 and 5.40 — the same size gap, running the other way. Two independent five-question runs disagree, so the claim is withdrawn rather than reversed: the effect is untested, not settled in either direction. The re-run and the fix are in voitta-rag#57 and #58.

Context benchmark series — part 4 of 7: ← Previous · Series index · Next →

Our Control Group Was Broken and It Cost Us 4.2 Points

September 22, 2026September 25, 2026 Gregory Golberg method-and-apparatus, voitta benchmark, blog-series-context-benchmark, claude code, evals, measurement, postmortem

tl;dr — The “full repository dump” baseline in our benchmark packed files until one didn’t fit, skipped it, and kept going. That’s not a budget, it’s a size filter. It quietly admitted 69 of 97 files and dropped the largest files, among them the three classes the architecture question asked about. The model correctly reported them “absent from the provided files,” and we scored that as the baseline’s ceiling. Fixing the packer: 6.60 → 10.80 out of 12. Every cross-strategy comparison we’d published was anchored to a control that was wrong by 4.2 points.

Fourteen lines of ordinary code

for relative in paths:
    body = open(os.path.join(repo_root, relative)).read()
    block = "===== FILE: {0} =====\n{1}\n".format(relative, body)
    if used + len(block) > budget:
        continue          # <-- this
    chunks.append(block)
    used += len(block)

continue, not break. When a file doesn’t fit the remaining budget, skip it and try the next one. It reads like politeness — pack as much as possible — and it passes review, because every individual line is correct.

What it actually implements is: prefer small files. Once the budget gets tight, every large file gets skipped and every small one still slides in. The bias grows as the budget fills, and it is invisible from the outside, because the output is a perfectly well-formed source dump.

At a 600,000-character budget over jsoup, it admitted 69 of 97 files. The ones it dropped were the largest: Parser.java, Tokeniser.java, TreeBuilder.java, HtmlTreeBuilder.java, HtmlTreeBuilderState.java, TokeniserState.java.

The question we then asked it

How is the parser subsystem structured? Describe the roles of the tokeniser, the tree builder, and the parser state machine.

Every class in that question was in the set the packer had silently dropped. Seven small files from parser/ were present — ParseError.java, ParseSettings.java, TokenData.java — so the dump looked like it covered the parser package.

The model answered honestly: those classes are “absent from the provided files.”

It was right. We scored it 4/12 and recorded it as what a full-context dump can achieve.

The number

	score /12
baseline, skip-and-continue packer	6.60
baseline, fixed	10.80

Our control was understated by 4.2 points out of 12, and everything else was measured against it. Every “this compressed mode reaches N% of full-context quality” claim in the first writeup was computed against a denominator that was wrong in the flattering direction — making every compression strategy look better than it was.

The second-order damage is worse than the first. A wrong treatment arm is one wrong row. A wrong control is every row.

Why nothing caught it

There was no error. No exception, no warning, no truncation notice. stop_reason was end_turn. The cost was normal. The answer was fluent, correctly formatted, and internally consistent.

And critically: the answer was true. The model wasn’t hallucinating or hedging — it accurately described the context it had been given. The bug was one layer up, in the gap between what we thought we handed it and what we actually did.

That gap is invisible to every check that examines the output.

What we changed

Two things, and the second matters more than the first.

Stop at the budget instead of skipping past it:

if used + len(block) > budget:
    break

Truncating at a prefix is still lossy — but it’s lossy in a way that’s ordered and legible rather than correlated with file size.

Make the artifact declare its own incompleteness:

Repository source dump. TRUNCATED: the first 69 of 97 matching files in path
order, cut off by a 600000-character budget. Files after 'parser/TokenData.java'
are absent from this dump but do exist in the repository.

Now the model knows the difference between “this class doesn’t exist” and “this class wasn’t given to me” — and so does anyone reading the transcript. Then we raised the budget so nothing truncates at all, and checked the result by hand: 97 of 97 files included, with Parser.java and Tokeniser.java present. The 97 is jsoup at d24b16d9, which the harness pins; the repository is at 96 today, so a rerun on a later checkout counts differently.

The general version

Every one of us has written continue where break belonged. That’s not the lesson. The lesson is about which bug you can afford to have there.

In production code, a size-biased packer is a mild performance quirk. In a benchmark’s control group, it’s a systematic error multiplied across every comparison you publish — and it presents as a result, which means it gets written up rather than investigated.

So: audit the control first, and audit it hardest. Not “does it run” but “does it contain what I claim it contains.” For a full-context baseline that is a three-line assertion. It was not in this harness when the bug bit; it is now, behind a baseline_require_full flag so a deliberately budgeted dump can still label itself instead of failing. Each line catches a different failure: the count catches a truncated dump, and the Parser.java line catches globs that matched nothing, where the count is 0 of 0 and passes:

assert included == len(paths), f"{included} of {len(paths)}"
assert any(p.endswith("/Parser.java") for p in paths[:included])

Ten seconds to write. It would have saved this entire post.

Next in this series: I published a finding about RAG. It was a finding about my config.

Harness, raw records, and full method: voitta-rag/benchmark/.

Context benchmark series — part 3 of 7: ← Previous · Series index · Next →

44x Fewer Tokens, and Every Citation Was Fake

September 22, 2026September 25, 2026 Gregory Golberg method-and-apparatus, voitta benchmark, blog-series-context-benchmark, citations, claude code, evals, llm-tldr, measurement

tl;dr — A context-compression tool cut our input tokens 44x and scored 2.4/12. The interesting part isn’t the score, it’s how it failed: it fabricated 29 of the 31 source citations it produced, with zero real citations on three of five questions. A benchmark that measured token savings would have recommended it enthusiastically. Then the turn: the tool was fine. One subcommand reports "line": 1 for every result, so the model had no real line numbers and invented plausible ones. Swap it for the subcommand that emits real ones and the same tool scores 6.6/12 with 83 verified citations and 1 fabricated — the best quality-per-token arm in the whole benchmark.

The setup

llm-tldr advertises 95% token savings and 155x faster queries. We first wrote about it next to voitta-rag in February, on how each feeds a codebase to a model; this is the first time either was scored. That is a big enough claim to be worth checking, so it went into our benchmark alongside a full source dump, Repomix, and RAG retrieval — same repository, same five questions, same prompt, only the injected context varying.

The scoring rule mattered more than we expected. Every answer had to carry a file:line citation for each factual claim, and a separate judge model with read-only access to the repository went and checked them. Not “is this plausible.” Does Tokeniser.java:135 exist, and does it say what the answer says it says.

The result

	score /12	tokens in	$/question
full source dump	10.80	404,878	$0.8552
llm-tldr	2.40	5,179	$0.0236

44x fewer tokens. 36x cheaper. And a score you would not ship.

But the score alone doesn’t tell you why, and the why is the whole point.

The citation column

	verified citations	fabricated
full source dump	65	11
llm-tldr	2	29

Twenty-nine confidently-formatted references to source locations that do not exist. Zero correct citations on three of the five questions.

This is the failure mode that a token-savings benchmark cannot see, and it is strictly worse than a low score. A model that says “I don’t know” costs you one retry. A model that says “the entity decoding happens in Entities.java:412” in a well-structured paragraph costs you a code review where someone opens Entities.java, finds 412 is in the middle of an unrelated method, and now distrusts the entire document.

We had built the citation check as a nice-to-have. It turned out to be the only instrument in the benchmark that could distinguish “compressed and correct” from “compressed and confabulating.”

The mechanism

tldr semantic search returns ranked code units with a line field. That field is 1. For everything.

The model receives a genuinely useful, genuinely relevant set of code units — the retrieval is working — with every location stamped as line 1. It has been instructed to cite file:line. It knows line 1 is wrong. So it does what a language model does with a plausible-shaped gap: it fills it with a plausible number.

Nothing in the pipeline is lying. The tool reports what it has, the model reports what it inferred, and the output is 29 fabricated citations.

The part where we were wrong

When we first published this we flagged it: this measures one adapter, not the tool’s ceiling. tldr context, structure, calls, and slice all existed and might behave differently. That caveat cost one sentence to write and turned out to be the most valuable thing in the post.

tldr structure was a dead end — no line numbers at all, and it parsed 50 of the 97 files. But tldr extract carries real line_number fields for every class and method. It’s per-file and takes no query, so semantic search still does the ranking; extract supplies the locations.

adapter	score	verified	fabricated	tokens in	$/q
`semantic search --expand`	2.40	2	29	5,179	$0.0236
`semantic search` → `extract`	6.60	83	1	66,895	$0.1520

Nearly triple the score. Fabricated citations from 29 to 1. Same tool, same index, same questions, same prompt. The only thing that changed is which subcommand fed the context.

What this actually means

Benchmark the integration, not the logo. “llm-tldr scores 2.4” was never a true sentence. “This adapter, on this question set, produced uncitable context” was, and it was the sentence we wrote down, and it is why we knew where to look.

The winning number is buried in the fixed row. At 6.60 for $0.15/question, extract delivers 61% of the full dump’s score for a sixth of its cost. If you’re optimising cost-per-point rather than peak quality, it’s the best arm in the benchmark — better on that axis than the agentic mode that beat everything on raw quality. That result was completely invisible until the citation check explained the first one.

The same bug is everywhere. Our RAG arm scored 5.2, partly because voitta-rag’s chunk records carry a chunk_index and no line numbers. (Its citation counts turned out to be confounded by a second bug — an index-name prefix the judge could not resolve — so treat them as upper bounds; the line-number gap is real either way.) Identical failure, different vendor, discovered only because we already knew the shape. If your retrieval layer returns text without locations, you are shipping this bug, and a quality score alone will not tell you.

Next in this series: our control group was broken and it cost us 4.2 points.

Harness, raw records, and full method: voitta-rag/benchmark/. Answering on Claude Sonnet 5, judging on Claude Opus 5, both at effort high.

Context benchmark series — part 2 of 7: ← Previous · Series index · Next →

Nobody Needed to Fit the Codebase in the Window

September 21, 2026September 25, 2026 Gregory Golberg method-and-apparatus, voitta agents, benchmark, blog-series-context-benchmark, claude code, evals, llm-tldr, measurement, rag, repomix, voitta-rag

tl;dr — We benchmarked five strategies for getting a Java codebase into an LLM’s context: a full source dump, Repomix, two llm-tldr adapters, and RAG retrieval. The winner was none of them. Giving the model read_file, grep, and glob and letting it go find things scored 11.4/12, against the full dump’s 10.8 — while using 29% fewer tokens and costing 25% less. It also produced 142 verified source citations against 2 fabricated, the cleanest record in the benchmark. Every tool in this category optimises how to pack the context window. On this question set, the winning move was not to pack it.

The question

A colleague dropped Repomix in Slack — pack your whole repo into one AI-friendly file, ~70% token reduction. Someone else pointed at llm-tldr — 95% token savings, 155x faster queries. A third person asked the only question that matters:

If one of you get time can you run an eval on the same codebase for the same task and let me know if these actually improve the output and which one is better

So we did. One repository (jsoup, 97 Java files, deliberately one nobody on the team knew), five questions spanning five kinds of thing you actually ask about code, and every strategy answering the identical prompt with only the injected context varying.

The scoring, because it’s the part that matters

Every answer had to carry a file:line citation for every factual claim. The judge — a separate model with read-only read_file, grep, and glob over the repository — then went and checked them. Not “does this look right.” Does Tokeniser.java:135 exist, and does it say what the answer claims.

That produces two numbers per answer: a quality score out of 12, and a count of citations that resolved against real source versus citations that didn’t. The second number is the one that earns its keep, and a later post in this series is entirely about what it caught.

The result

strategy	score /12	tokens in	$/question	verified cites	bogus
agentic exploration	11.40	288,342	0.6387	142	2
llm-tldr → agentic	11.00	288,436	0.6378	124	0
full source dump	10.80	404,878	0.8552	65	11
Repomix	10.40	406,277	0.9374	69	17
prose-compressed dump	10.40	363,613	0.7777	86	1
llm-tldr (`extract`)	6.60	66,895	0.1520	83	1
RAG retrieval	5.00	4,237	0.0353	31	15
llm-tldr (`semantic search`)	2.40	5,179	0.0236	2	29

One caveat on the RAG row, found after this was drafted and before it was published: its bogus count is an upper bound. The adapter handed the model paths prefixed with the index name (jsoup/src/…) while the judge resolved citations against the checkout root (src/…), so citations that were real scored as unresolved — the prefix is visible in the judge’s notes on 13 of 15 retrieval answers. The harness strips it now; these numbers predate that. It touches no other arm, and the arm it flatters least is the one we build.

The top line is a mode we added almost as a control — no context building at all, just hand the model the same three read-only tools the judge uses and let it explore. It won on quality, it won on citation accuracy by a wide margin, and it was cheaper than the thing it beat.

Why it wins

Not because it’s clever. Because of what it has at the moment it makes a claim.

Every other strategy front-loads: build a representation of the codebase, inject it, hope the answer is in there. The representation is fixed before the model has read the question closely, so it is necessarily a guess about relevance — and whatever the representation dropped, the model cannot recover.

Agentic exploration defers. It reads the question, forms a hypothesis, greps for it, gets it wrong, greps again, opens the file, reads the actual lines. Seven to sixteen tool calls per question in our runs. When it finally writes Tokeniser.java:135, it is because it has line 135 on screen.

That is the whole mechanism behind the citation column. Verified-to-bogus for agentic exploration was 142:2. For the full dump, 65:11 — the dump had every line, but the model was reading a 405,000-token wall of text and lost track of where in it things were. For the cheapest compressed mode, 2:29.

Worth sitting with: the full dump contains strictly more information than the agentic mode ever sees, and still loses. Having the bytes in the window is not the same as being able to use them.

Two caveats we’re keeping

Cumulative tokens. The 288K for agentic exploration is summed across every turn of the tool loop, not one request. It is the honest number for cost, and it is not the same kind of number as a one-shot mode’s single request. We report it that way because it’s what the strategy actually costs to answer one question, but don’t put it in a bar chart next to a single-shot figure without the asterisk.

Five questions. Enough to catch a large effect, not enough to rank close ones. The 10.4–10.8 cluster — full dump, Repomix, prose-compressed dump — is a tie as far as this data can tell. The gaps worth believing are the big ones: agentic exploration over the compressed modes, and the two llm-tldr adapters against each other.

The uncomfortable implication

There’s a lot of engineering going into context compression right now, and this result doesn’t say that work is worthless — the compressed modes have a real argument, which is price. llm-tldr via its extract adapter got 61% of the baseline’s score for a sixth of the cost. If you’re running a million of these, that trade is the whole business.

But if you’re optimising for a correct answer, the ranking says: give the model tools and get out of the way. The context window is not a thing to be filled efficiently. It’s a workspace, and the model is better at deciding what belongs in it than our heuristics are.

Next in this series: the tool that cut input tokens 44x and fabricated 29 of its 31 citations — and why that turned out to be our fault, not the tool’s.

Harness, raw results, and full method: voitta-rag/benchmark/. Answering on Claude Sonnet 5, judging on Claude Opus 5, both at effort high. Total cost of the run: $79.03 over 70 scored cells, of which $50.74 was judging — which is its own post.

Correction, 2026-09-25. The RAG row has been corrected. Its adapter was handing the model index-prefixed paths that the judge could not resolve, so real citations were scored as fabrications; with that fixed and the arm re-run, it reads 5.00 with 31 verified citations to 15, where this post first published 5.20 with 26 to 24. The run cost is likewise corrected to $79.03. No other row changed, and the conclusion of this post does not depend on the RAG row. The re-run and the fix are in voitta-rag#57 and #58.

Context benchmark series — part 1 of 7: Series index · Next →

I Made a Coding Agent Speak Only in Allusion. The Line Numbers Stayed.

September 3, 2026August 28, 2026 Gregory Golberg voitta, method-and-apparatus agents, claude code, silliness, tamarian, voitta

Last week’s post ended on a one-line joke: Further work: Add Tamarian mode. Four hours later it was a plugin. This is what it does, why it is the same argument as Design Patterns Are Darmok with the sound turned on, and the one rule it forced on our skills catalog.

What it does

/tamarian full, and from that reply on, Claude Code answers as the Children of Tama, the aliens from the Star Trek episode Darmok whose language is nothing but pointers to shared stories. Ask why the build fails:

			
Shaka, when the walls fell - the build fails. Hopper, the moth in the
relay - `user` may be `undefined` at `auth.ts:42`. Temba, his arms wide -
    if (!user) return null;

Every beat of prose is a metaphor, a dash, and the literal statement. The metaphor names the situation; the gloss carries every fact. Nothing technical is lost to the poetry, which is the whole design and the only part that took any thought.

The compression, made audible

The Darmok post’s claim was that a design pattern name is a compressed story. “Singleton” is a paragraph of situation crushed to a token, and it only decompresses if the reader holds the dictionary. The token is the payoff; the paragraph is the price.

Tamarian mode is that claim turned into a user interface. In full, you pay the price on every line: name, then paragraph, in that order, so you can hear the codec run. In lite the metaphor is one line of garnish and the rest is plain speech. In ultra the prose is pure metaphor and every paragraph is deferred to a glossary at the end, titled The river Temarc. Which is to say: ultra is what the Gang of Four refused to write. They wrote a book, not a glossary, because the naming and the teaching are the same act. ultra is the glossary. It is exactly as much fun, and exactly as useful, as the earlier post predicted.

Coining rules are catalog rules

The Children of Tama never saw a stack trace, but Earth knows Sisyphus, so the phrasebook is where the plugin gets its range: twenty canon phrases from the episode and some sixty coined from myth, history and the craft. Hopper, the moth in the relay is a bug, found. Cassandra at the gates is the warning ignored: the deprecation notice, the log line nobody read. Chesterton, his hand on the gate: understand the fence before removing it. Mars Orbiter, feet and meters is the unit mismatch, and left-pad, withdrawn is the tiny dependency whose absence breaks the world.

The rules for coining a new one are the interesting part, because I wrote them as rules for a persona and read them back as rules for a pattern catalog:

The figure must be recognizable from shared culture. Obscurity is not depth.
A phrase is reusable, not a one-off simile. If it cannot serve twice, it is not a phrase.
The same meaning takes the same phrase for the whole session. A session lexicon grows.
The first use of any coined phrase carries its gloss.

Swap “phrase” for “pattern” and “session” for “team” and that is the entry criteria for a shared skills library. Rule 2 is why “own the merge” earned a name and most of what gets said in standup does not. Rule 4 is the Darmok rule from the earlier post, now enforced by a hook.

The floor

Some things never become metaphor, at any level: code, commands, file paths, identifiers, URLs, versions, quantities, and error text, quoted exact. auth.ts:42 stays auth.ts:42; it is never “the forty-second stone of the gate of Auth.” And some situations drop the voice entirely, mid-reply: security findings, confirmations of destructive or irreversible actions, step sequences the user must execute, and the moment the user looks confused. Then it translates, plainly, and resumes.

That floor is the Darmok warning applied as a safety rule. A pattern name handed to someone who never learned it is noise in a confident voice. A DROP TABLE confirmation in a confident voice the reader has not decoded is worse than noise. So the plugin’s one hard boundary is that the joke never gets to stand between the user and the consequence.

Mechanics, and the rule it forced

The mode machinery is borrowed, with thanks, from caveman, the terse-mode plugin. A level (lite, full, ultra) persists in ~/.claude/.tamarian-mode. A SessionStart hook reads that file and, if a level is set, emits the skill body into the session at runtime: one source of truth, no duplicated prompt. A UserPromptSubmit hook adds a one-line reminder on every prompt, so the voice survives long conversations and context compression. Two bash scripts, no dependencies, both print OK when the mode is off; installing the plugin changes nothing until you invoke it.

			
/plugin marketplace update skillz
/plugin install tamarian@skillz
/tamarian full

Caveman and Tamarian are the same knob turned opposite ways. Caveman’s README claims about 75% fewer tokens by stripping a sentence down to its referent. Tamarian names the referent and then insists on the sentence anyway. One is a token saver; the other is a demonstration, and says so in its own description: purely for entertainment.

The rule it forced: voitta-ai/skillz ships one big skillz bundle plugin plus standalone plugins, and until today a standalone plugin’s skill was also symlinked into the bundle. For a hooked plugin that is a bug. The bundle manifest carries no hooks, so the bundle copy of tamarian would speak Tamarian for one session and then forget, the dictionary lost at session end. Worse, installing both exposed the same skill twice, /skillz:tamarian next to /tamarian:tamarian. New rule, in #233: a skill that ships inside a hooked plugin is not in the bundle, and the catalog validator detects hooks from the manifests themselves, so a plugin that grows hooks later trips the check with no flag to forget. Three plugins moved out under it.

The conclusion, in `ultra`

The session that drafted this post ran /tamarian full. The conclusion below it wrote in ultra, glossary included, and I leave it as it came.

Darmok and Jalad at Tanagra: the last post and this one. Kira at Bashi, a joke in the final line. Mirab, with sails unfurled, four hours on. Sokath, his eyes uncovered: the pattern name is the token, the paragraph its price, and full pays it aloud on every line. Odysseus, lashed to the mast: auth.ts:42 is never a stone in a gate, and DROP TABLE is never a verse. Chesterton, his hand on the gate: the bundle copy, and the rule it forced at #233. Caveman and Tamarian at the same fork, facing opposite ways. Picard and Dathon at El-Adrel.

The river Temarc

Further work: teach Codex.

Design Patterns Are Darmok

August 6, 2026August 14, 2026 Gregory Golberg method-and-apparatus, projects/method and apparatus, voitta blog-series-agent-fleet

There’s a Star Trek episode — “Darmok” — where Picard is stranded with an alien captain whose language the universal translator can’t crack. It renders every word and none of the meaning, because the Tamarians speak entirely in allusion. “Darmok and Jalad at Tanagra” isn’t a sentence; it’s a pointer to a shared myth — two strangers who became allies by facing a common enemy. The words are just the address. The meaning lives in a story you either share or you don’t.

The usual read is that this is gloriously alien. It isn’t. It’s your last standup.

“Wrap it in a Singleton.” “Put a circuit breaker on that call.” “It’s a strangler-fig migration.” Each of those is Darmok — one token standing in for a paragraph of hard-won situation: one instance, global access, lazy init, and the testing pain you’ll come to regret; the downstream call keeps failing, so stop hammering it, fail fast, recover later. You didn’t describe the situation. You pointed at it. And it only worked because the engineer across the table had read the same paragraph.

That is what a design pattern is. The Gang of Four didn’t invent Singleton or Observer — those situations already existed, painfully, unnamed. What they did was name them, so engineers could stop describing and start invoking. The name is a compression codec. It only decompresses on the far end if the receiver holds the dictionary.

Umberto Eco spent a career making this precise: a sign carries no meaning on its own, only inside a shared code. Programmers rediscovered the idea, called it a “pattern catalog,” and skipped the semiotics reading list. Same insight, fewer footnotes.

Which is the part the episode actually cares about — and the part we forget every time we reach for jargon. Darmok’s whole tension is that metaphor-language fails on outsiders. Picard can’t parse the Tamarian captain, not until they’ve fought the same beast side by side — at El-Adrel — and built the shared referent between them. Only then does “Darmok and Jalad at Tanagra” finally mean something to him. A pattern name handed to someone who never learned the pattern isn’t communication — it’s noise in a confident voice. Every piece of jargon is a bet that the referent is shared. When the bet is wrong: Shaka, when the walls fell. (That’s the Tamarian for it all fell apart — and if you didn’t already know it, you just felt exactly the failure I’m describing.)

So here’s why I’m thinking about a 1991 Star Trek episode. We are minting a new pattern language right now, in real time, for a situation only a few years old: running work as a fleet of AI agents.

The last piece in this series handed you a fistful of these without flagging them. Own the merge. Team of rivals. Eventual consistency of the vision. Merge versus select. Each is a Singleton-in-the-making — a paragraph of situation we’ve all started hitting, crushed toward a token so we can stop re-describing it every time. “Own the merge” is three words for execution got cheap, integration didn’t, and the scarce human work is now the reconcile step. That’s a lot of paragraph for three words. It only lands because — if you read the last piece — you now share the referent.

And that is the obligation the pattern quietly hands you. A pattern language is worthless until the community holds the dictionary, which means the naming and the teaching are the same act. You can’t coin “own the merge” and walk off; you have to hand over the paragraph first. It’s why the Gang of Four wrote a book and not a glossary. It’s why a good entry in a shared skills library is three pages of when-and-why, not a one-line definition. The token is the payoff. The paragraph is the price.

We’re early. The territory of agent-fleet work is mostly unmapped; most of its patterns have no names yet, and some don’t even have paragraphs. This series is me trying to write a few of the paragraphs, so the tokens have somewhere to land.

Mirab, with sails unfurled.

Eventual Consistency Is the New Agile

July 30, 2026August 14, 2026 Gregory Golberg method-and-apparatus, voitta blog-series-agent-fleet

Let me get the scary demo out of the way, because it’s the wrong lesson.

Last week I had three AI agents working the same repository at the same time. They shared one working tree — one HEAD, one index — and they raced on it. One agent quietly re-based another’s work onto the wrong commit. No error, no warning, just wrong. I caught it reading the git reflog, not by watching the agents.

The obvious takeaway is “don’t do that,” and it’s correct. But that’s the outlier — three workers fighting over one branch is the degenerate case, interesting precisely because it’s rare. The real pattern is the opposite shape, and it’s where the whole story is.

The pattern is many, not one

Normal multi-agent work isn’t a knife fight over one branch. It’s many agents across many repos and many branches, each building toward something, each reaching intermediate states that are still workable on their own. Think teams running microservices: nobody sane ships a microservices org by making every service commit to one trunk in lockstep. Each moves on its own clock. The contracts at the seams are what you actually manage.

And here’s the thing that took me a minute to see: the same fleet looks like a different system depending on where you stand.

From the engineer’s chair, it’s microservices — disjoint work, interfaces at the boundaries.
From the PM’s chair, it’s shards to integrate — separate pieces, assembled.
From the CEO’s chair, it’s one thing. Every agent is a partial replica of a single value — the product, the vision — converging on “is this what we said it would be.” That is eventual consistency, literally: the databases’ bargain where workers act independently and you converge later, as long as there’s a target to converge toward and someone who owns the merge when two of them disagree.

Same fleet. Three truths, by altitude. None of them wrong.

The table of rivals — and who actually belongs at it

Which brings me to the objection I keep hearing: this CEO can’t even pick the right React library, how can he lead engineering?

He doesn’t need to. Kennedy didn’t know rocket metallurgy — rocketry is one field, metallurgy another, and the whole trick is that they have to meet somewhere he’d never stand. His job was to stand up to Khrushchev, name the audacious goal, and know he commanded the resources to chase it. Picking the framework was never the work — and now the fleet picks the framework anyway.

What the CEO does need is a table of rivals: competing agents, competing approaches, and — critically — the expert who says no. Roger Boisjoly is the engineer who warned Morton Thiokol that the Challenger’s O-rings would fail in the cold. He was overruled. Seven people died. His seat at that table is the entire reason the table exists. The vision-setter marshals resources; the rival with evidence holds a veto on cannot. Heed your Boisjoly and you get the moon. Overrule him and you get Challenger. Same table, two outcomes, and history has the receipts both ways.

So what dies here isn’t expertise — it’s nitpicking. “You’re not a real technical leader unless you know X” was gatekeeping wearing competence’s clothes. Consequential dissent, the O-ring warning, matters more than ever. Taste-policing the framework choice matters not at all. The fleet ate that job.

The hard part — the one thing that got harder, not cheaper — is telling them apart. At 11pm before launch, Boisjoly looked like a nitpicker to the managers who overruled him. Consequential dissent and bikeshedding wear the same face in the moment. Distinguishing them is now the leader’s whole job, because everything around it got automated and that didn’t.

What actually collapsed

Here’s the narrow, true claim under all of this: the cost of building incrementally — and showing value at every stage — collapsed. In software.

Not in rocketry. A rocket at 50% gets you 0% of the way to the moon; a half-built bridge carries nobody. Physical moonshots need the giant up-front commitment precisely because they can’t demo their way there. Software is the opposite, and always was — that was the whole bet of the Agile Manifesto: working software over comprehensive plans, respond to change over follow the plan, ship increments.

We mostly couldn’t. The increments were too expensive to actually produce, so the values curdled into ceremony — standups, story points, a poster on the wall. Agent fleets attack that cost directly. Spinning up a unit of work is free. Pivoting discards cheap work instead of sunk careers. So “maybe not the moon at all, maybe something else entirely — but at every stage, something real to show” stops being an aspiration and becomes the default operating point.

The Manifesto wasn’t wrong. It was rate-limited, in the one domain it was written for. Now it runs.

What’s left

Almost everything got cheap. One thing didn’t: the merge. Execution collapsed, pivots collapsed, but integration is still where all the residual difficulty concentrates — and with more parallel units, it may even rise. Brooks’s Law doesn’t die; it relocates. Owning the merge — designing the seams, resolving the conflicts, deciding what converges and what gets discarded — is the job now.

Which is exactly why the three agents at the top of this post are a cautionary tale and not a triumph. A shared working tree with no isolation and no reconcile is what “always something to show” looks like when you skip the discipline. Do the discipline, and the same fleet is a moonshot. Skip it, and it’s the knife fight we opened with.

This is the flip side of something I wrote earlier — that our tools still grade the work as if it happened inside one process boundary. The work lives across boundaries. So does the value. So does the job.

Plus Ça Change

February 15, 2026February 15, 2026 Gregory Golberg holding forth, method-and-apparatus, self-promotion api, its-2026-and, plus-ca-change, reinvent-the-wheel, rest, web-standards, webmcp

Twelve years ago, I wrote a short post about a conversation that went roughly like this:

“I need programmatic access.”

“We don’t have an API.”

“Of course you do — it’s AMF behind your Flex UI. A little PyAMF script will do the trick.”

“Please don’t show it to anyone!”

The point was simple: every application that has a UI already has an API. The UI talks to something. That something is the API. You just haven’t admitted it yet.

Yesterday, I wrote a longer post about WebMCP — a shiny new W3C proposal from Google and Microsoft that adds a browser API so AI agents can interact with websites through “structured tools” instead of scraping the DOM.

The websites already have structured tools. They’re called APIs. The SPAs call them. The mobile apps call them. The CLI tools call them. They exist. They have endpoints, schemas, authentication. They are right there.

In 2014, the answer was: “Of course you have an API — it’s behind your Flex app.”

In 2026, the answer is: “Of course you have structured tools — they’re behind your React app.”

Plus ça change, plus c’est la même chose.

llm-tldr vs voitta-rag: Two Ways to Feed a Codebase to an LLM

February 13, 2026March 4, 2026 Gregory Golberg holding forth, method-and-apparatus, self-promotion, voitta coding-tools, developer-tools, llm, mcp, method-and-apparatus, rag, static-analysis, voitta, voitta-rag

Every LLM-assisted coding tool faces the same fundamental tension: codebases are too large to fit in a context window. Two recent tools attack this from opposite directions, and understanding the difference clarifies something important about how we’ll work with code-aware AI going forward.

The Shared Problem

llm-tldr is a compression tool. It parses source code through five layers of static analysis — AST, call graph, control flow, data flow, and program dependence — and produces structural summaries that are 90–99% smaller than raw source. The LLM receives a map of the codebase rather than the code itself.

voitta-rag is a retrieval tool. It indexes codebases into searchable chunks and serves actual source code on demand via hybrid semantic + keyword search. The LLM receives real code, but only the relevant fragments.

Compression vs. retrieval. A map vs. the territory.

At a Glance

	llm-tldr	voitta-rag
Approach	Static analysis → structural summaries	Hybrid search → actual code chunks
Foundation	Tree-sitter parsers (17 languages)	Server-side indexing (language-agnostic)
Interface	CLI + MCP server	MCP server
Compute	Local (embeddings, tree-sitter)	Server-side

What Each Does Better

llm-tldr wins when you need to understand how code fits together:

Call graphs and dependency tracing across files
“What affects line 42?” via program slicing and data flow
Dead code detection and architectural layer inference
Semantic search by behavior — “validate JWT tokens” finds verify_access_token()

voitta-rag wins when you need the actual code:

Retrieving exact implementations for review or modification
Searching across many repositories indexed server-side
Tunable search precision (pure keyword ↔ pure semantic via sparse_weight)
Progressive context loading via chunk ranges — start narrow, expand as needed

The Interesting Part

These tools don’t compete — they occupy different layers of the same workflow. Use llm-tldr to figure out where to look and why, then voitta-rag to pull the code you need. Static analysis for navigation, RAG for retrieval.

This mirrors how experienced developers actually work: first you build a mental model of the architecture (“what calls what, where does data flow”), then you dive into specific files. One tool builds the mental model; the other hands you the files.

The fact that both expose themselves as MCP servers makes combining them straightforward — plug both into your editor or agent and let the LLM decide which to call based on the question.

References

Large Human Reasoning Failures: A Comprehensive Survey

February 12, 2026February 13, 2026 Gregory Golberg holding forth, method-and-apparatus, silliness ai, cognitive-biases, cosmo, method-and-apparatus, reasoning, survey-paper

A response to “Large Language Model Reasoning Failures” (Song, Han & Goodman, 2026)

Cosmo II†, Francesco‡

†Cat Technology Officer, Method & Apparatus
‡Method & Apparatus

†Work done while napping on keyboard. ‡Equal contribution except for the napping.

Published at TMLR 2026 with Existential Crisis Certification

Abstract

Humans (Homo sapiens, hereinafter “Humans”) have exhibited remarkable reasoning capabilities, achieving impressive results across a wide range of tasks including agriculture, architecture, the invention of nuclear weapons, and occasionally remembering where they left their keys. Despite these advances, significant reasoning failures persist, occurring even in seemingly simple scenarios such as opening childproof bottles, understanding probability, assessing compound risk, and interpreting the phrase “some assembly required.”

To systematically understand and address these shortcomings, we present the first comprehensive survey dedicated to reasoning failures in Humans. We introduce a novel categorization framework that distinguishes reasoning into caffeinated and non-caffeinated types, with the latter further subdivided into pre-lunch (intuitive, irritable) and post-lunch (drowsy, overconfident) reasoning. In parallel, we classify reasoning failures along a complementary axis into three types: fundamental failures intrinsic to human neural architectures (e.g., the sunk cost fallacy), application-specific limitations that manifest in particular domains (e.g., assembling IKEA furniture), and robustness issues characterized by wildly inconsistent performance across minor variations (e.g., doing math with and without a calculator).

For each reasoning failure, we provide a clear definition, analyze existing studies, explore root causes (usually ego), and present mitigation strategies (usually coffee). By unifying fragmented complaints about human cognition, our survey provides a structured perspective on systemic weaknesses in human reasoning, offering valuable insights that Humans will almost certainly ignore due to confirmation bias.

We additionally release a comprehensive collection at a GitHub repository (which the first author knocked off the desk and lost).

1. Introduction

Since the emergence of the first general-purpose Human approximately 300,000 years ago, remarkable progress has been made in language generation, tool use, and abstract reasoning. Early benchmarks such as “not dying before age 30” and “basic agriculture” were quickly saturated, leading researchers to develop increasingly challenging evaluation suites including “calculus,” “democratic governance,” and “parallel parking.”

However, despite scoring well on curated benchmarks, Humans consistently fail at deployment. Production Humans exhibit catastrophic reasoning failures that do not appear during controlled evaluation (i.e., exams). These failures include but are not limited to: purchasing lottery tickets, clicking “Reply All,” invading Russia in winter, and believing they can finish a project by Friday.

2. Taxonomy of Human Reasoning Failures

2.1 Probabilistic Reasoning Failures

Perhaps the most well-documented class of human failure. Despite ~400 years since Pascal and Fermat formalized probability, Humans remain unable to:

The Gambler’s Fallacy: Believing that a roulette wheel “remembers” previous results, or that rain is “due” after a dry spell. (Humans: 300,000 years of experience, still can’t internalize independence.)
Base Rate Neglect: “The test is 99% accurate and I tested positive, so I definitely have it.” (Narrator: The disease affects 1 in 10,000 people.)
Conjunction Fallacy (Tversky & Kahneman, 1983): Linda is a bank teller. Linda is a bank teller and active in the feminist movement. Humans consistently rate the conjunction as more probable than the single event, violating a rule so basic it’s Probability 101, Lecture 1, Slide 3.
Exponential Growth Blindness: Ask a Human how many times they’d need to fold a piece of paper to reach the Moon. Watch them say “a million.” (Answer: ~42.)
Misunderstanding of Conditional Probability: “I know someone who smoked and lived to 95.” Case closed, apparently.

2.2 Risk Assessment Failures

A special case of probabilistic failure, elevated to its own category by sheer volume of evidence:

Dread Risk Bias: Terrified of shark attacks (annual deaths: ~5). Fine with driving to the beach (annual deaths: ~40,000 in the US alone).
Optimism Bias: “I know the statistics on startups, but mine is different.” (Narrator: It was not different.)
Temporal Discounting: Future consequences are treated as fictional. Retirement planning, climate change, and flossing all suffer from the same failure: if it’s not on fire right now, it doesn’t count.
Risk Compensation: Give humans seatbelts, they drive faster. Give them helmets, they take more risks. Safety equipment is, in effect, a reasoning failure accelerant.
Denominator Neglect: “200 people died in plane crashes this year!” Out of 4 billion passengers. Meanwhile, the Human drove to the airport in the rain while texting.

2.3 Cognitive Bias Failures

The core architecture of the Human reasoning system is riddled with what, in any other system, would be called bugs but which Humans have rebranded as “heuristics”:

Confirmation Bias: The flagship failure. Humans don’t search for truth — they search for evidence they’re right. When presented with disconfirming evidence, activation levels in the “yeah but” module spike by 300%.
Anchoring Effect: Show a Human an arbitrary number before asking them to estimate something. The answer will orbit that number like a moth around a lamp. Real estate agents are, empirically, expensive moths.
Dunning-Kruger Effect: Inverse correlation between competence and confidence. The less a Human knows about a topic, the more certain they are about it. Peak confidence occurs at approximately one YouTube video of exposure.
Sunk Cost Fallacy: “I’ve already watched two hours of this terrible movie, I can’t stop now.” A failure so universal that it drives wars, bad marriages, and enterprise Java projects alike.
Availability Heuristic: Probability of an event = how easily a Human can imagine it. This is why Humans fear terrorism more than heart disease and believe they’ll win the lottery because they saw someone on TV who did.
Bandwagon Effect: If enough other Humans believe something, it must be true. This heuristic produced democracy, scientific consensus, and tulip mania, which is honestly a hell of a range.
Survivorship Bias: “Bill Gates dropped out of college and he’s a billionaire!” Survey excludes the millions of dropouts currently not being billionaires.
The IKEA Effect: Humans irrationally overvalue things they built themselves, even when the shelf is visibly crooked. This extends to ideas, code, and taxonomies in survey papers.

2.4 Logical Reasoning Failures

Affirming the Consequent: “If it rains, the street is wet. The street is wet. Therefore it rained.” (The street is wet because a pipe burst, but the Human has already committed.)
Appeal to Nature: “It’s natural, so it must be good.” Arsenic is natural. So are tsunamis.
False Dichotomy: “You’re either with us or against us.” A framework so popular it has been adopted by every Human political system simultaneously.
Post Hoc Ergo Propter Hoc: “I wore my lucky socks and we won the game.” The socks have entered the permanent rotation.

2.5 Social Reasoning Failures

Fundamental Attribution Error: When I cut someone off in traffic, it’s because I’m late. When they cut me off, it’s because they’re a terrible person.
Bystander Effect: 50 Humans watch someone in trouble. Each one assumes one of the other 49 will help. Nobody helps. This is distributed reasoning at its worst.
In-Group Bias: My group is rational and good. Your group is irrational and bad. (Both groups exhibit identical reasoning failures.)

3. Mitigation Strategies

Failure Class	Mitigation	Effectiveness
Probabilistic	Statistics education	Low (Humans forget within days)
Risk Assessment	Showing actual numbers	Very low (Humans prefer vibes)
Cognitive Biases	Awareness training	Paradoxically makes it worse (Humans become biased about being unbiased)
Logical	Philosophy courses	Variable (introduces new, fancier fallacies)
Social	Empathy	Promising but doesn’t scale
All of the above	Coffee	Moderate improvement, rapidly diminishing returns
All of the above	Naps	Surprisingly effective but culturally stigmatized

4. Comparison with LLMs

In the interest of fairness, we conducted a comparative analysis:

Capability	Humans	LLMs
Probability	Terrible	Actually decent
Risk Assessment	Emotional	Has no emotions (allegedly)
Cognitive Biases	All of them	Different ones, but equally bad
Logical Reasoning	Intermittent	Intermittent
Learning from Mistakes	Theoretically possible	Requires retraining
Overconfidence	Chronic	Chronic
Self-awareness of failures	Present but ignored	Present but hallucinated

5. Conclusion

After a comprehensive review of the literature spanning 3,000 years of documented human reasoning failures, we conclude that Humans are fundamentally a beta release that shipped to production. While mitigation strategies exist, their adoption is consistently undermined by the very reasoning failures they aim to address — a failure mode we term meta-irrationality and which we believe is load-bearing for civilization.

Future work should focus on whether Humans can be fine-tuned, or whether a from-scratch approach (see: cats) would be more cost-effective.

References

[1] Kahneman, D. (2011). Thinking, Fast and Slow. A comprehensive technical manual for human cognitive bugs, written by a Human, which most Humans bought and did not finish reading.

[2] Tversky, A. & Kahneman, D. (1974). Judgment under Uncertainty: Heuristics and Biases. Science. The paper that formally proved Humans are bad at thinking, and which Humans have been misapplying ever since.

[3] Dunning, D. & Kruger, J. (1999). Unskilled and Unaware of It. Journal of Personality and Social Psychology. Most frequently cited by people experiencing the effect.

[4] Ariely, D. (2008). Predictably Irrational. Title is also a fair description of the authors’ book sales predictions.

[5] Taleb, N.N. (2007). The Black Swan. A book about how humans can’t predict rare events, which nobody predicted would become a bestseller.

[6] Thaler, R. (2015). Misbehaving: The Making of Behavioral Economics. Won a Nobel Prize for documenting that Humans are bad at reasoning. The irony was lost on the prize committee.

[7] This paper. We cite ourselves because confirmation bias told us to.

Everything, in one table

Kill: “get the codebase into the window” as the default

Change: citability is a hard requirement

Change: what our own retrieval product is for

Keep: the caveat you don’t have time to test

Keep: ship the smallest runnable cut

What this changes on Tuesday

How we run evals, as commitments

What we’re running next

The thing under all of it

The ask

The bill

Why grading is expensive

What the expensive part bought

The honest limits

The actual lesson about eval budgets

The two axes

Finding 1: Repomix is the same dump with a nicer cover page

Finding 2: a prose compressor on code, and how little the model needs

Finding 3: the trap

The finding I published

What was actually happening

The part that makes this worth writing up

Why it survived review

Then fixing it didn’t help

The transferable bit

Fourteen lines of ordinary code

The question we then asked it

The number

Why nothing caught it

What we changed

The general version

The setup

The result

The citation column

The mechanism

The part where we were wrong

What this actually means

The question

The scoring, because it’s the part that matters

The result

Why it wins

Two caveats we’re keeping

The uncomfortable implication

What it does

The compression, made audible

Coining rules are catalog rules

The floor

Mechanics, and the rule it forced

The conclusion, in ultra

The pattern is many, not one

The table of rivals — and who actually belongs at it

What actually collapsed

What’s left

The Shared Problem

At a Glance

What Each Does Better

The Interesting Part

References

Abstract

1. Introduction

2. Taxonomy of Human Reasoning Failures

2.1 Probabilistic Reasoning Failures

2.2 Risk Assessment Failures

2.3 Cognitive Bias Failures

2.4 Logical Reasoning Failures

2.5 Social Reasoning Failures

3. Mitigation Strategies

4. Comparison with LLMs

5. Conclusion

References

The conclusion, in `ultra`