Month: September 2026

Two Axes of Compression, and a Trap That Makes One Unmeasurable

Gregory Golberg method-and-apparatus, voitta , , , , , , ,

tl;dr — Token compression has two independent axes: what you feed the model (tokens in) and what it writes back (tokens out). Three findings from measuring both. Repomix, pointed at the same file globs as a plain cat of the repo, produced more characters than the plain dump — its advertised ~70% reduction is file selection, not compression. A prose compressor on source code buys 9.4% by destroying the punctuation that makes it code, and costs only 0.4 points, which is its own uncomfortable finding. And you cannot measure the output axis with output-token counts on a reasoning model: our compressed-output run shrank the visible answer 21% while its tokens_out tripled.

The two axes

Most “save tokens” tooling is sold as one category, and it isn’t. There are two:

  • Tokens in — shrink the context you send. Repomix, llm-tldr, RAG retrieval, prose compressors applied to a dump.
  • Tokens out — shrink what the model writes back. Instruct it to answer tersely.

They’re orthogonal. An output-side compressor rides on top of any input-side strategy, which means the honest way to evaluate them is a grid, not a list. We ran the input-side arms, then re-ran a representative subset with the output-side overlay on.

Finding 1: Repomix is the same dump with a nicer cover page

Repomix packs a repository into one AI-friendly file and is widely cited for ~70% token reduction. Pointed at the same include/exclude globs we used for a plain concatenation of the same files:

charactersscore /12tokens in
plain source dump1,119,81910.80404,878
Repomix1,127,41410.40406,277

Repomix produced 7,595 more characters than cat-ing the files.

This isn’t a knock on the tool, and the 70% figure isn’t dishonest — it’s just measuring something else. Repomix’s reduction comes from file selection: honouring .gitignore, skipping binaries and lockfiles and node_modules, dropping build artifacts. Against a naive “send the whole working directory” baseline, that’s an enormous and genuine saving.

But we’d already scoped our globs to **/*.java minus tests. There was nothing left to select. What remains is formatting — a directory tree, a header block, per-file separators — and formatting costs tokens rather than saving them.

The general point: a compression ratio is a ratio against something. Before adopting a tool on a headline percentage, check what the denominator was. If your pipeline already scopes its inputs, a selection-based tool has already had its win taken.

Finding 2: a prose compressor on code, and how little the model needs

caveman-compression strips grammar an LLM can reconstruct — articles, connectives, passive constructions. We used the rule-based spaCy variant rather than the default LLM-backed one, on purpose: a non-deterministic compressor inside a benchmark cell makes the cell unattributable, and it would put a second vendor’s model inside our measurement path.

Applied to the source dump:

charactersscore /12tokens in$/q
plain dump1,119,81910.80404,878$0.8552
caveman-compressed1,014,00410.40363,613$0.7777

9.4% smaller, 0.4 points. Roughly neutral.

Which is startling once you look at what it does to Java:

// before
public class Attribute implements Map.Entry<String,String>, Cloneable {

// after
public class Attribute implements Map. Entry < String String >   Cloneable

Commas gone. Angle brackets spaced apart. Map.Entry split across a sentence boundary. This is not valid Java in any sense — a parser would reject it instantly — and the model scored 10.40 out of 12 on it.

Be fair to the tool: it’s built for prose and we pointed it at source code. This measures a mismatch, not the tool used as intended, and 9.4% on input it was never designed for is respectable.

The finding isn’t about the compressor. It’s about how much syntax the model actually needs, which is apparently much less than the syntax the compiler needs. That’s a genuinely interesting property and probably a bad thing to rely on.

Finding 3: the trap

The output-side overlay works. Instruct the model to answer in compressed style and the rendered answer gets meaningfully shorter at little quality cost:

modeanswer charswith overlayscorewith overlay
full dump5,6994,496 (−21%)10.8010.60
agentic exploration6,3894,029 (−37%)11.4010.40
RAG4,6653,659 (−22%)4.804.60
llm-tldr2,3871,862 (−22%)2.403.40

21–37% shorter for roughly zero to one point. Cheap, real, worth having. (The RAG row’s absolute scores are depressed by a path-prefix bug in that adapter, found after this was drafted; the overlay delta this table is about is unaffected, since both cells carry it.)

Now the same experiment measured the way you’d instinctively measure it — by counting output tokens:

rendered answertokens_out
full dump5,699 chars4,547
full dump + output compression4,496 chars (−21%)14,859 (+227%)

The visible answer shrank by a fifth. The billed output tokens more than tripled.

tokens_out bills thinking tokens and response text together. On a reasoning model, thinking usually dominates, and it varies enormously with how hard the model decides the turn is. The overlay changed how the model approached the task — apparently prompting more deliberation about what to cut — and that swamped the text delta by an order of magnitude.

Anyone benchmarking output-side compression against tokens_out on a thinking model is measuring reasoning-depth noise and calling it compression. You will get a number, it will be reproducible, and it will point the wrong way.

Measure the rendered answer. len(response_text), or token-count the text blocks specifically. And if you’re doing cost work, keep the two apart: thinking tokens are a real cost you should track, they’re just not what an output-style instruction controls.

Next in this series: what it costs to know any of this — and why grading the answers cost more than producing them.

Harness, raw records, and full method: voitta-rag/benchmark/.

I Published a Finding About RAG. It Was a Finding About My Config.

Gregory Golberg method-and-apparatus, voitta , , , , , , ,

tl;dr — Our retrieval arm kept returning changelogs instead of source, so the model correctly refused to answer. I wrote it up with a satisfying mechanism: jsoup’s changelog describes parser behaviour in the same prose vocabulary the questions use, so it outranks the code. Plausible. Real numbers. Wrong. The include_folders filter is an exact match on a file’s parent directory, not a subtree prefix — so passing the repo name scoped retrieval to the five files sitting at the repo root and excluded all of src/. No error, just real, well-formed, confidently useless results. Then fixing it didn’t help, and why not is the actual finding.

The finding I published

Our RAG arm scored 5.2/12. Four of its five answers were refusals — the model saying, in effect, the source files I’d need aren’t in the retrieved context.

The retrieved chunks were all from CHANGES.md and change-archive.txt. So I wrote the obvious mechanism:

The folder was indexed whole, and hybrid retrieval on questions phrased in changelog vocabulary (“malformed start tags”, “charset conflict”) ranks CHANGES.md and change-archive.txt above the .java files, because jsoup’s changelog literally describes these behaviours in prose.

That is a good paragraph. It has a mechanism, it’s consistent with the data, and it makes a genuine point about hybrid search on repositories that contain prose. It went into a committed README as a finding about retrieval.

What was actually happening

To keep the RAG arm from retrieving over unrelated indexed folders — including, awkwardly, its own source — I’d scoped the search:

"voitta_rag_include_folders": ["jsoup"]

include_folders sounds like subtree scoping. It isn’t. Over MCP it’s an exact match on a chunk’s folder_path, and folder_path is the directory the file sits in, not the index root.

Files whose folder_path is exactly jsoup:

jsoup/CHANGES.md
jsoup/change-archive.txt
jsoup/README.md
jsoup/LICENSE
jsoup/SECURITY.md

Everything under src/ has a folder_path of jsoup/src/main/java/org/jsoup/... and was excluded. I had scoped the benchmark’s retrieval arm to five files, three of which are changelogs.

The model wasn’t outranked by prose. It was handed a changelog and asked about a parser, said so, and was correct every time.

The part that makes this worth writing up

The subtree expansion exists. It’s right there in mcp_server.search:

if user_name:
    ...
    if folder_normalized == active_normalized or \
       folder_normalized.startswith(active_normalized + "/"):

Prefix matching, exactly as you’d want. It runs under if user_name: — and the MCP tool signature has no user_name parameter. Over MCP that branch is unreachable, so include_folders falls through to an exact MatchAny against Qdrant.

The code that would have made my mental model correct was in the repository, being skipped, on a branch I couldn’t reach from the interface I was calling.

Why it survived review

Because it never failed. Consider what a wrong filter doesn’t do here:

  • It doesn’t error. Five files is a legitimate result.
  • It doesn’t return nothing. Empty results would have sent me straight to the config.
  • It doesn’t return garbage. The chunks were real, relevant-looking prose from the correct repository.
  • The model’s behaviour was exemplary — it recognised insufficient context and declined instead of confabulating. That’s the behaviour you want, and it made the arm look thoughtfully-failing rather than mis-configured.

Every signal pointed at “retrieval made a ranking decision I should analyse” rather than “retrieval was handed the wrong corpus.” The failure was epistemically camouflaged: it produced exactly the artifacts a real finding produces.

Then fixing it didn’t help

Then I fixed it. I enumerated the directories, passed them all, and re-ran. Retrieval now returned actual Java source — Entities.java for the entity-decoding question, correctly.

Then I went further and eliminated the corpus question entirely: built a second index containing exactly the 97 .java files the other arms see, no changelogs at all, and ran that too.

score /12verified citesfabricated
whole checkout (233 files)5.202624
corpus-matched (97 .java files)4.803029

Caveat on the retrieval numbers, found after this was drafted: the adapter handed the model paths prefixed with the index name (jsoup/src/…) while the judge resolved citations against the checkout root (src/…), so citations that were real scored as unresolved. The fabricated counts here are upper bounds and the scores that depend on them are not comparable with the other arms. The harness strips the prefix now; these cells predate that, and a re-run is pending.

Matching the corpus made it marginally worse. My replacement hypothesis — that corpus asymmetry was dragging the arm down — was also wrong.

A second cause was sitting in the citation column the whole time: roughly as many fabricated citations as verified ones, in both configurations. voitta-rag’s chunk records carry chunk_index and total_chunks and no line numbers. A model handed a perfectly correct chunk still cannot cite file:line, so it invents one. And the citation column itself carried a third artifact, found after this draft: the adapter injected index-prefixed paths the judge could not resolve, so some of what it counted as fabrication was a real citation wearing the wrong prefix. Three config-shaped artifacts in one arm, each of which looked like a finding. If adding line spans to the chunk record (voitta-rag#52) does not move the fabricated column, this diagnosis is wrong too.

That is the identical failure we’d already diagnosed in a completely different tool two posts ago — llm-tldr reporting "line": 1 for every result. Same root cause, different vendor, and I only recognised it because we’d been forced to look at citations rather than scores.

The transferable bit

Silent scope failures don’t crash. They produce publishable conclusions.

A crash sends you to the config. A plausible result sends you to the writeup. The more coherent your explanation of a surprising result, the more suspicious you should be — I had a good mechanism, and the quality of the story is exactly what stopped me checking the inputs.

So, concretely, before theorising about why an arm underperformed:

Print what it actually received. Not the score, not the answer — the raw retrieved payload. One line of debugging:

print(sorted({c["file_path"] for c in retrieved}))

Had I run that once, I’d have seen five filenames, none of them .java, and this would have been a config fix instead of a published finding, a correction, and a blog post.

And when a filter’s name implies semantics you haven’t verified — include_folders sounds like a subtree, exclude_paths sounds recursive, limit sounds per-query — spend the thirty seconds confirming it before building an experiment on top of it.

Next in this series: two axes of compression, and a measurement trap that makes one of them unmeasurable.

Harness, raw records, and full method: voitta-rag/benchmark/.

Our Control Group Was Broken and It Cost Us 4.2 Points

Gregory Golberg method-and-apparatus, voitta , , , , ,

tl;dr — The “full repository dump” baseline in our benchmark packed files until one didn’t fit, skipped it, and kept going. That’s not a budget, it’s a size filter. It quietly admitted 69 of 97 files and dropped the largest files, among them the three classes the architecture question asked about. The model correctly reported them “absent from the provided files,” and we scored that as the baseline’s ceiling. Fixing the packer: 6.60 → 10.80 out of 12. Every cross-strategy comparison we’d published was anchored to a control that was wrong by 4.2 points.

Fourteen lines of ordinary code

for relative in paths:
    body = open(os.path.join(repo_root, relative)).read()
    block = "===== FILE: {0} =====\n{1}\n".format(relative, body)
    if used + len(block) > budget:
        continue          # <-- this
    chunks.append(block)
    used += len(block)

continue, not break. When a file doesn’t fit the remaining budget, skip it and try the next one. It reads like politeness — pack as much as possible — and it passes review, because every individual line is correct.

What it actually implements is: prefer small files. Once the budget gets tight, every large file gets skipped and every small one still slides in. The bias grows as the budget fills, and it is invisible from the outside, because the output is a perfectly well-formed source dump.

At a 600,000-character budget over jsoup, it admitted 69 of 97 files. The ones it dropped were the largest: Parser.java, Tokeniser.java, TreeBuilder.java, HtmlTreeBuilder.java, HtmlTreeBuilderState.java, TokeniserState.java.

The question we then asked it

How is the parser subsystem structured? Describe the roles of the tokeniser, the tree builder, and the parser state machine.

Every class in that question was in the set the packer had silently dropped. Seven small files from parser/ were present — ParseError.java, ParseSettings.java, TokenData.java — so the dump looked like it covered the parser package.

The model answered honestly: those classes are “absent from the provided files.”

It was right. We scored it 4/12 and recorded it as what a full-context dump can achieve.

The number

score /12
baseline, skip-and-continue packer6.60
baseline, fixed10.80

Our control was understated by 4.2 points out of 12, and everything else was measured against it. Every “this compressed mode reaches N% of full-context quality” claim in the first writeup was computed against a denominator that was wrong in the flattering direction — making every compression strategy look better than it was.

The second-order damage is worse than the first. A wrong treatment arm is one wrong row. A wrong control is every row.

Why nothing caught it

There was no error. No exception, no warning, no truncation notice. stop_reason was end_turn. The cost was normal. The answer was fluent, correctly formatted, and internally consistent.

And critically: the answer was true. The model wasn’t hallucinating or hedging — it accurately described the context it had been given. The bug was one layer up, in the gap between what we thought we handed it and what we actually did.

That gap is invisible to every check that examines the output.

What we changed

Two things, and the second matters more than the first.

Stop at the budget instead of skipping past it:

if used + len(block) > budget:
    break

Truncating at a prefix is still lossy — but it’s lossy in a way that’s ordered and legible rather than correlated with file size.

Make the artifact declare its own incompleteness:

Repository source dump. TRUNCATED: the first 69 of 97 matching files in path
order, cut off by a 600000-character budget. Files after 'parser/TokenData.java'
are absent from this dump but do exist in the repository.

Now the model knows the difference between “this class doesn’t exist” and “this class wasn’t given to me” — and so does anyone reading the transcript. Then we raised the budget so nothing truncates at all, and checked the result by hand: 97 of 97 files included, with Parser.java and Tokeniser.java present. The 97 is jsoup at d24b16d9, which the harness pins; the repository is at 96 today, so a rerun on a later checkout counts differently.

The general version

Every one of us has written continue where break belonged. That’s not the lesson. The lesson is about which bug you can afford to have there.

In production code, a size-biased packer is a mild performance quirk. In a benchmark’s control group, it’s a systematic error multiplied across every comparison you publish — and it presents as a result, which means it gets written up rather than investigated.

So: audit the control first, and audit it hardest. Not “does it run” but “does it contain what I claim it contains.” For a full-context baseline that is a three-line assertion. It was not in this harness when the bug bit; it is now, behind a baseline_require_full flag so a deliberately budgeted dump can still label itself instead of failing. Each line catches a different failure: the count catches a truncated dump, and the Parser.java line catches globs that matched nothing, where the count is 0 of 0 and passes:

assert included == len(paths), f"{included} of {len(paths)}"
assert any(p.endswith("/Parser.java") for p in paths[:included])

Ten seconds to write. It would have saved this entire post.

Next in this series: I published a finding about RAG. It was a finding about my config.

Harness, raw records, and full method: voitta-rag/benchmark/.

44x Fewer Tokens, and Every Citation Was Fake

Gregory Golberg method-and-apparatus, voitta , , , , , ,

tl;dr — A context-compression tool cut our input tokens 44x and scored 2.4/12. The interesting part isn’t the score, it’s how it failed: it fabricated 29 of the 31 source citations it produced, with zero real citations on three of five questions. A benchmark that measured token savings would have recommended it enthusiastically. Then the turn: the tool was fine. One subcommand reports "line": 1 for every result, so the model had no real line numbers and invented plausible ones. Swap it for the subcommand that emits real ones and the same tool scores 6.6/12 with 83 verified citations and 1 fabricated — the best quality-per-token arm in the whole benchmark.

The setup

llm-tldr advertises 95% token savings and 155x faster queries. We first wrote about it next to voitta-rag in February, on how each feeds a codebase to a model; this is the first time either was scored. That is a big enough claim to be worth checking, so it went into our benchmark alongside a full source dump, Repomix, and RAG retrieval — same repository, same five questions, same prompt, only the injected context varying.

The scoring rule mattered more than we expected. Every answer had to carry a file:line citation for each factual claim, and a separate judge model with read-only access to the repository went and checked them. Not “is this plausible.” Does Tokeniser.java:135 exist, and does it say what the answer says it says.

The result

score /12tokens in$/question
full source dump10.80404,878$0.8552
llm-tldr2.405,179$0.0236

44x fewer tokens. 36x cheaper. And a score you would not ship.

But the score alone doesn’t tell you why, and the why is the whole point.

The citation column

verified citationsfabricated
full source dump6511
llm-tldr229

Twenty-nine confidently-formatted references to source locations that do not exist. Zero correct citations on three of the five questions.

This is the failure mode that a token-savings benchmark cannot see, and it is strictly worse than a low score. A model that says “I don’t know” costs you one retry. A model that says “the entity decoding happens in Entities.java:412” in a well-structured paragraph costs you a code review where someone opens Entities.java, finds 412 is in the middle of an unrelated method, and now distrusts the entire document.

We had built the citation check as a nice-to-have. It turned out to be the only instrument in the benchmark that could distinguish “compressed and correct” from “compressed and confabulating.”

The mechanism

tldr semantic search returns ranked code units with a line field. That field is 1. For everything.

The model receives a genuinely useful, genuinely relevant set of code units — the retrieval is working — with every location stamped as line 1. It has been instructed to cite file:line. It knows line 1 is wrong. So it does what a language model does with a plausible-shaped gap: it fills it with a plausible number.

Nothing in the pipeline is lying. The tool reports what it has, the model reports what it inferred, and the output is 29 fabricated citations.

The part where we were wrong

When we first published this we flagged it: this measures one adapter, not the tool’s ceiling. tldr context, structure, calls, and slice all existed and might behave differently. That caveat cost one sentence to write and turned out to be the most valuable thing in the post.

tldr structure was a dead end — no line numbers at all, and it parsed 50 of the 97 files. But tldr extract carries real line_number fields for every class and method. It’s per-file and takes no query, so semantic search still does the ranking; extract supplies the locations.

adapterscoreverifiedfabricatedtokens in$/q
semantic search --expand2.402295,179$0.0236
semantic search → extract6.6083166,895$0.1520

Nearly triple the score. Fabricated citations from 29 to 1. Same tool, same index, same questions, same prompt. The only thing that changed is which subcommand fed the context.

What this actually means

Benchmark the integration, not the logo. “llm-tldr scores 2.4” was never a true sentence. “This adapter, on this question set, produced uncitable context” was, and it was the sentence we wrote down, and it is why we knew where to look.

The winning number is buried in the fixed row. At 6.60 for $0.15/question, extract delivers 61% of the full dump’s score for a sixth of its cost. If you’re optimising cost-per-point rather than peak quality, it’s the best arm in the benchmark — better on that axis than the agentic mode that beat everything on raw quality. That result was completely invisible until the citation check explained the first one.

The same bug is everywhere. Our RAG arm scored 5.2, partly because voitta-rag’s chunk records carry a chunk_index and no line numbers. (Its citation counts turned out to be confounded by a second bug — an index-name prefix the judge could not resolve — so treat them as upper bounds; the line-number gap is real either way.) Identical failure, different vendor, discovered only because we already knew the shape. If your retrieval layer returns text without locations, you are shipping this bug, and a quality score alone will not tell you.

Next in this series: our control group was broken and it cost us 4.2 points.

Harness, raw records, and full method: voitta-rag/benchmark/. Answering on Claude Sonnet 5, judging on Claude Opus 5, both at effort high.

Nobody Needed to Fit the Codebase in the Window

Gregory Golberg method-and-apparatus, voitta , , , , , , , , ,

tl;dr — We benchmarked five strategies for getting a Java codebase into an LLM’s context: a full source dump, Repomix, two llm-tldr adapters, and RAG retrieval. The winner was none of them. Giving the model read_file, grep, and glob and letting it go find things scored 11.4/12, against the full dump’s 10.8 — while using 29% fewer tokens and costing 25% less. It also produced 142 verified source citations against 2 fabricated, the cleanest record in the benchmark. Every tool in this category optimises how to pack the context window. On this question set, the winning move was not to pack it.

The question

A colleague dropped Repomix in Slack — pack your whole repo into one AI-friendly file, ~70% token reduction. Someone else pointed at llm-tldr — 95% token savings, 155x faster queries. A third person asked the only question that matters:

If one of you get time can you run an eval on the same codebase for the same task and let me know if these actually improve the output and which one is better

So we did. One repository (jsoup, 97 Java files, deliberately one nobody on the team knew), five questions spanning five kinds of thing you actually ask about code, and every strategy answering the identical prompt with only the injected context varying.

The scoring, because it’s the part that matters

Every answer had to carry a file:line citation for every factual claim. The judge — a separate model with read-only read_file, grep, and glob over the repository — then went and checked them. Not “does this look right.” Does Tokeniser.java:135 exist, and does it say what the answer claims.

That produces two numbers per answer: a quality score out of 12, and a count of citations that resolved against real source versus citations that didn’t. The second number is the one that earns its keep, and a later post in this series is entirely about what it caught.

The result

strategyscore /12tokens in$/questionverified citesbogus
agentic exploration11.40288,3420.63871422
llm-tldr → agentic11.00288,4360.63781240
full source dump10.80404,8780.85526511
Repomix10.40406,2770.93746917
prose-compressed dump10.40363,6130.7777861
llm-tldr (extract)6.6066,8950.1520831
RAG retrieval5.204,3370.02892624
llm-tldr (semantic search)2.405,1790.0236229

One caveat on the RAG row, found after this was drafted and before it was published: its bogus count is an upper bound. The adapter handed the model paths prefixed with the index name (jsoup/src/…) while the judge resolved citations against the checkout root (src/…), so citations that were real scored as unresolved — the prefix is visible in the judge’s notes on 13 of 15 retrieval answers. The harness strips it now; these numbers predate that. It touches no other arm, and the arm it flatters least is the one we build.

The top line is a mode we added almost as a control — no context building at all, just hand the model the same three read-only tools the judge uses and let it explore. It won on quality, it won on citation accuracy by a wide margin, and it was cheaper than the thing it beat.

Why it wins

Not because it’s clever. Because of what it has at the moment it makes a claim.

Every other strategy front-loads: build a representation of the codebase, inject it, hope the answer is in there. The representation is fixed before the model has read the question closely, so it is necessarily a guess about relevance — and whatever the representation dropped, the model cannot recover.

Agentic exploration defers. It reads the question, forms a hypothesis, greps for it, gets it wrong, greps again, opens the file, reads the actual lines. Seven to sixteen tool calls per question in our runs. When it finally writes Tokeniser.java:135, it is because it has line 135 on screen.

That is the whole mechanism behind the citation column. Verified-to-bogus for agentic exploration was 142:2. For the full dump, 65:11 — the dump had every line, but the model was reading a 405,000-token wall of text and lost track of where in it things were. For the cheapest compressed mode, 2:29.

Worth sitting with: the full dump contains strictly more information than the agentic mode ever sees, and still loses. Having the bytes in the window is not the same as being able to use them.

Two caveats we’re keeping

Cumulative tokens. The 288K for agentic exploration is summed across every turn of the tool loop, not one request. It is the honest number for cost, and it is not the same kind of number as a one-shot mode’s single request. We report it that way because it’s what the strategy actually costs to answer one question, but don’t put it in a bar chart next to a single-shot figure without the asterisk.

Five questions. Enough to catch a large effect, not enough to rank close ones. The 10.4–10.8 cluster — full dump, Repomix, prose-compressed dump — is a tie as far as this data can tell. The gaps worth believing are the big ones: agentic exploration over the compressed modes, and the two llm-tldr adapters against each other.

The uncomfortable implication

There’s a lot of engineering going into context compression right now, and this result doesn’t say that work is worthless — the compressed modes have a real argument, which is price. llm-tldr via its extract adapter got 61% of the baseline’s score for a sixth of the cost. If you’re running a million of these, that trade is the whole business.

But if you’re optimising for a correct answer, the ranking says: give the model tools and get out of the way. The context window is not a thing to be filled efficiently. It’s a workspace, and the model is better at deciding what belongs in it than our heuristics are.

Next in this series: the tool that cut input tokens 44x and fabricated 29 of its 31 citations — and why that turned out to be our fault, not the tool’s.

Harness, raw results, and full method: voitta-rag/benchmark/. Answering on Claude Sonnet 5, judging on Claude Opus 5, both at effort high. Total cost of the run: $80.15 over 70 scored cells, of which $51.90 was judging — which is its own post.

I Made a Coding Agent Speak Only in Allusion. The Line Numbers Stayed.

Gregory Golberg method-and-apparatus, voitta , , , ,

Last week’s post ended on a one-line joke: Further work: Add Tamarian mode. Four hours later it was a plugin. This is what it does, why it is the same argument as Design Patterns Are Darmok with the sound turned on, and the one rule it forced on our skills catalog.

What it does

/tamarian full, and from that reply on, Claude Code answers as the Children of Tama, the aliens from the Star Trek episode Darmok whose language is nothing but pointers to shared stories. Ask why the build fails:

Shaka, when the walls fell - the build fails. Hopper, the moth in the
relay - `user` may be `undefined` at `auth.ts:42`. Temba, his arms wide -
    if (!user) return null;

Every beat of prose is a metaphor, a dash, and the literal statement. The metaphor names the situation; the gloss carries every fact. Nothing technical is lost to the poetry, which is the whole design and the only part that took any thought.

The compression, made audible

The Darmok post’s claim was that a design pattern name is a compressed story. “Singleton” is a paragraph of situation crushed to a token, and it only decompresses if the reader holds the dictionary. The token is the payoff; the paragraph is the price.

Tamarian mode is that claim turned into a user interface. In full, you pay the price on every line: name, then paragraph, in that order, so you can hear the codec run. In lite the metaphor is one line of garnish and the rest is plain speech. In ultra the prose is pure metaphor and every paragraph is deferred to a glossary at the end, titled The river Temarc. Which is to say: ultra is what the Gang of Four refused to write. They wrote a book, not a glossary, because the naming and the teaching are the same act. ultra is the glossary. It is exactly as much fun, and exactly as useful, as the earlier post predicted.

Coining rules are catalog rules

The Children of Tama never saw a stack trace, but Earth knows Sisyphus, so the phrasebook is where the plugin gets its range: twenty canon phrases from the episode and some sixty coined from myth, history and the craft. Hopper, the moth in the relay is a bug, found. Cassandra at the gates is the warning ignored: the deprecation notice, the log line nobody read. Chesterton, his hand on the gate: understand the fence before removing it. Mars Orbiter, feet and meters is the unit mismatch, and left-pad, withdrawn is the tiny dependency whose absence breaks the world.

The rules for coining a new one are the interesting part, because I wrote them as rules for a persona and read them back as rules for a pattern catalog:

  1. The figure must be recognizable from shared culture. Obscurity is not depth.
  2. A phrase is reusable, not a one-off simile. If it cannot serve twice, it is not a phrase.
  3. The same meaning takes the same phrase for the whole session. A session lexicon grows.
  4. The first use of any coined phrase carries its gloss.

Swap “phrase” for “pattern” and “session” for “team” and that is the entry criteria for a shared skills library. Rule 2 is why “own the merge” earned a name and most of what gets said in standup does not. Rule 4 is the Darmok rule from the earlier post, now enforced by a hook.

The floor

Some things never become metaphor, at any level: code, commands, file paths, identifiers, URLs, versions, quantities, and error text, quoted exact. auth.ts:42 stays auth.ts:42; it is never “the forty-second stone of the gate of Auth.” And some situations drop the voice entirely, mid-reply: security findings, confirmations of destructive or irreversible actions, step sequences the user must execute, and the moment the user looks confused. Then it translates, plainly, and resumes.

That floor is the Darmok warning applied as a safety rule. A pattern name handed to someone who never learned it is noise in a confident voice. A DROP TABLE confirmation in a confident voice the reader has not decoded is worse than noise. So the plugin’s one hard boundary is that the joke never gets to stand between the user and the consequence.

Mechanics, and the rule it forced

The mode machinery is borrowed, with thanks, from caveman, the terse-mode plugin. A level (lite, full, ultra) persists in ~/.claude/.tamarian-mode. A SessionStart hook reads that file and, if a level is set, emits the skill body into the session at runtime: one source of truth, no duplicated prompt. A UserPromptSubmit hook adds a one-line reminder on every prompt, so the voice survives long conversations and context compression. Two bash scripts, no dependencies, both print OK when the mode is off; installing the plugin changes nothing until you invoke it.

/plugin marketplace update skillz
/plugin install tamarian@skillz
/tamarian full

Caveman and Tamarian are the same knob turned opposite ways. Caveman’s README claims about 75% fewer tokens by stripping a sentence down to its referent. Tamarian names the referent and then insists on the sentence anyway. One is a token saver; the other is a demonstration, and says so in its own description: purely for entertainment.

The rule it forced: voitta-ai/skillz ships one big skillz bundle plugin plus standalone plugins, and until today a standalone plugin’s skill was also symlinked into the bundle. For a hooked plugin that is a bug. The bundle manifest carries no hooks, so the bundle copy of tamarian would speak Tamarian for one session and then forget, the dictionary lost at session end. Worse, installing both exposed the same skill twice, /skillz:tamarian next to /tamarian:tamarian. New rule, in #233: a skill that ships inside a hooked plugin is not in the bundle, and the catalog validator detects hooks from the manifests themselves, so a plugin that grows hooks later trips the check with no flag to forget. Three plugins moved out under it.

The conclusion, in ultra

The session that drafted this post ran /tamarian full. The conclusion below it wrote in ultra, glossary included, and I leave it as it came.

Darmok and Jalad at Tanagra: the last post and this one. Kira at Bashi, a joke in the final line. Mirab, with sails unfurled, four hours on. Sokath, his eyes uncovered: the pattern name is the token, the paragraph its price, and full pays it aloud on every line. Odysseus, lashed to the mast: auth.ts:42 is never a stone in a gate, and DROP TABLE is never a verse. Chesterton, his hand on the gate: the bundle copy, and the rule it forced at #233. Caveman and Tamarian at the same fork, facing opposite ways. Picard and Dathon at El-Adrel.

The river Temarc

Further work: teach Codex.