tl;dr — Token compression has two independent axes: what you feed the model (tokens in) and what it writes back (tokens out). Three findings from measuring both. Repomix, pointed at the same file globs as a plain cat of the repo, produced more characters than the plain dump — its advertised ~70% reduction is file selection, not compression. A prose compressor on source code buys 9.4% by destroying the punctuation that makes it code, and costs only 0.4 points, which is its own uncomfortable finding. And you cannot measure the output axis with output-token counts on a reasoning model: our compressed-output run shrank the visible answer 21% while its tokens_out tripled.
The two axes
Most “save tokens” tooling is sold as one category, and it isn’t. There are two:
- Tokens in — shrink the context you send. Repomix, llm-tldr, RAG retrieval, prose compressors applied to a dump.
- Tokens out — shrink what the model writes back. Instruct it to answer tersely.
They’re orthogonal. An output-side compressor rides on top of any input-side strategy, which means the honest way to evaluate them is a grid, not a list. We ran the input-side arms, then re-ran a representative subset with the output-side overlay on.
Finding 1: Repomix is the same dump with a nicer cover page
Repomix packs a repository into one AI-friendly file and is widely cited for ~70% token reduction. Pointed at the same include/exclude globs we used for a plain concatenation of the same files:
| characters | score /12 | tokens in | |
|---|---|---|---|
| plain source dump | 1,119,819 | 10.80 | 404,878 |
| Repomix | 1,127,414 | 10.40 | 406,277 |
Repomix produced 7,595 more characters than cat-ing the files.
This isn’t a knock on the tool, and the 70% figure isn’t dishonest — it’s just measuring something else. Repomix’s reduction comes from file selection: honouring .gitignore, skipping binaries and lockfiles and node_modules, dropping build artifacts. Against a naive “send the whole working directory” baseline, that’s an enormous and genuine saving.
But we’d already scoped our globs to **/*.java minus tests. There was nothing left to select. What remains is formatting — a directory tree, a header block, per-file separators — and formatting costs tokens rather than saving them.
The general point: a compression ratio is a ratio against something. Before adopting a tool on a headline percentage, check what the denominator was. If your pipeline already scopes its inputs, a selection-based tool has already had its win taken.
Finding 2: a prose compressor on code, and how little the model needs
caveman-compression strips grammar an LLM can reconstruct — articles, connectives, passive constructions. We used the rule-based spaCy variant rather than the default LLM-backed one, on purpose: a non-deterministic compressor inside a benchmark cell makes the cell unattributable, and it would put a second vendor’s model inside our measurement path.
Applied to the source dump:
| characters | score /12 | tokens in | $/q | |
|---|---|---|---|---|
| plain dump | 1,119,819 | 10.80 | 404,878 | $0.8552 |
| caveman-compressed | 1,014,004 | 10.40 | 363,613 | $0.7777 |
9.4% smaller, 0.4 points. Roughly neutral.
Which is startling once you look at what it does to Java:
// before
public class Attribute implements Map.Entry<String,String>, Cloneable {
// after
public class Attribute implements Map. Entry < String String > CloneableCommas gone. Angle brackets spaced apart. Map.Entry split across a sentence boundary. This is not valid Java in any sense — a parser would reject it instantly — and the model scored 10.40 out of 12 on it.
Be fair to the tool: it’s built for prose and we pointed it at source code. This measures a mismatch, not the tool used as intended, and 9.4% on input it was never designed for is respectable.
The finding isn’t about the compressor. It’s about how much syntax the model actually needs, which is apparently much less than the syntax the compiler needs. That’s a genuinely interesting property and probably a bad thing to rely on.
Finding 3: the trap
The output-side overlay works. Instruct the model to answer in compressed style and the rendered answer gets meaningfully shorter at little quality cost:
| mode | answer chars | with overlay | score | with overlay |
|---|---|---|---|---|
| full dump | 5,699 | 4,496 (−21%) | 10.80 | 10.60 |
| agentic exploration | 6,389 | 4,029 (−37%) | 11.40 | 10.40 |
| RAG | 4,665 | 3,659 (−22%) | 4.80 | 4.60 |
| llm-tldr | 2,387 | 1,862 (−22%) | 2.40 | 3.40 |
21–37% shorter for roughly zero to one point. Cheap, real, worth having. (The RAG row’s absolute scores are depressed by a path-prefix bug in that adapter, found after this was drafted; the overlay delta this table is about is unaffected, since both cells carry it.)
Now the same experiment measured the way you’d instinctively measure it — by counting output tokens:
| rendered answer | tokens_out | |
|---|---|---|
| full dump | 5,699 chars | 4,547 |
| full dump + output compression | 4,496 chars (−21%) | 14,859 (+227%) |
The visible answer shrank by a fifth. The billed output tokens more than tripled.
tokens_out bills thinking tokens and response text together. On a reasoning model, thinking usually dominates, and it varies enormously with how hard the model decides the turn is. The overlay changed how the model approached the task — apparently prompting more deliberation about what to cut — and that swamped the text delta by an order of magnitude.
Anyone benchmarking output-side compression against tokens_out on a thinking model is measuring reasoning-depth noise and calling it compression. You will get a number, it will be reproducible, and it will point the wrong way.
Measure the rendered answer. len(response_text), or token-count the text blocks specifically. And if you’re doing cost work, keep the two apart: thinking tokens are a real cost you should track, they’re just not what an output-style instruction controls.
Next in this series: what it costs to know any of this — and why grading the answers cost more than producing them.
Harness, raw records, and full method: voitta-rag/benchmark/.
