tl;dr — A context-compression tool cut our input tokens 44x and scored 2.4/12. The interesting part isn’t the score, it’s how it failed: it fabricated 29 of the 31 source citations it produced, with zero real citations on three of five questions. A benchmark that measured token savings would have recommended it enthusiastically. Then the turn: the tool was fine. One subcommand reports "line": 1 for every result, so the model had no real line numbers and invented plausible ones. Swap it for the subcommand that emits real ones and the same tool scores 6.6/12 with 83 verified citations and 1 fabricated — the best quality-per-token arm in the whole benchmark.
The setup
llm-tldr advertises 95% token savings and 155x faster queries. We first wrote about it next to voitta-rag in February, on how each feeds a codebase to a model; this is the first time either was scored. That is a big enough claim to be worth checking, so it went into our benchmark alongside a full source dump, Repomix, and RAG retrieval — same repository, same five questions, same prompt, only the injected context varying.
The scoring rule mattered more than we expected. Every answer had to carry a file:line citation for each factual claim, and a separate judge model with read-only access to the repository went and checked them. Not “is this plausible.” Does Tokeniser.java:135 exist, and does it say what the answer says it says.
The result
| score /12 | tokens in | $/question | |
|---|---|---|---|
| full source dump | 10.80 | 404,878 | $0.8552 |
| llm-tldr | 2.40 | 5,179 | $0.0236 |
44x fewer tokens. 36x cheaper. And a score you would not ship.
But the score alone doesn’t tell you why, and the why is the whole point.
The citation column
| verified citations | fabricated | |
|---|---|---|
| full source dump | 65 | 11 |
| llm-tldr | 2 | 29 |
Twenty-nine confidently-formatted references to source locations that do not exist. Zero correct citations on three of the five questions.
This is the failure mode that a token-savings benchmark cannot see, and it is strictly worse than a low score. A model that says “I don’t know” costs you one retry. A model that says “the entity decoding happens in Entities.java:412” in a well-structured paragraph costs you a code review where someone opens Entities.java, finds 412 is in the middle of an unrelated method, and now distrusts the entire document.
We had built the citation check as a nice-to-have. It turned out to be the only instrument in the benchmark that could distinguish “compressed and correct” from “compressed and confabulating.”
The mechanism
tldr semantic search returns ranked code units with a line field. That field is 1. For everything.
The model receives a genuinely useful, genuinely relevant set of code units — the retrieval is working — with every location stamped as line 1. It has been instructed to cite file:line. It knows line 1 is wrong. So it does what a language model does with a plausible-shaped gap: it fills it with a plausible number.
Nothing in the pipeline is lying. The tool reports what it has, the model reports what it inferred, and the output is 29 fabricated citations.
The part where we were wrong
When we first published this we flagged it: this measures one adapter, not the tool’s ceiling. tldr context, structure, calls, and slice all existed and might behave differently. That caveat cost one sentence to write and turned out to be the most valuable thing in the post.
tldr structure was a dead end — no line numbers at all, and it parsed 50 of the 97 files. But tldr extract carries real line_number fields for every class and method. It’s per-file and takes no query, so semantic search still does the ranking; extract supplies the locations.
| adapter | score | verified | fabricated | tokens in | $/q |
|---|---|---|---|---|---|
semantic search --expand | 2.40 | 2 | 29 | 5,179 | $0.0236 |
semantic search → extract | 6.60 | 83 | 1 | 66,895 | $0.1520 |
Nearly triple the score. Fabricated citations from 29 to 1. Same tool, same index, same questions, same prompt. The only thing that changed is which subcommand fed the context.
What this actually means
Benchmark the integration, not the logo. “llm-tldr scores 2.4” was never a true sentence. “This adapter, on this question set, produced uncitable context” was, and it was the sentence we wrote down, and it is why we knew where to look.
The winning number is buried in the fixed row. At 6.60 for $0.15/question, extract delivers 61% of the full dump’s score for a sixth of its cost. If you’re optimising cost-per-point rather than peak quality, it’s the best arm in the benchmark — better on that axis than the agentic mode that beat everything on raw quality. That result was completely invisible until the citation check explained the first one.
The same bug is everywhere. Our RAG arm scored 5.2, partly because voitta-rag’s chunk records carry a chunk_index and no line numbers. (Its citation counts turned out to be confounded by a second bug — an index-name prefix the judge could not resolve — so treat them as upper bounds; the line-number gap is real either way.) Identical failure, different vendor, discovered only because we already knew the shape. If your retrieval layer returns text without locations, you are shipping this bug, and a quality score alone will not tell you.
Next in this series: our control group was broken and it cost us 4.2 points.
Harness, raw records, and full method: voitta-rag/benchmark/. Answering on Claude Sonnet 5, judging on Claude Opus 5, both at effort high.
