llm-tldr

44x Fewer Tokens, and Every Citation Was Fake

Gregory Golberg method-and-apparatus, voitta , , , , , ,

tl;dr — A context-compression tool cut our input tokens 44x and scored 2.4/12. The interesting part isn’t the score, it’s how it failed: it fabricated 29 of the 31 source citations it produced, with zero real citations on three of five questions. A benchmark that measured token savings would have recommended it enthusiastically. Then the turn: the tool was fine. One subcommand reports "line": 1 for every result, so the model had no real line numbers and invented plausible ones. Swap it for the subcommand that emits real ones and the same tool scores 6.6/12 with 83 verified citations and 1 fabricated — the best quality-per-token arm in the whole benchmark.

The setup

llm-tldr advertises 95% token savings and 155x faster queries. We first wrote about it next to voitta-rag in February, on how each feeds a codebase to a model; this is the first time either was scored. That is a big enough claim to be worth checking, so it went into our benchmark alongside a full source dump, Repomix, and RAG retrieval — same repository, same five questions, same prompt, only the injected context varying.

The scoring rule mattered more than we expected. Every answer had to carry a file:line citation for each factual claim, and a separate judge model with read-only access to the repository went and checked them. Not “is this plausible.” Does Tokeniser.java:135 exist, and does it say what the answer says it says.

The result

score /12tokens in$/question
full source dump10.80404,878$0.8552
llm-tldr2.405,179$0.0236

44x fewer tokens. 36x cheaper. And a score you would not ship.

But the score alone doesn’t tell you why, and the why is the whole point.

The citation column

verified citationsfabricated
full source dump6511
llm-tldr229

Twenty-nine confidently-formatted references to source locations that do not exist. Zero correct citations on three of the five questions.

This is the failure mode that a token-savings benchmark cannot see, and it is strictly worse than a low score. A model that says “I don’t know” costs you one retry. A model that says “the entity decoding happens in Entities.java:412” in a well-structured paragraph costs you a code review where someone opens Entities.java, finds 412 is in the middle of an unrelated method, and now distrusts the entire document.

We had built the citation check as a nice-to-have. It turned out to be the only instrument in the benchmark that could distinguish “compressed and correct” from “compressed and confabulating.”

The mechanism

tldr semantic search returns ranked code units with a line field. That field is 1. For everything.

The model receives a genuinely useful, genuinely relevant set of code units — the retrieval is working — with every location stamped as line 1. It has been instructed to cite file:line. It knows line 1 is wrong. So it does what a language model does with a plausible-shaped gap: it fills it with a plausible number.

Nothing in the pipeline is lying. The tool reports what it has, the model reports what it inferred, and the output is 29 fabricated citations.

The part where we were wrong

When we first published this we flagged it: this measures one adapter, not the tool’s ceiling. tldr context, structure, calls, and slice all existed and might behave differently. That caveat cost one sentence to write and turned out to be the most valuable thing in the post.

tldr structure was a dead end — no line numbers at all, and it parsed 50 of the 97 files. But tldr extract carries real line_number fields for every class and method. It’s per-file and takes no query, so semantic search still does the ranking; extract supplies the locations.

adapterscoreverifiedfabricatedtokens in$/q
semantic search --expand2.402295,179$0.0236
semantic search → extract6.6083166,895$0.1520

Nearly triple the score. Fabricated citations from 29 to 1. Same tool, same index, same questions, same prompt. The only thing that changed is which subcommand fed the context.

What this actually means

Benchmark the integration, not the logo. “llm-tldr scores 2.4” was never a true sentence. “This adapter, on this question set, produced uncitable context” was, and it was the sentence we wrote down, and it is why we knew where to look.

The winning number is buried in the fixed row. At 6.60 for $0.15/question, extract delivers 61% of the full dump’s score for a sixth of its cost. If you’re optimising cost-per-point rather than peak quality, it’s the best arm in the benchmark — better on that axis than the agentic mode that beat everything on raw quality. That result was completely invisible until the citation check explained the first one.

The same bug is everywhere. Our RAG arm scored 5.2, partly because voitta-rag’s chunk records carry a chunk_index and no line numbers. (Its citation counts turned out to be confounded by a second bug — an index-name prefix the judge could not resolve — so treat them as upper bounds; the line-number gap is real either way.) Identical failure, different vendor, discovered only because we already knew the shape. If your retrieval layer returns text without locations, you are shipping this bug, and a quality score alone will not tell you.

Next in this series: our control group was broken and it cost us 4.2 points.

Harness, raw records, and full method: voitta-rag/benchmark/. Answering on Claude Sonnet 5, judging on Claude Opus 5, both at effort high.

Nobody Needed to Fit the Codebase in the Window

Gregory Golberg method-and-apparatus, voitta , , , , , , , , ,

tl;dr — We benchmarked five strategies for getting a Java codebase into an LLM’s context: a full source dump, Repomix, two llm-tldr adapters, and RAG retrieval. The winner was none of them. Giving the model read_file, grep, and glob and letting it go find things scored 11.4/12, against the full dump’s 10.8 — while using 29% fewer tokens and costing 25% less. It also produced 142 verified source citations against 2 fabricated, the cleanest record in the benchmark. Every tool in this category optimises how to pack the context window. On this question set, the winning move was not to pack it.

The question

A colleague dropped Repomix in Slack — pack your whole repo into one AI-friendly file, ~70% token reduction. Someone else pointed at llm-tldr — 95% token savings, 155x faster queries. A third person asked the only question that matters:

If one of you get time can you run an eval on the same codebase for the same task and let me know if these actually improve the output and which one is better

So we did. One repository (jsoup, 97 Java files, deliberately one nobody on the team knew), five questions spanning five kinds of thing you actually ask about code, and every strategy answering the identical prompt with only the injected context varying.

The scoring, because it’s the part that matters

Every answer had to carry a file:line citation for every factual claim. The judge — a separate model with read-only read_file, grep, and glob over the repository — then went and checked them. Not “does this look right.” Does Tokeniser.java:135 exist, and does it say what the answer claims.

That produces two numbers per answer: a quality score out of 12, and a count of citations that resolved against real source versus citations that didn’t. The second number is the one that earns its keep, and a later post in this series is entirely about what it caught.

The result

strategyscore /12tokens in$/questionverified citesbogus
agentic exploration11.40288,3420.63871422
llm-tldr → agentic11.00288,4360.63781240
full source dump10.80404,8780.85526511
Repomix10.40406,2770.93746917
prose-compressed dump10.40363,6130.7777861
llm-tldr (extract)6.6066,8950.1520831
RAG retrieval5.204,3370.02892624
llm-tldr (semantic search)2.405,1790.0236229

One caveat on the RAG row, found after this was drafted and before it was published: its bogus count is an upper bound. The adapter handed the model paths prefixed with the index name (jsoup/src/…) while the judge resolved citations against the checkout root (src/…), so citations that were real scored as unresolved — the prefix is visible in the judge’s notes on 13 of 15 retrieval answers. The harness strips it now; these numbers predate that. It touches no other arm, and the arm it flatters least is the one we build.

The top line is a mode we added almost as a control — no context building at all, just hand the model the same three read-only tools the judge uses and let it explore. It won on quality, it won on citation accuracy by a wide margin, and it was cheaper than the thing it beat.

Why it wins

Not because it’s clever. Because of what it has at the moment it makes a claim.

Every other strategy front-loads: build a representation of the codebase, inject it, hope the answer is in there. The representation is fixed before the model has read the question closely, so it is necessarily a guess about relevance — and whatever the representation dropped, the model cannot recover.

Agentic exploration defers. It reads the question, forms a hypothesis, greps for it, gets it wrong, greps again, opens the file, reads the actual lines. Seven to sixteen tool calls per question in our runs. When it finally writes Tokeniser.java:135, it is because it has line 135 on screen.

That is the whole mechanism behind the citation column. Verified-to-bogus for agentic exploration was 142:2. For the full dump, 65:11 — the dump had every line, but the model was reading a 405,000-token wall of text and lost track of where in it things were. For the cheapest compressed mode, 2:29.

Worth sitting with: the full dump contains strictly more information than the agentic mode ever sees, and still loses. Having the bytes in the window is not the same as being able to use them.

Two caveats we’re keeping

Cumulative tokens. The 288K for agentic exploration is summed across every turn of the tool loop, not one request. It is the honest number for cost, and it is not the same kind of number as a one-shot mode’s single request. We report it that way because it’s what the strategy actually costs to answer one question, but don’t put it in a bar chart next to a single-shot figure without the asterisk.

Five questions. Enough to catch a large effect, not enough to rank close ones. The 10.4–10.8 cluster — full dump, Repomix, prose-compressed dump — is a tie as far as this data can tell. The gaps worth believing are the big ones: agentic exploration over the compressed modes, and the two llm-tldr adapters against each other.

The uncomfortable implication

There’s a lot of engineering going into context compression right now, and this result doesn’t say that work is worthless — the compressed modes have a real argument, which is price. llm-tldr via its extract adapter got 61% of the baseline’s score for a sixth of the cost. If you’re running a million of these, that trade is the whole business.

But if you’re optimising for a correct answer, the ranking says: give the model tools and get out of the way. The context window is not a thing to be filled efficiently. It’s a workspace, and the model is better at deciding what belongs in it than our heuristics are.

Next in this series: the tool that cut input tokens 44x and fabricated 29 of its 31 citations — and why that turned out to be our fault, not the tool’s.

Harness, raw results, and full method: voitta-rag/benchmark/. Answering on Claude Sonnet 5, judging on Claude Opus 5, both at effort high. Total cost of the run: $80.15 over 70 scored cells, of which $51.90 was judging — which is its own post.

One Search Surface: Teaching voitta-rag to Speak Architecture

Gregory Golberg holding forth, projects/method and apparatus , , , , , , ,

Back in February, I wrote that llm-tldr and voitta-rag were complementary. One builds a map of a codebase through static analysis. The other retrieves the actual code you need. My conclusion then was basically: great, wire both into the agent and let it choose.

That works, but it still leaves the agent doing tool-routing. It has to know that one question wants architecture and another wants source. It has to bounce between surfaces. So we collapsed the distinction.

voitta-rag can now index llm-tldr‘s static-analysis output as companion documents alongside the raw code chunks it already stores for Git sources. Turn on the new gh_llm_tldr flag for a repo, sync it, and the same search surface now returns two different kinds of context:

  • raw code chunks for the implementation itself, and
  • structural analysis chunks describing callers, callees, imports, signatures, and relationships.

One query. One index. No “which tool should I call?” moment.

The old split was clean, but inconvenient

The original split between the two tools made conceptual sense.

llm-tldr is good at questions like:

  • What calls this function?
  • What depends on this module?
  • Where does this piece of data flow?
  • What parts of the codebase are structurally central?

voitta-rag is good at questions like:

  • Show me the implementation of token verification.
  • Find the code that handles OAuth callbacks.
  • Search across this repo, that wiki, and those tickets.
  • Give me the actual file I need to edit.

That’s a nice division of labor for a human. It is less nice for an agent, because agents do not merely need information; they need the right shape of information without extra orchestration. The more routing logic you make them do, the more failure modes you introduce.

The latest voitta-rag implementation removes that choice entirely. Static analysis stops being a separate destination and becomes part of retrieval.

What actually shipped

When a Git source has gh_llm_tldr enabled, sync now runs llm-tldr over each supported source file and stores the results in the same Qdrant collection as the ordinary code chunks.

Those analysis chunks are tagged as source_type="llm-tldr-analysis" and linked back to their origin file with related_file. That sounds like plumbing, and it is, but it matters: the search layer now knows that an analysis chunk about verify_token() belongs to a specific source file rather than floating around as a free-standing summary.

The first proof of concept indexed file-level summaries. The more interesting version goes further: it now stores one overview chunk per file plus one chunk per top-level function and class method. Each function-level chunk can carry structured payload fields such as:

  • function name
  • class name
  • callers
  • callees
  • caller count
  • callee count
  • imports

That means this is not just “RAG, but with bigger summaries.” The call graph is queryable metadata now. You can filter for things like “functions with more than five callers” or “functions importing module X” without standing up a separate graph database just to answer what are, in practice, glorified indexing questions.

GitNexus

GitNexus is interesting, but it is licensed under PolyForm Noncommercial. That’s a non-starter for a lot of consulting and commercial work. By contrast, both llm-tldr and voitta-rag are AGPL v3.

Why function-level chunks beat file-level blobs

The biggest design improvement was moving from file-level rendered analysis to function-level structural chunks.

On voitta-rag indexing itself, that produced 647 stored analysis chunks: 70 file-overview chunks and 577 function chunks. That sounds like more pieces, but it is actually a better unit of retrieval. Agents rarely need a whole philosophical treatise about a file. They need to know that foo() is called from three handlers, imports sqlalchemy.orm, and sits on the hot path for authentication. Function-level chunks make that retrievable directly.

It is also a cheaper way to approximate code intelligence than hauling in a dedicated graph stack. You keep the retrieval surface the agent already understands, but enrich the payload enough to answer the structural questions that retrieval alone cannot.

Related reading: llm-tldr vs voitta-rag: Two Ways to Feed a Codebase to an LLM