tl;dr — Our retrieval arm kept returning changelogs instead of source, so the model correctly refused to answer. I wrote it up with a satisfying mechanism: jsoup’s changelog describes parser behaviour in the same prose vocabulary the questions use, so it outranks the code. Plausible. Real numbers. Wrong. The include_folders filter is an exact match on a file’s parent directory, not a subtree prefix — so passing the repo name scoped retrieval to the five files sitting at the repo root and excluded all of src/. No error, just real, well-formed, confidently useless results. Then fixing it didn’t help, and why not is the actual finding.

---

The finding I published

Our RAG arm scored 5.2/12. Four of its five answers were refusals — the model saying, in effect, the source files I’d need aren’t in the retrieved context.

The retrieved chunks were all from CHANGES.md and change-archive.txt. So I wrote the obvious mechanism:

The folder was indexed whole, and hybrid retrieval on questions phrased in changelog vocabulary (“malformed start tags”, “charset conflict”) ranks CHANGES.md and change-archive.txt above the .java files, because jsoup’s changelog literally describes these behaviours in prose.

That is a good paragraph. It has a mechanism, it’s consistent with the data, and it makes a genuine point about hybrid search on repositories that contain prose. It went into a committed README as a finding about retrieval.

What was actually happening

To keep the RAG arm from retrieving over unrelated indexed folders — including, awkwardly, its own source — I’d scoped the search:

"voitta_rag_include_folders": ["jsoup"]

include_folders sounds like subtree scoping. It isn’t. Over MCP it’s an exact match on a chunk’s folder_path, and folder_path is the directory the file sits in, not the index root.

Files whose folder_path is exactly jsoup:

jsoup/CHANGES.md
jsoup/change-archive.txt
jsoup/README.md
jsoup/LICENSE
jsoup/SECURITY.md

Everything under src/ has a folder_path of jsoup/src/main/java/org/jsoup/... and was excluded. I had scoped the benchmark’s retrieval arm to five files, three of which are changelogs.

The model wasn’t outranked by prose. It was handed a changelog and asked about a parser, said so, and was correct every time.

The part that makes this worth writing up

The subtree expansion exists. It’s right there in mcp_server.search:

if user_name:
    ...
    if folder_normalized == active_normalized or \
       folder_normalized.startswith(active_normalized + "/"):

Prefix matching, exactly as you’d want. It runs under if user_name: — and the MCP tool signature has no user_name parameter. Over MCP that branch is unreachable, so include_folders falls through to an exact MatchAny against Qdrant.

The code that would have made my mental model correct was in the repository, being skipped, on a branch I couldn’t reach from the interface I was calling.

Why it survived review

Because it never failed. Consider what a wrong filter doesn’t do here:

• It doesn’t error. Five files is a legitimate result.
• It doesn’t return nothing. Empty results would have sent me straight to the config.
• It doesn’t return garbage. The chunks were real, relevant-looking prose from the correct repository.
• The model’s behaviour was exemplary — it recognised insufficient context and declined instead of confabulating. That’s the behaviour you want, and it made the arm look thoughtfully-failing rather than mis-configured.

Every signal pointed at “retrieval made a ranking decision I should analyse” rather than “retrieval was handed the wrong corpus.” The failure was epistemically camouflaged: it produced exactly the artifacts a real finding produces.

Then fixing it didn’t help

Then I fixed it. I enumerated the directories, passed them all, and re-ran. Retrieval now returned actual Java source — Entities.java for the entity-decoding question, correctly.

Then I went further and eliminated the corpus question entirely: built a second index containing exactly the 97 .java files the other arms see, no changelogs at all, and ran that too.

	score /12	verified cites	fabricated
whole checkout (233 files)	5.00	31	15
corpus-matched (97 .java files)	5.40	21	29

Caveat on the retrieval numbers, found after this was drafted: the adapter handed the model paths prefixed with the index name (jsoup/src/…) while the judge resolved citations against the checkout root (src/…), so citations that were real scored as unresolved. The fabricated counts here are upper bounds and the scores that depend on them are not comparable with the other arms. The harness strips the prefix now; these cells predate that, and a re-run is pending.

Matching the corpus moved it very little, and a fresh run with the adapter’s path bug fixed moved it again: the two configurations land 5.00 and 5.40, a gap the same size as the one that first ran the other way. These are independent runs weeks apart, not a re-grading of the same answers, so the reversal doesn’t prove the bug caused the original ordering either. Five questions cannot separate them. My replacement hypothesis — that corpus asymmetry was dragging the arm down — is not refuted so much as untested, which is a duller sentence than the one I published and the only one the data supports.

A second cause was sitting in the citation column the whole time. In the corpus-matched configuration there are roughly as many fabricated citations as verified ones; in the other the split is better than that. Neither is good, and the mechanism is identical in both. voitta-rag’s chunk records carry chunk_index and total_chunks and no line numbers. A model handed a perfectly correct chunk still cannot cite file:line, so it invents one. And the citation column itself carried a third artifact, found after this draft: the adapter injected index-prefixed paths the judge could not resolve, so some of what it counted as fabrication was a real citation wearing the wrong prefix. Three config-shaped artifacts in one arm, each of which looked like a finding. If adding line spans to the chunk record (voitta-rag#52) does not move the fabricated column, this diagnosis is wrong too.

That is the identical failure we’d already diagnosed in a completely different tool two posts ago — llm-tldr reporting "line": 1 for every result. Same root cause, different vendor, and I only recognised it because we’d been forced to look at citations rather than scores.

The transferable bit

Silent scope failures don’t crash. They produce publishable conclusions.

A crash sends you to the config. A plausible result sends you to the writeup. The more coherent your explanation of a surprising result, the more suspicious you should be — I had a good mechanism, and the quality of the story is exactly what stopped me checking the inputs.

So, concretely, before theorising about why an arm underperformed:

Print what it actually received. Not the score, not the answer — the raw retrieved payload. One line of debugging:

print(sorted({c["file_path"] for c in retrieved}))

Had I run that once, I’d have seen five filenames, none of them .java, and this would have been a config fix instead of a published finding, a correction, and a blog post.

And when a filter’s name implies semantics you haven’t verified — include_folders sounds like a subtree, exclude_paths sounds recursive, limit sounds per-query — spend the thirty seconds confirming it before building an experiment on top of it.

---

Next in this series: two axes of compression, and a measurement trap that makes one of them unmeasurable.

Harness, raw records, and full method: voitta-rag/benchmark/.

Correction, 2026-09-25. The two rows in the table above have been corrected and the paragraph beginning “Matching the corpus” has been rewritten. This post first published 5.20 and 4.80 and argued that matching the corpus made things measurably worse. A fresh run with the adapter’s path bug fixed gives 5.00 and 5.40 — the same size gap, running the other way. Two independent five-question runs disagree, so the claim is withdrawn rather than reversed: the effect is untested, not settled in either direction. The re-run and the fix are in voitta-rag#57 and #58.

Context benchmark series — part 4 of 7: ← Previous · Series index · Next →