postmortem

I Published a Finding About RAG. It Was a Finding About My Config.

Gregory Golberg method-and-apparatus, voitta , , , , , , ,

tl;dr — Our retrieval arm kept returning changelogs instead of source, so the model correctly refused to answer. I wrote it up with a satisfying mechanism: jsoup’s changelog describes parser behaviour in the same prose vocabulary the questions use, so it outranks the code. Plausible. Real numbers. Wrong. The include_folders filter is an exact match on a file’s parent directory, not a subtree prefix — so passing the repo name scoped retrieval to the five files sitting at the repo root and excluded all of src/. No error, just real, well-formed, confidently useless results. Then fixing it didn’t help, and why not is the actual finding.

The finding I published

Our RAG arm scored 5.2/12. Four of its five answers were refusals — the model saying, in effect, the source files I’d need aren’t in the retrieved context.

The retrieved chunks were all from CHANGES.md and change-archive.txt. So I wrote the obvious mechanism:

The folder was indexed whole, and hybrid retrieval on questions phrased in changelog vocabulary (“malformed start tags”, “charset conflict”) ranks CHANGES.md and change-archive.txt above the .java files, because jsoup’s changelog literally describes these behaviours in prose.

That is a good paragraph. It has a mechanism, it’s consistent with the data, and it makes a genuine point about hybrid search on repositories that contain prose. It went into a committed README as a finding about retrieval.

What was actually happening

To keep the RAG arm from retrieving over unrelated indexed folders — including, awkwardly, its own source — I’d scoped the search:

"voitta_rag_include_folders": ["jsoup"]

include_folders sounds like subtree scoping. It isn’t. Over MCP it’s an exact match on a chunk’s folder_path, and folder_path is the directory the file sits in, not the index root.

Files whose folder_path is exactly jsoup:

jsoup/CHANGES.md
jsoup/change-archive.txt
jsoup/README.md
jsoup/LICENSE
jsoup/SECURITY.md

Everything under src/ has a folder_path of jsoup/src/main/java/org/jsoup/... and was excluded. I had scoped the benchmark’s retrieval arm to five files, three of which are changelogs.

The model wasn’t outranked by prose. It was handed a changelog and asked about a parser, said so, and was correct every time.

The part that makes this worth writing up

The subtree expansion exists. It’s right there in mcp_server.search:

if user_name:
    ...
    if folder_normalized == active_normalized or \
       folder_normalized.startswith(active_normalized + "/"):

Prefix matching, exactly as you’d want. It runs under if user_name: — and the MCP tool signature has no user_name parameter. Over MCP that branch is unreachable, so include_folders falls through to an exact MatchAny against Qdrant.

The code that would have made my mental model correct was in the repository, being skipped, on a branch I couldn’t reach from the interface I was calling.

Why it survived review

Because it never failed. Consider what a wrong filter doesn’t do here:

  • It doesn’t error. Five files is a legitimate result.
  • It doesn’t return nothing. Empty results would have sent me straight to the config.
  • It doesn’t return garbage. The chunks were real, relevant-looking prose from the correct repository.
  • The model’s behaviour was exemplary — it recognised insufficient context and declined instead of confabulating. That’s the behaviour you want, and it made the arm look thoughtfully-failing rather than mis-configured.

Every signal pointed at “retrieval made a ranking decision I should analyse” rather than “retrieval was handed the wrong corpus.” The failure was epistemically camouflaged: it produced exactly the artifacts a real finding produces.

Then fixing it didn’t help

Then I fixed it. I enumerated the directories, passed them all, and re-ran. Retrieval now returned actual Java source — Entities.java for the entity-decoding question, correctly.

Then I went further and eliminated the corpus question entirely: built a second index containing exactly the 97 .java files the other arms see, no changelogs at all, and ran that too.

score /12verified citesfabricated
whole checkout (233 files)5.003115
corpus-matched (97 .java files)5.402129

Caveat on the retrieval numbers, found after this was drafted: the adapter handed the model paths prefixed with the index name (jsoup/src/…) while the judge resolved citations against the checkout root (src/…), so citations that were real scored as unresolved. The fabricated counts here are upper bounds and the scores that depend on them are not comparable with the other arms. The harness strips the prefix now; these cells predate that, and a re-run is pending.

Matching the corpus moved it very little, and a fresh run with the adapter’s path bug fixed moved it again: the two configurations land 5.00 and 5.40, a gap the same size as the one that first ran the other way. These are independent runs weeks apart, not a re-grading of the same answers, so the reversal doesn’t prove the bug caused the original ordering either. Five questions cannot separate them. My replacement hypothesis — that corpus asymmetry was dragging the arm down — is not refuted so much as untested, which is a duller sentence than the one I published and the only one the data supports.

A second cause was sitting in the citation column the whole time. In the corpus-matched configuration there are roughly as many fabricated citations as verified ones; in the other the split is better than that. Neither is good, and the mechanism is identical in both. voitta-rag’s chunk records carry chunk_index and total_chunks and no line numbers. A model handed a perfectly correct chunk still cannot cite file:line, so it invents one. And the citation column itself carried a third artifact, found after this draft: the adapter injected index-prefixed paths the judge could not resolve, so some of what it counted as fabrication was a real citation wearing the wrong prefix. Three config-shaped artifacts in one arm, each of which looked like a finding. If adding line spans to the chunk record (voitta-rag#52) does not move the fabricated column, this diagnosis is wrong too.

That is the identical failure we’d already diagnosed in a completely different tool two posts ago — llm-tldr reporting "line": 1 for every result. Same root cause, different vendor, and I only recognised it because we’d been forced to look at citations rather than scores.

The transferable bit

Silent scope failures don’t crash. They produce publishable conclusions.

A crash sends you to the config. A plausible result sends you to the writeup. The more coherent your explanation of a surprising result, the more suspicious you should be — I had a good mechanism, and the quality of the story is exactly what stopped me checking the inputs.

So, concretely, before theorising about why an arm underperformed:

Print what it actually received. Not the score, not the answer — the raw retrieved payload. One line of debugging:

print(sorted({c["file_path"] for c in retrieved}))

Had I run that once, I’d have seen five filenames, none of them .java, and this would have been a config fix instead of a published finding, a correction, and a blog post.

And when a filter’s name implies semantics you haven’t verified — include_folders sounds like a subtree, exclude_paths sounds recursive, limit sounds per-query — spend the thirty seconds confirming it before building an experiment on top of it.

Next in this series: two axes of compression, and a measurement trap that makes one of them unmeasurable.

Harness, raw records, and full method: voitta-rag/benchmark/.

Correction, 2026-09-25. The two rows in the table above have been corrected and the paragraph beginning “Matching the corpus” has been rewritten. This post first published 5.20 and 4.80 and argued that matching the corpus made things measurably worse. A fresh run with the adapter’s path bug fixed gives 5.00 and 5.40 — the same size gap, running the other way. Two independent five-question runs disagree, so the claim is withdrawn rather than reversed: the effect is untested, not settled in either direction. The re-run and the fix are in voitta-rag#57 and #58.

Context benchmark series — part 4 of 7: ← Previous · Series index · Next →

Our Control Group Was Broken and It Cost Us 4.2 Points

Gregory Golberg method-and-apparatus, voitta , , , , ,

tl;dr — The “full repository dump” baseline in our benchmark packed files until one didn’t fit, skipped it, and kept going. That’s not a budget, it’s a size filter. It quietly admitted 69 of 97 files and dropped the largest files, among them the three classes the architecture question asked about. The model correctly reported them “absent from the provided files,” and we scored that as the baseline’s ceiling. Fixing the packer: 6.60 → 10.80 out of 12. Every cross-strategy comparison we’d published was anchored to a control that was wrong by 4.2 points.

Fourteen lines of ordinary code

for relative in paths:
    body = open(os.path.join(repo_root, relative)).read()
    block = "===== FILE: {0} =====\n{1}\n".format(relative, body)
    if used + len(block) > budget:
        continue          # <-- this
    chunks.append(block)
    used += len(block)

continue, not break. When a file doesn’t fit the remaining budget, skip it and try the next one. It reads like politeness — pack as much as possible — and it passes review, because every individual line is correct.

What it actually implements is: prefer small files. Once the budget gets tight, every large file gets skipped and every small one still slides in. The bias grows as the budget fills, and it is invisible from the outside, because the output is a perfectly well-formed source dump.

At a 600,000-character budget over jsoup, it admitted 69 of 97 files. The ones it dropped were the largest: Parser.java, Tokeniser.java, TreeBuilder.java, HtmlTreeBuilder.java, HtmlTreeBuilderState.java, TokeniserState.java.

The question we then asked it

How is the parser subsystem structured? Describe the roles of the tokeniser, the tree builder, and the parser state machine.

Every class in that question was in the set the packer had silently dropped. Seven small files from parser/ were present — ParseError.java, ParseSettings.java, TokenData.java — so the dump looked like it covered the parser package.

The model answered honestly: those classes are “absent from the provided files.”

It was right. We scored it 4/12 and recorded it as what a full-context dump can achieve.

The number

score /12
baseline, skip-and-continue packer6.60
baseline, fixed10.80

Our control was understated by 4.2 points out of 12, and everything else was measured against it. Every “this compressed mode reaches N% of full-context quality” claim in the first writeup was computed against a denominator that was wrong in the flattering direction — making every compression strategy look better than it was.

The second-order damage is worse than the first. A wrong treatment arm is one wrong row. A wrong control is every row.

Why nothing caught it

There was no error. No exception, no warning, no truncation notice. stop_reason was end_turn. The cost was normal. The answer was fluent, correctly formatted, and internally consistent.

And critically: the answer was true. The model wasn’t hallucinating or hedging — it accurately described the context it had been given. The bug was one layer up, in the gap between what we thought we handed it and what we actually did.

That gap is invisible to every check that examines the output.

What we changed

Two things, and the second matters more than the first.

Stop at the budget instead of skipping past it:

if used + len(block) > budget:
    break

Truncating at a prefix is still lossy — but it’s lossy in a way that’s ordered and legible rather than correlated with file size.

Make the artifact declare its own incompleteness:

Repository source dump. TRUNCATED: the first 69 of 97 matching files in path
order, cut off by a 600000-character budget. Files after 'parser/TokenData.java'
are absent from this dump but do exist in the repository.

Now the model knows the difference between “this class doesn’t exist” and “this class wasn’t given to me” — and so does anyone reading the transcript. Then we raised the budget so nothing truncates at all, and checked the result by hand: 97 of 97 files included, with Parser.java and Tokeniser.java present. The 97 is jsoup at d24b16d9, which the harness pins; the repository is at 96 today, so a rerun on a later checkout counts differently.

The general version

Every one of us has written continue where break belonged. That’s not the lesson. The lesson is about which bug you can afford to have there.

In production code, a size-biased packer is a mild performance quirk. In a benchmark’s control group, it’s a systematic error multiplied across every comparison you publish — and it presents as a result, which means it gets written up rather than investigated.

So: audit the control first, and audit it hardest. Not “does it run” but “does it contain what I claim it contains.” For a full-context baseline that is a three-line assertion. It was not in this harness when the bug bit; it is now, behind a baseline_require_full flag so a deliberately budgeted dump can still label itself instead of failing. Each line catches a different failure: the count catches a truncated dump, and the Parser.java line catches globs that matched nothing, where the count is 0 of 0 and passes:

assert included == len(paths), f"{included} of {len(paths)}"
assert any(p.endswith("/Parser.java") for p in paths[:included])

Ten seconds to write. It would have saved this entire post.

Next in this series: I published a finding about RAG. It was a finding about my config.

Harness, raw records, and full method: voitta-rag/benchmark/.

Context benchmark series — part 3 of 7: ← Previous · Series index · Next →

The Call Was Coming From Inside the House

Gregory Golberg holding forth, voitta , , , , , , , , , ,

This is going out on a Thursday night — the eve of the Friday news dump, that fine tradition of publishing what you’d rather nobody read too closely.

That is not why. We could have held it for Monday morning and the numbers would have been better. We didn’t want to sit on it, and the joke lands a day early. Look at what we’re willing to put in front of you.

Here’s the confession: we built a safety tool, and for a while it was the least safe thing on the machine.

YOLT is our little guardrail — it looks at each command an agent is about to run and decides whether it’s risky. Useful. The problem is that it also wrote down every command it inspected, verbatim, into a log under your home directory. Forever. No rotation. On by default.

You can see where this goes. Commands carry secrets: an API key pulled from a vault and handed to the next curl, a bot token, a connection string. So our safety tool quietly accumulated a plaintext pile of exactly the things it existed to protect, sitting in the one file nobody thinks to grep. Because who audits their seatbelt.

It got better. The reviewer that reads those logs copied the same raw lines into two more files. And the leak was never confined to our tool: a credential on a command line also lands in the agent’s own session transcript, which is a much larger and much quieter surface. The blast radius was bigger than the bug.

Then we swept everything, and it got worse

Once we started looking properly — every agent directory on every machine — the pattern that came back wasn’t carelessness. It was housekeeping.

A collaborator’s cloud key was rotated by an agent session. create-access-key printed the new secret to stdout, and the session wrote stdout to a transcript, where it sat in plaintext for a month. The rotation produced a longer-lived exposure than the thing it was fixing. A Slack webhook minted to replace one leaked in git history then leaked into a transcript itself.

The dangerous moments are the tidy ones. Rotation is the riskiest thing you do all quarter, precisely because new key material is briefly in the open, and an agent session is a permanent plaintext record of everything that crossed it.

Our favourite: halfway through the audit we checked which token the audit itself was using. It was the leaked one. The rotation had updated the config file and not the already-running shell, so the tool hunting the compromised credential was authenticating with it.

The retirement gap

Most of what we found was in sessions of software we had already decided to kill.

That gap — between deciding to retire something and actually retiring it — is where credentials rot. Nobody audits the tool on its way out. Nobody rotates for it. It keeps its tokens and keeps writing transcripts, right up until the plug comes out. Every team reading this has that gap open right now.

Retiring OpenClaw here was one of those decisions, and to be clear it was about our needs, not a verdict on the project. Vi versus emacs, and who cares. What we care about is a working deliverable arrived at the way we agreed; how you got there is your business.

Everything failed quietly

The cleanup taught us more than the leak did, mostly about no-ops that look like success.

Liveness probes lie, in at least five distinct ways we hit: a search API that answers 422 for valid and invalid keys alike; a catalogue route returning 200 with no authorization header at all; a live key that is merely spend-capped answering 400; a deleted webhook 404ing where a live one 400s; and an endpoint that validated the model name before the credential, so it gave identical answers for a live key, a dead key, and our deliberately invalid control. Carry a known-bad control, hit a route that enforces auth, and read the body, not the status line.

Then the same lie turned up somewhere we weren’t even looking. Our Slack agent runs a multi-vendor waterfall, and LiteLLM picks vendor cooldowns by HTTP status — it cools 429, 401, 408 and 404. Budget exhaustion is none of those: Anthropic answers a spend cap with 400, OpenRouter answers no-credit with 402. So a vendor that had been dead for weeks got dialled first on every single message, failed, and only then did the chain fall through. Fallback worked perfectly; nothing ever learned. That one is filed upstream. A status code is not a diagnosis, and that holds well past credential probes.

The scrubbing was the same shape. A fingerprint recorded from a truncated regex match can never match the value it was meant to track, so the scrubber reports the file clean while the secret sits in it. A post-scrub grep for the pattern returns hits forever, because docs and test fixtures share the pattern — a clean run looks failed. Nothing errors. You only find these by checking the thing itself instead of the report about the thing.

What’s shipped

The redaction that should have been there on day one is now day one for real: YOLT redacts before it writes, and the current release carries it. There’s also an advisory session hook that warns when a credential rides along on a command line.

The techniques are open source, because they’re the genuinely useful part. Our skills catalog, skillz, shipped v1.15.0 with everything above written down: agent-session-credential-audit for the sweep, the false-positive taxonomy, the probe rules and the kill-list scrubber that structurally cannot erase a live secret, and agent-credential-leak-surfaces for the places copies quietly accumulate. Install them, or just read them and steal the parts you want.

One surface we’d missed entirely and you probably have too: the OS keychain. No filesystem sweep will ever see it. Ours held a stale token and handed it over to a pipe with no prompt at all.

shmobster, the Slack agent we introduced in July, took the brunt of it. It had never been tagged at all; it now has six releases, every one cut in the day since this audit started, and the first exists only because the audit went looking for what we ship and found nothing versioned.

The one that matters here is redaction. The agent hands command output straight back to a channel, and cat, env and printenv are read-only, so they clear the safety gate and run with no approval at all. Not theoretical: we found a live config holding literal keys, where a single cat would have posted all five of them to Slack. Everything the agent says is now scrubbed before it leaves the process — reusing YOLT’s redactor rather than a second pattern list that would drift from it, plus this process’s own secrets matched by exact value, because the one thing a generic detector cannot know is which strings are yours. Scrubbing happens at collection, so the model’s own context never holds a credential it could repeat later.

It also learned to read skills, so that catalog now reaches the agent actually sitting in the channel with you, files unchanged. And v0.5.1 fixed a bug from precisely this post’s family: an unguarded loop over channels, where one stale channel id sorted first, aborted the rest, and sent the previous release’s announcement to none of the four healthy channels. The only trace was a traceback about the one channel that failed.

On our end we’re rotating and scrubbing. Assume-compromised is cheaper than assume-fine.

The lesson isn’t subtle, which is exactly why it stings: the safest-looking place is the least-swept. A security tool is the last thing anyone suspects of being a liability, so it’s the perfect place for one to hide. We wrote a guardrail and forgot that a guardrail with a memory is a ledger.

So, two asks. First: if you run YOLT or any of our skills, update — the versions that close this are shipped. Second: come pound on us. Find the next hole, open the issue, tell us where else we’re being careless. We would much rather hear it from you than from a log file.

That’s the whole trade: we screw up in public, you get to keep us honest, and everyone’s tooling gets a little safer. We take the work seriously. Ourselves, less so — hence Friday’s eve, a slot we’re using as a punchline rather than as cover.

Blow-by-blow in the copious links above. Have a good weekend.

A post-mortem of a project: Wildboard

Gregory Golberg projects , , , , , , , , , , ,

Once upon a time, I was working at Snaplogic, and at that time its office was in downtown San Mateo. Pretty much across the street from a great coffee shop, Kaffeehaus.

If you’ve been there, or even if you just looked at the website, you’d realize that the owner put quite an effort into it being a Viennese-style coffee house, with all the interior design decisions that go with it.

Now, a local coffee shop is often a place where people expect to post some local notices and ads (“lost dog”, “handyman available”, “local church choir concert”, etc). And here’s a conundrum. A simple cork bulletin board with a bunch of papers pinned to it just did not seem to fit the overall mood/interior/decor of the cafe:

Yet the cafe does want to serve local community and become an institution.

This being Silicon Valley, Val, the Kaffeehaus owner, had a vision — what about a virtual board, as a touch-screen.

The name was quickly chosen to be Wildboard — because it is, well, a bulletin board and in honor of the boar’s head that is prominently featured on the wall:

.

A multi-touch-based virtual bulletin board sounded interesting. Most touch-screen kiosks I’ve seen so far — in hotels and malls, for instance, or things like ImageSurge — only allow tap, not true multi-touch. (To be honest, multi-touch may or may not be useful — but see below and see also P.S. — but it is a very nice “pizzazz”).

And we — that is, myself and Vio — got to work. And in short order we had:

  • Fully multi-touch (with rotation, zoom, etc) web UI — as a Windows 8 CSS/Javascript app (source).
  • Wildboard “board server” — a Python app running on the same computer as the UI. It is responsible for polling the web server (below) and serving information to the UI (source).
  • Wildboard web server — a PHP app based on an existing web classified application(source). This allows users to submit ads (or they can do it via a mobile app, as below). It is also modified to automatically create QR codes based on user-provided information (map, contact, calendar, etc) and adds them to an ad.
  • Wildboard mobile app — PhoneGap/Cordova based app for both Android and iPhone (source)
    This app allows one to:

    • Post an ad
    • Scan an ad’s QR code
    • And, finally, for the “Wow!” effect during the demo, one can drag an ad from the screen into the phone. Here it is, in action:

  • Wildboard orchestrator — a Node.js app (source) designed to coordinate interactions between the mobile app and the board. It is the one that is determines which mobile app is near which board and orchestrates the fancy “drag” operation shown above.
  • For more information, check out spec and the writeup.

Charismatic Val somehow managed to get a big touch screen from Elo Touch. Here’s how it fit in the decor:

A network of such bulletin boards, allowing hyper-local advertising, seems like a good idea. Monetization can be done in a number of ways:

  • Charging for additional QR codes — e.g., map, contact, schedule.
  • Custom ad design (including interactive and advanced multimedia features — sound, animation, video).
  • A CPA (cost-per-acquisition) model, while tracking interaction via an app — per saved contact, per scheduled appointment, per phone call.
  • Premium section.

But… alas… This is as far as we got.

P.S. One notable exception is a touch-screen showing suggestions in Whole Foods in Redwood City.