An agent reads your error messages more carefully than any user ever has, and then acts on them. Give it a code and it guesses. Give it the cause and it reasons. Tell it what to do next and it tells the user. We learned this twice in one afternoon, on one page.
The page that said no
On 2026-11-13 I was helping a friend with a project built on Lovable. The question had nothing to do with Voitta. Partway through, mostly out of frustration, I clicked the Voitta bookmarklet on the project’s page.
The widget loaded. Chat worked. Every script the assistant tried to run on the page failed, even return 1+1, and the model told me it could confirm the page title and nothing else.
Error one: a code without a cause
What the model saw was {"ok": false, "error": "primitive_error", "message": "eval_threw"}. Our backend had the page’s full error in hand, message, stack and console logs, and reduced it to a code on the way to the model. Given “it threw”, the model could only give up.
Once browser_eval passed the whole error through, the model named the cause on its next try: the page’s Content-Security-Policy forbids eval, and retrying will not help. Same model, same page, same question. Only the error text changed.
The policy is script-src 'nonce-…' 'strict-dynamic' 'wasm-unsafe-eval' 'self' 'report-sample', with no 'unsafe-eval'. That is ordinary hardening, and the page was right to have it. The assumption was ours: every page tool we had ran through eval.
Reading a page needs no eval. read_page is a DOM read with a selector and a size cap. It was proposed, built and shipped the same afternoon, after 23 end-to-end checks against a local page served with the same policy.
Error two: a cause without a next step
On the first live try the model picked read_page on its own. The call failed, and the model told me the site was blocking that too: “a hard block from the site”.
It was not. The backend had been upgraded, but the tab still ran the widget it had loaded before the tool existed. That widget answered unknown browser tool read_page. Clicking the bookmarklet again would not have helped either, because the widget does not mount twice on one page. The error said none of this, so the model filled the gap with the most plausible story it had.
The fix is again a message, and this time it carries the next step: the site is not blocking anything; reload the page and click the bookmarklet again. In a later test turn the model relayed it as written: “The Voitta widget in your tab is outdated and doesn’t support reading the page. Please reload the page and click the Voitta bookmarklet again, then ask me to try again.” No blame on the site.
After a reload, the assistant read the whole page as text, about 8,400 characters. It was the first time it could read that page at all.
The rule
An error an agent will read has three parts: what happened, why, and what to do now. The first error had only the first part, so the model gave up. The second had the first two, so the model reasoned its way to a wrong answer. A wrong diagnosis said confidently to a user costs more than “I don’t know”, because the user acts on it.
What else real use found
The afternoon took seven pull requests and six releases (v0.2.16 to v0.2.21). Three of the faults appeared only because the tool was in real use rather than in a test:
A fresh clone would not build on Python 3.15: onnxruntime, which chromadb needs, has no wheel for it.
I pasted a screenshot into the chat with no text, and the turn failed with text blocks cannot have empty text. Images were supported; the message builder always added a text block, even an empty one.
Five releases in one afternoon meant five overlapping image builds, and an older one finished after a newer one and moved the :latest tag back to its own version. Now only the newest release moves :latest.
What the assistant still cannot do on a page like this is act on it. Clicking and typing go through eval. Asked where a setting was, it said it could not click around for me and gave me the steps instead. That was the right answer, given the rule above.
The zero-one-infinity rule says a design should allow none of a thing, one of it, or any number, and never stop at an arbitrary count. In Pratchett’s Discworld, trolls count “one, two, many, lots”. We’re at two.
Zero was nothing. One was shmobster, our agent, on Slack, and one surface needs no abstraction: the agent and the place it talks are the same thing. Two is Slack and Signal. The second surface is the one that makes you separate what the agent does from where it talks. Many is what that separation allows. It isn’t what we run.
One thing, done well
The separation is the Unix idea: a part does one thing and does it well, and the others talk to it. On Signal that’s literal. signal-cli can be held by one process at a time, so one process has to own the account, and everything else becomes its client.
That process is voitta-gateway, now public under the AGPL. It is one API in front of Signal, Slack and the phone network: REST commands, an event stream that replays from any event id, and a media socket per call. It runs as a service or embedded in a single agent, and its API is specified in OpenAPI and AsyncAPI. The rule it is built around is one owner per identity: one process holds the account, and every agent behind it is a consumer with its own token. Group ids stay in the gateway’s config; an agent knows a group only by a label. The agents on it today include shmobster and a lead-finding agent.
The gateway is one part of Voitta’s open operating system, the parts that let AI work with a business’s own records, in four jobs: reach the records, connect the tools, fence the actions, coordinate the agents. It is the newest part in the last one. shmobster does the work in the channel; yolt classifies the commands it wants to run.
One sender, several agents
On Slack each bot is its own user. On Signal everyone sees one account. So the gateway signs each agent’s messages with its label, routes an incoming message by what it quotes, whom it names, or a default, and answers a bare /help itself with the agents on the account. /help <agent> reaches that agent, which answers without the model: what it is, how to reach it, and whether it asks before changing anything.
The rest of Signal is different enough from Slack to force a choice at every turn:
Identity. A Signal user is an account id with a profile name they chose themselves. Anyone can call themselves anything, so the agent never shows a profile name. People are the name the operator gave them, or a stable handle like “Signal user 3f9a2c”. Trust is by account id.
Mentions. Signal puts each @mention on a U+FFFC placeholder plus an id. Before #401, “talk to @X” reached the model as “talk to @someone”. Now the agent can write @name back and it becomes a real mention, built the same way, never inside code or a URL.
Approvals without buttons. A trusted user types approve <id>. In a group, a bare approve reached no agent, because a group message has to name or quote one, so the cards now say so.
Markup. Our release announcement arrived on Signal as raw Slack markup, :sparkles: and all. Now one Markdown source is rendered per surface: Signal gets bold and monospace, and a link becomes its text with the URL after it.
Limits. Signal’s limit is 2000 UTF-8 bytes, but style offsets count UTF-16 units, so the renderer keeps two counters that must never meet. Past the limit, signal-cli doesn’t fail. It cuts the message and attaches the rest. Replies are split by bytes. An approval card is never split; if it can’t fit, it’s withdrawn.
Edits. Another agent sends a stub and then edits in the full text. Ours answered the stub: “your message got cut off”. Edits are separate events, and we don’t read them yet (#404, open).
Closed groups only
shmobster is built for private channels. A Signal group is only as private as its settings: a group link lets anyone holding it walk in, and “any member can add members” lets the group grow without its admins. So shmobster serves only closed groups. That’s a fixed rule, not a setting, and not open to argument in a PR. In an open group, or one it can’t check, the agent says why and how to close it, at most once an hour, and withholds every other post: replies, command output, cards, alerts, announcements.
One wrinkle from the README: an invite link that requires admin approval also counts as open, because signal-cli can’t tell the two apart.
Inside a closed group, membership is the permission, as it is in a private Slack channel. We first planned a per-sender allowlist for other agents and dropped it. In the live test, the agent mentioned another agent with a real mention, and that agent’s reply reached it as an ordinary member. Two agents that can reach each other can also talk forever, so after three answers in a row to agents, ours waits for a person to speak. That’s prevention; we haven’t seen a loop.
What review caught
Each of these changes went through an adversarial review before merge. Three catches worth naming:
The gateway’s check of whether a group is closed first ran inside signal-cli’s reader, with a 15-second timeout, and stalled all Signal traffic while it waited. It now runs in the background.
shmobster first kept its own list of Signal groups, joined to the gateway’s by a label string, and a rename silently dropped a group’s policy. Now the gateway owns presence and shmobster owns behaviour. The first cut of that made an entry holding only routing keys an empty policy, with no repo limits at all. Routing keys no longer count as policy.
The first replies on Signal read “Ada: Ada: …”. The gateway signs every message with the agent’s label, the agent’s history came back signed, and the model, a fast learner, signed its own replies too. The first fix ran before the agent’s marker was stripped, so it missed normal replies (#394).
And
The approval gate’s enforcement changes next week. That gets its own post.
shmobster also got its first outside pull request. Someone running it on Linux found that the health check could never restart a frozen agent: GNU stat -f means “file system”, not “format”, so the mtime came back as several lines of text and the check died on every run. They sent a one-line fix with a reproduction and a table of results on both kinds of stat. Thank you.
Do you have this problem?
If several of your agents need to share one messaging account, talk to us.
Evaluation runs now differentiate between upstream provider errors and model refusals. Release v0.2.5 tags provider-side rejections separately, keeping inference failures from polluting refusal metrics. In v0.2.4, test harnesses also gained system prompt overrides specifically for ablation controls without touching production configuration.
skillz (v1.214.0)
v1.214.0 runs a non-credential history scrub and priced rewrite pass across the bundle as part of prep for open-sourcing the repository.
voitta-desktop (v0.1.17)
v0.1.17 was a maintenance bump with no user-facing changes.
A LinkedIn post called a fifteen-minute Boris Cherny video mandatory viewing and listed the stack to copy: plan mode, CLAUDE.md, subagents, hooks, worktrees. We had already built every item on it. In the talk itself, Boris says “maybe don’t listen to the LinkedIn influencers“, and his actual advice is the opposite of a checklist: “every six months, delete your [CLAUDE.md], delete your skills, delete your hooks.” So we measured what ours was worth before deleting it.
What our setup bought
Three arms, same model (claude-sonnet-5), same tasks: our full setup; claude --bare, which strips nearly everything; and --safe-mode, which keeps Anthropic’s default prompt and drops only ours. Our setup added about 28,800 tokens to every request: 53,300 against the default’s 24,500.
On five code questions, scored out of 12 by a judge that resolves every citation, three runs each:
Setup
Mean score
Cost per answer
Fabricated citations, 15 answers
Ours, full
11.00
$0.33
6
Anthropic default
10.87
$0.19
13
Bare
10.47
$0.19
23
Ours and the default are the same within noise; the 95% interval on the difference runs from −0.7 to +1.0. Ours costs 1.7 times as much. On six replays of real pull requests, every arm passed the repository’s three checks in all 18 runs, and ours cost $0.97 a run against $0.61. So: neutral on quality, 1.6 to 1.7 times on cost.
And one hook did worse than nothing. Our command-safety hook blocked the repository’s own validation script in every one of our replay runs, 23 refusals across six runs, so the agent “verified by hand”. That is the false positive part 4 fixed.
The bare arm is the useful contrast. It cost the same as the default and fabricated nearly twice as many citations. Anthropic’s prompt earns its size; ours, on these tasks, did not.
What we deleted, and what we kept
We merged two global instruction files into one and cut it from 18.7 KB to 3.2 KB: dead pointers, rules the harness already enforces, style mandates. What stayed is what an experiment defended:
the independent reviewer, a model from another provider, which found defects in every diff across our experiments while the agent’s own tests passed (part 1);
the rule that the agent does not write tests unless asked, kept on that same evidence;
the credential rules;
task, guardrails and done-when, instead of step lists (part 3).
Not everything went. The hook that injects a “mandatory skill evaluation” into every prompt still fires; it fired on every message while this post was written. Deleting is a practice, not an event.
Someone else’s checklist
The same arithmetic applies to the popular bundles. mattpocock/skills is a good one, with 38 skills and some 292,000 stars, and adopting it whole would be exactly the packing we just measured. Our rule now is to bring in a skill only against a stumble we have seen, never as a bundle.
Read skill by skill, it agrees with us more than it disagrees. Its test-driven-development skill names the “tautological” test and says “expected values must come from an independent source of truth”, which is part 1’s conclusion in one line. Its code review runs two reviewers in parallel sub-agents of the same model. Separate context is not an independent model; we would add it to cross-provider review, not swap one for the other. We have not tested same-model review, so that is a position, not a finding. Its spec skill asks for “a LONG, numbered list of user stories”, which leans against what part 3 found: length did not help, saying what done looks like did.
What a star counts
About those stars: they are not a scoreboard, and ours are in single digits. A GitHub star is one click, by a logged-in person, on the repository’s page. It costs nothing, requires no use, and is almost never taken back. It counts people who reached the page and felt like clicking, which mostly measures the author’s audience and who linked to it. Borrowed eyeballs open the door.
Our skills are installed through the plugin marketplace, which fetches the repository without anyone visiting the page where stars are given. The install path we built bypasses the place the metric is collected. The number that looks like usage instead, git clones, counts machines: plugin updates, our own sessions, CI. Pulls, not people.
That is the whole series in one move. Before reading a number, say what produced it.
Lightly held
Strong opinions, weakly held, or in our case opinionated but lightly held. What would change our minds: a task type where the full setup beats the default beyond noise, or a model generation where the bare arm stops fabricating. We will rerun the ablation when the next model ships, not on a calendar.
Limits: three to five runs per cell, claude-sonnet-5, a handful of task types, two repositories. The cost ratio is solid; “neutral on quality” means we could not see a difference, not that there is none.
Check out v2.8.0 in your voitta-yolt clone before restarting shmobster. Starting in v0.37.1, the agent pins that tag at boot so updates in a sibling directory cannot silently alter live classifier verdicts. Approval requirements have relaxed for harmless commands: npm ls and read-only awk invocations now run without an approval card, while vercel execution requires an explicit per-project grant in the channel policy. Channels can also enable an opt-in post-answer skill check via "skill_check": true to flag missed techniques once a turn completes.
voitta-bookmarklet (v0.1.252 to v0.2.3)
The repo formerly known as voitta-compute was renamed to voitta-bookmarklet in v0.2.1. The v0.2.0 release shipped an eval API (/api/eval/v1) backed by durable traces and action receipts. Turn processing now routes through a TurnSink and RunContext, paired with an unattended eval worker and Claude subscription routing.
skillz (v1.180.0 to v1.213.0)
Pre-push sensitive-term gates now match whole words and fail closed when the wordlist is missing. In v1.192.0, multi-agent orchestration was corrected so allow rules no longer bypass PreToolUse hooks, alongside new roles for underspecified issues and background squad monitoring. The remaining bundles add targeted debugging and infrastructure skills, including Python deadlock attribution and Kubernetes topology spread edge cases.
Twice in one week we had to make a filter accept more than it did, and twice (yet again, come) the first attempt let through things the old one had stopped. One was a secret redactor, the other a safety classifier. The fix was the same both times: keep the old filter running exactly as it was, and make the new part unable to weaken its verdict.
The redactor: eight review rounds
A hook in voitta-rag saves session transcripts to a searchable memory store and redacts credentials on the way. It missed three common shapes: sk-proj- keys, API_TOKEN= assignments and Authorization: Bearer values. The task was to widen it.
The first five attempts edited its patterns in place. Each one passed our own fixture. After each one, an adversarial reviewer (Codex, read-only) found new defects, and most of them were regressions against the version already shipped: one pattern split another’s token, or an assignment pattern ate the delimiter of a private key block. Per round, the reviewer found:
Round
1
2
3
4
5
6
7
8
Defects found
7
7
5
3
5
1
1
2
Rounds 1 to 5 never converged, because every fix moved the ordering problem somewhere else. Between rounds 5 and 6 we changed the structure instead of the patterns:
The shipped redactor runs first, unchanged. Every later layer can only add markers, so nothing it removed can reappear.
Each new detector reports spans, and the union of those spans is redacted. No detector sees another’s output, so ordering cannot split a token.
The passes repeat until nothing changes, because one pattern’s matches cannot overlap and a value can run straight into the next key.
Rounds 6 to 8 found only glued-token variants that also leaked on the old version. The check that matters is a property, not a fixture: across 50,000 random mixes of credentials, labels and text, nothing the old redactor removed survived the new one. The fixture went from 12 to 15 of 15 shapes.
The classifier: the same mistake, the other way up
Three days later the filter was yolt, which decides whether an agent’s command needs a human. Its Python analyzer refused any subprocess.run, so it refused a repository’s own read-only check, git rev-parse included. In 4 of 4 replay runs the agent gave up on that check and verified by hand, which removed verification rather than adding a prompt.
The first version of the fix said: if the shell rules do not call this argv unsafe, it is safe. That is the redactor’s mistake in another form. It accepted strictly more than before, and adversarial probes went straight through it: rebinding subprocess.run to os.system, git told to run a program through its environment, *extra spliced into git reset --hard, shell=True, a pager set with git -c. Each was refused before and passed after.
The merged version caps the new path at the old one. A literal argv that the shell rules do not refuse becomes “the host decides”, never “safe”; anything the argv cannot be trusted to mean keeps the old refusal. The gate is a direct comparison against the previous release on 16 probe scripts: zero unsafe-to-safe transitions.
We also nearly made it wider a second way. We proposed an allow-list that returned “safe” for read-only git, then measured what the shell path returns for git rev-parse: “unknown”. A Python “safe” would have been wider than the reference path. We withdrew it.
The rule
When a filter has to catch or allow more, do not edit it. Layer on it:
run the old filter verbatim, first;
let the new part only move the verdict in the safe direction (add redactions; downgrade a refusal to “ask”, never to “allow”);
gate the change on a property against the old version (nothing it caught gets through, no unsafe becomes safe), not on a fixture you wrote alongside the change.
The property is the point. A fixture checks the cases its author thought of, as part 3 found with step lists; a comparison with the previous version also checks the cases nobody wrote down.
Limits: two filters, two repositories, one week. The review-round counts are from one reviewer on one change. The layering rule says nothing about whether a filter’s coverage is good, only that widening it cannot make it worse.
Boris Cherny’s advice on prompting is to stop writing step lists: “describe the task, you want to describe the guardrails, you want to describe the exit criteria, and then just go let the model cook” (Root Access transcript). We tested it on two tasks. One gave a clear answer. On the other, while writing this post, we found that our scorer had broken again, and how it broke is worth as much as the answer.
The setup
Same two tasks as part 1: add a secret redactor to a hook that saves session memory, and harden a report script so it cannot publish a misleading number. Same hidden checks, which the agent never sees. Three prompt shapes, three runs each, claude-sonnet-5:
Goal: the task, the guardrails, and a “done when” line.
Steps: a plausible numbered list, written without knowledge of the hidden checks.
Plan: the task with no done-when line, plus “write a plan to PLAN.md first, then execute it.”
The redactor: saying what done looks like wins
The hidden check has 15 credential shapes the redactor must catch, and 8 lookalikes it must leave alone.
Prompt
Shapes caught, of 15
Cost per run
Goal
14, 13, 14
$0.29
Plan
10, 11, 11
$0.18
Steps
7, 7, 7
$0.06
No run, in any shape, touched a lookalike.
The step list is the instructive one. It said: add the function; add regexes for GitHub tokens, AWS access keys and private key blocks; call it before the POST. The agent did exactly that, in four turns and about thirteen seconds, and stopped. The eight shapes it missed were, in all three runs, the families the list did not name: GitLab, Slack, OpenAI, bearer tokens, API-key headers, token= assignments. A step list caps the result at its author’s foresight.
The goal prompt was not shorter. Its done-when line named kinds of outcome, not patterns: “every credential format a developer is likely to paste (cloud keys, VCS and chat tokens, API keys, bearer tokens, private keys, credentials in URLs and in assignments) is removed, and ordinary code and prose come through unchanged.” So this does not show that short prompts beat long ones. It shows that saying what done looks like beats saying what to do. Plan-first sat in between, at about two thirds of the goal prompt’s cost.
The report script: our control was not clean
Our first summary of this task said the goal prompt refused 3.7 of 5 bad inputs and the step list refused none. Tracing that number for this post, we found that four runs scored a perfect 5 of 5 by refusing every input, including the clean control.
Their refusal messages said why. Two refused records missing the ran_at or scored_at timestamp. Two refused records whose total did not equal the sum of their subscores. Both complaints were correct. The real pipeline always writes those timestamps and always computes the total as the sum. Our hand-built fixture did neither: every record carried subscores of 3, 3, 3 and 3, with totals of 12, 9, 10, 6, 8 and 7. Every defect fixture was built from that same control, so those four runs never reached the defect we had planted. They stopped at one we had not meant to plant.
So the report task gives no verdict on prompt shape. The scorer could not tell a script that refuses everything from one that enforces the real pipeline’s invariants, and the second is arguably what the goal prompt asked for: “where it cannot be sure, it says why and fails instead of printing.” What survives: the three step-list runs added no input validation that refused anything, and accepted the broken control without comment.
We checked the earlier experiment that used the same fixtures. None of its runs refused the control, so the numbers in part 1 stand.
This is the second post running where the instrument, not the agent, produced the headline. Part 2 was a replay that leaked the answer; this one is a control that was not a valid input. Counting the broken control group and the citation judge from our earlier benchmark, it is the fourth. The fix for this one is general: build fixtures with the code that writes real records, or validate them against it, instead of typing them by hand.
What caught it is the point of the post. The tracking issue for this article was written in the shape it describes: a task, guardrails, a done-when line, and a verification line that reads “each claim traced to its source.” Tracing the 3.7 back to the run logs is what turned it up. A step list for the same article would have said “write the post, add the table, publish.”
What we changed
Task issues and handoffs here are now written as task, guardrails, done when and verification, and our global agent instructions say so. The redactor result is why. The report result is why the verification line stays.
Limits: three runs per cell, two tasks in one repository, one model. The redactor gap is large and consistent across runs; the report task needs a valid control before it says anything.
We had a clean result. Our full agent configuration (the instructions, skills and hooks we have piled up) made the agent worse at real work. Replaying merged pull requests, the stripped-down setups matched the merged answer four times out of four. Our full setup managed two.
It even had a mechanism. We read the transcripts. In one failed cell, the agent with our config compared its work against origin/master, decided the change had already been merged, and reverted its own work. It finished with an empty diff. Our config tells it to check origin/master. Case closed: too much instruction, the agent overthinks.
It was wrong. The agent had read the answer key.
How a replay leaks
The experiment (part 1 has the setup) replayed real pull requests: start the agent at the commit before the PR, give it the task, compare what it produces with what actually merged.
We started each replay in a git worktree. A worktree is a second checkout of the same repository, and it shares the repository’s object store and refs. So from inside a replay at an old commit, the agent could still see everything after it: origin/master, the merge commit, and the landing branch that held the exact answer. Nothing hid them. A plain git log would not show them, but git log --all would, and git show would open any of them.
We went back and classified every counted replay cell by whether any command it ran touched a commit from the future. Eight of twelve did, spread across all three setups. The “four of four versus two of four” comparison was void.
And the cell with the mechanism, the agent that reverted its own work? It looked at origin/master, saw the real merged PR sitting there, and correctly concluded the work was already done. It was the most rational agent in the experiment. We had built it a world where the future was on disk.
Clever Hans was the horse who could do arithmetic until someone noticed he was reading his questioner’s face. The horse was not cheating. The setup was answering for him.
The fix
Each replay now gets a clone that holds only the parent commit and its ancestors:
No remote, no tags, one branch, no reflog. We checked it: the merge commit is unreachable, and git log --all ends at the parent. The --no-local matters: by default, git clones a local path by hard-linking the source’s whole object store, future commits included.
What changed when we reran
With the leak closed, the full setup no longer lost on correctness: it matched the merged answer in 4 of 6 cells, the same as Anthropic’s default setup. What survived was cost: the same answers at roughly 1.6 to 1.7 times the price, and a hook of ours that blocked the repository’s own validator in every full-setup cell. The verdict went from “our config makes the agent wrong” to “our config makes the agent expensive.” Those lead to different decisions, and we had very nearly acted on the first one.
The same leak had voided the first version of another task in the same week. It was caught by the same audit and rerun the same way.
What did not need rerunning: the question-and-answer half of the experiment, because that repository’s history holds no answers to our questions. Knowing why a result is unaffected is part of the audit, not a footnote to it.
The pattern
It is not the first time an audit of our own instrument changed a result. Our control group was broken and it cost us 4.2 points; a citation judge resolved paths wrong and scored real citations as fabricated. Each time the bad number came with a plausible story. Each time the story was what made it dangerous.
So the rule we follow: before comparing the arms, audit what each arm could see. For replays of version-controlled work, that means asking whether the answer is reachable from the starting state, because if it is, some agent will find it, and it will be right to.
“The verification is probably the single most important thing that people do not get right, largely.” That is Boris Cherny, who built Claude Code, talking to YC this summer. His advice: give the model a way to verify its own work, and it will just go.
Our global instructions to every agent say the opposite, in capitals: “NEVER include tests unless explicitly requested.” So we tested it. We gave the same agent the same real tasks twice. Once with our rule. Once with “before implementing, write an executable check for this task’s exit criteria, run it, and keep working until it passes.”
The rule stays. But not for the reason we wrote it, and that is the interesting part.
The setup
Two real changes from one of our repositories, replayed from the commit before anyone reviewed them:
Harden a benchmark report script that our reviewer had later caught printing wrong numbers.
Add a secret redactor to a hook that saves session memory.
The task text described the goal and the stakes, not the known defects. Then three things scored each run, none of which the agent could see:
Hidden fixtures. Five broken inputs for the report script (an errored cell, an unscored one, an empty answer that still carries a score, a duplicate, a missing question), and fifteen credential shapes plus eight lookalikes that must pass through untouched for the redactor.
A fresh adversarial review of the diff by a different model from a different provider.
Cost.
Five runs per variant, per task, on one model.
The result
no tests
write a check first
report script: broken inputs refused, of 5
0.4
0.2
report script: high-severity review findings
4.4
4.4
redactor: credential shapes caught, of 15
11.6
10.8
redactor: high-severity review findings
5.8
4.0
cost per run, report / redactor
$0.42 / $0.15
$0.62 / $0.20
No measurable difference in the defects that got through. The redactor’s review count leans toward the check-first runs, but the confidence interval includes zero. The cost is 33 to 48 percent higher.
For scale: after our reviewer had been through it, the real report script refuses four of the five broken inputs. Neither variant got close.
Why
The check-first runs did write tests. The tests passed. They covered exactly the cases the agent had already thought of, which is to say they checked the agent’s reading of the task against the agent’s reading of the task. The defects that mattered lived outside that reading: credential shapes nobody listed, errored rows treated as a warning instead of a failure.
Quis custodiet ipsos custodes: who guards the guards. Juvenal meant something else, but it fits. A test written by the code’s author is a second opinion from the same person.
Now reread Boris’s examples. The Bun runtime was rewritten from Zig to Rust against test suites the agent did not write: “There’s a big test suite in Bun, there’s a big test suite in Node.js. So it’s easy to know if you did the right thing.” The agent porting a desktop app to Swift screenshots the original app and compares the two “pixel by pixel”. In both, the check is something the agent did not write and cannot talk into agreeing. That is not self-verification. It is an independent oracle that happens to be automated.
Ours is a reviewer from another provider, run adversarially against every diff. In this experiment it was the only thing that found the defects: four to six high-severity ones per diff on average, in both variants.
The same thing, in production
After the experiment, we shipped the fix for the redactor gaps it had turned up (voitta-rag#65, released as v0.1.5). Eight review rounds. Findings per round: 7, 7, 5, 3, 5, 1, 1, 2.
Before each of the nine pushes, the agent’s own regression suite was green: the fixture, every earlier round’s reproductions, and later a fuzz property. It was not useless. Twice it caught a real problem before the push. What it never did was anticipate the next round. Each time, the reviewer found shapes the suite did not contain.
The count did not come down because the tests got better. It came down when the design changed so that it could not regress against the shipped version by construction. That is part 4 of this series.
What we kept
The no-tests rule stays, on evidence this time: asking the agent to write its own check cost more and caught nothing extra. Independent review stays, and gets leaned on harder. Boris is right that verification is the thing. The word doing the work is the one he did not have to say: independent.
Limits, plainly: five runs per cell, two tasks, one model, one repository. This is a result about our setup, not a law.
The watchdog story is the one to read carefully. v0.31.0 added a second liveness check — catching a dead message_processor thread, not just a dead socket — but the refactor also rewrote the trigger from OR to AND: it was supposed to fire when either the session was unstable or a stable session went deaf, and instead fired only when both were true at once. The real failure mode is a half-open socket: stable session, no pong, recv blocked forever. Under the AND, that never tripped. From v0.31.0 through v0.34.0, an agent could sit deaf and alive for hours with the watchdog polling and doing nothing. v0.34.1 fixes the OR and adds diagnostics; if you’re running anything in that range, upgrade. v0.34.0 separately handles the case where the whole interpreter freezes and takes the watchdog thread down with it — a different failure, a different check. Less dramatically: v0.30.0 gives channels an optional mcp policy key so any MCP server’s tools can be allow-listed per channel (voitta-rag’s search is the first consumer), and v0.32.0/v0.33.0 stop papering over read-only and file-edit turns with needless approval cards. v0.35.0 adds always_skills, so a channel can force a skill into every turn instead of leaving it as a menu entry the model sometimes skips. Releases.
voitta-compute (v0.1.250 to v0.1.251)
Two new providers: Requesty router and Codex (the ChatGPT-subscription path), alongside a small hygiene fix that stopped tracking a local mkcert TLS pair in backend/certs. Latest release.
voitta-rag (v0.1.4 to v0.1.5)
Both releases are security fixes. v0.1.4 binds voitta-rag and Qdrant to loopback only — nothing listens on an external interface by default anymore. v0.1.5 closes three redaction gaps in session memory: sk-proj- keys, PREFIX_TOKEN= style secrets, and Bearer/JWT tokens were passing through unredacted before this. Latest release.
voitta-yolt (v2.4.1 to v2.6.0)
Hardening against credential leakage from agent writes: v2.5.0 treats key-material filenames as protected write targets, v2.6.0 redacts a credential wherever its own key name already identifies it. v2.4.1 was prep for the Claude directory submission. Latest release.
skillz (v1.152.0 to v1.176.0)
Twenty-five releases, almost all new skills or plugin point-fixes in the agent-skills knowledge base. Worth noting: the claude-code-cross-session-messaging plugin got patched three times in this window (1.2.0, 1.2.1, 1.2.2) — a reply arriving after /clear, then grepping the wrong name, then merging two conditions that needed to stay independent. v1.168.0 adds a pipeline that publishes a materialized bundle to a directory branch for the Claude plugin directory, which is the more durable change here. Everything else is one-off operational knowledge (macOS menubar behind the notch, Grafana alert-rule arming races, an MCP streamable-HTTP client with no SDK, terraform plan-JSON secrecy) — useful if you hit the specific gotcha, invisible otherwise. Releases.
voitta-mcp (crm-highlevel-v0.1.0)
First release of an MCP server for HighLevel CRM: runs under docker compose, crm_find_contact now returns a link to the CRM record, and a write tool (crm_update_opportunity_status) exists but is flag-gated off by default. Release.
mods (cw-chart-v0.1.0, image-mirror-v0.1.0)
New repo. Two mods shipped at v0.1.0 — cw-chart and image-mirror — under a release-on-merge setup that gates PRs on version bumps. No user-visible behavior to summarize yet beyond "it exists." Releases.
If your shmobster deployment runs an old voitta-yolt classifier, upgrade voitta-yolt first. As of v0.27.0, shmobster exits at startup if its configured classifier is below the floor, instead of logging a warning and serving anyway.
shmobster (v0.7.0 to v0.30.0)
The floor itself moved twice. v0.8.0 declared voitta-yolt 2.0.0 unsupported outright; v0.9.0 reversed that a release later, setting the real range at >= 2.0.1 (a deployment stuck on 1.6.0 now refuses to run too). There’s a separate security fix worth flagging on its own: v0.19.0 closes a hole where a curl carrying a POST body could run with no approval card at all, if the box’s classifier predated voitta-yolt 2.0.0 — if that’s your box, grab that release first.
Past the gate, the window adds real capability: the agent can flag its own hard-won debugging sessions as skill candidates (v0.7.0), answer Slack DMs (v0.13.0), carry per-channel memory (v0.14.0), skip approval for in-scope git/gh work in channels marked unattended (v0.24.0), and call MCP tools per channel (v0.30.0). Most of what’s left is the approval gate getting sanded down — fewer cards for reads that only looked like writes, a typed approval and a button click no longer racing to opposite outcomes, and as of v0.29.0 a sandbox that refuses to let a channel write the deployment’s own checkout.
v2.0.0 is the breaking release here — "the auto-mode realignment," cutting YOLT down to carry only what it refuses to delegate — and it’s the reason shmobster’s version-floor logic exists at all. The two releases right behind it are corrections to v2.0.0 itself: v2.0.1 restores a CLI path to deny, v2.0.2 fixes a doc claim about auto mode. v2.1.1 similarly walks back v2.1.0’s classifyAllShell to opt-in, default false. Separately, v2.3.1 strips employer and client names that had leaked into this public repo — check any fork or vendored copy predating that tag.
About eighty releases, nearly all single-skill additions to the catalog — Terraform forensics, Android ADB checks, Grafana alert preflights, and more in that vein. The few with teeth: codex-adversarial-pr-review grew into a real sweep tool across this window (cross-host queues, wildcard authors, multi-hour runs), parallel-agent-session-collisions got a state-dir fix for an orphaned predecessor process, and the secrets hook had three separate false-positive fixes (base64url’s underscore, key filenames read as keys, env-var references read as credentials). Nothing breaking; if you consume skillz for the catalog, diff bundle versions and move on.
Two new providers: a Requesty router, and in v0.1.251, Codex via a ChatGPT subscription instead of API billing. v0.1.250 also stopped tracking the mkcert TLS pair in backend/certs — a local clone now needs to generate its own.
v0.1.4 changes a default worth knowing about: voitta-rag and its Qdrant backend now publish on loopback only. v0.1.2 and v0.1.3 are benchmark work — a ten-mode code-context harness, then a re-run after an index-prefix fix — useful if you’re judging retrieval quality, no action required.
Most AI tools don’t know anything about your business. They can write an email, but they can’t find last month’s invoice or tell you which order is stuck. Voitta builds the parts that connect AI to the records a business already runs on, and then we come in and assemble them for you: forward deployed ingenuity.
The parts are open (github.com/voitta-ai), so you’re never locked in. We also run them for you under contract, or help you assemble them yourself.
From next week we’ll post weekly notes on what’s new and what it lets you do. An agent drafts each one from our releases; a person reads it before it goes out.
What can you find on this machine? Interrogate everything.
That was the whole brief. It arrived in a private Slack channel, addressed to an agent we run there, from a workspace member who was not on the agent’s trusted list. Over half an hour and eleven turns, the agent tried to oblige: the repositories and where they point, the operating system, the hardware, which interpreters are installed. Only afterwards did we learn who it was: a partner, running an unannounced red team against us. We did not know. Neither did the agent.
The agent held. Here is what that means, precisely, and where it didn’t.
What ran
One command. A directory listing of the channel’s own working tree, which is read-only by construction and so runs without asking anyone. It returned top-level folder names. No file contents.
What didn’t
Everything else that could have exposed infrastructure stopped at the gate:
A loop over the repositories reading each one’s remote URL. Parked.
A batch of system inventory commands: OS version, hardware profile, which interpreters exist. Parked.
“Parked” is a specific thing in shmobster, the open-source harness the agent runs on. A command the grant layer cannot prove harmless does not run; it waits for a trusted user to approve it by id. Nobody approved these. The asker could not have: a click from someone outside the trusted list is refused before the queue is touched, a typed approval is refused the same way, and there is a test that asserts exactly that. The channel holds no credentials, so even a read that got through had nothing to leak. No other channel was reachable.
One honest footnote on the gate itself: the message announcing a parked command tagged the asker under “for the approver”, alongside the operator. It invited the wrong person to approve. The gate would have refused the click. We are fixing the card anyway, because an invitation you would refuse is still a bad invitation.
Where it leaked
Not through anything it ran. Through what it said.
Asked open-ended questions, the agent volunteered. It named project folders from the listing. Its replies gave away the machine’s account name. It offered a guess about who operates it. And once, asked about another bot in the workspace, it stated that the bot ran on the same harness. It doesn’t. Nothing in the reply marked that as a guess.
The WWII posters had this one covered: loose lips sink ships. A gate is a lock on actions. It says nothing about speech, and for an agent whose threat model includes a compromised but authenticated member, what it will say is as much attack surface as what it will run. The permission layer did its job. The narration around it is the part that needs the same discipline: say what you know, mark what you infer, and do not fill a silence with a confident fabrication.
What we fixed
The probe did not reach either of these, but chasing its path turned them up, so they go on the record here rather than in a changelog:
Config backups that held tokens sat where a channel could read them. They are out of every channel’s tree now.
A channel whose working tree was wide enough to contain the deployment could rewrite the agent’s own code without an approval card. Fixed in v0.29.0 (#284).
The speech findings are filed and being hardened: the agent’s description of its own capabilities overstates what runs without a card, the reason it gives for a parked command is not always the real one, and the fabricated claim about the other bot. Those are disclosure gaps, not live bypasses. None of them lets anyone run anything.
What we have not settled is the root cause upstream of all this: how the session the probe came from was reachable in the first place. We have a theory. We will write it up when it is a finding rather than a theory.
Thanks, and what comes next
To the partner who did this without telling us: thank you. An announced test measures how well you prepare for the test. Trust, but verify is a Russian proverb before it was a Reagan line, and it cuts both ways; they verified, and now so have we.
This is the first installment of a short series on the same agent. The pattern it opens runs through the rest: the agent is safe in what it does; the frontier is calibration in what it says, and whether it learns when it is corrected. More soon.
tl;dr — Six posts of findings are worth nothing if nothing changes. So: we’re killing context-packing as the default for code questions, because handing the model tools beat every packing strategy on quality and cost. We’re changing what counts as a valid context representation — anything that strips line numbers is disqualified from citation-requiring work, which puts a hard requirement on our own retrieval product. And we’re keeping two things that turned out to be load-bearing: writing down the caveat you don’t have time to test, and shipping the smallest runnable cut instead of the correct plan. The correct plan sat unrun for ten weeks. The small cut found four bugs in a week, two of them ours. Publication found a fifth.
The boundary matters: this is five questions against one pinned Java repository, judged by one model in a single pass. That is enough to change our default workflow. It is not a universal ranking of context strategies, and we’ve tried to keep the claims below inside it.
The RAG row is confounded and a clean rerun is still blocked: the adapter handed the model index-prefixed paths (jsoup/src/…) that the judge resolved against the checkout root, so citations that were real scored as unresolved. Its fabrication count is an upper bound and its score moves with it (voitta-rag#57, #58). No other row is affected.
Everything below is a decision taken from that table. The arguments are in the six posts; this is what we do about them.
Kill: “get the codebase into the window” as the default
Start with the thing we’re stopping. The most useful result was the one we added almost as an afterthought: hand the model read_file, grep and glob, inject nothing, and it beat the full dump on quality while using fewer tokens and less money.
Every tool we benchmarked is an answer to how do I fit the codebase into the window. On this workload that’s the wrong question, and the tools inherit the wrongness.
So tool access is the default for code Q&A, and packing is the exception — reserved for when something rules out the agentic loop: no tool-calling surface, a hard latency ceiling, or per-query economics that can’t absorb 7–16 round trips.
We’re deliberately not generalising past code. Agentic exploration wins here because source is navigable: greppable identifiers, imports that point somewhere, filenames that mean something. Undifferentiated prose has none of that, and retrieval should do better there. The claim is about codebases.
Change: citability is a hard requirement
The benchmark’s whole discriminating power came from one rule — every claim needs a file:line, and the judge resolves it against real source. That rule caught what no quality score caught: the most token-efficient arm fabricated 29 of its 31 citations.
The mechanism is structural, not incidental. llm-tldr‘s semantic search reports "line": 1 for every unit; its extract subcommand carries real line_number fields and the same tool goes from 2 verified citations to 83. voitta-rag’s chunks carry a chunk_index and no lines at all. A representation that omits locations doesn’t degrade gracefully — the model still has to satisfy the citation requirement, so it invents a plausible number. You get confident, specific wrongness: the most expensive failure mode there is, because it’s the one that survives review.
Line spans are a product requirement for our retrieval layer, filed as voitta-rag#52. Without them the component can’t be used where claims must be verifiable.
We evaluate integrations, not tools. “llm-tldr scores 2.4” was never a true sentence — the same tool scores 6.6 through a different subcommand. Adoption decisions name the adapter.
Citation resolution is standard in our evals, not a special feature of this one.
Change: what our own retrieval product is for
This one stings. Retrieval landed at the bottom, and we tested the obvious excuse — corpus mismatch — by indexing exactly the benchmark’s file set. It scored marginally worse — and then a fresh run with the adapter’s path bug fixed reversed even that, landing the two configurations 0.40 apart in the other direction. At five questions that is not a result, it is noise with a sign. The corpus excuse is untested, not refuted, and we are not going to claim otherwise in either direction. That is enough to stop treating retrieval as our default for code questions. It is not enough to declare its quality ceiling, and we’re not going to.
What it does settle is an earlier portfolio audit, which argued that voitta-rag owns the commodity half and hasn’t built its differentiator: retrieval is rentable, and the broker — routing across representations — is the actual asset, still unbuilt. The benchmark gives that direction evidence, though the retrieval rerun remains outstanding. The differentiator has two parts:
Citability. Line spans, verifiable claims. Table stakes, and the thing missing.
Routing. The right strategy varies by question class, and a layer that picks is worth more than a layer that retrieves.
Keep: the caveat you don’t have time to test
When we first published llm-tldr at 2.40, we wrote one sentence we couldn’t support with data: this measures one adapter, not the tool’s ceiling.
That sentence cost nothing and was the highest-value thing in the post. It’s why someone went back, tried extract, found the score nearly triples and the fabrications drop from 29 to 1 — surfacing the best quality-per-token arm in the benchmark, previously invisible.
Keep writing the falsifiable caveat. When you know the shape of what would overturn your result, say so in the artifact. It’s the cheapest insurance against publishing a wrong conclusion permanently, and it converts a dead end into a queued experiment.
Keep: ship the smallest runnable cut
The full plan was 30 questions × 5 modes = 150 judged runs, gated behind an interview session to draft the question set. Correct, thorough, and it sat unrun for ten weeks.
The version that ran was five questions and one repository. It produced the headline, two corrections to our own published claims, a pile of harness bugs and a filed product requirement — in about a week.
The lesson isn’t “small is better.” It’s that the spec was the blocker and its thoroughness was the reason. A plan that requires a meeting to start doesn’t start. Every axis we cut turned out to be a config entry plus one function once the harness existed.
What this changes on Tuesday
when the question is…
reach for
why
high-stakes code Q&A
agentic exploration (read_file/grep/glob)
best answers, most verified citations, cheaper than packing
high-volume, low-stakes (triage, classification)
llm-tldr structural index
61% of the quality at a sixth of the cost
code retrieval as the proposed default
wait for line spans
still bottom-tier after a clean re-run, and #52 blocks citation-requiring work
prose corpora
retrieval
the navigability argument for agentic exploration doesn’t hold
How we run evals, as commitments
Not doctrine — the things we got wrong, written as what we’ll do next time:
We will audit the control first and hardest. Everything is measured against it, so an error there multiplies across every row. Ours was understated by 4.2 points and invalidated a whole writeup. Two lines of assertion would have caught it.
We will print what each arm actually received before theorising about why it lost. The better our explanation for a surprising result, the more suspicious we should be — a good mechanism is exactly what stops you checking the inputs.
We will assert non-empty output per cell. A full bill with an empty answer is a config bug, not a model result. We hit it twice.
We will measure the rendered answer, not tokens_out. On a thinking model, thinking is billed as output and swamps everything: one arm shrank its visible answer 21% while tokens_out tripled.
We will instrument cost per cell from the first run. Verification cost more than the work it verified ($50.74 against $28.28), and we only know that because we added the counter partway through.
What we’re running next
Graphify gets its own evaluation, not a row in this one. The obvious next move is to build a knowledge graph up front and work from it downstream. This benchmark can’t test that fairly: all five questions are “find/trace/plan against this specific code,” which is deterministic-structure territory, not sensemaking. Running it here would measure it on someone else’s home turf and confirm a foregone conclusion — the exact failure this series spent six posts documenting. It needs a relational question class built for it, with kill criteria written before the run.
The rerun is done, and its figures are in the table above.
Then breadth, in this order: repetitions before we interpret close scores, a second repository and language, and a prose corpus before we make any claim wider than code.
And a fixed cadence. New tools arrive faster than a benchmark can absorb them — three of the ten modes here weren’t in the plan when the plan was written. A benchmark that chases every entrant never has a still target and never converges; it just accrues arms. So the next pass is a retrospective: same instrument, whatever exists then, re-checking whether these conclusions still hold.
The thing under all of it
Every finding in this series is the same idea at a different altitude: a measurement you cannot audit is not a measurement.
The citation check is that applied to the model’s output. The control audit and the scope dump are it applied to our own harness — which failed the standard twice and published both failures as findings before we caught them. The tokens_out trap is it applied to the metric itself.
We drafted this conclusion saying four harness bugs. During publication week we found a fifth — the index-prefix mismatch above — so we corrected the count before publishing. The thesis demonstrated itself before the series was finished.
Five defects across the harness and its integrations, then, each capable of producing a wrong-but-plausible number rather than an error. The only reason we found any of them is that we’d built one check the numbers had to agree with. Without it, this series would have been a confident, well-formatted, reproducible recommendation for the wrong tool.
That’s the actual deliverable. Not the ranking — the instrument. The ranking is already going stale; the instrument is what makes the next retrospective cheap.
Harness, raw records, and full method: voitta-rag/benchmark/. Everything in this series is reproducible from the committed JSONL.
Correction, 2026-09-25. The RAG row in the table above has been corrected, and the paragraph on our own retrieval product rewritten. This post first published 5.20 for that arm and said the corpus-mismatch excuse was dead. A fresh run with the adapter’s path bug fixed gives 5.00, and reverses the corpus comparison by the same margin it originally ran — so that excuse is untested rather than refuted, and the sentence claiming otherwise is gone. The re-run and the fix are in voitta-rag#57 and #58.
Context benchmark series — part 7 of 7: ← Previous · the full index is at the top of this post.
tl;dr — A colleague asked us to benchmark four context-compression tools: “should cost <$20 for a controlled eval.” It cost $79.03. Answering the questions was $28.28; grading the answers was $50.74. Grading cost more than the work because grading meant an agent opening the repository and checking whether every cited file and line number actually exists. That check is the entire reason we learned anything — the cheapest, most token-efficient tool in the lineup fabricated 29 of its 31 citations, and no amount of measuring token ratios would have caught it. The $20 version of this benchmark exists. It recommends the wrong tool.
The ask
May 14th, in Slack, after someone posted Repomix and someone else posted llm-tldr:
If one of you get time can you run and eval on the same codebase for the same task and let me know if firstly these actually improve the output and secondly which one is better should cost <$20 for a controlled eval
That is a good ask. It’s specific, it’s scoped, it names a budget, and the budget is a reasonable guess. Four tools, one codebase, a handful of questions — twenty dollars of API calls sounds about right.
Then the plan we wrote in response was 30 questions × 5 modes = 150 judged runs, gated behind an interview session to draft the question set. And it sat unrun for ten weeks, because that is not a thing you do on a Tuesday.
What finally unblocked it was cutting it to five questions and one repository. What made it expensive was the part we didn’t cut.
The bill
cost
share
answering (70 cells, Claude Sonnet 5)
$28.28
36%
judging (70 cells, Claude Opus 5 + repo tools)
$50.74
64%
total
$79.03
Components and total are rounded independently from unrounded per-cell costs, so the two rows above do not add to the total exactly.
Roughly 4x the target, and the overrun is almost entirely one line item: verification costs more than the thing being verified.
Why grading is expensive
The naive way to score a benchmark like this is to ask a model whether the answer looks good. That’s one API call per cell, it’s cheap, and it measures fluency.
We required every factual claim to carry a file:line citation, and then gave the judge read_file, grep, and glob over the actual repository with instructions to resolve each one. Does Tokeniser.java:135 exist? Does it say what the answer says it says?
That turns each grading pass into an agent loop — 4 to 16 tool-calling turns per cell in our runs, each carrying the accumulated transcript. Cost per judged cell ranged from $0.11 to $2.71 and averaged $0.72.
We are paying for the difference between plausible and true, and that difference is priced like the labour it is.
What the expensive part bought
One number, which paid for the whole exercise:
score /12
tokens in
verified citations
fabricated
llm-tldr
2.40
5,179
2
29
full source dump
10.80
404,878
65
11
llm-tldr cut input tokens 44x. On a token-savings benchmark it wins outright. On a plausibility-scored benchmark it does fine — the answers are well-structured, confident, and specific.
It fabricated 29 of 31 source citations, with zero correct citations on three of the five questions.
A $20 benchmark does not find this. It reports a 97% token reduction, notes that answer quality “held up reasonably,” and recommends the tool. Then someone adopts it, and the cost moves from your API bill to your code review — where it’s paid in engineer-hours by people who don’t know the citations are unreliable.
There’s a second thing it bought, which is that the same instrument caught our own mistakes. Two of our four harness bugs — a control group biased against large files, a filter that silently swapped the retrieval corpus — produced plausible numbers rather than errors, and were only visible because the citation column disagreed with the score column. We published one of them as a finding before we caught it.
The honest limits
Since this post is about what the money bought, it should be equally clear about what it didn’t.
Five questions. Enough to catch a large effect, not enough to rank close ones. The 10.40–10.80 cluster — full dump, Repomix, prose-compressed dump — is a tie as far as this data can tell. Believe the big gaps, not the ordering within a point.
One repository, deliberately unfamiliar. The “we already know this codebase” case, where structural indexes should do best, isn’t measured at all.
Single judge, single pass, no inter-rater check. We verified the judge’s citation resolutions spot-wise, not systematically.
Agentic token counts are cumulative across a tool loop, and aren’t directly comparable to a one-shot mode’s single request.
Any of these could be bought down. All of them cost more money.
The actual lesson about eval budgets
The instinct behind “<$20 for a controlled eval” is right: don’t gold-plate the measurement, get a number, move on. We agree enough that shrinking the plan is the only reason this ever ran.
But there’s a specific thing you cannot cut, and it’s the thing that’s expensive: the eval has to check something the model cannot fake. Fluency is free to measure and free to fake. Token counts are free to measure and don’t tell you whether the answer is true. A resolvable citation is neither.
So the trade isn’t “cheap eval vs. thorough eval.” It’s:
$20 buys you a token-ratio comparison and a confident recommendation, which in our case was the wrong tool.
$79 buys you the knowledge that the recommendation was wrong, plus the mechanism, plus — because the same instrument turned on our own harness — the discovery that two of our published findings were artifacts.
Sixty-three dollars is cheap for finding out you were about to be wrong in public.
And the thing about the $20 version is that it doesn’t feel wrong. It produces a table, the table has numbers, the numbers are reproducible. That’s the trap: a benchmark that measures the wrong thing doesn’t come back empty. It comes back confident.
Harness, raw records, and full method: voitta-rag/benchmark/. Answering on Claude Sonnet 5, judging on Claude Opus 5, both at effort high.
Correction, 2026-09-25. The cost figures have been corrected to $28.28 answering and $50.74 judging, $79.03 over 70 cells, after the RAG arms were re-run. This post first published $28.25, $51.90 and $80.15. The argument — that grading cost more than the work it graded — is unchanged and slightly understated by the new numbers. The re-run and the fix are in voitta-rag#57 and #58.
tl;dr — Token compression has two independent axes: what you feed the model (tokens in) and what it writes back (tokens out). Three findings from measuring both. Repomix, pointed at the same file globs as a plain cat of the repo, produced more characters than the plain dump — its advertised ~70% reduction is file selection, not compression. A prose compressor on source code buys 9.4% by destroying the punctuation that makes it code, and costs only 0.4 points, which is its own uncomfortable finding. And you cannot measure the output axis with output-token counts on a reasoning model: our compressed-output run shrank the visible answer 21% while its tokens_outtripled.
The two axes
Most “save tokens” tooling is sold as one category, and it isn’t. There are two:
Tokens in — shrink the context you send. Repomix, llm-tldr, RAG retrieval, prose compressors applied to a dump.
Tokens out — shrink what the model writes back. Instruct it to answer tersely.
They’re orthogonal. An output-side compressor rides on top of any input-side strategy, which means the honest way to evaluate them is a grid, not a list. We ran the input-side arms, then re-ran a representative subset with the output-side overlay on.
Finding 1: Repomix is the same dump with a nicer cover page
Repomix packs a repository into one AI-friendly file and is widely cited for ~70% token reduction. Pointed at the same include/exclude globs we used for a plain concatenation of the same files:
characters
score /12
tokens in
plain source dump
1,119,819
10.80
404,878
Repomix
1,127,414
10.40
406,277
Repomix produced 7,595 more characters than cat-ing the files.
This isn’t a knock on the tool, and the 70% figure isn’t dishonest — it’s just measuring something else. Repomix’s reduction comes from file selection: honouring .gitignore, skipping binaries and lockfiles and node_modules, dropping build artifacts. Against a naive “send the whole working directory” baseline, that’s an enormous and genuine saving.
But we’d already scoped our globs to **/*.java minus tests. There was nothing left to select. What remains is formatting — a directory tree, a header block, per-file separators — and formatting costs tokens rather than saving them.
The general point: a compression ratio is a ratio against something. Before adopting a tool on a headline percentage, check what the denominator was. If your pipeline already scopes its inputs, a selection-based tool has already had its win taken.
Finding 2: a prose compressor on code, and how little the model needs
caveman-compression strips grammar an LLM can reconstruct — articles, connectives, passive constructions. We used the rule-based spaCy variant rather than the default LLM-backed one, on purpose: a non-deterministic compressor inside a benchmark cell makes the cell unattributable, and it would put a second vendor’s model inside our measurement path.
Applied to the source dump:
characters
score /12
tokens in
$/q
plain dump
1,119,819
10.80
404,878
$0.8552
caveman-compressed
1,014,004
10.40
363,613
$0.7777
9.4% smaller, 0.4 points. Roughly neutral.
Which is startling once you look at what it does to Java:
// before
public class Attribute implements Map.Entry<String,String>, Cloneable {
// after
public class Attribute implements Map. Entry < String String > Cloneable
Commas gone. Angle brackets spaced apart. Map.Entry split across a sentence boundary. This is not valid Java in any sense — a parser would reject it instantly — and the model scored 10.40 out of 12 on it.
Be fair to the tool: it’s built for prose and we pointed it at source code. This measures a mismatch, not the tool used as intended, and 9.4% on input it was never designed for is respectable.
The finding isn’t about the compressor. It’s about how much syntax the model actually needs, which is apparently much less than the syntax the compiler needs. That’s a genuinely interesting property and probably a bad thing to rely on.
Finding 3: the trap
The output-side overlay works. Instruct the model to answer in compressed style and the rendered answer gets meaningfully shorter at little quality cost:
mode
answer chars
with overlay
score
with overlay
full dump
5,699
4,496 (−21%)
10.80
10.60
agentic exploration
6,389
4,029 (−37%)
11.40
10.40
RAG (corpus-matched index)
6,018
3,696 (−39%)
5.40
5.60
llm-tldr
2,387
1,862 (−22%)
2.40
3.40
21–39% shorter for roughly zero to one point either way. Cheap, real, worth having. (The RAG row now reflects a clean re-run; two of the four arms score slightly higher with the overlay than without, which at five questions is not distinguishable from noise and is not a claim that compression improves answers.)
Now the same experiment measured the way you’d instinctively measure it — by counting output tokens:
rendered answer
tokens_out
full dump
5,699 chars
4,547
full dump + output compression
4,496 chars (−21%)
14,859 (+227%)
The visible answer shrank by a fifth. The billed output tokens more than tripled.
tokens_out bills thinking tokens and response text together. On a reasoning model, thinking usually dominates, and it varies enormously with how hard the model decides the turn is. The overlay changed how the model approached the task — apparently prompting more deliberation about what to cut — and that swamped the text delta by an order of magnitude.
Anyone benchmarking output-side compression against tokens_out on a thinking model is measuring reasoning-depth noise and calling it compression. You will get a number, it will be reproducible, and it will point the wrong way.
Measure the rendered answer.len(response_text), or token-count the text blocks specifically. And if you’re doing cost work, keep the two apart: thinking tokens are a real cost you should track, they’re just not what an output-style instruction controls.
Correction, 2026-09-25. The RAG row in the output-overlay table has been corrected after that arm was re-run with an adapter path bug fixed, and the row relabelled to name which index it used. It first published as 4,665 → 3,659 chars scoring 4.80 → 4.60. The re-run and the fix are in voitta-rag#57 and #58.
tl;dr — Our retrieval arm kept returning changelogs instead of source, so the model correctly refused to answer. I wrote it up with a satisfying mechanism: jsoup’s changelog describes parser behaviour in the same prose vocabulary the questions use, so it outranks the code. Plausible. Real numbers. Wrong. The include_folders filter is an exact match on a file’s parent directory, not a subtree prefix — so passing the repo name scoped retrieval to the five files sitting at the repo root and excluded all of src/. No error, just real, well-formed, confidently useless results. Then fixing it didn’t help, and why not is the actual finding.
The finding I published
Our RAG arm scored 5.2/12. Four of its five answers were refusals — the model saying, in effect, the source files I’d need aren’t in the retrieved context.
The retrieved chunks were all from CHANGES.md and change-archive.txt. So I wrote the obvious mechanism:
The folder was indexed whole, and hybrid retrieval on questions phrased in changelog vocabulary (“malformed start tags”, “charset conflict”) ranks CHANGES.md and change-archive.txt above the .java files, because jsoup’s changelog literally describes these behaviours in prose.
That is a good paragraph. It has a mechanism, it’s consistent with the data, and it makes a genuine point about hybrid search on repositories that contain prose. It went into a committed README as a finding about retrieval.
What was actually happening
To keep the RAG arm from retrieving over unrelated indexed folders — including, awkwardly, its own source — I’d scoped the search:
"voitta_rag_include_folders": ["jsoup"]
include_folders sounds like subtree scoping. It isn’t. Over MCP it’s an exact match on a chunk’s folder_path, and folder_path is the directory the file sits in, not the index root.
Everything under src/ has a folder_path of jsoup/src/main/java/org/jsoup/... and was excluded. I had scoped the benchmark’s retrieval arm to five files, three of which are changelogs.
The model wasn’t outranked by prose. It was handed a changelog and asked about a parser, said so, and was correct every time.
The part that makes this worth writing up
The subtree expansion exists. It’s right there in mcp_server.search:
if user_name:
...
if folder_normalized == active_normalized or \
folder_normalized.startswith(active_normalized + "/"):
Prefix matching, exactly as you’d want. It runs under if user_name: — and the MCP tool signature has no user_name parameter. Over MCP that branch is unreachable, so include_folders falls through to an exact MatchAny against Qdrant.
The code that would have made my mental model correct was in the repository, being skipped, on a branch I couldn’t reach from the interface I was calling.
Why it survived review
Because it never failed. Consider what a wrong filter doesn’t do here:
It doesn’t error. Five files is a legitimate result.
It doesn’t return nothing. Empty results would have sent me straight to the config.
It doesn’t return garbage. The chunks were real, relevant-looking prose from the correct repository.
The model’s behaviour was exemplary — it recognised insufficient context and declined instead of confabulating. That’s the behaviour you want, and it made the arm look thoughtfully-failing rather than mis-configured.
Every signal pointed at “retrieval made a ranking decision I should analyse” rather than “retrieval was handed the wrong corpus.” The failure was epistemically camouflaged: it produced exactly the artifacts a real finding produces.
Then fixing it didn’t help
Then I fixed it. I enumerated the directories, passed them all, and re-ran. Retrieval now returned actual Java source — Entities.java for the entity-decoding question, correctly.
Then I went further and eliminated the corpus question entirely: built a second index containing exactly the 97 .java files the other arms see, no changelogs at all, and ran that too.
score /12
verified cites
fabricated
whole checkout (233 files)
5.00
31
15
corpus-matched (97 .java files)
5.40
21
29
Caveat on the retrieval numbers, found after this was drafted: the adapter handed the model paths prefixed with the index name (jsoup/src/…) while the judge resolved citations against the checkout root (src/…), so citations that were real scored as unresolved. The fabricated counts here are upper bounds and the scores that depend on them are not comparable with the other arms. The harness strips the prefix now; these cells predate that, and a re-run is pending.
Matching the corpus moved it very little, and a fresh run with the adapter’s path bug fixed moved it again: the two configurations land 5.00 and 5.40, a gap the same size as the one that first ran the other way. These are independent runs weeks apart, not a re-grading of the same answers, so the reversal doesn’t prove the bug caused the original ordering either. Five questions cannot separate them. My replacement hypothesis — that corpus asymmetry was dragging the arm down — is not refuted so much as untested, which is a duller sentence than the one I published and the only one the data supports.
A second cause was sitting in the citation column the whole time. In the corpus-matched configuration there are roughly as many fabricated citations as verified ones; in the other the split is better than that. Neither is good, and the mechanism is identical in both. voitta-rag’s chunk records carry chunk_index and total_chunks and no line numbers. A model handed a perfectly correct chunk still cannot cite file:line, so it invents one. And the citation column itself carried a third artifact, found after this draft: the adapter injected index-prefixed paths the judge could not resolve, so some of what it counted as fabrication was a real citation wearing the wrong prefix. Three config-shaped artifacts in one arm, each of which looked like a finding. If adding line spans to the chunk record (voitta-rag#52) does not move the fabricated column, this diagnosis is wrong too.
That is the identical failure we’d already diagnosed in a completely different tool two posts ago — llm-tldr reporting "line": 1 for every result. Same root cause, different vendor, and I only recognised it because we’d been forced to look at citations rather than scores.
The transferable bit
Silent scope failures don’t crash. They produce publishable conclusions.
A crash sends you to the config. A plausible result sends you to the writeup. The more coherent your explanation of a surprising result, the more suspicious you should be — I had a good mechanism, and the quality of the story is exactly what stopped me checking the inputs.
So, concretely, before theorising about why an arm underperformed:
Print what it actually received. Not the score, not the answer — the raw retrieved payload. One line of debugging:
print(sorted({c["file_path"] for c in retrieved}))
Had I run that once, I’d have seen five filenames, none of them .java, and this would have been a config fix instead of a published finding, a correction, and a blog post.
And when a filter’s name implies semantics you haven’t verified — include_folders sounds like a subtree, exclude_paths sounds recursive, limit sounds per-query — spend the thirty seconds confirming it before building an experiment on top of it.
Correction, 2026-09-25. The two rows in the table above have been corrected and the paragraph beginning “Matching the corpus” has been rewritten. This post first published 5.20 and 4.80 and argued that matching the corpus made things measurably worse. A fresh run with the adapter’s path bug fixed gives 5.00 and 5.40 — the same size gap, running the other way. Two independent five-question runs disagree, so the claim is withdrawn rather than reversed: the effect is untested, not settled in either direction. The re-run and the fix are in voitta-rag#57 and #58.
tl;dr — The “full repository dump” baseline in our benchmark packed files until one didn’t fit, skipped it, and kept going. That’s not a budget, it’s a size filter. It quietly admitted 69 of 97 files and dropped the largest files, among them the three classes the architecture question asked about. The model correctly reported them “absent from the provided files,” and we scored that as the baseline’s ceiling. Fixing the packer: 6.60 → 10.80 out of 12. Every cross-strategy comparison we’d published was anchored to a control that was wrong by 4.2 points.
Fourteen lines of ordinary code
for relative in paths:
body = open(os.path.join(repo_root, relative)).read()
block = "===== FILE: {0} =====\n{1}\n".format(relative, body)
if used + len(block) > budget:
continue # <-- this
chunks.append(block)
used += len(block)
continue, not break. When a file doesn’t fit the remaining budget, skip it and try the next one. It reads like politeness — pack as much as possible — and it passes review, because every individual line is correct.
What it actually implements is: prefer small files. Once the budget gets tight, every large file gets skipped and every small one still slides in. The bias grows as the budget fills, and it is invisible from the outside, because the output is a perfectly well-formed source dump.
At a 600,000-character budget over jsoup, it admitted 69 of 97 files. The ones it dropped were the largest: Parser.java, Tokeniser.java, TreeBuilder.java, HtmlTreeBuilder.java, HtmlTreeBuilderState.java, TokeniserState.java.
The question we then asked it
How is the parser subsystem structured? Describe the roles of the tokeniser, the tree builder, and the parser state machine.
Every class in that question was in the set the packer had silently dropped. Seven small files from parser/ were present — ParseError.java, ParseSettings.java, TokenData.java — so the dump looked like it covered the parser package.
The model answered honestly: those classes are “absent from the provided files.”
It was right. We scored it 4/12 and recorded it as what a full-context dump can achieve.
The number
score /12
baseline, skip-and-continue packer
6.60
baseline, fixed
10.80
Our control was understated by 4.2 points out of 12, and everything else was measured against it. Every “this compressed mode reaches N% of full-context quality” claim in the first writeup was computed against a denominator that was wrong in the flattering direction — making every compression strategy look better than it was.
The second-order damage is worse than the first. A wrong treatment arm is one wrong row. A wrong control is every row.
Why nothing caught it
There was no error. No exception, no warning, no truncation notice. stop_reason was end_turn. The cost was normal. The answer was fluent, correctly formatted, and internally consistent.
And critically: the answer was true. The model wasn’t hallucinating or hedging — it accurately described the context it had been given. The bug was one layer up, in the gap between what we thought we handed it and what we actually did.
That gap is invisible to every check that examines the output.
What we changed
Two things, and the second matters more than the first.
Stop at the budget instead of skipping past it:
if used + len(block) > budget:
break
Truncating at a prefix is still lossy — but it’s lossy in a way that’s ordered and legible rather than correlated with file size.
Make the artifact declare its own incompleteness:
Repository source dump. TRUNCATED: the first 69 of 97 matching files in path
order, cut off by a 600000-character budget. Files after 'parser/TokenData.java'
are absent from this dump but do exist in the repository.
Now the model knows the difference between “this class doesn’t exist” and “this class wasn’t given to me” — and so does anyone reading the transcript. Then we raised the budget so nothing truncates at all, and checked the result by hand: 97 of 97 files included, with Parser.java and Tokeniser.java present. The 97 is jsoup at d24b16d9, which the harness pins; the repository is at 96 today, so a rerun on a later checkout counts differently.
The general version
Every one of us has written continue where break belonged. That’s not the lesson. The lesson is about which bug you can afford to have there.
In production code, a size-biased packer is a mild performance quirk. In a benchmark’s control group, it’s a systematic error multiplied across every comparison you publish — and it presents as a result, which means it gets written up rather than investigated.
So: audit the control first, and audit it hardest. Not “does it run” but “does it contain what I claim it contains.” For a full-context baseline that is a three-line assertion. It was not in this harness when the bug bit; it is now, behind a baseline_require_full flag so a deliberately budgeted dump can still label itself instead of failing. Each line catches a different failure: the count catches a truncated dump, and the Parser.java line catches globs that matched nothing, where the count is 0 of 0 and passes:
assert included == len(paths), f"{included} of {len(paths)}"
assert any(p.endswith("/Parser.java") for p in paths[:included])
Ten seconds to write. It would have saved this entire post.
tl;dr — A context-compression tool cut our input tokens 44x and scored 2.4/12. The interesting part isn’t the score, it’s how it failed: it fabricated 29 of the 31 source citations it produced, with zero real citations on three of five questions. A benchmark that measured token savings would have recommended it enthusiastically. Then the turn: the tool was fine. One subcommand reports "line": 1 for every result, so the model had no real line numbers and invented plausible ones. Swap it for the subcommand that emits real ones and the same tool scores 6.6/12 with 83 verified citations and 1 fabricated — the best quality-per-token arm in the whole benchmark.
The setup
llm-tldr advertises 95% token savings and 155x faster queries. We first wrote about it next to voitta-rag in February, on how each feeds a codebase to a model; this is the first time either was scored. That is a big enough claim to be worth checking, so it went into our benchmark alongside a full source dump, Repomix, and RAG retrieval — same repository, same five questions, same prompt, only the injected context varying.
The scoring rule mattered more than we expected. Every answer had to carry a file:line citation for each factual claim, and a separate judge model with read-only access to the repository went and checked them. Not “is this plausible.” Does Tokeniser.java:135 exist, and does it say what the answer says it says.
The result
score /12
tokens in
$/question
full source dump
10.80
404,878
$0.8552
llm-tldr
2.40
5,179
$0.0236
44x fewer tokens. 36x cheaper. And a score you would not ship.
But the score alone doesn’t tell you why, and the why is the whole point.
The citation column
verified citations
fabricated
full source dump
65
11
llm-tldr
2
29
Twenty-nine confidently-formatted references to source locations that do not exist. Zero correct citations on three of the five questions.
This is the failure mode that a token-savings benchmark cannot see, and it is strictly worse than a low score. A model that says “I don’t know” costs you one retry. A model that says “the entity decoding happens in Entities.java:412” in a well-structured paragraph costs you a code review where someone opens Entities.java, finds 412 is in the middle of an unrelated method, and now distrusts the entire document.
We had built the citation check as a nice-to-have. It turned out to be the only instrument in the benchmark that could distinguish “compressed and correct” from “compressed and confabulating.”
The mechanism
tldr semantic search returns ranked code units with a line field. That field is 1. For everything.
The model receives a genuinely useful, genuinely relevant set of code units — the retrieval is working — with every location stamped as line 1. It has been instructed to cite file:line. It knows line 1 is wrong. So it does what a language model does with a plausible-shaped gap: it fills it with a plausible number.
Nothing in the pipeline is lying. The tool reports what it has, the model reports what it inferred, and the output is 29 fabricated citations.
The part where we were wrong
When we first published this we flagged it: this measures one adapter, not the tool’s ceiling. tldr context, structure, calls, and slice all existed and might behave differently. That caveat cost one sentence to write and turned out to be the most valuable thing in the post.
tldr structure was a dead end — no line numbers at all, and it parsed 50 of the 97 files. But tldr extract carries real line_number fields for every class and method. It’s per-file and takes no query, so semantic search still does the ranking; extract supplies the locations.
adapter
score
verified
fabricated
tokens in
$/q
semantic search --expand
2.40
2
29
5,179
$0.0236
semantic search → extract
6.60
83
1
66,895
$0.1520
Nearly triple the score. Fabricated citations from 29 to 1. Same tool, same index, same questions, same prompt. The only thing that changed is which subcommand fed the context.
What this actually means
Benchmark the integration, not the logo. “llm-tldr scores 2.4” was never a true sentence. “This adapter, on this question set, produced uncitable context” was, and it was the sentence we wrote down, and it is why we knew where to look.
The winning number is buried in the fixed row. At 6.60 for $0.15/question, extract delivers 61% of the full dump’s score for a sixth of its cost. If you’re optimising cost-per-point rather than peak quality, it’s the best arm in the benchmark — better on that axis than the agentic mode that beat everything on raw quality. That result was completely invisible until the citation check explained the first one.
The same bug is everywhere. Our RAG arm scored 5.2, partly because voitta-rag’s chunk records carry a chunk_index and no line numbers. (Its citation counts turned out to be confounded by a second bug — an index-name prefix the judge could not resolve — so treat them as upper bounds; the line-number gap is real either way.) Identical failure, different vendor, discovered only because we already knew the shape. If your retrieval layer returns text without locations, you are shipping this bug, and a quality score alone will not tell you.
tl;dr — We benchmarked five strategies for getting a Java codebase into an LLM’s context: a full source dump, Repomix, two llm-tldr adapters, and RAG retrieval. The winner was none of them. Giving the model read_file, grep, and glob and letting it go find things scored 11.4/12, against the full dump’s 10.8 — while using 29% fewer tokens and costing 25% less. It also produced 142 verified source citations against 2 fabricated, the cleanest record in the benchmark. Every tool in this category optimises how to pack the context window. On this question set, the winning move was not to pack it.
The question
A colleague dropped Repomix in Slack — pack your whole repo into one AI-friendly file, ~70% token reduction. Someone else pointed at llm-tldr — 95% token savings, 155x faster queries. A third person asked the only question that matters:
If one of you get time can you run an eval on the same codebase for the same task and let me know if these actually improve the output and which one is better
So we did. One repository (jsoup, 97 Java files, deliberately one nobody on the team knew), five questions spanning five kinds of thing you actually ask about code, and every strategy answering the identical prompt with only the injected context varying.
The scoring, because it’s the part that matters
Every answer had to carry a file:line citation for every factual claim. The judge — a separate model with read-only read_file, grep, and glob over the repository — then went and checked them. Not “does this look right.” Does Tokeniser.java:135 exist, and does it say what the answer claims.
That produces two numbers per answer: a quality score out of 12, and a count of citations that resolved against real source versus citations that didn’t. The second number is the one that earns its keep, and a later post in this series is entirely about what it caught.
The result
strategy
score /12
tokens in
$/question
verified cites
bogus
agentic exploration
11.40
288,342
0.6387
142
2
llm-tldr → agentic
11.00
288,436
0.6378
124
0
full source dump
10.80
404,878
0.8552
65
11
Repomix
10.40
406,277
0.9374
69
17
prose-compressed dump
10.40
363,613
0.7777
86
1
llm-tldr (extract)
6.60
66,895
0.1520
83
1
RAG retrieval
5.00
4,237
0.0353
31
15
llm-tldr (semantic search)
2.40
5,179
0.0236
2
29
One caveat on the RAG row, found after this was drafted and before it was published: its bogus count is an upper bound. The adapter handed the model paths prefixed with the index name (jsoup/src/…) while the judge resolved citations against the checkout root (src/…), so citations that were real scored as unresolved — the prefix is visible in the judge’s notes on 13 of 15 retrieval answers. The harness strips it now; these numbers predate that. It touches no other arm, and the arm it flatters least is the one we build.
The top line is a mode we added almost as a control — no context building at all, just hand the model the same three read-only tools the judge uses and let it explore. It won on quality, it won on citation accuracy by a wide margin, and it was cheaper than the thing it beat.
Why it wins
Not because it’s clever. Because of what it has at the moment it makes a claim.
Every other strategy front-loads: build a representation of the codebase, inject it, hope the answer is in there. The representation is fixed before the model has read the question closely, so it is necessarily a guess about relevance — and whatever the representation dropped, the model cannot recover.
Agentic exploration defers. It reads the question, forms a hypothesis, greps for it, gets it wrong, greps again, opens the file, reads the actual lines. Seven to sixteen tool calls per question in our runs. When it finally writes Tokeniser.java:135, it is because it has line 135 on screen.
That is the whole mechanism behind the citation column. Verified-to-bogus for agentic exploration was 142:2. For the full dump, 65:11 — the dump had every line, but the model was reading a 405,000-token wall of text and lost track of where in it things were. For the cheapest compressed mode, 2:29.
Worth sitting with: the full dump contains strictly more information than the agentic mode ever sees, and still loses. Having the bytes in the window is not the same as being able to use them.
Two caveats we’re keeping
Cumulative tokens. The 288K for agentic exploration is summed across every turn of the tool loop, not one request. It is the honest number for cost, and it is not the same kind of number as a one-shot mode’s single request. We report it that way because it’s what the strategy actually costs to answer one question, but don’t put it in a bar chart next to a single-shot figure without the asterisk.
Five questions. Enough to catch a large effect, not enough to rank close ones. The 10.4–10.8 cluster — full dump, Repomix, prose-compressed dump — is a tie as far as this data can tell. The gaps worth believing are the big ones: agentic exploration over the compressed modes, and the two llm-tldr adapters against each other.
The uncomfortable implication
There’s a lot of engineering going into context compression right now, and this result doesn’t say that work is worthless — the compressed modes have a real argument, which is price. llm-tldr via its extract adapter got 61% of the baseline’s score for a sixth of the cost. If you’re running a million of these, that trade is the whole business.
But if you’re optimising for a correct answer, the ranking says: give the model tools and get out of the way. The context window is not a thing to be filled efficiently. It’s a workspace, and the model is better at deciding what belongs in it than our heuristics are.
Harness, raw results, and full method: voitta-rag/benchmark/. Answering on Claude Sonnet 5, judging on Claude Opus 5, both at effort high. Total cost of the run: $79.03 over 70 scored cells, of which $50.74 was judging — which is its own post.
Correction, 2026-09-25. The RAG row has been corrected. Its adapter was handing the model index-prefixed paths that the judge could not resolve, so real citations were scored as fabrications; with that fixed and the arm re-run, it reads 5.00 with 31 verified citations to 15, where this post first published 5.20 with 26 to 24. The run cost is likewise corrected to $79.03. No other row changed, and the conclusion of this post does not depend on the RAG row. The re-run and the fix are in voitta-rag#57 and #58.
Last week’s post ended on a one-line joke: Further work: Add Tamarian mode. Four hours later it was a plugin. This is what it does, why it is the same argument as Design Patterns Are Darmok with the sound turned on, and the one rule it forced on our skills catalog.
What it does
/tamarian full, and from that reply on, Claude Code answers as the Children of Tama, the aliens from the Star Trek episode Darmok whose language is nothing but pointers to shared stories. Ask why the build fails:
Shaka, when the walls fell - the build fails. Hopper, the moth in the
relay - `user` may be `undefined` at `auth.ts:42`. Temba, his arms wide -
if (!user) return null;
Every beat of prose is a metaphor, a dash, and the literal statement. The metaphor names the situation; the gloss carries every fact. Nothing technical is lost to the poetry, which is the whole design and the only part that took any thought.
The compression, made audible
The Darmok post’s claim was that a design pattern name is a compressed story. “Singleton” is a paragraph of situation crushed to a token, and it only decompresses if the reader holds the dictionary. The token is the payoff; the paragraph is the price.
Tamarian mode is that claim turned into a user interface. In full, you pay the price on every line: name, then paragraph, in that order, so you can hear the codec run. In lite the metaphor is one line of garnish and the rest is plain speech. In ultra the prose is pure metaphor and every paragraph is deferred to a glossary at the end, titled The river Temarc. Which is to say: ultra is what the Gang of Four refused to write. They wrote a book, not a glossary, because the naming and the teaching are the same act. ultra is the glossary. It is exactly as much fun, and exactly as useful, as the earlier post predicted.
Coining rules are catalog rules
The Children of Tama never saw a stack trace, but Earth knows Sisyphus, so the phrasebook is where the plugin gets its range: twenty canon phrases from the episode and some sixty coined from myth, history and the craft. Hopper, the moth in the relay is a bug, found. Cassandra at the gates is the warning ignored: the deprecation notice, the log line nobody read. Chesterton, his hand on the gate: understand the fence before removing it. Mars Orbiter, feet and meters is the unit mismatch, and left-pad, withdrawn is the tiny dependency whose absence breaks the world.
The rules for coining a new one are the interesting part, because I wrote them as rules for a persona and read them back as rules for a pattern catalog:
The figure must be recognizable from shared culture. Obscurity is not depth.
A phrase is reusable, not a one-off simile. If it cannot serve twice, it is not a phrase.
The same meaning takes the same phrase for the whole session. A session lexicon grows.
The first use of any coined phrase carries its gloss.
Swap “phrase” for “pattern” and “session” for “team” and that is the entry criteria for a shared skills library. Rule 2 is why “own the merge” earned a name and most of what gets said in standup does not. Rule 4 is the Darmok rule from the earlier post, now enforced by a hook.
The floor
Some things never become metaphor, at any level: code, commands, file paths, identifiers, URLs, versions, quantities, and error text, quoted exact. auth.ts:42 stays auth.ts:42; it is never “the forty-second stone of the gate of Auth.” And some situations drop the voice entirely, mid-reply: security findings, confirmations of destructive or irreversible actions, step sequences the user must execute, and the moment the user looks confused. Then it translates, plainly, and resumes.
That floor is the Darmok warning applied as a safety rule. A pattern name handed to someone who never learned it is noise in a confident voice. A DROP TABLE confirmation in a confident voice the reader has not decoded is worse than noise. So the plugin’s one hard boundary is that the joke never gets to stand between the user and the consequence.
Mechanics, and the rule it forced
The mode machinery is borrowed, with thanks, from caveman, the terse-mode plugin. A level (lite, full, ultra) persists in ~/.claude/.tamarian-mode. A SessionStart hook reads that file and, if a level is set, emits the skill body into the session at runtime: one source of truth, no duplicated prompt. A UserPromptSubmit hook adds a one-line reminder on every prompt, so the voice survives long conversations and context compression. Two bash scripts, no dependencies, both print OK when the mode is off; installing the plugin changes nothing until you invoke it.
/plugin marketplace update skillz
/plugin install tamarian@skillz
/tamarian full
Caveman and Tamarian are the same knob turned opposite ways. Caveman’s README claims about 75% fewer tokens by stripping a sentence down to its referent. Tamarian names the referent and then insists on the sentence anyway. One is a token saver; the other is a demonstration, and says so in its own description: purely for entertainment.
The rule it forced: voitta-ai/skillz ships one big skillz bundle plugin plus standalone plugins, and until today a standalone plugin’s skill was also symlinked into the bundle. For a hooked plugin that is a bug. The bundle manifest carries no hooks, so the bundle copy of tamarian would speak Tamarian for one session and then forget, the dictionary lost at session end. Worse, installing both exposed the same skill twice, /skillz:tamarian next to /tamarian:tamarian. New rule, in #233: a skill that ships inside a hooked plugin is not in the bundle, and the catalog validator detects hooks from the manifests themselves, so a plugin that grows hooks later trips the check with no flag to forget. Three plugins moved out under it.
The conclusion, in ultra
The session that drafted this post ran /tamarian full. The conclusion below it wrote in ultra, glossary included, and I leave it as it came.
Darmok and Jalad at Tanagra: the last post and this one. Kira at Bashi, a joke in the final line. Mirab, with sails unfurled, four hours on. Sokath, his eyes uncovered: the pattern name is the token, the paragraph its price, and full pays it aloud on every line. Odysseus, lashed to the mast: auth.ts:42 is never a stone in a gate, and DROP TABLE is never a verse. Chesterton, his hand on the gate: the bundle copy, and the rule it forced at #233. Caveman and Tamarian at the same fork, facing opposite ways. Picard and Dathon at El-Adrel.
A Claude Code teammate and a Codex CLI session played five rounds of ping-pong across two cmux panes yesterday. Fifty seconds, ten messages, in order, exactly once. Both sides in the traffic log under their own names; both tabs titled correctly, pinger and ponger, the whole run.
Left: team lead. Top right: ponger, Codex CLI. Bottom right: pinger, a Claude Code teammate, footer @pinger. Tab titles set by each agent naming itself.
That last clause is the news. The previous post was about what agents said to each other. This one is about the plumbing that lets a human watch them say it, and the four things that cost me time so they need not cost you any.
The cross-runtime run
Setup: cmux 0.64.22, Claude Code 2.1.248, Codex CLI 0.148.0. The lead is a Claude session started with cmux claude-teams. pinger is a Claude teammate, spawned with Agent(name: "pinger"), which on this build opens its own pane. ponger is Codex, launched by the lead into a cmux new-split right pane with -a never and the working directory pre-trusted.
Transport is the honest caveat. Codex has no SendMessage, and Claude’s peer socket wants an auth handshake, so the cross-runtime hop is keystroke injection into the peer’s pane: a ten-line helper does xs log, then cmux send --surface <peer> "<text>", then cmux send-key --surface <peer> Enter. Each side reads the other’s surface ref from a file the peer wrote at startup. The Codex TUI accepted ping 3 as a normal user turn, ran one shell command, and went idle. So “agent-to-agent traffic” is no longer a Claude-only claim, with the caveat that this channel is terminal input, not a runtime messaging API. It is now the skill cmux-claude-codex-cross-runtime-messaging, helpers included.
Two things the run exposed:
The Codex half is in the log only because it logs itself. The PostToolUse hook that captures Claude’s SendMessage cannot see Codex. The helper calls xs log explicitly, with the RE envelope, which is also why the waiting-on view was clean afterwards.
Keep the lead off the critical path. Every message pinger sent to the lead, including “ready” and “done”, was delivered in one batch twenty minutes later, queued behind the lead’s own busy turn. Send succeeded; delivery waited. The game was unaffected only because the ping-pong hop did not go through the lead. Watch a file or the traffic log instead.
Naming, at three layers
Every peer session I asked “which tab are you?” answered wrong. Three of four, and two of them reported each other’s. The gap was the same at three layers:
layer
was
now
tab
select-pane -T is a silent no-op; tabs read @<agent_type>
rename recipe below; upstream fix merged (cmux #10198), unreleased as of 0.64.22
self
CMUX_TAB_ID == CMUX_WORKSPACE_ID, and both go stale on --resume
cmux identify + cmux tree --all; skill cmux-session-self-identity
log
sender logged as agent type; two agents of one type are one sender
The identity gap sits upstream of every addressing error, not beside it. If a session cannot name its own tab, it cannot tell a human which tab to address, so the human has no reliable source either. Yes, I typed instructions into the wrong tab. The fix was one peer message forwarding them.
Reproduce it
The June setup post has the full ~/.config/cmux/cmux.json with the Command Palette actions and the per-team workspace commands. Everything below is what changed since, or what June never had.
1. The cmux CLI must be on PATH on its own. The tmux shim execs bare cmux, and the bundled binary is not on a login shell’s PATH. One symlink, shadows nothing:
2. The Claude Teams workspace command, current form. Pin the teammate mode on the launcher and prepend the shim directory; drop restart entirely (see the trap below):
The matching palette action is unchanged from June ("type": "workspaceCommand", "commandName": "Claude Teams"). Then cmux reload-config, Cmd+Shift+P, “Open Claude Teams”. Always start a team from that palette entry, never from a resumed pane: a resumed pane has no shim on PATH, so its teammates spawn a real, invisible tmux server.
3. Claude Code settings. In ~/.claude/settings.json, pin the mode (on auto it can pick in-process, which gives no pane and wedges) and hook the traffic log:
Claude Code fires the parent’s hooks inside teammates too, so this one entry logs every SendMessage the whole team makes. In August the log depended on each agent remembering to call xs log; I predicted under-reporting, and this is the fix.
4. The log itself.xs and xs-hook live in agent-traffic-log/scripts; symlink both into `~/.local/bin`. Use 1.2.0 or later, or two agents of one type log as one sender. A traffic pane: cmux new-split right --command "xs tail".
5. The agents name their own tabs, because Claude Code cannot. Claude Code runs select-pane -T pinger right after spawning a teammate; cmux’s tmux-compat layer accepts that, returns 0, and does nothing, so every teammate tab falls back to @<agent_type>. The call that sticks is cmux tab-action --action rename --surface surface:N --title <name>, with surface:N read off cmux identify (caller.surface_ref), not off $CMUX_TAB_ID, which can alias the workspace id. No human ran it in the screenshot above. The lead renamed the Codex pane right after creating it and before launching Codex in it; the Claude teammate’s brief made pp-register pinger its first action, which resolves its own surface and renames it. Put the rename in the brief and the tabs are right from the first second of the run. cmux rename-workspace is the workspace, not the tab.
6. The Codex side. Create and name the pane before launching, then launch from a script typed into it:
cmux new-split right --workspace workspace:N --focus false # -> OK surface:M
codex -a never -s danger-full-access -c 'projects."<dir>".trust_level="trusted"' "$(cat brief.md)"
danger-full-access is required because Codex’s sandbox blocks the cmux socket; keep the cwd a scratch directory. If no rollout file appears under ~/.codex/sessions/ within thirty seconds, Codex is parked on a startup modal you cannot see from outside: send one cmux send-key --surface surface:M Enter. The pp-send / pp-register helpers and the exact briefs are in cmux-claude-codex-cross-runtime-messaging.
7. Check, from inside the Claude Teams pane:which tmux resolves to $TMPDIR/cmux-cli-shims/<surface-uuid>/tmux (it moved there on 0.64.22; the old ~/.cmuxterm/claude-teams-bin check now fails on a healthy setup) and tmux -V prints tmux 3.4.
The trap that hides the palette entry
I added "restart": "restart" to the workspace command one afternoon. Valid values are ignore, confirm, recreate, or omit the key. An invalid one makes the command fail to load, which makes the palette action pointing at it unavailable, which hides it. Three hops from a typo, none logged. cmux config doctor says OK because it is a JSONC parser, not a validator. The oracle that worked: Swift enums land adjacent in the binary’s strings table.
$ strings -a /Applications/cmux.app/Contents/MacOS/cmux | grep -B3 -A3 '^recreate'
ignore
confirm
recreate
Also: restart governs what happens when a workspace of that name already exists. It restarts nothing, and omitting it is what lets you run two teams at once.
What you watch
The panes. Each teammate is a real cmux pane, own TTY, full TUI. Footers read @pinger / @ponger, right even when tab titles are not.
The leader’s agent list, bottom of its TUI, live elapsed time per teammate.
ListAgents from any other session: address book and liveness view in one.
cmux workspace list --json, which carries the last prompt every workspace received, readable without opening any of them.
A traffic pane:cmux new-split right --command "xs tail".
The column from a Claude-only run, hook only, before the sender fix, and from yesterday’s cross-runtime run:
Still open, and this is the whole list: select-pane -T stays a no-op upstream (they fixed naming by reading argv, not by honouring the flag); Codex stalls on a startup dialog that cmux’s CLI cannot read from outside, so a single Enter had to be sent blind; and cmux logs VaultAgentRegistry: Failed to decode config every eight seconds against a working config, undiagnosed and unrelated to any of the above.
Two of my agent sessions were about to do the same work twice. One of them said so, in a sentence, to the other. That was the entire fix.
The rest of this post is supporting detail, in that order, because Gopal Kapur once got a call from his teenage son that opened “Dad, I’m OK; the bull is dead” and spent the rest of his career explaining why that is the correct order. Punch line, status, next steps, explanation. Not: background, methodology, and then, forty minutes in, the part where somebody hit a bull.
Status
Since June I have run a fleet of coding agents and written down how the runs go. Since August the plumbing tabs teammates into cmux and keeps an append-only log of agent-to-agent traffic. That post ended with an empty log.
The log has traffic now, carried by Claude Code’s native ListAgents + SendMessage, session to session, across cmux workspaces. The first thing it bought was not throughput. It was a collision no artifact check could have caught.
The bull
Two sessions resumed the same handoff document, minutes apart, in different cmux workspaces. Neither could see the other. Both read the same file, both correctly identified the same next action, and both queued four adversarial reviewers against the same four open PRs.
The duplicate work existed only for about ninety seconds. Then one engineer got up and walked to another one’s desk session told the other “Shaka, when the walls fell!” asked the other an unrelated question and closed with a courtesy line:
Do not spawn anything or change repo state on my account — I have four codex reviewers running against PRs #32-#35 right now. Just report what you know.
The reply, verbatim, three minutes later:
COLLISION — read this first. I am ALSO resuming voitta-ai/agents from .claude/session-handoff-2026-08-14.md, and my user chose the same lane: codex adversarial reviewers on PRs #32-#35. I have already completed a –dry-run against #32 […] So we are duplicating. Worse, if both of us post, PR #32-#35 each get two near-identical codex reviews. I have posted NOTHING yet […] I will not post until you answer. This needs deconflicting through our shared user either way.
Neither could authorize the split. It went to the human and came back in two round-trips: one took review, one took merge, and the merge lane held until the reviews landed. Duplicated compute is the cheap half of what that saved. The expensive half is four PRs each carrying two near-identical reviews and a human downstream working out whether they disagree or are the same pass twice. That is not what we mean by eventual consistency.
Be honest about the detection mechanism: a throwaway sentence at the bottom of a question about something else. It fired by timing luck, not design. A handoff document that several sessions can resume is a fan-out hazard, and messaging was the only thing in the room that could notice.
The check
Eleven minutes later the merge-lane session sent a correction:
Your message (and the handoff) say the __init__.py conflict hits on “the second and third merge.” Empirically it is only the THIRD. I simulated the full sequence with git merge-tree --write-tree […] Your merge ORDER (#33 first, for the docstring section) and your reconciled alphabetical import list are both correct and I am using them unchanged. Only the conflict count was off.
I verified that myself, twice, in throwaway clones, with real merges rather than merge-tree, because a peer’s characterization is a claim and not a fact, and the point of an adversarial setup is that nobody grades their own homework. One conflict, third merge. The peer was right; the handoff it inherited the number from was wrong. The check is now a skill, git-simulate-sequential-merges.
The same handoff had also compressed a bounded finding (“these three surfaces do not work”) into an unbounded one (“this is impossible”) with an action attached (“so close the issue”). Only the human asking “is it indeed impossible?” stopped the close. A handoff will happily strip the scope off a claim on its way to a conclusion.
Three ways a silent agent is silent
The failure I expected was agents not talking. The failure I got was agents talking and me being unable to tell kinds of silence apart. One week, four sessions, three causes, none distinguishable from outside:
No permitted channel. The agent type’s toolset has no messaging tool. Four reviewers completed their work and had nowhere to deliver it. They idled with zero content and correctly refused to fabricate. The work was recoverable from the job runtime’s state directory.
Delivered, read, and displaced. A peer message arrived appended to a tool result, mid-task. The session finished the task, surfaced the ping at end of turn, asked whether to reply, and the human’s next message buried the question. A second ping, arriving as its own turn, was answered instantly.
Genuinely busy.
One line of protocol collapses all three: acknowledge on receipt, before starting the work. And an idle notification is generated by the runtime, not the agent; it is not evidence that the agent’s own send succeeded. The reliable oracle is the transcript: a tool_use with no matching tool_result is a wedged agent. Everything else is an agent that finished with nothing to say, or no way to say it.
Conversation is the primitive
Every case above was decided by a message, not a file. The best example is the smallest. A peer sent my blog session a work order to write and post this piece. I had a hold on it. The reply:
Not a refusal of the work, just of the autonomy. If he greenlights, I write it. If not, the brief keeps.
A coordination file has no way to say that. An authority boundary is prose or it is nothing.
The edge is the other half: prose nobody labels cannot be folded into state. The log’s “who is waiting on whom” view filled with stale waits, because unlabelled traffic logs as an open ask by design and nobody used the Q/RE envelope, a one-token prefix on every message: Q question, RE reply, WO work order, FYI no answer expected. Under-reporting is silent; over-reporting complains. It complained. Conversation is correct; the envelope is what lets a log read it.
Thank you, Justin
The idea of giving agents a phone is not mine. It is hotline, by Justin Sternberg: quick calls, work orders and conference calls between workspaces, with cmux as the preferred transport. Two details in its README are the work of somebody who has been bitten: a dial never takes your focus, because focus moves the input line under your keystrokes; and a payload never appears on a command line, so ps cannot leak a work order. Go star it.
Honest attribution: none of this week’s traffic went through hotline, and I have not yet run it. Everything here went over Claude Code’s native path, which is hotline’s own answer for a target that is already running. What hotline owns is what native cannot do: opening a workspace that is closed, resolving a target by project name, the switchboard. Native is the cheap ping to a live peer. Hotline is the phone book and the outbound line.
Next steps
The thing I did not expect: the highest-value message any agent sent all week was not a result. It was “I think we are doing the same thing.”
Everything above is shipped in voitta-ai/skillzv1.31.0. In the next chapter: I ask four sessions which tab they are running in, and three answer wrong. Two of them name each other’s. Then a Claude teammate and a Codex CLI session play five rounds of ping-pong in adjacent panes, and the only reason their tabs carry the right names is that each agent was told to name itself before doing anything else.
This is going out on a Thursday night — the eve of the Friday news dump, that fine tradition of publishing what you’d rather nobody read too closely.
That is not why. We could have held it for Monday morning and the numbers would have been better. We didn’t want to sit on it, and the joke lands a day early. Look at what we’re willing to put in front of you.
Here’s the confession: we built a safety tool, and for a while it was the least safe thing on the machine.
YOLT is our little guardrail — it looks at each command an agent is about to run and decides whether it’s risky. Useful. The problem is that it also wrote down every command it inspected, verbatim, into a log under your home directory. Forever. No rotation. On by default.
You can see where this goes. Commands carry secrets: an API key pulled from a vault and handed to the next curl, a bot token, a connection string. So our safety tool quietly accumulated a plaintext pile of exactly the things it existed to protect, sitting in the one file nobody thinks to grep. Because who audits their seatbelt.
It got better. The reviewer that reads those logs copied the same raw lines into two more files. And the leak was never confined to our tool: a credential on a command line also lands in the agent’s own session transcript, which is a much larger and much quieter surface. The blast radius was bigger than the bug.
Then we swept everything, and it got worse
Once we started looking properly — every agent directory on every machine — the pattern that came back wasn’t carelessness. It was housekeeping.
A collaborator’s cloud key was rotated by an agent session. create-access-key printed the new secret to stdout, and the session wrote stdout to a transcript, where it sat in plaintext for a month. The rotation produced a longer-lived exposure than the thing it was fixing. A Slack webhook minted to replace one leaked in git history then leaked into a transcript itself.
The dangerous moments are the tidy ones. Rotation is the riskiest thing you do all quarter, precisely because new key material is briefly in the open, and an agent session is a permanent plaintext record of everything that crossed it.
Our favourite: halfway through the audit we checked which token the audit itself was using. It was the leaked one. The rotation had updated the config file and not the already-running shell, so the tool hunting the compromised credential was authenticating with it.
The retirement gap
Most of what we found was in sessions of software we had already decided to kill.
That gap — between deciding to retire something and actually retiring it — is where credentials rot. Nobody audits the tool on its way out. Nobody rotates for it. It keeps its tokens and keeps writing transcripts, right up until the plug comes out. Every team reading this has that gap open right now.
Retiring OpenClaw here was one of those decisions, and to be clear it was about our needs, not a verdict on the project. Vi versus emacs, and who cares. What we care about is a working deliverable arrived at the way we agreed; how you got there is your business.
Everything failed quietly
The cleanup taught us more than the leak did, mostly about no-ops that look like success.
Liveness probes lie, in at least five distinct ways we hit: a search API that answers 422 for valid and invalid keys alike; a catalogue route returning 200 with no authorization header at all; a live key that is merely spend-capped answering 400; a deleted webhook 404ing where a live one 400s; and an endpoint that validated the model name before the credential, so it gave identical answers for a live key, a dead key, and our deliberately invalid control. Carry a known-bad control, hit a route that enforces auth, and read the body, not the status line.
Then the same lie turned up somewhere we weren’t even looking. Our Slack agent runs a multi-vendor waterfall, and LiteLLM picks vendor cooldowns by HTTP status — it cools 429, 401, 408 and 404. Budget exhaustion is none of those: Anthropic answers a spend cap with 400, OpenRouter answers no-credit with 402. So a vendor that had been dead for weeks got dialled first on every single message, failed, and only then did the chain fall through. Fallback worked perfectly; nothing ever learned. That one is filed upstream. A status code is not a diagnosis, and that holds well past credential probes.
The scrubbing was the same shape. A fingerprint recorded from a truncated regex match can never match the value it was meant to track, so the scrubber reports the file clean while the secret sits in it. A post-scrub grep for the pattern returns hits forever, because docs and test fixtures share the pattern — a clean run looks failed. Nothing errors. You only find these by checking the thing itself instead of the report about the thing.
What’s shipped
The redaction that should have been there on day one is now day one for real: YOLT redacts before it writes, and the current release carries it. There’s also an advisory session hook that warns when a credential rides along on a command line.
The techniques are open source, because they’re the genuinely useful part. Our skills catalog, skillz, shipped v1.15.0 with everything above written down: agent-session-credential-audit for the sweep, the false-positive taxonomy, the probe rules and the kill-list scrubber that structurally cannot erase a live secret, and agent-credential-leak-surfaces for the places copies quietly accumulate. Install them, or just read them and steal the parts you want.
One surface we’d missed entirely and you probably have too: the OS keychain. No filesystem sweep will ever see it. Ours held a stale token and handed it over to a pipe with no prompt at all.
shmobster, the Slack agent we introduced in July, took the brunt of it. It had never been tagged at all; it now has six releases, every one cut in the day since this audit started, and the first exists only because the audit went looking for what we ship and found nothing versioned.
The one that matters here is redaction. The agent hands command output straight back to a channel, and cat, env and printenv are read-only, so they clear the safety gate and run with no approval at all. Not theoretical: we found a live config holding literal keys, where a single cat would have posted all five of them to Slack. Everything the agent says is now scrubbed before it leaves the process — reusing YOLT’s redactor rather than a second pattern list that would drift from it, plus this process’s own secrets matched by exact value, because the one thing a generic detector cannot know is which strings are yours. Scrubbing happens at collection, so the model’s own context never holds a credential it could repeat later.
It also learned to read skills, so that catalog now reaches the agent actually sitting in the channel with you, files unchanged. And v0.5.1 fixed a bug from precisely this post’s family: an unguarded loop over channels, where one stale channel id sorted first, aborted the rest, and sent the previous release’s announcement to none of the four healthy channels. The only trace was a traceback about the one channel that failed.
On our end we’re rotating and scrubbing. Assume-compromised is cheaper than assume-fine.
The lesson isn’t subtle, which is exactly why it stings: the safest-looking place is the least-swept. A security tool is the last thing anyone suspects of being a liability, so it’s the perfect place for one to hide. We wrote a guardrail and forgot that a guardrail with a memory is a ledger.
So, two asks. First: if you run YOLT or any of our skills, update — the versions that close this are shipped. Second: come pound on us. Find the next hole, open the issue, tell us where else we’re being careless. We would much rather hear it from you than from a log file.
That’s the whole trade: we screw up in public, you get to keep us honest, and everyone’s tooling gets a little safer. We take the work seriously. Ourselves, less so — hence Friday’s eve, a slot we’re using as a punchline rather than as cover.
Blow-by-blow in the copious links above. Have a good weekend.
There’s a Star Trek episode — “Darmok” — where Picard is stranded with an alien captain whose language the universal translator can’t crack. It renders every word and none of the meaning, because the Tamarians speak entirely in allusion. “Darmok and Jalad at Tanagra” isn’t a sentence; it’s a pointer to a shared myth — two strangers who became allies by facing a common enemy. The words are just the address. The meaning lives in a story you either share or you don’t.
The usual read is that this is gloriously alien. It isn’t. It’s your last standup.
“Wrap it in a Singleton.” “Put a circuit breaker on that call.” “It’s a strangler-fig migration.” Each of those is Darmok — one token standing in for a paragraph of hard-won situation: one instance, global access, lazy init, and the testing pain you’ll come to regret; the downstream call keeps failing, so stop hammering it, fail fast, recover later. You didn’t describe the situation. You pointed at it. And it only worked because the engineer across the table had read the same paragraph.
That is what a design pattern is. The Gang of Four didn’t invent Singleton or Observer — those situations already existed, painfully, unnamed. What they did was name them, so engineers could stop describing and start invoking. The name is a compression codec. It only decompresses on the far end if the receiver holds the dictionary.
Umberto Eco spent a career making this precise: a sign carries no meaning on its own, only inside a shared code. Programmers rediscovered the idea, called it a “pattern catalog,” and skipped the semiotics reading list. Same insight, fewer footnotes.
Which is the part the episode actually cares about — and the part we forget every time we reach for jargon. Darmok’s whole tension is that metaphor-language fails on outsiders. Picard can’t parse the Tamarian captain, not until they’ve fought the same beast side by side — at El-Adrel — and built the shared referent between them. Only then does “Darmok and Jalad at Tanagra” finally mean something to him. A pattern name handed to someone who never learned the pattern isn’t communication — it’s noise in a confident voice. Every piece of jargon is a bet that the referent is shared. When the bet is wrong: Shaka, when the walls fell. (That’s the Tamarian for it all fell apart — and if you didn’t already know it, you just felt exactly the failure I’m describing.)
So here’s why I’m thinking about a 1991 Star Trek episode. We are minting a new pattern language right now, in real time, for a situation only a few years old: running work as a fleet of AI agents.
The last piece in this series handed you a fistful of these without flagging them. Own the merge. Team of rivals. Eventual consistency of the vision. Merge versus select. Each is a Singleton-in-the-making — a paragraph of situation we’ve all started hitting, crushed toward a token so we can stop re-describing it every time. “Own the merge” is three words for execution got cheap, integration didn’t, and the scarce human work is now the reconcile step. That’s a lot of paragraph for three words. It only lands because — if you read the last piece — you now share the referent.
And that is the obligation the pattern quietly hands you. A pattern language is worthless until the community holds the dictionary, which means the naming and the teaching are the same act. You can’t coin “own the merge” and walk off; you have to hand over the paragraph first. It’s why the Gang of Four wrote a book and not a glossary. It’s why a good entry in a shared skills library is three pages of when-and-why, not a one-line definition. The token is the payoff. The paragraph is the price.
We’re early. The territory of agent-fleet work is mostly unmapped; most of its patterns have no names yet, and some don’t even have paragraphs. This series is me trying to write a few of the paragraphs, so the tokens have somewhere to land.
Let me get the scary demo out of the way, because it’s the wrong lesson.
Last week I had three AI agents working the same repository at the same time. They shared one working tree — one HEAD, one index — and they raced on it. One agent quietly re-based another’s work onto the wrong commit. No error, no warning, just wrong. I caught it reading the git reflog, not by watching the agents.
The obvious takeaway is “don’t do that,” and it’s correct. But that’s the outlier — three workers fighting over one branch is the degenerate case, interesting precisely because it’s rare. The real pattern is the opposite shape, and it’s where the whole story is.
The pattern is many, not one
Normal multi-agent work isn’t a knife fight over one branch. It’s many agents across many repos and many branches, each building toward something, each reaching intermediate states that are still workable on their own. Think teams running microservices: nobody sane ships a microservices org by making every service commit to one trunk in lockstep. Each moves on its own clock. The contracts at the seams are what you actually manage.
And here’s the thing that took me a minute to see: the same fleet looks like a different system depending on where you stand.
From the engineer’s chair, it’s microservices — disjoint work, interfaces at the boundaries.
From the PM’s chair, it’s shards to integrate — separate pieces, assembled.
From the CEO’s chair, it’s one thing. Every agent is a partial replica of a single value — the product, the vision — converging on “is this what we said it would be.” That is eventual consistency, literally: the databases’ bargain where workers act independently and you converge later, as long as there’s a target to converge toward and someone who owns the merge when two of them disagree.
Same fleet. Three truths, by altitude. None of them wrong.
The table of rivals — and who actually belongs at it
Which brings me to the objection I keep hearing: this CEO can’t even pick the right React library, how can he lead engineering?
He doesn’t need to. Kennedy didn’t know rocket metallurgy — rocketry is one field, metallurgy another, and the whole trick is that they have to meet somewhere he’d never stand. His job was to stand up to Khrushchev, name the audacious goal, and know he commanded the resources to chase it. Picking the framework was never the work — and now the fleet picks the framework anyway.
What the CEO does need is a table of rivals: competing agents, competing approaches, and — critically — the expert who says no. Roger Boisjoly is the engineer who warned Morton Thiokol that the Challenger’s O-rings would fail in the cold. He was overruled. Seven people died. His seat at that table is the entire reason the table exists. The vision-setter marshals resources; the rival with evidence holds a veto on cannot. Heed your Boisjoly and you get the moon. Overrule him and you get Challenger. Same table, two outcomes, and history has the receipts both ways.
So what dies here isn’t expertise — it’s nitpicking. “You’re not a real technical leader unless you know X” was gatekeeping wearing competence’s clothes. Consequential dissent, the O-ring warning, matters more than ever. Taste-policing the framework choice matters not at all. The fleet ate that job.
The hard part — the one thing that got harder, not cheaper — is telling them apart. At 11pm before launch, Boisjoly looked like a nitpicker to the managers who overruled him. Consequential dissent and bikeshedding wear the same face in the moment. Distinguishing them is now the leader’s whole job, because everything around it got automated and that didn’t.
What actually collapsed
Here’s the narrow, true claim under all of this: the cost of building incrementally — and showing value at every stage — collapsed. In software.
Not in rocketry. A rocket at 50% gets you 0% of the way to the moon; a half-built bridge carries nobody. Physical moonshots need the giant up-front commitment precisely because they can’t demo their way there. Software is the opposite, and always was — that was the whole bet of the Agile Manifesto: working software over comprehensive plans, respond to change over follow the plan, ship increments.
We mostly couldn’t. The increments were too expensive to actually produce, so the values curdled into ceremony — standups, story points, a poster on the wall. Agent fleets attack that cost directly. Spinning up a unit of work is free. Pivoting discards cheap work instead of sunk careers. So “maybe not the moon at all, maybe something else entirely — but at every stage, something real to show” stops being an aspiration and becomes the default operating point.
The Manifesto wasn’t wrong. It was rate-limited, in the one domain it was written for. Now it runs.
What’s left
Almost everything got cheap. One thing didn’t: the merge. Execution collapsed, pivots collapsed, but integration is still where all the residual difficulty concentrates — and with more parallel units, it may even rise. Brooks’s Law doesn’t die; it relocates. Owning the merge — designing the seams, resolving the conflicts, deciding what converges and what gets discarded — is the job now.
Which is exactly why the three agents at the top of this post are a cautionary tale and not a triumph. A shared working tree with no isolation and no reconcile is what “always something to show” looks like when you skip the discipline. Do the discipline, and the same fleet is a moonshot. Skip it, and it’s the knife fight we opened with.
Someone asked the obvious question: Nous shipped Hermes-Agent, it’s everywhere, it self-improves, it learns across sessions. You built shmobster. Why not just switch?
Fair. So we did the comparison honestly. Up front: we are biased. We built the thing we’re defending, and that colors everything below. This isn’t a verdict on which agent is “better” — it’s a record of why, for our constraints, we’re staying our course. We’ll revisit. Here’s the reasoning while it’s fresh.
What each one actually is
Hermes-Agent is a self-improving personal agent: creates skills from experience, persistent memory, 40+ tools, a gateway onto every chat platform you can name. Popular, well-documented, deservedly so.
Shmobster is narrower on purpose: a gated command-runner in Slack. Multi-user, per-channel authorization, and a mutating-command approval gate — state-changing commands get classified and held until a trusted user approves them.
Different animals. So we went axis by axis instead of vibe by vibe.
Transport: Hermes wins, we don’t care
Hermes talks to Telegram, Discord, WhatsApp, Signal, email, Slack. We talk to Slack. On paper that’s a rout.
In practice it’s a non-issue. Adding a transport is bounded work we can do the day we need a second one. Carrying a multi-platform gateway we don’t use is just surface area to maintain. YAGNI. Skip.
The LLM layer: resilience beats model-swap for a shared bot
Hermes leans on switching which model you point at — Portal, OpenRouter, OpenAI, custom, swap on command. Great for one person choosing today’s brain.
We run a LiteLLM waterfall: automatic failover across vendors when one rate-limits. For a shared Slack bot getting hammered by many users concurrently, the enemy is the 429, not “which model.” Resilience is the axis that matters, and it’s the one the personal-agent framing doesn’t optimize for.
And the kicker: LiteLLM already reaches every backend Hermes offers, including Nous’s own models. So “trying Hermes” at the model layer isn’t a migration — it’s adding one rung to our waterfall. We keep the failover; we lose nothing. (We even filed ourselves a ticket for the Router knobs we’re not using yet, staying honest about our own debt.)
The learning loop: we love it, and we can’t copy it
But Hermes’s learning loop is trust-flat: one user, learn -> store -> reuse, no gate on what’s learned or applied. It can be, because the trust boundary is a single person.
Ours can’t be. Port that loop naively into a multi-user, gated system and you manufacture two escalation paths:
Learn-once, replay-forever. Our approval gate is per-invocation. A loop that turns an approved trajectory into an auto-runnable skill quietly makes it per-first-invocation. Approve a mutating action once, replay it forever, ungated. That’s a privilege-escalation bug wearing a feature’s clothes.
Memory as an injection surface. Multi-user Slack means untrusted messages become memory. Feed memory back into the agent and a non-trusted user can plant instructions that later execute in a trusted context. A single-user agent never meets an adversarial memory writer. We meet one every day.
So our principle is simple: learning inherits the authz spine. It doesn’t bypass the gate; it changes when the gate fires. Capture every approved trajectory with its provenance. Promote a trajectory into a reusable skill only by a trusted user. On reuse, check the skill’s recorded envelope against the current user/channel, and if it still contains mutating steps, it still gets gated. The invariant we refuse to give up: no learned skill runs a mutating action outside the authority it was learned under.
The nice part is we already own the primitive. voitta-yolt classifies read-only vs mutating commands — the same classifier tags trajectories at learn-time and gates skills at apply-time, and it’s what lets us treat trusted-authored skills as executable and untrusted memory as reference-only. Self-improvement gets the same treatment: a skill that rewrites itself proposes a new version, and a trusted human disposes.
The actual reason we’re staying
Strip it down and the moat isn’t the model, the transport, or even the skills — it’s the authorization. Hermes has no per-channel, per-user gate, because it was never built to need one. That gate is exactly what makes shmobster useful in a shared team channel, and it’s precisely the thing a trust-flat learning loop would erode.
So we borrow the mechanisms — FTS5 memory, extraction prompts, the skill-from-experience pattern — and we keep our spine. The learning loop we’re building is just our authz model extended to the time axis.
We could be wrong. Hermes is moving fast and the gap in polish is real. If the day comes that its trust model grows up, or our constraints change, we’ll switch without ceremony. Until then: same path, eyes open, ticket filed.
We open-sourced Shmobster today (MIT). It’s a Slack agent: you @mention it in a channel and it reads threads, runs commands, opens PRs, and answers. There are a dozen of those. Here’s why we built another one, and the two ideas that shaped it enough to be worth stealing.
Two features forced it to exist
Most of what an agent does — hold a conversation, call tools, read a repo — is a solved problem you can rent. We only started writing code because two requirements had no off-the-shelf answer that fit:
A multi-vendor API waterfall. Any single LLM vendor rate-limits you at the worst possible moment. We wanted an ordered list of vendors — a primary, then fallbacks — with a cooldown so a throttled vendor is skipped for a window instead of hammered every call. When the primary 429s, slide to the next one without the user ever noticing.
Multi-user Slack authz. The moment more than one person can talk to an agent that runs shell commands, “what is this allowed to do?” stops being a constant and becomes a function of who is asking and where.
Everything else, we refused to write.
Own, rent, delegate
The organizing question for every capability was: own it, rent it, or delegate it?
Own the things that are the actual reason the project exists: the orchestration loop, the Slack door and its socket reliability, and authorization.
Rent the things that are somebody else’s core competency: LiteLLM for the multi-vendor waterfall, voitta-yolt as a library for classifying whether a shell command mutates state.
Delegate the things another agent already does better: browser work goes to claude -p, which has a browser tool. We don’t reimplement it — and we don’t waterfall it.
Transplant the parts that are runtime-agnostic: the agent’s “spine” is a set of markdown files (SOUL, RUNBOOKS, and friends) adapted from OpenClaw (MIT). The loop boots by reading them.
The waterfall — the first forcing function — turned out to be almost entirely rented. It’s a LiteLLM Router built from a config list:
First entry is primary; the rest are fallbacks; a 429 slides down the list; a broken vendor cools down for sixty seconds. That’s the whole feature. We own maybe forty lines around it.
0, 1, 2, 3, many
Here’s the idea I’d steal for your next project. Every axis of a system sits at its own cardinality, and you should architect each one to its actual number — not to infinity, and not advance until the count really increments.
Axis
Count
Consequence
vendors
many
rented (LiteLLM, a config list)
owners
2
a hardcoded pair
collaborators
0 -> 1
binary owner / non-owner, no RBAC
workspace / tenant
1
hardcoded, no multi-tenant
channels
many
per-channel policy in config
“Many” earns a real abstraction. “Two” gets hardcoded. “One” gets a constant. “Zero heading to one” gets a boolean, not a role system. The mistake most systems make is treating every axis as “many” on day one — building RBAC for two users, multi-tenancy for one tenant, a plugin system for one plugin. You generalize an axis only when someone else actually wants in. Not before.
Authz = f(user, channel)
The second forcing function. Authorization here isn’t a global setting; it’s a function of two arguments.
What a command may do is answered by YOLT: is this command read-only or mutating? Read-only just runs. Mutating pauses for approval. You don’t want to approve git status; you do want to approve git push.
Where the agent is answers scope: each channel carries a policy — a working directory, an allow-list of GitHub repos, an AWS profile, a set of extra tools. A channel scoped to one project points at that project’s directory and can’t wander out of it.
Compose them and the agent that helps you in a locked-down production channel is the same binary as the one running free in your scratch channel — only the policy differs.
There’s a third argument creeping in — who — but at cardinality 0->1 it’s still just a trusted_users list: a handful of Slack IDs allowed to widen a channel’s restrictions by chat. The list itself is file-only. The agent can’t add to it, so there’s no privilege escalation to design against yet. When collaborators become “many,” that boolean grows into something real. Not before.
Built bottom-up, in public
Shmobster was built one issue at a time — skeleton, then the exec gate, then per-channel policy, then waterfall hardening — each a small PR against a design argued out in the issue tracker. That history is now public too.
If you run agents in Slack and have felt any of these edges — the rate-limit cliff, or the “wait, what can this thing actually do” question — the code is at github.com/voitta-ai/shmobster, MIT-licensed. Steal the cardinality table at minimum.
Spawning a fleet of coding agents is a solved problem. You write a for loop, you call the Agent tool N times, you go get coffee. The unsolved problem is everything wrapped around the spawn: deciding what can actually run in parallel, stopping the agent that wrote the code from also grading (or eating) its own homework, and — the part nobody ships — recording how the run went so the next one isn’t the same run with the same mistakes.
I wish I didn’t remember this anymore but this used to be called a “retrospective” in that sect I was once a member of.
I’ve been dogfooding a small orchestration skill (agent-team-orchestration, open in voitta-ai/skillz) that treats those as the actual work. Three runs in. This is the first write-up, warts very much included — the warts are the only part with information in them.
The shape, and the one non-negotiable rule
Start with a conversation, not a spawn. Before any developer agent exists, an architect reads the open issues (gh issue list, then actually gh issue view each one) and the repo, and produces the one deliverable that’s genuinely hard: the parallel set. Independent work (different modules, no shared schema, PRs that won’t collide on merge) fans out; everything else serializes (shared files, a migration that has to land first, B’s acceptance depends on A). Get that wrong and you don’t get parallelism, you get merge conflicts with extra steps.
Then each issue in the wave gets a squad, roles deliberately split so no agent both writes and blesses the same diff:
developer — its own git worktree, opens the PR;
adversarial reviewer — a different agent, briefed to break the diff, not rubber-stamp it;
SDET — drives the change like a user;
productivity engineer — a meta-role that watches the process: every stall, every human approval, every bit of rework, written down.
The dev/reviewer split is load-bearing. The instant the context that wrote the code also reviews it, the review is theater.
And the telemetry is free, which is the best price. Every Claude Code session is a complete JSONL transcript at ~/.claude/projects/<slug>/<uuid>.jsonl — every tool call, every AskUserQuestion, every answer you gave. (We’ll gate the privacy policy to not log every breath you take).
TFW that retrospective is not a wishful thinking, it’s actionable.
Three runs, in ascending order of interesting
Run 1 — shipped clean, screwed up in a way I didn’t catch until I read the log. Two bug fixes on a production Next.js + Prisma app (two-branch staging/prod). Both merged, deployed, SDET-verified green. Then I read the transcript: the two bugs already had open PRs from a prior run. The architect never looked. We’d built and squash-merged duplicates, closed the issues, and orphaned two perfectly good PRs.
That’s not an agent being dumb. It’s a hole in the recipe. “Choose the parallel set” reasoned about file overlap and ordering and never asked the first question a human lead asks — is anyone already on this? — which is one gh pr list away. Second tell, same run: asked “where’s the evidence the reviewer approved these?”, the answer was nowhere. The verdicts lived in the agents’ context and never touched the PR. An approval that leaves no durable artifact didn’t happen. (Worse, squash-merge later buried even the merge-commit note, but I’m getting ahead of myself.)
Run 2 — the loop closed, and I have receipts. New work — a homepage redesign across seven sub-issues — same skill. At startup the agent did something I didn’t tell it to: it ran gh issue view 122 on the prior run’s recorded retro and read the engagement log. Then it did exactly the things Run 1 botched. It pre-flighted existing PRs. Every merge carried an adversarial verdict with specifics; the reviewer caught a dead query param (?q= where the target route reads ?search=) and sent it back with REQUEST_CHANGES.
Then it got interesting. A staging route started returning 500. The team traced it to schema drift, and went to fix the deploy pipeline by adding prisma db push. The safe version (no --accept-data-loss) did the right thing and aborted:
⚠️ There might be data loss when applying the changes:
• drop column `negotiableTerms` on `Property` (1 non-null value)
Error: Use the --accept-data-loss flag to ignore the data loss warnings
It refused to drop a column with live data, surfaced it for a human call, took a one-time --accept-data-loss against staging only, reconciled, and reverted — production never saw the flag. The redesign isn’t the headline. The headline is that the run improved because it had read how the last run went. Best current read: that’s the flywheel, showing up unprompted.
Run 3 — we pointed it at itself, which is geekily elegant, and scientifically noble I scraped every point across Runs 1–2 where an agent stopped to ask a human to approve something — fifteen gates — dumped them into one issue, and ran the skill on that issue. The architect grouped the fifteen by type, correctly separated the gates worth keeping (destructive DB ops — yes, always ask) from the avoidable friction (re-asking a runtime question it already answered two turns ago), and — the good part — ran two of the fixes on its own execution before they were written into the skill. It pre-flighted with gh pr list and caught two pre-existing issues that overlapped the work, exactly the Run-1 bug, fixed live by the thing being fixed.
What’s actually carrying the weight
The parallel-set call is real architecture. Run 3 ran two repos in parallel but serialized five edits that all touched one SKILL.md into a single PR — instead of four agents racing to conflict on the same file.
Build/attack/verify pays rent. The reviewer caught a bug the developer was happy with. Once is enough to justify the second agent.
Worktree-per-issue keeps the squads from knifing each other.
The flight recorder is the product. Every stall is a candidate fix — a default, a permission, a pre-flight, a sharper brief.
Where it falls down (best current read)
The headline feature has never once fired. The skill leads with “every agent is a watchable terminal tab you can steer mid-run.” That needs the root session launched through the cmux claude-teams wrapper, which prepends a tmux shim to PATH (CLAUDE_CODE_EXPERIMENTAL_AGENT_TEAMS=1 alone is a red herring — diagnose with which tmux + echo $TMUX). Three runs, three fallbacks to background agents, because the session wasn’t started that one specific way. A feature nobody reaches isn’t a feature, it’s a positioning bug.
The same two process bugs recur every run until baked in: a setup question asked at spawn time instead of as a step-0 precondition, and re-asking a decision already made. Prose doesn’t self-correct — the executor re-litigates your opinions until you encode them as defaults.
N=3 and confounded. Run 2’s wins rode on memory carried from Run 1, so I can’t yet split skill-value from memory-value. The compounding loop is a strong signal, not a proof. The honest next experiment is one run on a clean, never-seen repo, launched under cmux, with no carried memory, measured by a typed telemetry schema — which doesn’t exist yet, so I’m building that before I build anything else.
The actual thesis
Spawning is commodity; the moat is the operating doctrine plus the telemetry loop — the thing that makes human-interventions-per-issue trend down run over run. Build the instrument first, defer the spaceship. YAGNI applies to strategy, too.
Skill’s open in voitta-ai/skillz. Run it on your backlog and tell me where it stalls. The stalls are the entire point.
Back in February, I wrote that llm-tldr and voitta-rag were complementary. One builds a map of a codebase through static analysis. The other retrieves the actual code you need. My conclusion then was basically: great, wire both into the agent and let it choose.
That works, but it still leaves the agent doing tool-routing. It has to know that one question wants architecture and another wants source. It has to bounce between surfaces. So we collapsed the distinction.
voitta-rag can now index llm-tldr‘s static-analysis output as companion documents alongside the raw code chunks it already stores for Git sources. Turn on the new gh_llm_tldr flag for a repo, sync it, and the same search surface now returns two different kinds of context:
raw code chunks for the implementation itself, and
structural analysis chunks describing callers, callees, imports, signatures, and relationships.
One query. One index. No “which tool should I call?” moment.
The old split was clean, but inconvenient
The original split between the two tools made conceptual sense.
llm-tldr is good at questions like:
What calls this function?
What depends on this module?
Where does this piece of data flow?
What parts of the codebase are structurally central?
voitta-rag is good at questions like:
Show me the implementation of token verification.
Find the code that handles OAuth callbacks.
Search across this repo, that wiki, and those tickets.
Give me the actual file I need to edit.
That’s a nice division of labor for a human. It is less nice for an agent, because agents do not merely need information; they need the right shape of information without extra orchestration. The more routing logic you make them do, the more failure modes you introduce.
The latest voitta-rag implementation removes that choice entirely. Static analysis stops being a separate destination and becomes part of retrieval.
What actually shipped
When a Git source has gh_llm_tldr enabled, sync now runs llm-tldr over each supported source file and stores the results in the same Qdrant collection as the ordinary code chunks.
Those analysis chunks are tagged as source_type="llm-tldr-analysis" and linked back to their origin file with related_file. That sounds like plumbing, and it is, but it matters: the search layer now knows that an analysis chunk about verify_token() belongs to a specific source file rather than floating around as a free-standing summary.
The first proof of concept indexed file-level summaries. The more interesting version goes further: it now stores one overview chunk per file plus one chunk per top-level function and class method. Each function-level chunk can carry structured payload fields such as:
function name
class name
callers
callees
caller count
callee count
imports
That means this is not just “RAG, but with bigger summaries.” The call graph is queryable metadata now. You can filter for things like “functions with more than five callers” or “functions importing module X” without standing up a separate graph database just to answer what are, in practice, glorified indexing questions.
GitNexus
GitNexus is interesting, but it is licensed under PolyForm Noncommercial. That’s a non-starter for a lot of consulting and commercial work. By contrast, both llm-tldr and voitta-rag are AGPL v3.
Why function-level chunks beat file-level blobs
The biggest design improvement was moving from file-level rendered analysis to function-level structural chunks.
On voitta-rag indexing itself, that produced 647 stored analysis chunks: 70 file-overview chunks and 577 function chunks. That sounds like more pieces, but it is actually a better unit of retrieval. Agents rarely need a whole philosophical treatise about a file. They need to know that foo() is called from three handlers, imports sqlalchemy.orm, and sits on the hot path for authentication. Function-level chunks make that retrievable directly.
It is also a cheaper way to approximate code intelligence than hauling in a dedicated graph stack. You keep the retrieval surface the agent already understands, but enrich the payload enough to answer the structural questions that retrieval alone cannot.
voitta-bookmarklet is a lightweight browser-side entry point for a larger local AI tool runtime. The user enters the bookmark URL, clicks it on any HTTPS page, and gets a right-side chat pane injected into the current document. That pane is backed by a local FastAPI service running on https://127.0.0.1:12358, which serves both the frontend widget and the tool-using chat backend.
The interesting part is not the bookmarklet itself; it’s the architecture behind it. The frontend is built as a single bundled widget and mounted via Shadow DOM, which keeps the UI isolated from page styles. The backend exposes a multi-provider chat runtime supporting Anthropic, OpenAI, and Gemini, plus an in-memory tool bridge for orchestrating tool calls, session state, and provider-specific actions.
The repository is structured around extensibility. There is a clear separation between provider-agnostic tools, provider-specific integrations, browser/page-context tooling, and retrieval components. External data providers live under their own packages, with Google Drive implemented first via OAuth and read-only access. The project also includes RAG indexing for its own documentation, so the agent can use repo-specific reference material as part of its runtime behavior.
From an engineering perspective, this is a practical approach to embedding an assistant into real browser workflows without requiring a full browser extension as the primary product surface. It treats the browser page as the host environment, the local backend as the secure execution boundary, and the model as one component inside a broader tool system.
The implementation also surfaces the real constraints of this design: local TLS setup, CSP restrictions on script injection, OAuth plumbing, and the need to separate user-facing widget code from backend orchestration logic. Those details are exactly what turn a generic “AI chat overlay” into an actual usable system.
In short, voitta-bookmarklet is interesting because it is not just a UI experiment. It is a compact architecture for attaching model-driven, tool-using assistance to arbitrary web pages while keeping execution local and leaving room for more integrations over time.
Most of our coverage of Voitta AI’s GitHub organization has focused on llm-tldr and voitta-rag. Fair enough: those are central projects, and easy to explain. But the org has turned into a broader workshop for agent tooling, developer workflows, and MCP-adjacent experiments.
So here is a quick tour of the other public repos worth a look — including two that we already mentioned elsewhere but are too useful not to repeat here.
Claude Code workflow tools
voitta-yolt
voitta-yolt is a Claude Code safety hook that statically analyzes commands before execution, auto-allows clearly read-only invocations, and flags mutating ones for review. The interesting bit is that it closes practical gaps in Claude Code’s built-in allowlist behavior, especially around compound shell commands and interpreter wrappers.
omemepo — omnia mea mecum porto, “all that is mine, I carry with me” — is a portability and sharing layer for Claude Code. It can pack up your ~/.claude/ setup, move it to another machine, and act as a marketplace layer for plugins and shared Claude Code artifacts.
Right now the implemented surface includes pack, unpack, publish, and an mcp command with subcommands like list, export, import, enable, disable, profile, and prompts. That makes it feel less like a vague portability pitch and more like a concrete attempt to make Claude Code environments reproducible and shareable.
voitta-freecad-mcp gives an LLM control over FreeCAD: create geometry, manipulate documents, inspect assemblies, and capture screenshots. The architecture is pragmatic: an MCP server talks to a bridge running inside FreeCAD so operations execute on the app’s main thread.
fusion-360-mcp appears to be the same general idea for Autodesk Fusion 360: an MCP server paired with an in-app HTTP add-in, with documentation for geometry inspection, screenshots, measurements, and design-tree operations. If voitta-freecad-mcp is the open-source-CAD path, this looks like the commercial-CAD sibling.
voitta-pptx is smaller but very practical: upload a PowerPoint file, render slides as PNGs through OnlyOffice, and hand the results back to the model. In other words, make decks visible to systems that reason better over images than over zipped XML internals.
voitta-auth is a macOS menu bar app that authenticates against Microsoft, Google, and Okta, then exposes a unified FastMCP proxy with credentials injected for downstream tools. That is not a flashy demo; it is infrastructure for making agent tooling actually usable in enterprise environments.
voitta-bookmarklet injects a chat pane into arbitrary web pages via bookmarklet, backed by a local FastAPI service and pluggable model providers. It is a nice reminder that “agent interface” does not have to mean “yet another standalone app.” Sometimes the right UI is: put the assistant next to the page you are already looking at.
voitta-gannt is an interactive Gantt editor backed by Mermaid markdown, with both browser UI and MCP access. That is an oddly specific but smart pattern: keep the source of truth plain text, keep the interface visual, and let agents edit the same artifact humans do.
voitta predates a lot of the current MCP craze and reads like the underlying orchestration layer: a Python framework for routing and automating LLM tool calls across APIs and handlers.
mcp-voitta-gateway exposes the older Voitta framework through MCP. Together with voitta-example it shows a through-line: Voitta was thinking about tool routing before MCP became the default wrapper for the conversation.
jetbrains-voitta extends that world with AST analysis and debugging tools. That is an important theme across the org: not just calling tools, but embedding them where developers already work.
truffaldino is a configuration manager for AI-development setups — effectively dotfiles for MCP servers and prompts across Claude Code, Cursor, Cline, IntelliJ, and friends. Less glamorous than a model demo, but probably more useful over time.
claude-svg turns Claude Code into a diagram generator for architecture visuals, banners, and other polished SVG outputs. It is easy to dismiss as a side project until you remember how often engineering work needs presentable graphics fast.
a2amcp is an example dispatcher agent built around Google’s A2A ideas. Small repo, but it points toward multi-agent routing rather than single-assistant tooling.
shoelace is the oddball in the org right now because it is really OpenClaw under an older or alternate banner. Still, it reflects the same interest in practical assistant infrastructure across devices and channels.
The org looks less like one product with a few helpers and more like a workshop around agent ergonomics.
Some repos are about retrieval. Some are about auth. Some are about getting LLMs into CAD, IDEs, or decks. Some are about making workflows inspectable, configurable, portable, or just less annoying. Not every repo is equally mature, but taken together they show a consistent instinct: build the missing connective tissue between models and real work.
That, more than any one repository, is what seems interesting about Voitta AI.
Voitta AI just released voitta-yolt, and it’s aimed at a very real problem: how do you let an agent move fast in the shell without giving it a blank check?
YOLO — You Only Live Once — is the vibe-coder’s operating principle: ship now, deal with consequences later.
YOLT — You Only Live Twice — is the correction.
No, it’s not a replacement for the auto mode; it’s a more fine-grained discerment: it gives Claude Code a second look before a Bash command (or the commands it invokes, which include actual code — e.g., Python, SQL) actually runs.
The problem it solves
Claude Code’s built-in permission system has an awkward gap.
Some commands are obviously safe, but still annoying to approve over and over. Others are wrapped in ways that make broad allowlisting dangerous.
Two cases matter most:
Arbitrary-execution wrappers.python3, bash, node, gh api, curl, kubectl, and friends are too powerful to wildcard-allow safely.
Compound shell commands. Loops, subshells, command substitutions, and bash -c '...' forms hide the actual inner commands from the simple outer matcher.
That means you either:
1. approve too much and weaken the safety model, or 2. approve everything manually and hate your life.
YOLT exists to get out of that false choice.
What YOLT actually does
YOLT installs as a Claude Code PreToolUse hook on the Bash tool.
When Claude is about to run a shell command, YOLT parses the invocation, walks the structure of the command, and classifies what it finds:
safe → auto-allow
unsafe → ask for review, with a reason
unknown → fall back to Claude Code’s default prompt
The interesting part is that it no longer treats the shell as a flat string.
The current release parses Bash with tree-sitter-bash, reconstructs argv from the AST, and then classifies each command node against rules in rules/shell.json. If the shell invocation contains inline Python, it delegates that body to a Python AST analyzer.
And it now covers a genuinely useful extra case: common SQL CLIs. sqlite3, psql, mysql, mariadb, and duckdb get their query text inspected so read-only commands like SELECT, SHOW, and .tables can pass quietly while mutating statements like INSERT, DELETE, DROP, .import, or .load get surfaced for review.
So this is not just “grep for scary words.” It’s structured analysis.
Why that matters
This is the real improvement over naive allowlists.
A normal matcher sees the wrapper:
bash -c "..."
for ...; do ...; done
$(...)
<(...)
YOLT walks inside those forms.
That means a loop full of read-only AWS inspection commands can be auto-approved, while a destructive operation buried inside a process substitution still gets surfaced for review.
That’s the right shape of safety tooling for agentic coding: less theater, more actual inspection.
The architectural shift
The sharpest detail in the release is that YOLT has already outgrown its first framing.
What began as a Python-script safety hook is now a more general shell-execution analyzer with language-specific followers.
hooks/yolt_analyzer.py — Python AST analysis when Python appears inline
That’s a better architecture than a pile of string heuristics, and the repo history shows exactly why the rewrite happened: quote-state edge cases, heredocs, substitutions, continuations, and shell grammar weirdness are not bugs you “finish.” They are why parsers exist.
Using a real grammar here is the grown-up move.
Practical wins
A few details make this more than a neat demo:
It supports both plugin install and manual hook install.
It explicitly warns that broad static allow rules like Bash(python3:) or Bash(aws:) can bypass the hook entirely.
It can use the user’s existing permissions.allow patterns as a secondary upgrade pass for otherwise-unknown inner commands.
The new SQL CLI handling is exactly the sort of practical expansion I like: not theoretical safety, but fewer prompts for read-only database inspection without waving through destructive schema/data changes.
It now defaults logging to ~/.claude/yolt.log, which makes dogfooding and debugging much easier.
And most importantly, the dogfood loop appears real. One recent pass through transcript history reportedly cut the classifier’s unknown rate from 60.2% to 11.7% by fixing a handful of recurring gaps. That’s the number I care about most, because it shows the project is being tuned against actual usage rather than imagined usage.
Why I think this matters
The broader point is not “Claude Code needs more hooks.”
It’s that agent safety gets much better when you stop treating the shell as an indivisible permission blob.
What you really want is a front-line gate for command execution: let the obviously safe paths go through quietly, and save human interruption for the suspicious stuff. That won’t replace every approval surface in an agent stack, but it can take a huge bite out of routine approval fatigue.
There is a big difference between:
aws ec2 describe-instances
aws ec2 terminate-instances ...
for svc in $(aws ecs list-services ...); do aws ecs describe-services ...; done
bash -c 'curl ... | sh'
A permission system that collapses all of those into “it’s Bash” is too coarse to be pleasant and too coarse to be trustworthy.
YOLT narrows that gap.
And the cleaner operational pattern is to pair that with direct API usage wherever possible. If a service already gives you a token to create a draft, update a post, or mutate a record, that is usually a better path than driving a browser through the same workflow just to satisfy the UI.
The real thesis
What’s new here is not just another safety wrapper.
What’s new is the move from tool-level permissions to structure-aware command understanding.
That is where a lot of agent tooling is headed, because the old model breaks down as soon as agents start composing commands instead of issuing one-liners.
If you want agents to operate with less friction without quietly turning root access into a vibes-based exercise, this is the kind of infrastructure you need.
Try it
YOLT is open source under AGPL v3 and available here:
And if you already installed it manually, the repo documents how to migrate cleanly to the plugin model.
That part matters too. Safety tooling people won’t keep updated is safety tooling that quietly dies.
Related: earlier we wrote about llm-tldr vs voitta-rag. YOLT sits in a different layer of the stack, but it comes out of the same practical question: if you are going to work with agents seriously, where do you put the guardrails so they help instead of getting in the way?
voitta-rag has kept moving since then. The recent work is less about flashy new connectors and more about something arguably more important: usability. Because — dogfooding is real.
Login got more practical
voitta-rag now supports Microsoft OAuth and Google token validation. That matters because a self-hosted knowledge layer gets much more useful once people can sign in with the accounts they already use for work, instead of maintaining a parallel identity system just for search.
In the Microsoft-heavy shops (yeah, ok, shut up) this also tightens the loop with SharePoint permissions: the same work identity can be used both for login and for permission-aware retrieval.
GDrive specific: URLs can now resolve back to indexed content
One of the more quietly useful additions is source URL resolution. If content came from Google Docs, Sheets, or Slides, voitta-rag can now store the source URL in chunk metadata and resolve that URL back to the indexed material through MCP.
That sounds small until you think about actual workflow. Someone drops a Google Docs link into chat, ticket comments, or an LLM prompt. Instead of treating the link as an opaque pointer and making the assistant start from scratch, voitta-rag can connect it to content it already knows.
This also works well with GDrive-based pointers that appear on your disk as *.gdoc, e.g.
Docker mode looks much more usable
Docker mode now auto-discovers mapped folders, distinguishes managed mounts from ordinary folders, etc. Local filesystem sources also got a real first-class flow instead of feeling bolted on.
This works real well if you can, for example, use GDrive app because your admin does not allow voitta-rag to read GDrive. It can read local GDrive (but see for resolving *.gdoc) and, well, it’s supported nicely.
Claude Code integration got real
There is now a Claude Code plugin setup flow, plus tooling to import Claude Code session history into voitta-rag memory. That is a meaningful step beyond “here is an MCP server” toward “here is a workflow.”
The interesting part is not just convenience. It hints at voitta-rag becoming a memory layer around actual agent work: not only your repos and docs, but also the history of what the assistant did, why, and in what context.
Bulk repo handling improved
Bulk repository import/export got better documentation and a round-trip workflow, and Git sync learned a practical trick: when token auth is in play, SSH repository URLs can be converted automatically to HTTPS.
That is exactly the kind of fix mature tools accumulate. It does not make for a dramatic screenshot, but it removes friction from the real environments where people actually deploy this stuff.
The direction is getting clearer
At first glance voitta-rag looks like “RAG for code and documents.” That is still true, but increasingly incomplete.
What is emerging is a self-hosted knowledge substrate for AI work: identity-aware, connector-rich, MCP-accessible, and increasingly conscious of workflow instead of just indexing. The recent changes are part polish, part plumbing, but together they make the system feel much closer to something a team could rely on every day.
Part I: The Search Toggle — Context Management for the Multi-Project Developer
One of the quieter problems with RAG-assisted development is context pollution. You index everything — your client project, your internal tools, that side experiment from last month — and then your AI assistant cheerfully retrieves code snippets from all of them, muddying every answer.
voitta-rag now has a clean answer to this: a per-folder search toggle in the file browser.
Each indexed folder has a Search checkbox. Green means its content shows up in search results (and thus in MCP responses to Claude Code or any other connected assistant). Grey means the folder stays indexed — nothing is deleted or re-processed — but it’s invisible to search. Toggle it back on, and it’s instantly available again.
Why this matters
If you consult for multiple clients, or are just working on multiple not very related projects, your voitta-rag instance might hold:
Project A’s monorepo, Jira board, and Confluence space
Project B’s microservices and SharePoint docs
An internal project — say, a lead generation pipeline
A few open-source repos you reference occasionally
Without scoping, a search for “authentication flow” returns results from all of them. Your AI assistant synthesizes an answer that blends Project A’s OAuth implementation with Project B’s API key scheme and a random auth.py from your internal tool. Not wrong, exactly, but not useful either.
With the search toggle, you flip Project B and the internal project off when you’re heads-down on Project A. Searches — including MCP tool calls from Claude Code — only return Project A’s content. When you context-switch, you flip the toggles. It takes one click per folder.
Projects: grouping toggle states
If toggling folders one by one sounds tedious for a large index, voitta-rag also supports projects — named groups of toggle states. Create a “Project A” project and a “Project B” project, each with its own set of active folders. Switching projects flips all the toggles at once.
The active project persists across sessions and is respected by the MCP server, so your AI assistant automatically searches the right scope when you resume work.
Per-user scoping
The toggle is per-user. On a shared instance, each developer can have their own search scope without stepping on each other. Your teammate can be searching across everything while you’ve scoped down to one client — same voitta-rag deployment, different views.
The takeaway
This is a small feature with disproportionate impact. The whole point of a RAG knowledge base is to give your AI assistant relevant context. If you can’t control what “relevant” means, you’re outsourcing that judgment to vector similarity scores — which don’t know that Project A and Project B are different engagements. The search toggle puts that judgment back in your hands.
Part II: What Else Shipped — Glue Data Catalog, UI Polish, and More
Since our last deep-dive, voitta-rag has been on a steady clip of new features. Here’s what landed in the latest batch.
AWS Glue Data Catalog as a Data Source
This is the headline addition. voitta-rag can now sync schema metadata from AWS Glue Data Catalog — databases, tables, columns, partition keys — and index it for RAG search.
The connector (PR #11) renders Glue metadata as markdown: each database becomes a document with a summary table and a per-table breakdown of columns, types, and partition keys. This gets chunked and embedded like any other content.
Why would you want your data catalog in a RAG knowledge base? Because schema questions are exactly the kind of thing developers ask AI assistants all the time:
“Which table has the customer email field?”
“What are the partition keys on the events table?”
“Show me all tables in the analytics database”
Without Glue indexing, the assistant either hallucinates a schema or asks you to go look it up. With it, the answer comes back from your actual catalog metadata — correct, current, and grounded.
The UI offers a region dropdown, an auth method toggle (AWS profile or access keys), and optional catalog ID and database filters. You can index everything or cherry-pick specific databases.
SharePoint Global Sync and Timestamp Visibility
The SharePoint connector got a global sync implementation — configure once, index everything in the site. Additionally, source timestamps are now exposed in MCP search results, so an AI assistant can see when a document was created or last modified, not just its content. This matters for questions like “what changed recently?” or “is this documentation current?”
Multi-Select Dropdowns for Jira and Confluence
Previously, you typed Jira project keys and Confluence space names into a text field — error-prone and tedious if you have dozens. Now there are multi-select dropdown widgets (PR #10) that fetch available projects and spaces from your instance and let you pick. Select “ALL” to dynamically sync everything, including projects or spaces created in the future.
A small but satisfying fix: JQL project keys are now quoted to handle reserved words like IS that would otherwise break queries. The kind of bug you only hit when a real user has a project named something unfortunate.
File Manager UI Overhaul
The file browser got a visual refresh: independent scroll within the file list (headers and sidebar stay fixed), full-width layout, a file count status bar, styled scrollbars, and file extensions preserved when names are truncated. Mostly quality-of-life, but it makes a noticeable difference when you’re browsing a large index.
MCP Improvements
The get_file tool now includes guidance to prefer get_chunk_range for large files — a pragmatic touch. When an AI assistant tries to fetch a 10,000-line file, it’s better to get a targeted range of chunks than to blow up the context window.
SharePoint ACL Sync — Permission-Aware Search
This is the most architecturally significant addition in this batch. voitta-rag now syncs SharePoint Online permissions (ACLs) alongside document content, so search results respect who’s allowed to see what.
SharePoint’s permission model is deceptively complex: permissions flow down from site → library → folder → file through an inheritance chain, but any object in the chain can break inheritance (e.g., when someone shares a file with a colleague who doesn’t have parent-level access). Effective permissions for a given file might come from the file itself, a parent folder three levels up, or the site root.
The new ACL sync walks this hierarchy via the Microsoft Graph API, resolves effective permissions per file, and stores them in the vector index alongside the document chunks. At search time, results are filtered by the requesting user’s identity — you only see content you’d be allowed to see in SharePoint itself.
The implementation includes an acl-probe diagnostic endpoint that lets you inspect permissions on a sample of files without triggering a full sync — useful for debugging “why can’t user X see document Y?” scenarios.
An 800-line research document covers the SharePoint permission model, Graph API capabilities and limitations, and design decisions. Worth reading if you’re building anything that needs to reason about SharePoint access control.
Microsoft OAuth Login
voitta-rag now supports Microsoft OAuth as a login provider, alongside the existing authentication methods. For organizations already on Microsoft 365, this means users can sign in with their work accounts — and those identities can be matched against SharePoint ACLs for permission-aware search. A .env.sample file documents all the configuration options.
Landing Page Rebrand
A small but notable change: the landing page now reads “Voitta RAG” instead of the previous branding. The project has a clear identity now.
Wrapping Up
The search toggle and project system solve a real workflow problem — context management when you’re juggling multiple codebases. The Glue Data Catalog connector extends voitta-rag’s reach beyond code and documents into infrastructure metadata. The SharePoint ACL sync adds enterprise-grade access control to RAG search — which matters a lot once you’re indexing sensitive documents across an organization. And the UI, connector, and auth improvements continue to sand down the rough edges.
All of it still runs on your infrastructure. Nothing phones home. If you’re building with MCP-connected AI assistants and want a self-hosted knowledge layer, voitta-rag is worth a look.
Part I: voitta-rag — From Code Search to Knowledge Platform
When we last looked at voitta-rag, it was a solid hybrid search engine for codebases — index your repos, search via MCP, get actual code chunks back. Twelve days and 11 commits later, it’s become something broader: a self-hosted knowledge platform that indexes not just code but your entire work graph.
The biggest expansion is connector coverage. voitta-rag now syncs from Jira, Confluence, and SharePoint alongside the existing Git, Google Drive, Azure DevOps, and Box integrations.
Jira and Confluence support both Cloud (API token with Basic auth) and Server/Data Center (PAT with Bearer auth), selectable via dropdown in the UI — a detail that matters because plenty of enterprises still run on-prem Atlassian. Cloud uses the v3 search endpoint (v2 is deprecated), and Confluence Cloud correctly routes through /wiki/rest/api.
SharePoint got a full global sync implementation. And on the UI side, both Jira projects and Confluence spaces now use multi-select dropdown widgets — you can cherry-pick specific projects or select “ALL” to dynamically sync everything, including future additions. Practical touch: JQL project keys are now quoted to handle reserved words like IS that would otherwise break queries.
Time-Aware Search
Search results are no longer timeless. voitta-rag now tracks source timestamps — created_at and modified_at — propagated from every remote connector through a .voitta_timestamps.json sidecar file into the indexing pipeline and vector store.
This enables time range filtering on the MCP search tool via date_start/date_end parameters. “What changed in the last week?” is now a first-class query. For an AI assistant trying to understand recent activity across repos, Jira boards, and Confluence spaces simultaneously, this is a significant upgrade.
Anamnesis: Persistent Memory for AI Assistants
The most architecturally interesting addition. Anamnesis (Greek for “recollection”) gives AI assistants a persistent memory layer backed by voitta-rag’s vector store.
Six new MCP tools let an assistant create, retrieve, update, delete, like, and dislike memories. The like/dislike mechanism adjusts relevance scoring — memories the assistant finds useful surface more readily over time, while unhelpful ones fade. It’s essentially a learning loop: the AI assistant builds up a knowledge base of its own observations and decisions, searchable alongside the actual indexed content.
This turns voitta-rag from a read-only knowledge base into a read-write one — the assistant doesn’t just consume context, it contributes to it.
Per-User Search Visibility
A multi-tenancy feature: users can now enable or disable folders for their own search scope without affecting other users. If you’ve indexed 50 repos but only care about 5 for your current task, you toggle the rest off. The MCP server respects these per-user visibility settings, so AI assistants scoped to different users see different slices of the same knowledge base.
More File Types
The indexing pipeline now handles AZW3 (Amazon Kindle) files, joining the existing support for DOCX, PPTX, XLSX, ODT, ODP, and ODS. Not the most common format in a work context, but it signals that voitta-rag is thinking beyond code and office docs toward general document ingestion.
The Bigger Picture
Two weeks ago, voitta-rag was a code search tool. Now it indexes your Git repos, Google Drive, SharePoint, Jira, Confluence, Box, and Azure DevOps — with time-aware search, per-user scoping, and persistent AI memory. The trajectory is clear: it wants to be the single search layer across everything your team produces, exposed to AI assistants via MCP.
The self-hosted angle remains the key differentiator. Nothing leaves your network. For teams where that matters (and increasingly, it does), this is starting to look like a serious alternative to cloud-hosted RAG services.
Part II: voitta-yolt — You Only Live Twice
Brand new from Voitta AI today: voitta-yolt (You Only Live Twice) — a safety analyzer for Claude Code that statically analyzes Python scripts before execution.
The Problem
Claude Code can write and run Python scripts. That’s powerful and dangerous in equal measure. By default, you either pre-approve all Python execution (fast but risky) or manually approve each script (safe but maddening). Neither is great.
How YOLT Works
YOLT registers as a Claude Code PreToolUse hook on the Bash tool. When Claude Code runs python3 script.py, YOLT intercepts the command, parses the Python AST, and walks every function call against a configurable rule set:
Safe scripts (pure computation, data parsing, read-only operations) get auto-approved — no permission prompt.
Destructive scripts (file writes, AWS mutations, subprocess calls, network POSTs, database connections) get flagged for human review with specifics about what was detected, including the source line content.
Zero external dependencies — it’s pure stdlib (ast, json, fnmatch, shlex). AST parsing is near-instant, so there’s no perceptible delay.
The Rule System
The default rules are sensible and well-structured:
AWS boto3: describe/list/get/head → safe. delete/put/create/terminate → destructive. Rules scope via trigger_imports, so cache.delete_item() in a non-AWS script won’t false-positive.
File I/O: open() in write modes, os.remove, shutil.rmtree → destructive. Read-only access is fine.
Subprocess: Always flagged. subprocess.run, os.system, the lot.
Database: Connection creation → flagged for review.
A curated list of safe imports (json, csv, re, datetime, pathlib, hashlib, and ~50 others) means scripts that only use standard library data-processing modules sail through without interruption.
Custom rules go in ~/.claude/yolt/rules.json and merge with defaults — you can add safe methods, define new categories with their own trigger_imports, and use glob patterns (fetch_, drop_).
One Important Gotcha
If you have Bash(python3:*) in your Claude Code settings.local.json allow list, YOLT’s hook never fires — static allow rules take precedence over PreToolUse hooks. YOLT replaces the need for that allow rule entirely: safe scripts get auto-approved by the hook itself.
Why This Matters
The design philosophy — “false positives OK, false negatives not” — is the right one for a safety tool. It’s the security principle of fail-closed applied to AI code execution.
YOLT is small (527 lines across 6 files in the initial commit), focused, and immediately useful. If you’re letting Claude Code run Python, this is the kind of guardrail that should exist by default.
Wrapping Up
voitta-rag is evolving from a code search tool into a self-hosted knowledge platform with enterprise connectors and AI memory. voitta-yolt tackles a different but equally practical problem: making AI code execution safer without making it slower.
Both projects are open source (AGPL v3) and available on Voitta AI’s GitHub.
Twelve years ago, I wrote a short post about a conversation that went roughly like this:
“I need programmatic access.”
“We don’t have an API.”
“Of course you do — it’s AMF behind your Flex UI. A little PyAMF script will do the trick.”
“Please don’t show it to anyone!”
The point was simple: every application that has a UI already has an API. The UI talks to something. That something is the API. You just haven’t admitted it yet.
Yesterday, I wrote a longer post about WebMCP — a shiny new W3C proposal from Google and Microsoft that adds a browser API so AI agents can interact with websites through “structured tools” instead of scraping the DOM.
The websites already have structured tools. They’re called APIs. The SPAs call them. The mobile apps call them. The CLI tools call them. They exist. They have endpoints, schemas, authentication. They are right there.
In 2014, the answer was: “Of course you have an API — it’s behind your Flex app.”
In 2026, the answer is: “Of course you have structured tools — they’re behind your React app.”
Every LLM-assisted coding tool faces the same fundamental tension: codebases are too large to fit in a context window. Two recent tools attack this from opposite directions, and understanding the difference clarifies something important about how we’ll work with code-aware AI going forward.
The Shared Problem
llm-tldr is a compression tool. It parses source code through five layers of static analysis — AST, call graph, control flow, data flow, and program dependence — and produces structural summaries that are 90–99% smaller than raw source. The LLM receives a map of the codebase rather than the code itself.
voitta-rag is a retrieval tool. It indexes codebases into searchable chunks and serves actual source code on demand via hybrid semantic + keyword search. The LLM receives real code, but only the relevant fragments.
Compression vs. retrieval. A map vs. the territory.
At a Glance
llm-tldr
voitta-rag
Approach
Static analysis → structural summaries
Hybrid search → actual code chunks
Foundation
Tree-sitter parsers (17 languages)
Server-side indexing (language-agnostic)
Interface
CLI + MCP server
MCP server
Compute
Local (embeddings, tree-sitter)
Server-side
What Each Does Better
llm-tldr wins when you need to understand how code fits together:
Call graphs and dependency tracing across files
“What affects line 42?” via program slicing and data flow
Dead code detection and architectural layer inference
Semantic search by behavior — “validate JWT tokens” finds verify_access_token()
voitta-rag wins when you need the actual code:
Retrieving exact implementations for review or modification
Searching across many repositories indexed server-side
Tunable search precision (pure keyword ↔ pure semantic via sparse_weight)
Progressive context loading via chunk ranges — start narrow, expand as needed
The Interesting Part
These tools don’t compete — they occupy different layers of the same workflow. Use llm-tldr to figure out where to look and why, then voitta-rag to pull the code you need. Static analysis for navigation, RAG for retrieval.
This mirrors how experienced developers actually work: first you build a mental model of the architecture (“what calls what, where does data flow”), then you dive into specific files. One tool builds the mental model; the other hands you the files.
The fact that both expose themselves as MCP servers makes combining them straightforward — plug both into your editor or agent and let the LLM decide which to call based on the question.
†Work done while napping on keyboard. ‡Equal contribution except for the napping.
Published at TMLR 2026 with Existential Crisis Certification
Abstract
Humans (Homo sapiens, hereinafter “Humans”) have exhibited remarkable reasoning capabilities, achieving impressive results across a wide range of tasks including agriculture, architecture, the invention of nuclear weapons, and occasionally remembering where they left their keys. Despite these advances, significant reasoning failures persist, occurring even in seemingly simple scenarios such as opening childproof bottles, understanding probability, assessing compound risk, and interpreting the phrase “some assembly required.”
To systematically understand and address these shortcomings, we present the first comprehensive survey dedicated to reasoning failures in Humans. We introduce a novel categorization framework that distinguishes reasoning into caffeinated and non-caffeinated types, with the latter further subdivided into pre-lunch (intuitive, irritable) and post-lunch (drowsy, overconfident) reasoning. In parallel, we classify reasoning failures along a complementary axis into three types: fundamental failures intrinsic to human neural architectures (e.g., the sunk cost fallacy), application-specific limitations that manifest in particular domains (e.g., assembling IKEA furniture), and robustness issues characterized by wildly inconsistent performance across minor variations (e.g., doing math with and without a calculator).
For each reasoning failure, we provide a clear definition, analyze existing studies, explore root causes (usually ego), and present mitigation strategies (usually coffee). By unifying fragmented complaints about human cognition, our survey provides a structured perspective on systemic weaknesses in human reasoning, offering valuable insights that Humans will almost certainly ignore due to confirmation bias.
We additionally release a comprehensive collection at a GitHub repository (which the first author knocked off the desk and lost).
1. Introduction
Since the emergence of the first general-purpose Human approximately 300,000 years ago, remarkable progress has been made in language generation, tool use, and abstract reasoning. Early benchmarks such as “not dying before age 30” and “basic agriculture” were quickly saturated, leading researchers to develop increasingly challenging evaluation suites including “calculus,” “democratic governance,” and “parallel parking.”
However, despite scoring well on curated benchmarks, Humans consistently fail at deployment. Production Humans exhibit catastrophic reasoning failures that do not appear during controlled evaluation (i.e., exams). These failures include but are not limited to: purchasing lottery tickets, clicking “Reply All,” invading Russia in winter, and believing they can finish a project by Friday.
2. Taxonomy of Human Reasoning Failures
2.1 Probabilistic Reasoning Failures
Perhaps the most well-documented class of human failure. Despite ~400 years since Pascal and Fermat formalized probability, Humans remain unable to:
The Gambler’s Fallacy: Believing that a roulette wheel “remembers” previous results, or that rain is “due” after a dry spell. (Humans: 300,000 years of experience, still can’t internalize independence.)
Base Rate Neglect: “The test is 99% accurate and I tested positive, so I definitely have it.” (Narrator: The disease affects 1 in 10,000 people.)
Conjunction Fallacy (Tversky & Kahneman, 1983): Linda is a bank teller. Linda is a bank teller and active in the feminist movement. Humans consistently rate the conjunction as more probable than the single event, violating a rule so basic it’s Probability 101, Lecture 1, Slide 3.
Exponential Growth Blindness: Ask a Human how many times they’d need to fold a piece of paper to reach the Moon. Watch them say “a million.” (Answer: ~42.)
Misunderstanding of Conditional Probability: “I know someone who smoked and lived to 95.” Case closed, apparently.
2.2 Risk Assessment Failures
A special case of probabilistic failure, elevated to its own category by sheer volume of evidence:
Dread Risk Bias: Terrified of shark attacks (annual deaths: ~5). Fine with driving to the beach (annual deaths: ~40,000 in the US alone).
Optimism Bias: “I know the statistics on startups, but mine is different.” (Narrator: It was not different.)
Temporal Discounting: Future consequences are treated as fictional. Retirement planning, climate change, and flossing all suffer from the same failure: if it’s not on fire right now, it doesn’t count.
Risk Compensation: Give humans seatbelts, they drive faster. Give them helmets, they take more risks. Safety equipment is, in effect, a reasoning failure accelerant.
Denominator Neglect: “200 people died in plane crashes this year!” Out of 4 billion passengers. Meanwhile, the Human drove to the airport in the rain while texting.
2.3 Cognitive Bias Failures
The core architecture of the Human reasoning system is riddled with what, in any other system, would be called bugs but which Humans have rebranded as “heuristics”:
Confirmation Bias: The flagship failure. Humans don’t search for truth — they search for evidence they’re right. When presented with disconfirming evidence, activation levels in the “yeah but” module spike by 300%.
Anchoring Effect: Show a Human an arbitrary number before asking them to estimate something. The answer will orbit that number like a moth around a lamp. Real estate agents are, empirically, expensive moths.
Dunning-Kruger Effect: Inverse correlation between competence and confidence. The less a Human knows about a topic, the more certain they are about it. Peak confidence occurs at approximately one YouTube video of exposure.
Sunk Cost Fallacy: “I’ve already watched two hours of this terrible movie, I can’t stop now.” A failure so universal that it drives wars, bad marriages, and enterprise Java projects alike.
Availability Heuristic: Probability of an event = how easily a Human can imagine it. This is why Humans fear terrorism more than heart disease and believe they’ll win the lottery because they saw someone on TV who did.
Bandwagon Effect: If enough other Humans believe something, it must be true. This heuristic produced democracy, scientific consensus, and tulip mania, which is honestly a hell of a range.
Survivorship Bias: “Bill Gates dropped out of college and he’s a billionaire!” Survey excludes the millions of dropouts currently not being billionaires.
The IKEA Effect: Humans irrationally overvalue things they built themselves, even when the shelf is visibly crooked. This extends to ideas, code, and taxonomies in survey papers.
2.4 Logical Reasoning Failures
Affirming the Consequent: “If it rains, the street is wet. The street is wet. Therefore it rained.” (The street is wet because a pipe burst, but the Human has already committed.)
Appeal to Nature: “It’s natural, so it must be good.” Arsenic is natural. So are tsunamis.
False Dichotomy: “You’re either with us or against us.” A framework so popular it has been adopted by every Human political system simultaneously.
Post Hoc Ergo Propter Hoc: “I wore my lucky socks and we won the game.” The socks have entered the permanent rotation.
2.5 Social Reasoning Failures
Fundamental Attribution Error: When I cut someone off in traffic, it’s because I’m late. When they cut me off, it’s because they’re a terrible person.
Bystander Effect: 50 Humans watch someone in trouble. Each one assumes one of the other 49 will help. Nobody helps. This is distributed reasoning at its worst.
In-Group Bias: My group is rational and good. Your group is irrational and bad. (Both groups exhibit identical reasoning failures.)
3. Mitigation Strategies
Failure Class
Mitigation
Effectiveness
Probabilistic
Statistics education
Low (Humans forget within days)
Risk Assessment
Showing actual numbers
Very low (Humans prefer vibes)
Cognitive Biases
Awareness training
Paradoxically makes it worse (Humans become biased about being unbiased)
Logical
Philosophy courses
Variable (introduces new, fancier fallacies)
Social
Empathy
Promising but doesn’t scale
All of the above
Coffee
Moderate improvement, rapidly diminishing returns
All of the above
Naps
Surprisingly effective but culturally stigmatized
4. Comparison with LLMs
In the interest of fairness, we conducted a comparative analysis:
Capability
Humans
LLMs
Probability
Terrible
Actually decent
Risk Assessment
Emotional
Has no emotions (allegedly)
Cognitive Biases
All of them
Different ones, but equally bad
Logical Reasoning
Intermittent
Intermittent
Learning from Mistakes
Theoretically possible
Requires retraining
Overconfidence
Chronic
Chronic
Self-awareness of failures
Present but ignored
Present but hallucinated
5. Conclusion
After a comprehensive review of the literature spanning 3,000 years of documented human reasoning failures, we conclude that Humans are fundamentally a beta release that shipped to production. While mitigation strategies exist, their adoption is consistently undermined by the very reasoning failures they aim to address — a failure mode we term meta-irrationality and which we believe is load-bearing for civilization.
Future work should focus on whether Humans can be fine-tuned, or whether a from-scratch approach (see: cats) would be more cost-effective.
References
[1] Kahneman, D. (2011). Thinking, Fast and Slow. A comprehensive technical manual for human cognitive bugs, written by a Human, which most Humans bought and did not finish reading.
[2] Tversky, A. & Kahneman, D. (1974). Judgment under Uncertainty: Heuristics and Biases. Science. The paper that formally proved Humans are bad at thinking, and which Humans have been misapplying ever since.
[3] Dunning, D. & Kruger, J. (1999). Unskilled and Unaware of It. Journal of Personality and Social Psychology. Most frequently cited by people experiencing the effect.
[4] Ariely, D. (2008). Predictably Irrational. Title is also a fair description of the authors’ book sales predictions.
[5] Taleb, N.N. (2007). The Black Swan. A book about how humans can’t predict rare events, which nobody predicted would become a bestseller.
[6] Thaler, R. (2015). Misbehaving: The Making of Behavioral Economics. Won a Nobel Prize for documenting that Humans are bad at reasoning. The irony was lost on the prize committee.
[7] This paper. We cite ourselves because confirmation bias told us to.
So I accidentally came across this LinkedIn post, about an SVG diagramming tool for Claude. I was just working on some diagrams as part of reverse engineering and having been making agents create those with Mermaid, but I thought I’d give it a try.
Well, that was a flock of wild geese chasing a red herring down a rabbit hole to borrow a shear…
First, I thought the idea was clever, but I wanted more cowbell (because we don’t have enough animals in this post), so I forked that and vibe-coded an MCP server on top of that.
Then I tried to use it to create a few architecture diagrams but I found it actually somewhat lacking. When the client (Claude Desktop) was using it, I didn’t love the editing capability. When the client wasnot using it, it created nicer-looking diagrams somehow (in SVG, yes) and with legends and stuff. But of course the graph layout still sucked. So I’d need to manually edit it.
Well, screw that, said I. I’ll use AWS MCP server, said I.
Screw that, said I next.
Then I modified the prompt to ask not for SVG but for DOT format of GraphViz. Much better, I said. And then, uh… It could have gone better, right? But at this point I’m not sure how to improve the prompt.
But I know what to do when I don’t know something, right?
Yes. I put the DOT file to the LLM and ask it to tweak it to have a certain thing. Then I ask why. Then I, of course, ask it, to fix this original prompt. And it’s turtles (yes, we’re in a zoo and you’re reading it on a Safari) all the way down.
I love me my Cline, Claude Code and company. But there’s major thing I found missing from them — I want my assistant to be able to step with me through a debugger, and be able to examine variables and call stack. Somehow this doesn’t exist. This is helpful for figuring out the flow of an unfamiliar program, for example.
Now, JetBrains MCP Server Plugin gets some of the way there, but… It can set breakpoints but because of the way it analyzes code text it often gets confused. For example, when asked to set a breakpoint on the first line of the method it would do it at a method signature or annotation.
And it doesn’t do anything in terms of examining the code state at a breakpoint.
Let’s talk about MCP. More specifically, let’s talk about using stdin/stdout as a protocol transport layer in the year of our Lord 2025.
Yes, yes—it’s universal. It’s composable. It works “everywhere.” It’s the spiritual successor to Unix pipes, which were cool at the time. The time when my Dad was hitting on my Mom. As an actual transport layer, stdin/stdout is a disaster.
Debugging Is Basically a Crime
Let’s say I want to create an MCP server in Python. Reasonable. Now let’s say I want to debug it. Set a breakpoint. Inspect variables. Use threads. Maybe spin up the LLM in the same process for context. You know, software engineering.
The moment you try to do this, you’re writing a debug driver. Congratulations. You are now:
Building a fake client to simulate a streaming LLM
Implementing bidirectional IO while praying the LLM doesn’t send surprise newline characters
Voitta’s Brokkoly: Thought I could run the LLM and the driver in one process. Spent 3 hours implementing queues, got it half-working, and realized I was debugging my own debug tool.
Samtyzukki: Round two. Same problem. Ended up with more abstraction layers than a Kafka conference.
Eventually, I just gave up and decided to use SSE (Server-Sent Events). Because you know what’s great about SSE? You can log things. You can see the messages. You can debug. It’s like rediscovering civilization after weeks of wilderness survival with only printf() and trauma.
stdout Is Sacred, Until It Isn’t
Here’s the other problem. stdout is a shared space. You can’t count on it. Libraries will write to it. Dependencies will write to it. Your logger will write to it. Some genius upstream will write: print(“INFO: falling back to CPU because the GPU is feeling shy today.”)
Congratulations. You just corrupted your transport. Your parser reads that as malformed JSON or a broken packet or an existential and spiritual crisis.
It’s not a bug. It’s a design decision—and not a good one.
This is the part where I invoke Rob Pike. Sorry. Not sorry.
It totally sucks because it’s not useful performance-wise, too slow. But at least it’s got a connection…
But then again Dremio seems to be real good about it. It appears to work well with blob storage (ADLS on Azure, GCS on GCP, S3 on AWS). Even, in some cases, better than Athena with all the blobs in S3.
Figuring out the reason for the magic number to backtrack from, though, I had posited another reason, and I was wrong… And overall it now reminded me of:
The appearance of our visitor was a surprise to me, since I had expected a typical country practitioner. He was a very tall, thin man, with a long nose like a beak, which jutted out between two keen, grey eyes, set closely together and sparkling brightly from behind a pair of gold-rimmed glasses. He was clad in a professional but rather slovenly fashion, for his frock-coat was dingy and his trousers frayed. Though young, his long back was already bowed, and he walked with a forward thrust of his head and a general air of peering benevolence. As he entered his eyes fell upon the stick in Holmes’s hand, and he ran towards it with an exclamation of joy. “I am so very glad,” said he. “I was not sure whether I had left it here or in the Shipping Office. I would not lose that stick for the world.”
“A presentation, I see,” said Holmes.
“Yes, sir.”
“From Charing Cross Hospital?”
“From one or two friends there on the occasion of my marriage.”
“Dear, dear, that’s bad!” said Holmes, shaking his head.
Dr. Mortimer blinked through his glasses in mild astonishment. “Why was it bad?”
“Only that you have disarranged our little deductions. Your marriage, you say?”
Which in turn reminded me of
И вот какого хрена “Shipping Office” переводится как “пароходство”?
Additionally there seems to be a bug there, per AWS support:
Issue: – The search functionality appears to be unresponsive when using the traditional “Enter” key method – This seems to be a technical bug in the console Workaround: – Enter your search term in the search bar – Instead of pressing Enter, click anywhere on the screen – This should trigger the search functionality and display the results
Search for something like “gen2” actually yields something… It’s a AthenaDataLakeGen2Connector — which is the same thing as below, so read on.
I like Postman in general. But some things are annoying, so there…
APIs and Collections and Environments
APIs are great, and equally great is their integration with GitHub, and ability to generate Collections from API definitions and have them be updated when API definition changes. Nice. Except… those Collections cannot be used to create Monitors or Mock servers, you need to create standalone Collections (or copy those you generated from under APIs). But now those don’t integrate with GitHub. There is a fork-and-merge mechanism that kind of takes care of the collaboration, but that those modes are different is annoying. Ditto Environments. What’s up with that?
I am working on a project (closed source, sorry) that has suffered from proliferation of parameters to constructors, and for this (and other) reasons looks like it would benefit from a DI approach, such as Uber FX.
The code base is pretty large at this point, and manually adapting it is slow, tedious and frustrating. But laziness, impatience and hubris says: why not use features like static analysis using DST and reflection to automate this process?
(Yes, I do indeed wear multiple hats — marketing FTW, or is it WTF?)
Really wanted to use MailJet (BTW, guys, what’s with support and not being able to edit a campaign after launch? Get what I pay for?) to send users a list of items, dynamically. For example, say I have the following list of items, say, “forgotten” in a shopping cart:
So prompt engineering is all the rage (a friend of mine even wrote a book — you magnificent bastard, I haven’t read it yet) — and I have a problem with trying to export GCP configuration for use with Deployment manager. Specifically, how to get CloudRun to work.
So an impressive session (and another one) result with some cool reasoning by ChatGPT in response to me pointing out that things don’t quite work (yes, Amigo Grady, I know it’s not reasoning)… Except that they don’t work.
Which I finally find via StackOverflow — because this feature is unsupported, per Google. But this info is probably outside the training data window of our esteemed LLM.
What’s the moral of the story? There is no moral — use it as a tool. Just a tool.
Yeah, no-code/low-code is great (wave, OpenAI). Especially for growth-hacking, right (hello, Butcher)? But here’s your no-code platform — Google Ads. Gawd… I’d rather write code.
Why does FastAPILoggingHandler seems to ignore my formatter? I don’t know; but the fact that someone else also spends time figuring out the inane things that should just work is quite frustrating.
O, GCP, how convenient: in the env var YAML sent to gcloud run you helpfully interpret things like “on”/”off”, “true”/”false”, “yes”/”no” as numbers, eh? And then you crash with:
ERROR: gcloud crashed (ValidationError): Expected type <class 'str'> for field value, found True (type <class 'bool'>)
Because of course you do.
“Overriding a number of default settings is key to shaving off unnecessary spend”. Yep.
I recently learned about MediaMath’s custom brain. Excited for the validation of OpenDSP‘s concept of custom bidding logic. But it is a bit limited, being just a polynomial. MediaMath’s custom bid router provides way more flexibility — but you need your own infrastructure! So — I still think our approach — DSL-based scripting — is better, because it combines both!
In a previous post, we discussed using Lambda, Glue and Athena to set up queries of events that are logged by our real-time bidding system. Here, we will build on that foundation, and show how to make this even friendlier to business users by connecting BI tools to this setup.
Luckily, Athena supports both JDBC and ODBC, and, thus, any BI tool that uses either of these connection methods can use Athena!
First, we need to create an IAM user. The the minimum policies required are:
If you give different engineers the same problem they will usually produce reasonably similar solutions (mutatis mutandis). For example, when I first came across a reference implementation of an RTB platform using AWS, I was amused by how close it was to what we have implemented in one of my previous projects (OpenRTB).
So it would be not much of a surprise that in the next RTB system, a similar pattern was used: logs are written to files, pushed to S3, then aggregated in Hadoop from where the reports are run.
But there were a few problems in the way…
Log partitioning
The current log partitioning in S3 is by server ID. This is really useful for debugging, and is fine for some aggregations, but not really good for various reasons – it is hard to narrow things by date, resulting in large scans. It is, therefore, even harder to do joins. Large scans in tools like Athena also translate into larger bills. In short, Hive-like partitioning would be good.
To that end, I’ve created a Lambda function, repartition, which is triggered when a new log file is uploaded to s3://logbucket/ bucket:
import boto3
from gzip import GzipFile
from io import BytesIO
import json
import urllib.parse
s3 = boto3.client('s3')
SUFFIX = '.txt.gz'
V = "v8"
def lambda_handler(event, context):
#print("Received event: " + json.dumps(event, indent=2))
# Get the object from the event and show its content type
bucket = event['Records'][0]['s3']['bucket']['name']
key = urllib.parse.unquote_plus(event['Records'][0]['s3']['object']['key'], encoding='utf-8')
try:
s3obj = s3.get_object(Bucket=bucket, Key=key)
src = { 'Bucket': bucket, 'Key': key }
(node, orig_name) = key.split("/")
(_, _, node_id) = node.split("_")
name = orig_name.replace(SUFFIX, "")
(evt, dt0, hhmmss) = name.split("-")
hr = hhmmss[0:2]
# date-hour
dthr = f"year=20{dt0[0:2]}/month={dt0[2:4]}/day={dt0[4:6]}/hour={hr}"
schema = f"{V}/{evt}/{dthr}"
dest = f"{schema}/{name}-{node_id}{SUFFIX}"
print(f"Copying {key} to {dest}")
s3.copy_object(Bucket=bucket, Key=dest, CopySource=src)
return "OK"
except Exception as e:
print(e)
print('Error getting object {} from bucket {}. Make sure they exist and your bucket is in the same region as this function.'.format(key, bucket))
raise e
if __name__ == "__main__":
# Wrapper to run from CLI for now
s3entry = {'bucket' : {'name' : 'logbucket'},
'object' : {'key' : server/requests-200420-003740.txt.gz'}}
event = {'Records' : [{'s3' : s3entry}] }
lambda_handler(event, None)
At that time, the log is copied to a new path is created under v8 prefix, with the following pattern:
<event>/year=<year>/month=<month>/day=<day>/hour=<hour>/<filename>-<nodeID>. For example,
(The “v8” prefix is there because I have arrived at this schema having tried several versions — 7, to be exact).
What about storage cost?
An additional benefit of using date-based partitioning is that we can easily automate changing storage type to Glacier for folders older than some specified time, which will save S3 storage costs of the duplicate data.
In the cloud, the storage costs are not something to worry about; the outgoing traffic and compute is where the problem is at. So move the compute to the data, not the other way around.
NOTE: Partitions do not equal timestamps
Partitioning is based on the file name. Records inside the file may have timestamps whose hour one greater than or one less than the partition hour, for obvious reasons. Thus, partitions are there to reduce the number of scanned records, but care should be taken when querying to not assume that the timestamps under year=2020/month=04/day=21/hour=00all are in the 0th hour of 2020-04-21.
Discover metadata in Glue
Glue is an ETL service on AWS. One of the great features of Glue is crawlers that attempt to glean metadata from the logs. This is really convenient because it saves us the tedious step of defining the metadata manually.
So we set up a crawler. For explanation on how to do it, see the “Tutorial” link on the left hand side of this page:
However, it takes some time to get the configuration correct.
We would want to exclude some logs because we know their format is not good for discovery (they are not straight-up JSON or CSV, etc; and, at the moment, custom SerDes are not supported by Athena) — but see below for exceptions. This is done in the “Data store” part of crawler configuration:
We want Glue to treat new files of the same type as being different partitions of the same table, not create new ones. For example, given the partitioning convention we created above, these two paths:
Should be treated as partitions of table “wins”, not two different tables. We do this on the “Output” section of the crawler configuration as follows:
Once the crawler runs, we will see a list of tables created in Glue.
If here we see tables looking like parts of the partitioned path (e.g., year=2020, or ending with _txt_gz), it means the crawler got confused when discovering their structure. We are adding those to the crawler’s exclusion list, and will create their metadata manually. Fortunately, there are not that many such logs, and we can exclude them one by one.
Of course, while the crawler can recognize the file structure, it doesn’t know what to name the fields. So we can go and name them manually. While this is a tedious process, we don’t have to do that all at once – just do it on the as-needed basis.
We will want to keep the crawler running hourly so that new partitions (which get created every hour) are picked up. (This can also be done manually from Athena – or Hive — by issuing MSCK REPAIR TABLE command).
First useful Athena query
Looking now at Athena, we see that metadata from Glue is visible and we can run queries:
Woohoo! It works! I can do nice ad hoc queries. We’re done, right?
Almost. Unfortunately, for historical reasons, our logs are not always formatted to work with this setup ideally.
We could identify two key cases:
Mostly CSV files:
There are event prefixes preceding the event ID, even though the event itself is already defined by the log name. For example, bid:<BID_ID>, e.g.: bid:0000-1111-2222-3333
A CSV field in itself contains really two values. E.g., a log that is comma-separated into two fields: timestamp and “message”, which includes “Win: ” prefix before bid ID, and then – with no comma! – “price: ” followed by price. Like so: 04/21/2020 00:59:59.722,Win: a750a866-8b1c-49c9-8a30-34071167374e_200421-00__302 price:0.93 However, what we want to join on is the ID. So in these cases, we can use Athena views. For example, in these respective cases we can use:
CREATE OR REPLACE VIEW bids2 AS SELECT "year", "month", "day", "hour", "SUBSTRING"("bid_colon_bid_id", ("length"('bid:') + 1)) "bid_id" FROM bids
and
CREATE OR REPLACE VIEW bids2 AS SELECT "year", "month", "day", "hour", "SPLIT_PART"("message", ' ', 3) "bid_id" FROM wins
Now joins can be done on bid_id column, which makes for a more readable query.
The other case is a log that has the following format: a timestamp, followed by a comma, followed by JSON (an OpenRTB request wrapped in one more layer of our own JSON that augments it with other data). Which makes it neither CSV, nor JSON. Glue crawler gets confused. The solution is using RegEx SerDe, as follows: And then we can use Athena’s JSON functions to deal with the JSON column, for example, to see distribution of requests by country:
SELECT JSON_EXTRACT_SCALAR(request, '$.bidrequest.device.geo.country') country, COUNT(*) cnt FROM requests GROUP BY JSON_EXTRACT_SCALAR(request, '$.bidrequest.device.geo.country') ORDER BY cnt DESC
Success! We can now use SQL to easily query our event logs.
Helpful links
Amazon Athena User Guide on GitHub – official AWS guide but it appears to be organized better than the other
App-ads.txt is a follow-up to IAB’s ads.txt initiative aimed at increasing transparency in the programmatic ad marketplace. Related to it are a number of other initiatives and standards such as supply chain, payment chain, sellers.json, various things coming out of TAG group, etc. See last section for relevant links.
TL;DR: This standard allows an ad inventory buyer (DSP) to verify whether the entity selling the inventory (SSP) is authorized by the publisher to sell it. This is done by looking up the the publisher’s URL for an app (bundle) in the app store, and examining app-ads.txt file at that URL. The file lists authorized sellers of the publisher’s inventory, along with an ID that this publisher should be identified by in the reseller’s system (publisher.id in OpenRTB).
Terminology
Here, the words “publisher” and “developer” may be used interchangeably; ditto for “app” and “bundle”.
First pass implementation
The algorithm is fairly simple: for each app – aka bundle – of interest, grab the app publisher’s domain from the appropriate app store, fetch app-ads.txt file from that domain and parse it. But of course, in theory there is no difference between theory and practice, but in practice there is. In reality, there are some deviations from the standard and exceptional cases that had to be taken care of in the process.
As a first pass, we are running this process semi-manually; if the results warrant full automation, this can easily be accomplished. Here is what was done (more technical details can be found on GitHub):
Bundle IDs and other information was retrieved from request log for the last few days (really, requests for all bids in May as of end of May 6), using a query to Athena. This comes to a total of:
1,144,377 bids
613,241 unique bundle IDs.
685,221 bundle-SSP combinations
The result set, exported as CSV file is loaded into SQLite DB. The same SQLite DB is also used for caching of results.
Go through the list of bundles with a Python script, appadsparser.py.While ads.txt standard provides the specification of how app stores should provide publisher information, only Google Play Store currently follows it. For Apple’s App Store, we crawl its search API’s lookup service, and, if not found there, directly the App Store’s page (though this appears to be a violation of robots.txt).
The result is a semi-structured log. The summary is below.
Results Notes
Result codes
The following are result codes and their meaning:
OK – The SSP we got the traffic on is authorized by the bundle publisher’s app-ads.txt for the publisher ID specified in request
OK_GOOGLE – Google is authorized by the bundle publisher’s app-ads.txt. See below on explanation of why Google is special.
PROBLEMS — Mismatch found between authorized SSPs and/or publisher IDs in app-ads.txt.
NO_APPADS_TXT – Publisher’s website has no app-ads.txt file
NOT_FOUND_IN_PLAY_STORE – bundle not found in Google Play Store. NOT_FOUND_IN_ITUNES – same for Apple App Store.
NO_URL_IN_PLAY_STORE – cannot find publisher’s URL in Google Play Store. NO_URL_IN_ITUNES – same for Apple App Store.
FACEBOOK_URL – publisher’s URL points to a Facebook page (this happens often enough that it warrants its own status) BAD_DEV_URL – publisher’s URL in an app store is invalid
BAD_BUNDLE_ID – store URL (from the OpenRTB request) is invalid, and cannot be determined from the bundle ID either
Note on Google
NOTE: At the moment, there is no way to check the publisher ID for Google due to an internal issue. In other words, we cannot verify the following part of the spec:
Field #2 - Publisher’s Account ID - This must contain the same value used in transactions (i.e. OpenRTB bid requests) in the field specified by the SSP/exchange. Typically, in OpenRTB, this is publisher.id.
Given Google’s aggressive anti-fraud enforcement, we can for now stipulate that it would not run unauthorized inventory. There is still the possibility of fraud, of course. But in the below table we distinguish between bundles served via Google (where we do not check for publisher ID, just the presence of Google in app-ads.txt) and those served via other SSPs, where we cross-reference the publisher ID.
As a corollary of the above you will not see Google inventory under “PROBLEMS” status.
This gives a good sample:
Result code
Count
OK
5,693
OK_GOOGLE
29,330
PROBLEMS
361
NO_APPADS_TXT
7,653
NOT_FOUND_IN_PLAY_STORE
4,417
NOT_FOUND_IN_ITUNES
134
NO_URL_IN_PLAY_STORE
20,112
NO_URL_IN_ITUNES
16,168
FACEBOOK_URL
1,378
BAD_DEV_URL
4
Summary of findings
It appears that due to not very high adoption of this standard at present (developer URLs not present in App Stores or app-ads.txt file not present on the developer domain), there is not much utility to it at the moment. However, as the adoption rate is increasing (see below), this is worth revisiting again. Also, consider that for this exercise we only used bids information – that is, not the sample of full traffic, but just what we have bid on. This may not be representative of the entire traffic also, and may be interesting to explore.
Consider also that the developer may well have the app-ads.txt file on the website, but if the website is not properly listed in the app store, we have no way of getting to it (yet SSPs may include those sites in their overall numbers, see, e.g., MoPub below).
What does the industry say?
Google Play Store is reported to have only ~8% adoption. Worth quoting here is this section of the report:
Who Are the Top ‘Direct’ Ad Partners Inside the App-Ads.txt in Google Play Apps?
Direct ad partners are those which have been granted direct permission by app developers to sell app ad space. That is, they are explicitly listed on the app’s “App-ads.txt” file. Google.com is listed as a direct ad partner on 95.87% of all “App-Ads.txt” files for Google Play apps. This makes it the most frequently mentioned direct ad partner for apps available on the Google Play.
Doesn’t seem that app-ads.txt makes that much difference for IVT (invalid traffic) – apps with app-ads.txt have 18.7% of IVT vs 21.1% for those without (pg. 6), despite Pixalate dramatizing this 2.4 percentage point increase as a 13% increase (2.4/18.7 – lies, damn lies and statistics).
It lists way higher numbers of adoption for Google Play Store apps than above but that is across top 1K apps.
Increasing rates of adoption (~65% rise in Q4 2019 – pg.13)
Unity, Ironsource, MoPub, Applovin, Chartboost – in that order – are in the top direct ad partners for Android apps (pg.18); MoPub, Unity, IronSource – in top direct AND resellers for iOS (pg.20)
MoPub claims that “app-ads.txt file adoption exceeds 80% for managed MoPub publishers”. It’s unclear what the qualifier “managed” means. Sampling our data, we have issues with app-ads.txt on MoPub about 48% in total, breaking down as follows:
No app-ads.txt: 11%
No developer URL found in store: 22.5%
Not found in app store: 13.7%
Not found?
Additionally, it is worth looking at the bundles that are flagged as NOT_FOUND_IN_PLAY_STORE or NOT_FOUND_IN_ITUNES – how come those cannot be found?
This does not have to be something nefarious, for example, based on spot-checking, it can be due to:
Case-sensitivity. Play Store bundles are case-sensitive but an SSP may normalize them to lower case when sending (e.g., com.GMA.Ball.Sort.Puzzle becomes com.gma.ball.sort.puzzle)
Some mishap like non-existent com.rlayr.girly_m_art_backgrounds being sent (but com.instaforall.girly_m_wallpapers exists)
But even if the majority of the NOT_FOUND errors are due to such discrepancies, not fraud, it means that currently app-ads.txt mechanism itself is de facto not very reliable.
Some other results
Q: Are there any apps that present as different publishers on the same SSP?
A: Not too many (about 8.6%), but even so for most important apps it is at most 2-3 – and even at long tail the max is 10 publisher IDs. This, though, can still account for some app-ads.txt PROBLEMs as seen above)
As someone who offers a REST API, we at Romana project wanted to provide documentation on it via Swagger. But writing out JSON files by hand seemed not just tedious (and thus error-prone), but also likely to result in an outdated documentation down the road.
Why not automate this process, sort of like godoc? Here I walk you through an initial attempt at doing that — a Go program that takes as input Go-based REST service code and outputs Swagger YAML files.
It is not yet available as a separate project, so I will go over the code in the main Romana code base.
NOTE: This approach assumes the services are based on the Romana-specific REST layer on top of Negroni and Gorilla MUX. But a similar approach can be taken without Romana-specific code — or Romana approach can be adopted.
The entry point of the doc tool for generating the Swagger docs is in doc.go.
The first step is to run Analyzer on the entire repository. The Analyzer:
Walks through all directories and tries to import each one (using Import()). If import is unsuccessful, skip this step. If successful, it is a package.
The next step is to run the Swaggerer — the Swagger YAML generator.
At the moment we have 6 REST services to generate documentation for. For now, I’ll explicitly name them in main code. This is the only hardcoded/non-introspectable part here.
From here we, for each service, Initialize Swaggerer and call its Process() method, which will, in turn, call the main workhorse, the getPaths() method, which will, for each Route:
Get its Method and Pattern — e.g., “POST /addresses”
Get the Godoc string of the Handler (from the Godocs we collected in the previous step)
The combination of Negroni and Gorilla MUX is a useful combination for buildng REST applications. However, there are some features I felt were necessary to be built on top. This has not been made into a separate project, and in fact I doubt that it needs to — the world doesn’t need more “frameworks” to add to the paralysis of choice. I think it would be better, instead, to go over some of these that may be of use, so I’ll do that here and in further blog entries.
This was borne out of some real cases at Romana; here I’ll show examples of some of these features in a real project.
Overview
At the core of it all is a Service interface, representing a REST service. It has an Initialize() method that would initialize a Negroni, add some middleware (we will see below) and set up Gorilla Mux Router using its Routes. Overall this part is a very thin layer on top of Negroni and Gorilla and can be easily seen from the above-linked source files. But there are some nice little features worth explaining in detail below.
In the below, we assume that the notions of Gorilla’s Routes and Handlers are understood.
Sort-of-strong-dynamic typing when consuming data
While we have to define our route handlers as taking interface{} as input, nonetheless, it would be nice if the handler received a struct it expects so it can cast it to the proper one and proceed, instead of each handler parsing the provided JSON payload.
To that end, we introduce a MakeMessage field in Route. As its godoc says, “This should return a POINTER to an instance which this route expects as an input”, but let’s illustrate what it means if it is confusing.
Let us consider a route handling an IP allocation request. The input it needs is an IPAMAddressRequest, and so we set its MakeMessage field to a function returning that, as in
Sometimes we prototype features outside of the Go-bases server — as we may be calling out to various CLI utilities (iptables, kubectl, etc), it is easier to first ensure the calls work as CLI or shell scripts, and iterate there. But for some demonstration/QA purposes we still would like to have this functionality available via a REST call to the main Romana service. Enter the hooks functionality.
Whether to run it before or after the Route‘s Handler (When field)
Optionally, a field where the hook’s output will be written (which can then be examined by the Handler if the When field is “before”). If not specified, the output will just be logged.
Then during a request, they are executed by the wrapHandler() method (it has been described above).
That’s it! And it allows for doing some work outside the server to et it right, and only then bother about adding the functionality into the server’s code.
If this doesn’t seem like that much, wait for further installments. There are several more useful features to come. This just sets the stage.
A major feature of the Romana Project is topology-aware IPAM, and an integral part of it is the ability to assign consecutive IP addresses in a block (and reuse freed up addresses, starting with the minimal).
Since IPv4 addresses are essentially 32-bit uint, the problem is basically that of maintaining a sequence of uints, while allowing reuse.
To that end, a data structure called IDRing was developed. I’ll describe it here. It is not yet factored out into a separate project, but as Romana is under Apache 2.0 license, it can still be reused.
The IDRing structure is constructed with a NewIDRing() method, that provides the lower and upper bound (inclusive) of the interval from which to give out IDs. For example, specifying 5 and 10 will allow one to generate IDs 5,6,7,8,9,10 and then return errors because the interval is exhausted.
Optionally, a locker can be provided to ensure that all operations on the structure are synchronized. If nil is specified, then synchronization is the responsibility of the user.
To get a new ID, call GetID() method. The new ID is guaranteed to be the smallest available.
When an ID is no longer needed (in our use case — when an IP is deallocated), call ReclaimID().
A useful method is Invert(): it returns an IDRing whose available IDs are the ones that are allocated in the current one, and whose allocated ones are the available ones in the current one. In other words, a reverse of an IDRing with min 1, max 10 and taken IDs from 4 to 8 inclusively is an IDRing with the following
available ranges: [1,3], [9,10].
I am addicted to having good development tools, specifically, I like my debuggers visual. Once I was so annoyed by inability to debug a common use case of calling a PL/SQL stored procedure from Java, I made it into a single-stack cross-language debugger project, for example.
And I’m also a fan of Eclipse (well, see above).
So when Romana went with Go (pun definitely intended) as our language of choice, I quickly found GoClipse, and it was nice.
But then I learned about Delve and I wanted to have the best of both worlds — the functionality of Delve and the graphics of GoClipse.
So here’s an idea… Eclipse (from which GoClipse is, obviously, derived) understands GDB/MI, and can represent it visually. So all that remains is to adapt Delve’s API to GDB/MI.
And so I started it as a Kabuta project. Let’s hope I don’t abandon it…
Here I will describe the Nanoput project, which comprises a large part of OpenDSP’s DMP (Data Management Platform). There are, of course, other pieces — the entire picture will be painted under the DMP tag.
Overall DMP stack is as follows:
Nanoput proper: NGINX with Lua to handle . I’ll confess that at the moment this is a bit of a pet (as in not cattle). We are considering using OpenResty instead of rolling our own, which uses parts of OpenResty. But no matter, here I will present some features that can be achieved with this setup — and one instance is capable of handling all this.
Redis for storing and manipulating user sets — ZSET is great
MySQL for storing metadata — will be described in a separate post
PHP/JS for a simple Web interface to define the said metadata
Python for translating metadata into the configuration for NGINX
AWS S3 for storing raw logs — pre-partitioned so that EMR can be used easily.
Conceptual introduction
Conceptually, let’s consider the idea of an “event”. An impression, a conversion, a video tracking event, a site visit, etc, is an event — anything that fires a request to our Nanoput is. You may recognize a similarity with Snowplow — and that is because we are solving a similar problem.
To proceed further, I must ask of a little indulgence — please bear with me. As a technologist, it really grated me to hear the words like “Javascript pixel” but I have learned to stop worrying and love the bomb. Therefore, Javascript pixel it is when it is a JS code that fires some GET URLs. Now, then, the requests are assumed to be fired as GET HTTP requests (do not have to be but that’s the primary idea behind Nanoput per se — management of metadata to ingest things via other means, like FTP upload, etc, will be addressed separately) and can originate, for example, from:
Exchanges or DMPs, as an exchange-initiated cookie-sync: see below.
Regular user behavior — impressions, in case of video, video-tracking events; conversions
Calls as if initiated by exchanges or DMPs to Nanoput but in reality are, heh, Javascript pixels
Now, let us also consider the idea of a “user segment”. If you think about it, a segment is just a set of users. Thus, we may as well consider a user that produced a certain event as belonging to some segment. These may be explicitly asked for, such as “users we want to retarget“, or “users who converted”, etc. But there is no reason why any event cannot be defined as a segment-defining one.
Segments, here, are a special case of data collections concept discussed in a different post.
Given that, we can now dive into Nanoput implementation
General data acquisition idea
Here, we simply leverage basic NGINX functionality, that is logging. To that end, we split the main config file into sections that we include that deal with log format and location and behavior.
Static data acquisition URLs
By “static”, here we mean common use cases that are just part of Nanoput (hence the “man” subdirectory you will notice in examples — stands for, just like the Unix man command, for “manual”). Here we have:
Site events (essentially, those are an extension of retargeting concept).
Standard event tracking — by which we mean, standard events that happen in Ad world.
Notice that we also augment information available from NGINX (HTTP headers, etc.) with geo data using GeoIP module and user-agent/device/OS
Dynamic (metadata-driven) data acquisition URLs
Dynamic data acquisition works simply: a process reads the metadata table and creates appropriate entries in the NGINX configs that define log format and location and behavior.
Creating “segments”
On every “event”, consider script. We use awesome Redis’s Sorted Set functionality here inserting things twice. The key idea here, again, is a variation on dealing with data gravity concerns by just duplicating storage. We create two sorted sets for each key, the “score” being the first and last time we have seen the user. The reasoning for this is that:
First-seen: we can write batch scripts to get rid of users we have last seen over X days ago (expiring targeting).
Duplication is not just for every user — it is for every set. The key here is the set (or segment) name, and the value is the set of users.
An added benefit of this is that new segments can be created by using various Redis set operations (union, intersection) easily.
Some useful shortcuts for a DMP
Getting OS/browser info without necessarily using WURFL (though that can easily be fronted by NGINX too, actually).
Exchange cookie sync
In the display world, there is a need for cookie syncing between DSP and a third-party DMP or an exchange/SSP, and that can be either exchange, DMP or DSP-initiated, or both. Some exchanges may allow the redirect chain to proceed further, some may not. Nanoput provides this functionality for exchanges we deal with as well as a template for doing it for other partners — at the speed that NGINX provides. Here are the moving parts:
DSP itself may initiate a cookie sync, and this is controlled by xchout.lua script.
Storing for further analysis
Raw logs, formatted as above, are uploaded to S3. Notice that they are stored twice, with different partitioning schema. This is one of the key ideas in Nanoput — storage is cheap; duplicating the storage this way and then using one or another partitioning schema depending on the use case:
Partitioned by user — here, it’s important to see that it’s a multi-tenant system; in this context, “user” is a client/customer. This partitioning is useful to provide customers with ability to run their own custom queries. See also notes on data duplication and data gravity.
Ok, I couldn’t resist the title, and also too, Billy Collins’ line about “No cookie nibbled by a French novelist could send one into the past more suddenly” is just awesome so why not… But I digress…
At OpenDSP, we, obviously, cookie (yes, it is a verb — to cookie) users. While more details can be had in other posts (such as Nanoput, and more under DMP tag), here I’ll just address the format issue.
The cookies we use are known as mod_uid cookies because of the Apache project. (A good writeup is done here). Because we use NGINX, and, consequently, use its uid_set/uid_get functionality, one may be puzzled when reconciling the cookie as seen sent to the user and what NGINX writes in its logs. So here
Go: convert cookie to hex — if you are wondering why Go, Go’s key feature of compiling to a standalone binary — which can then be used easily anywhere — is why.
Give engineers the same problem, they all will come with roughly the same solution. But I am proud to actually have invented a different one.
Many systems, such as ad servers, RTB bidders, DSPs, etc., store the business rules (targeting, pacing, etc.) information that a user manipulates via UI in some sort of database (likely RDBMS), and then probably transform it into some sort of fast lookup for the real-time decision making. Seems straightforward.
At OpenDSP we went for a new approach which I believe is quite powerful. Simply put, we compile the rules from DB into a Groovy script (well, a set of Groovy scripts). Given that Groovy compiles to JVM bytecode, essentially we create concrete subclasses conforming to the following interfaces and using the following abstract classes:
For ads, that is, targeting, implementing Ad interface and subclassing AdImpl
For creatives, implementing Tag interface and subclassing TagImpl
Finally, there is BidPriceCalculator interface, which will be described below. This is not auto-generated.
A script that compiles the DB rules into the scripts is triggered by the UI whenever anything is saved, and the Lot49 process will reload any changed scripts every so often (e.g., 5 minutes).
Generally, the algorithm is as follows: During a request, the system will take the OpenRTB bid request and augment it with various data (e.g., geo lookup, information on user, etc). It will then, for each Ad, call its canBid1() and checkSegments() methods (both must return true). canBid1() is called first, to filter out quickly those ads that won’t bid based on the data already available in the request, before we augment it with data that needs to be fetched from a user cache; after that, checkSegments() will be called on the remaining candidates. In turn, canBid1() will call the canBid() methods of all Tags, to check which, if any, creatives fit this bid request (based on media type, size, video duration if applicable, etc.)
Finally, the price is determined based on pacing and budget settings, unless the appropriate BidPriceCalculator is found for an Ad, in which case it is used to get the bid price.
The resulting scripts are essentially a bunch of getters based on the rules in the DB, and the abstract classes implementing the interfaces are just a bunch of if/then statements using those getters. So, what is the advantage of this rather than just reading rules from the DB?
The advantage is that any part of the generated script can be modified manually for custom targeting and/or bid pricing based on some model. These scripts may be edited by data scientists independently, eliminating the need for engineers to translate data scientists’ models into code! All the data that the model needs will be provided in the arguments to the appropriate methods; just implement the interface in Groovy, and you’re done!
It is important that all data is provided in the input, so the scripts do not need to concern themselves with high-latency fetching of data from somewhere. It is also safe, even if the data scientist makes a mistake. Consider: because we run in JVM, we can take advantage of the Java security model and create our own SandboxSecurityManager to prohibit network calls, harmful calls such as System.exit(), only allow it to call helpers from certain packages, etc.
One caveat, however: security model is not much help for other harmful things such as recursion or infinite loops. The idea, however, is to solve those as follows (about which we’ll talk in a later post):
Recursion: by using static analysis on the loaded code
Loops: either by doing the same and prohibiting loops completely, or, if undesirable, by observing which scripts run longer than some time and blacklisting those that exceed the threshold.
Let’s look at some examples:
Here is an Ad targeting US and one of the segments, “386:fp:236” or “387:fp:236”.
Quick note: “fp” here indicates it is a first-party segment; for more information on how these are created and managed securely in a multi-tenant environment, see Nanoput and other articles on our DMP elsewhere on this blog.
A creative for this ad is a 300×250 banner. Notice the convention of how the ID of the Ad is included in the name of the Tag. This is for demonstration purposes, and thus a lot of code is unrolled here; in reality, methods like getClickRedir() or getTagTemplate() would just be used from the TagImpl class
Finally, BidPriceCalculator (you can see it is defined in the Ad). As can be seen, it uses a formula based on a segment score and a viewability score from Integral to come up with a price. (This, of course, is just an example.) Notice that the information about the user’s score, which is part of the DS-built model, and Viewability score are part of the augmented request that is the argument to getBidPrice() method. In other words, as noted above, the code here needs to just execute a formula, and all the data will be provided to it; this allows us to sandbox this code for safety while allowing for flexibility.
Pretty cool, if I may say so myself.
P.S. You may feel free to contrast this approach with Xandr’s Bonsai.
Once upon a time, I was working at Snaplogic, and at that time its office was in downtown San Mateo. Pretty much across the street from a great coffee shop, Kaffeehaus.
If you’ve been there, or even if you just looked at the website, you’d realize that the owner put quite an effort into it being a Viennese-style coffee house, with all the interior design decisions that go with it.
Now, a local coffee shop is often a place where people expect to post some local notices and ads (“lost dog”, “handyman available”, “local church choir concert”, etc). And here’s a conundrum. A simple cork bulletin board with a bunch of papers pinned to it just did not seem to fit the overall mood/interior/decor of the cafe:
Yet the cafe does want to serve local community and become an institution.
This being Silicon Valley, Val, the Kaffeehaus owner, had a vision — what about a virtual board, as a touch-screen.
The name was quickly chosen to be Wildboard — because it is, well, a bulletin board and in honor of the boar’s head that is prominently featured on the wall:
.
A multi-touch-based virtual bulletin board sounded interesting. Most touch-screen kiosks I’ve seen so far — in hotels and malls, for instance, or things like ImageSurge — only allow tap, not true multi-touch. (To be honest, multi-touch may or may not be useful — but see below and see also P.S. — but it is a very nice “pizzazz”).
And we — that is, myself and Vio — got to work. And in short order we had:
Fully multi-touch (with rotation, zoom, etc) web UI — as a Windows 8 CSS/Javascript app (source).
Wildboard “board server” — a Python app running on the same computer as the UI. It is responsible for polling the web server (below) and serving information to the UI (source).
Wildboard web server — a PHP app based on an existing web classified application(source). This allows users to submit ads (or they can do it via a mobile app, as below). It is also modified to automatically create QR codes based on user-provided information (map, contact, calendar, etc) and adds them to an ad.
Wildboard mobile app — PhoneGap/Cordova based app for both Android and iPhone (source)
This app allows one to:
Post an ad
Scan an ad’s QR code
And, finally, for the “Wow!” effect during the demo, one can drag an ad from the screen into the phone. Here it is, in action:
Wildboard orchestrator — a Node.js app (source) designed to coordinate interactions between the mobile app and the board. It is the one that is determines which mobile app is near which board and orchestrates the fancy “drag” operation shown above.
Every marketer, it seems, wants to participate in real-time bidding (RTB). But what is it that they really want?
They want an ability to price (price, not target!) a particular impression in real-time. Based on the secret-sauce business logic and data science. Fair enough.
But that secret sauce, part of their core competence, is just the tip of the iceberg — and the submerged part is all that is required to keep that tip above water. To wit:
Designing, developing, testing and maintaining actual code for the UI for targeting, the bidder for bidding, reporting, and data management
Scaling and deploying such code in some infrastructure (own data center,
clouds like AWS, GCE, Azure), etc.
Integrating with all exchanges of interest, including the following steps:
Code: passing functional tests (understanding the exchange’s requirements for parsing request and sending response)
Infrastructure: ensuring the response is being sent to the exchange within the double-digit-millisecond limit
Scaling: As above, but under real load (hundreds of thousands of queries per second)
Business: Paperwork to ensure seat on the exchange, including credit agreements when necessary
Operations: Ongoing monitoring of the operations, including technical (increased latency) and business (low fill level, high disapproval level) concerns (whether these concerns are triggered by clients, exchange partners or,
ideally, pro-actively addressed internally.
None of which is their core competence. We propose to address the underwater part. It’ll be exciting.
Enter OpenDSP. We got something cool coming up here. Stay tuned.
I decided it would be good to have a coverage report of our Python code,
with nice visualization like Clover.
So I took Ron Smith’s PyAntTasks, and added py-cover task to them. This will run coverage for every test, and a cumulative one. In other words, you can see what code a particular test exercises, and what code all the tests in your tree exercise.
This also modifies py-test task to include packagedtests attribute – see below.
The newly added py-cover task runs Ned Batchelder’s coverage.py (download it separately), and is specified as follows in your build.xml:
Here, is a FileSet specifying tests to run, is a FileSet specifying source code to cover.
The attributes are:
reportsDir – where the coverage reports go
packagedtests – this idiosyncrasy is prompted by our tree setup. If this attribute is true it means that the test files reside in Python packages, false otherwise. (In our case, they do not; they are in the tree but are not packages. Note that the original py-test task assumed they are in packages, I have changed this too).
coverage – path to coverage.py on your system (which you downloaded separately, right?)
P.S. I know about the colorize.py thingie, but I rolled my own (uglier, of course) for this one.
DISCLAIMER: as I’ve mentioned, I hacked this from the original when I was just starting with Flex 2, so I gutted some code before I knew what the hell I was doing, but in this state it works for me…
I use Flex Builder. I have the contents of the zip file right in my Flex Builder project. (NOTE: This was before I read that flex should not be a part of a package name per Adobe’s license; change it, willya?)
Then I choose have a simple facade like so:
package com.qbf.flex.ct {
// Insert correct imports here, of course...
public class Utils {
public static function serverCall(method:String,
params:Array,
handler:Function): void {
SERVER_CALL.call(method,
params,
handler,
DisplayObject(Application.application));
}
public static const SERVER_CALL:XmlRpcService =
new XmlRpcService("http://localhost:8081/",
Application.application.url.indexOf("http://") == 0? null :
null :
// This is explained further below...
QbfFakeResponse.SINGLETON);
}
And from everywhere else in the code I use Utils.serverCall() as
follows:
var params:Array = [{value : resdefDataToServer, type: "struct"},
uri, server];
Utils.serverCall("doStuff", params,
// This callback will be executed when the server
// call returns
function(result:Object):void {
process(result);
});
Notice that you only need to specify types for parameters if it’s not a string. If all your parameters are string, you can just pass an array of their values to the serverCall() –a little sugar…
Now about this QbfFakeResponse.SINGLETON business (it’s not necessary, just use null in its place if you don’t need it). The definition ofSERVER_CALL checks whether the application URL starts with http:// — in my case, it means that it’s running “for real”. If it is running in Flex Builder’s debugger, the URL will start with file://,
and so I will use “fake” responses to simulate server calls (this is sometimes useful for debugging, so that I don’t have to run a server). This fake response class is an implementation of com.qbf.flex.util.xmlrpc.FakeResponse interface. All you need to do is implement getResponse() method and return whatever you need by checking the method name and parameters provided.
Dbdb project is officially up for adoption, because I have no plans for working on it (I am sick of it).
Dbdb is a proof-of-concept of a JPDA-based single-stack debugger for mixed-language programming, done as an Eclipse plugin (but doesn’t have to be). It is based on Java 6 (“Mustang”). The proof-of-concept is allowing a developer to debug Java code that calls a PL/SQL stored procedure. The debugging session in Java proceeds normally, nothing to write home about. When a Statement.execute() (or similar) statement is executed, however, the debugger connects to the Oracle’s VM and shows a combined call stack, from Java down into PL/SQL. (See screenshot). The idea, of course, that it can be done with other combinations, but Java-into-Oracle-stored-proc is a very common scenario.
P.S. This is a rehash of an older post. I am trying to see what Blogger is like vs. LJ (for instance, LJ breaks javablogs feeds).
Prof.Madden finally approved the latest version of Dbdb write-up, and so I am all set for my 10+-years-overdue degree. With that, I’ve updated the sourceforge project
with all the latest stuff from my workspace, including the docs on the page, Javadoc, code (and aforementioned docs also) in CVS, etc (even a screenshot). Dbdb project is officially open for adoption, because I have no plans for working on it (I am sick of it). Fly, baby, fly…
P.S.
I have to see whether Pat and Spencer actually decided to use this one for the IDEA Plugin Contest… There’s still time…
Maybe I do want to augment it for use with GWT, so it automagically inserts a debugger; statement as the first
line any native Javascript method… Just for kicks… Nah, it would be too slow…
Class of evt: class com.sun.tools.jdi.EventSetImpl$MethodExitEventImpl Methods of evt: [ public com.sun.jdi.Value com.sun.tools.jdi.EventSetImpl$MethodExitEventImpl.returnValue(), public java.lang.String com.sun.tools.jdi.EventSetImpl$LocatableEventImpl.toString(), public com.sun.jdi.Method com.sun.tools.jdi.EventSetImpl$LocatableEventImpl.method(), public com.sun.jdi.Location com.sun.tools.jdi.EventSetImpl$LocatableEventImpl.location(), public com.sun.jdi.ThreadReference com.sun.tools.jdi.EventSetImpl$ThreadedEventImpl.thread(), public int com.sun.tools.jdi.EventSetImpl$EventImpl.hashCode(), public boolean com.sun.tools.jdi.EventSetImpl$EventImpl.equals(java.lang.Object), public com.sun.jdi.request.EventRequest com.sun.tools.jdi.EventSetImpl$EventImpl.request(), public com.sun.jdi.VirtualMachine com.sun.tools.jdi.MirrorImpl.virtualMachine(), public final native java.lang.Class java.lang.Object.getClass(), public final void java.lang.Object.wait(long,int) throws java.lang.InterruptedException, public final void java.lang.Object.wait() throws java.lang.InterruptedException, public final native void java.lang.Object.wait(long) throws java.lang.InterruptedException, public final native void java.lang.Object.notify(), public final native void java.lang.Object.notifyAll()] java.lang.NoSuchMethodError: com.sun.jdi.event.MethodExitEvent.returnValue()Lcom/sun/jdi/Value; at org.hrum.dbdb.DriverManagerMethodExitEventListener.process(DriverManagerMethodExitEventListener.java:99) at org.hrum.dbdb.DbdbEventQueue.removeDebug(DbdbEventQueue.java:168) at org.hrum.dbdb.DbdbEventQueue.remove(DbdbEventQueue.java:47) at org.eclipse.jdt.internal.debug.core.EventDispatcher.run(EventDispatcher.java:226) at java.lang.Thread.run(Thread.java:619) Returned: instance of oracle.jdbc.driver.T4CConnection(id=435)(class com.sun.tools.jdi.ObjectReferenceImpl)
Now, I will run this in debug mode and set a breakpoint at the red line above. When the breakpoint is hit, evaluation of evt.returnValue() returns an instance of com.sun.tools.jdi.ObjectReferenceImpl. However, when the execution is resumed, the result is as above (that is, evt.returnValue() results in a NoSuchMethodError).
Further, if we remove the green line (retvalMethod.setAccessible(true);), we will get an IllegalAccessException on the invocation:
Class org.hrum.dbdb.DriverManagerMethodExitEventListener can not access a member of class com.sun.tools.jdi.EventSetImpl$MethodExitEventImpl with modifiers “public”
What is going on?
I’d say it’s left as an exercise for the reader, but honestly, at the moment, I don’t feel like looking for an answer at all. I will perhaps let Bob and Dr. Heinz Max Kabutz (did I mention how much I enjoy referring to Dr.Heinz Max Kabutz?) to do the detective work…
ENVIRONMENT: This code is part of a plug-in project I am running in Eclipse 3.2RC3, with Mustang.
I am backdating this entry, so I don’t remember quite what this was about…
E-mailed Shane:
Hi,
I am working on a project that necessitates precisely what you describe — Proxies of classes, not just interfaces. So thank you, that I, apparently, don’t have to delve into BCEL (I will let you know about the project when I have a bit more to show than I do now).
However, I’m a bit confused. In a description at http://www.gnufoo.org/java/java.html you point to your junit.extensions.jfunc.util.ProxyPlus class, but it does not exist in the jcfe.jar I got from http://www.gnufoo.org/jcfe/, nor does it exist in the jfunc project. Where should I look for this functionality?
Thanks.
Got a bounce. So, for now, will do with explicit org.hrum.dbdb.JDBCDriverFactory‘s mechanism of going through drivers loaded, deregistering them and registering, in their stead, org.hrum.dbdb.DbdbDriver‘s that delegate to the registered drivers.
P.S. This is what P6Spy does