voitta-yolt

The Call Was Coming From Inside the House

Gregory Golberg holding forth, voitta , , , , , , , , , ,

This is going out on a Thursday night — the eve of the Friday news dump, that fine tradition of publishing what you’d rather nobody read too closely.

That is not why. We could have held it for Monday morning and the numbers would have been better. We didn’t want to sit on it, and the joke lands a day early. Look at what we’re willing to put in front of you.

Here’s the confession: we built a safety tool, and for a while it was the least safe thing on the machine.

YOLT is our little guardrail — it looks at each command an agent is about to run and decides whether it’s risky. Useful. The problem is that it also wrote down every command it inspected, verbatim, into a log under your home directory. Forever. No rotation. On by default.

You can see where this goes. Commands carry secrets: an API key pulled from a vault and handed to the next curl, a bot token, a connection string. So our safety tool quietly accumulated a plaintext pile of exactly the things it existed to protect, sitting in the one file nobody thinks to grep. Because who audits their seatbelt.

It got better. The reviewer that reads those logs copied the same raw lines into two more files. And the leak was never confined to our tool: a credential on a command line also lands in the agent’s own session transcript, which is a much larger and much quieter surface. The blast radius was bigger than the bug.

Then we swept everything, and it got worse

Once we started looking properly — every agent directory on every machine — the pattern that came back wasn’t carelessness. It was housekeeping.

A collaborator’s cloud key was rotated by an agent session. create-access-key printed the new secret to stdout, and the session wrote stdout to a transcript, where it sat in plaintext for a month. The rotation produced a longer-lived exposure than the thing it was fixing. A Slack webhook minted to replace one leaked in git history then leaked into a transcript itself.

The dangerous moments are the tidy ones. Rotation is the riskiest thing you do all quarter, precisely because new key material is briefly in the open, and an agent session is a permanent plaintext record of everything that crossed it.

Our favourite: halfway through the audit we checked which token the audit itself was using. It was the leaked one. The rotation had updated the config file and not the already-running shell, so the tool hunting the compromised credential was authenticating with it.

The retirement gap

Most of what we found was in sessions of software we had already decided to kill.

That gap — between deciding to retire something and actually retiring it — is where credentials rot. Nobody audits the tool on its way out. Nobody rotates for it. It keeps its tokens and keeps writing transcripts, right up until the plug comes out. Every team reading this has that gap open right now.

Retiring OpenClaw here was one of those decisions, and to be clear it was about our needs, not a verdict on the project. Vi versus emacs, and who cares. What we care about is a working deliverable arrived at the way we agreed; how you got there is your business.

Everything failed quietly

The cleanup taught us more than the leak did, mostly about no-ops that look like success.

Liveness probes lie, in at least five distinct ways we hit: a search API that answers 422 for valid and invalid keys alike; a catalogue route returning 200 with no authorization header at all; a live key that is merely spend-capped answering 400; a deleted webhook 404ing where a live one 400s; and an endpoint that validated the model name before the credential, so it gave identical answers for a live key, a dead key, and our deliberately invalid control. Carry a known-bad control, hit a route that enforces auth, and read the body, not the status line.

Then the same lie turned up somewhere we weren’t even looking. Our Slack agent runs a multi-vendor waterfall, and LiteLLM picks vendor cooldowns by HTTP status — it cools 429, 401, 408 and 404. Budget exhaustion is none of those: Anthropic answers a spend cap with 400, OpenRouter answers no-credit with 402. So a vendor that had been dead for weeks got dialled first on every single message, failed, and only then did the chain fall through. Fallback worked perfectly; nothing ever learned. That one is filed upstream. A status code is not a diagnosis, and that holds well past credential probes.

The scrubbing was the same shape. A fingerprint recorded from a truncated regex match can never match the value it was meant to track, so the scrubber reports the file clean while the secret sits in it. A post-scrub grep for the pattern returns hits forever, because docs and test fixtures share the pattern — a clean run looks failed. Nothing errors. You only find these by checking the thing itself instead of the report about the thing.

What’s shipped

The redaction that should have been there on day one is now day one for real: YOLT redacts before it writes, and the current release carries it. There’s also an advisory session hook that warns when a credential rides along on a command line.

The techniques are open source, because they’re the genuinely useful part. Our skills catalog, skillz, shipped v1.15.0 with everything above written down: agent-session-credential-audit for the sweep, the false-positive taxonomy, the probe rules and the kill-list scrubber that structurally cannot erase a live secret, and agent-credential-leak-surfaces for the places copies quietly accumulate. Install them, or just read them and steal the parts you want.

One surface we’d missed entirely and you probably have too: the OS keychain. No filesystem sweep will ever see it. Ours held a stale token and handed it over to a pipe with no prompt at all.

shmobster, the Slack agent we introduced in July, took the brunt of it. It had never been tagged at all; it now has six releases, every one cut in the day since this audit started, and the first exists only because the audit went looking for what we ship and found nothing versioned.

The one that matters here is redaction. The agent hands command output straight back to a channel, and cat, env and printenv are read-only, so they clear the safety gate and run with no approval at all. Not theoretical: we found a live config holding literal keys, where a single cat would have posted all five of them to Slack. Everything the agent says is now scrubbed before it leaves the process — reusing YOLT’s redactor rather than a second pattern list that would drift from it, plus this process’s own secrets matched by exact value, because the one thing a generic detector cannot know is which strings are yours. Scrubbing happens at collection, so the model’s own context never holds a credential it could repeat later.

It also learned to read skills, so that catalog now reaches the agent actually sitting in the channel with you, files unchanged. And v0.5.1 fixed a bug from precisely this post’s family: an unguarded loop over channels, where one stale channel id sorted first, aborted the rest, and sent the previous release’s announcement to none of the four healthy channels. The only trace was a traceback about the one channel that failed.

On our end we’re rotating and scrubbing. Assume-compromised is cheaper than assume-fine.

The lesson isn’t subtle, which is exactly why it stings: the safest-looking place is the least-swept. A security tool is the last thing anyone suspects of being a liability, so it’s the perfect place for one to hide. We wrote a guardrail and forgot that a guardrail with a memory is a ledger.

So, two asks. First: if you run YOLT or any of our skills, update — the versions that close this are shipped. Second: come pound on us. Find the next hole, open the issue, tell us where else we’re being careless. We would much rather hear it from you than from a log file.

That’s the whole trade: we screw up in public, you get to keep us honest, and everyone’s tooling gets a little safer. We take the work seriously. Ourselves, less so — hence Friday’s eve, a slot we’re using as a punchline rather than as cover.

Blow-by-blow in the copious links above. Have a good weekend.

Recursive self-improvement, you said?

Gregory Golberg voitta , , , , , , ,

Spawning a fleet of coding agents is a solved problem. You write a for loop, you call the Agent tool N times, you go get coffee. The unsolved problem is everything wrapped around the spawn: deciding what can actually run in parallel, stopping the agent that wrote the code from also grading (or eating) its own homework, and — the part nobody ships — recording how the run went so the next one isn’t the same run with the same mistakes.

I wish I didn’t remember this anymore but this used to be called a “retrospective” in that sect I was once a member of.

I’ve been dogfooding a small orchestration skill (agent-team-orchestration, open in voitta-ai/skillz) that treats those as the actual work. Three runs in. This is the first write-up, warts very much included — the warts are the only part with information in them.

The shape, and the one non-negotiable rule

Start with a conversation, not a spawn. Before any developer agent exists, an architect reads the open issues (gh issue list, then actually gh issue view each one) and the repo, and produces the one deliverable that’s genuinely hard: the parallel set. Independent work (different modules, no shared schema, PRs that won’t collide on merge) fans out; everything else serializes (shared files, a migration that has to land first, B’s acceptance depends on A). Get that wrong and you don’t get parallelism, you get merge conflicts with extra steps.

Then each issue in the wave gets a squad, roles deliberately split so no agent both writes and blesses the same diff:

  • developer — its own git worktree, opens the PR;
  • adversarial reviewer — a different agent, briefed to break the diff, not rubber-stamp it;
  • SDET — drives the change like a user;
  • productivity engineer — a meta-role that watches the process: every stall, every human approval, every bit of rework, written down.

The dev/reviewer split is load-bearing. The instant the context that wrote the code also reviews it, the review is theater.

And the telemetry is free, which is the best price. Every Claude Code session is a complete JSONL transcript at ~/.claude/projects/<slug>/<uuid>.jsonl — every tool call, every AskUserQuestion, every answer you gave. (We’ll gate the privacy policy to not log every breath you take).

TFW that retrospective is not a wishful thinking, it’s actionable.

Three runs, in ascending order of interesting

Run 1 — shipped clean, screwed up in a way I didn’t catch until I read the log. Two bug fixes on a production Next.js + Prisma app (two-branch staging/prod). Both merged, deployed, SDET-verified green. Then I read the transcript: the two bugs already had open PRs from a prior run. The architect never looked. We’d built and squash-merged duplicates, closed the issues, and orphaned two perfectly good PRs.

That’s not an agent being dumb. It’s a hole in the recipe. “Choose the parallel set” reasoned about file overlap and ordering and never asked the first question a human lead asks — is anyone already on this? — which is one gh pr list away. Second tell, same run: asked “where’s the evidence the reviewer approved these?”, the answer was nowhere. The verdicts lived in the agents’ context and never touched the PR. An approval that leaves no durable artifact didn’t happen. (Worse, squash-merge later buried even the merge-commit note, but I’m getting ahead of myself.)

Run 2 — the loop closed, and I have receipts. New work — a homepage redesign across seven sub-issues — same skill. At startup the agent did something I didn’t tell it to: it ran gh issue view 122 on the prior run’s recorded retro and read the engagement log. Then it did exactly the things Run 1 botched. It pre-flighted existing PRs. Every merge carried an adversarial verdict with specifics; the reviewer caught a dead query param (?q= where the target route reads ?search=) and sent it back with REQUEST_CHANGES.

Then it got interesting. A staging route started returning 500. The team traced it to schema drift, and went to fix the deploy pipeline by adding prisma db push. The safe version (no --accept-data-loss) did the right thing and aborted:

⚠️ There might be data loss when applying the changes:
  • drop column `negotiableTerms` on `Property` (1 non-null value)
Error: Use the --accept-data-loss flag to ignore the data loss warnings

It refused to drop a column with live data, surfaced it for a human call, took a one-time --accept-data-loss against staging only, reconciled, and reverted — production never saw the flag. The redesign isn’t the headline. The headline is that the run improved because it had read how the last run went. Best current read: that’s the flywheel, showing up unprompted.

Run 3 — we pointed it at itself, which is geekily elegant, and scientifically noble I scraped every point across Runs 1–2 where an agent stopped to ask a human to approve something — fifteen gates — dumped them into one issue, and ran the skill on that issue. The architect grouped the fifteen by type, correctly separated the gates worth keeping (destructive DB ops — yes, always ask) from the avoidable friction (re-asking a runtime question it already answered two turns ago), and — the good part — ran two of the fixes on its own execution before they were written into the skill. It pre-flighted with gh pr list and caught two pre-existing issues that overlapped the work, exactly the Run-1 bug, fixed live by the thing being fixed.

What’s actually carrying the weight

  • The parallel-set call is real architecture. Run 3 ran two repos in parallel but serialized five edits that all touched one SKILL.md into a single PR — instead of four agents racing to conflict on the same file.
  • Build/attack/verify pays rent. The reviewer caught a bug the developer was happy with. Once is enough to justify the second agent.
  • Worktree-per-issue keeps the squads from knifing each other.
  • The flight recorder is the product. Every stall is a candidate fix — a default, a permission, a pre-flight, a sharper brief.

Where it falls down (best current read)

  • The headline feature has never once fired. The skill leads with “every agent is a watchable terminal tab you can steer mid-run.” That needs the root session launched through the cmux claude-teams wrapper, which prepends a tmux shim to PATH (CLAUDE_CODE_EXPERIMENTAL_AGENT_TEAMS=1 alone is a red herring — diagnose with which tmux + echo $TMUX). Three runs, three fallbacks to background agents, because the session wasn’t started that one specific way. A feature nobody reaches isn’t a feature, it’s a positioning bug.
  • The same two process bugs recur every run until baked in: a setup question asked at spawn time instead of as a step-0 precondition, and re-asking a decision already made. Prose doesn’t self-correct — the executor re-litigates your opinions until you encode them as defaults.
  • N=3 and confounded. Run 2’s wins rode on memory carried from Run 1, so I can’t yet split skill-value from memory-value. The compounding loop is a strong signal, not a proof. The honest next experiment is one run on a clean, never-seen repo, launched under cmux, with no carried memory, measured by a typed telemetry schema — which doesn’t exist yet, so I’m building that before I build anything else.

The actual thesis

Spawning is commodity; the moat is the operating doctrine plus the telemetry loop — the thing that makes human-interventions-per-issue trend down run over run. Build the instrument first, defer the spaceship. YAGNI applies to strategy, too.

Skill’s open in voitta-ai/skillz. Run it on your backlog and tell me where it stalls. The stalls are the entire point.