shmobster

The Call Was Coming From Inside the House

Gregory Golberg holding forth, voitta , , , , , , , , , ,

This is going out on a Thursday night — the eve of the Friday news dump, that fine tradition of publishing what you’d rather nobody read too closely.

That is not why. We could have held it for Monday morning and the numbers would have been better. We didn’t want to sit on it, and the joke lands a day early. Look at what we’re willing to put in front of you.

Here’s the confession: we built a safety tool, and for a while it was the least safe thing on the machine.

YOLT is our little guardrail — it looks at each command an agent is about to run and decides whether it’s risky. Useful. The problem is that it also wrote down every command it inspected, verbatim, into a log under your home directory. Forever. No rotation. On by default.

You can see where this goes. Commands carry secrets: an API key pulled from a vault and handed to the next curl, a bot token, a connection string. So our safety tool quietly accumulated a plaintext pile of exactly the things it existed to protect, sitting in the one file nobody thinks to grep. Because who audits their seatbelt.

It got better. The reviewer that reads those logs copied the same raw lines into two more files. And the leak was never confined to our tool: a credential on a command line also lands in the agent’s own session transcript, which is a much larger and much quieter surface. The blast radius was bigger than the bug.

Then we swept everything, and it got worse

Once we started looking properly — every agent directory on every machine — the pattern that came back wasn’t carelessness. It was housekeeping.

A collaborator’s cloud key was rotated by an agent session. create-access-key printed the new secret to stdout, and the session wrote stdout to a transcript, where it sat in plaintext for a month. The rotation produced a longer-lived exposure than the thing it was fixing. A Slack webhook minted to replace one leaked in git history then leaked into a transcript itself.

The dangerous moments are the tidy ones. Rotation is the riskiest thing you do all quarter, precisely because new key material is briefly in the open, and an agent session is a permanent plaintext record of everything that crossed it.

Our favourite: halfway through the audit we checked which token the audit itself was using. It was the leaked one. The rotation had updated the config file and not the already-running shell, so the tool hunting the compromised credential was authenticating with it.

The retirement gap

Most of what we found was in sessions of software we had already decided to kill.

That gap — between deciding to retire something and actually retiring it — is where credentials rot. Nobody audits the tool on its way out. Nobody rotates for it. It keeps its tokens and keeps writing transcripts, right up until the plug comes out. Every team reading this has that gap open right now.

Retiring OpenClaw here was one of those decisions, and to be clear it was about our needs, not a verdict on the project. Vi versus emacs, and who cares. What we care about is a working deliverable arrived at the way we agreed; how you got there is your business.

Everything failed quietly

The cleanup taught us more than the leak did, mostly about no-ops that look like success.

Liveness probes lie, in at least five distinct ways we hit: a search API that answers 422 for valid and invalid keys alike; a catalogue route returning 200 with no authorization header at all; a live key that is merely spend-capped answering 400; a deleted webhook 404ing where a live one 400s; and an endpoint that validated the model name before the credential, so it gave identical answers for a live key, a dead key, and our deliberately invalid control. Carry a known-bad control, hit a route that enforces auth, and read the body, not the status line.

Then the same lie turned up somewhere we weren’t even looking. Our Slack agent runs a multi-vendor waterfall, and LiteLLM picks vendor cooldowns by HTTP status — it cools 429, 401, 408 and 404. Budget exhaustion is none of those: Anthropic answers a spend cap with 400, OpenRouter answers no-credit with 402. So a vendor that had been dead for weeks got dialled first on every single message, failed, and only then did the chain fall through. Fallback worked perfectly; nothing ever learned. That one is filed upstream. A status code is not a diagnosis, and that holds well past credential probes.

The scrubbing was the same shape. A fingerprint recorded from a truncated regex match can never match the value it was meant to track, so the scrubber reports the file clean while the secret sits in it. A post-scrub grep for the pattern returns hits forever, because docs and test fixtures share the pattern — a clean run looks failed. Nothing errors. You only find these by checking the thing itself instead of the report about the thing.

What’s shipped

The redaction that should have been there on day one is now day one for real: YOLT redacts before it writes, and the current release carries it. There’s also an advisory session hook that warns when a credential rides along on a command line.

The techniques are open source, because they’re the genuinely useful part. Our skills catalog, skillz, shipped v1.15.0 with everything above written down: agent-session-credential-audit for the sweep, the false-positive taxonomy, the probe rules and the kill-list scrubber that structurally cannot erase a live secret, and agent-credential-leak-surfaces for the places copies quietly accumulate. Install them, or just read them and steal the parts you want.

One surface we’d missed entirely and you probably have too: the OS keychain. No filesystem sweep will ever see it. Ours held a stale token and handed it over to a pipe with no prompt at all.

shmobster, the Slack agent we introduced in July, took the brunt of it. It had never been tagged at all; it now has six releases, every one cut in the day since this audit started, and the first exists only because the audit went looking for what we ship and found nothing versioned.

The one that matters here is redaction. The agent hands command output straight back to a channel, and cat, env and printenv are read-only, so they clear the safety gate and run with no approval at all. Not theoretical: we found a live config holding literal keys, where a single cat would have posted all five of them to Slack. Everything the agent says is now scrubbed before it leaves the process — reusing YOLT’s redactor rather than a second pattern list that would drift from it, plus this process’s own secrets matched by exact value, because the one thing a generic detector cannot know is which strings are yours. Scrubbing happens at collection, so the model’s own context never holds a credential it could repeat later.

It also learned to read skills, so that catalog now reaches the agent actually sitting in the channel with you, files unchanged. And v0.5.1 fixed a bug from precisely this post’s family: an unguarded loop over channels, where one stale channel id sorted first, aborted the rest, and sent the previous release’s announcement to none of the four healthy channels. The only trace was a traceback about the one channel that failed.

On our end we’re rotating and scrubbing. Assume-compromised is cheaper than assume-fine.

The lesson isn’t subtle, which is exactly why it stings: the safest-looking place is the least-swept. A security tool is the last thing anyone suspects of being a liability, so it’s the perfect place for one to hide. We wrote a guardrail and forgot that a guardrail with a memory is a ledger.

So, two asks. First: if you run YOLT or any of our skills, update — the versions that close this are shipped. Second: come pound on us. Find the next hole, open the issue, tell us where else we’re being careless. We would much rather hear it from you than from a log file.

That’s the whole trade: we screw up in public, you get to keep us honest, and everyone’s tooling gets a little safer. We take the work seriously. Ourselves, less so — hence Friday’s eve, a slot we’re using as a punchline rather than as cover.

Blow-by-blow in the copious links above. Have a good weekend.

We looked at Hermes-Agent. We’re keeping shmobster’s path (for now).

Gregory Golberg voitta , , , , , , ,

Someone asked the obvious question: Nous shipped Hermes-Agent, it’s everywhere, it self-improves, it learns across sessions. You built shmobster. Why not just switch?

Fair. So we did the comparison honestly. Up front: we are biased. We built the thing we’re defending, and that colors everything below. This isn’t a verdict on which agent is “better” — it’s a record of why, for our constraints, we’re staying our course. We’ll revisit. Here’s the reasoning while it’s fresh.

What each one actually is

Hermes-Agent is a self-improving personal agent: creates skills from experience, persistent memory, 40+ tools, a gateway onto every chat platform you can name. Popular, well-documented, deservedly so.

Shmobster is narrower on purpose: a gated command-runner in Slack. Multi-user, per-channel authorization, and a mutating-command approval gate — state-changing commands get classified and held until a trusted user approves them.

Different animals. So we went axis by axis instead of vibe by vibe.

Transport: Hermes wins, we don’t care

Hermes talks to Telegram, Discord, WhatsApp, Signal, email, Slack. We talk to Slack. On paper that’s a rout.

In practice it’s a non-issue. Adding a transport is bounded work we can do the day we need a second one. Carrying a multi-platform gateway we don’t use is just surface area to maintain. YAGNI. Skip.

The LLM layer: resilience beats model-swap for a shared bot

Hermes leans on switching which model you point at — Portal, OpenRouter, OpenAI, custom, swap on command. Great for one person choosing today’s brain.

We run a LiteLLM waterfall: automatic failover across vendors when one rate-limits. For a shared Slack bot getting hammered by many users concurrently, the enemy is the 429, not “which model.” Resilience is the axis that matters, and it’s the one the personal-agent framing doesn’t optimize for.

And the kicker: LiteLLM already reaches every backend Hermes offers, including Nous’s own models. So “trying Hermes” at the model layer isn’t a migration — it’s adding one rung to our waterfall. We keep the failover; we lose nothing. (We even filed ourselves a ticket for the Router knobs we’re not using yet, staying honest about our own debt.)

The learning loop: we love it, and we can’t copy it

This is the part worth thinking hard about, and it’s where we’re already moving — see the recursive-self-improvement post, voitta-yolt, the agent-team-orchestration plugin. We want skills-from-experience and persistent memory. Hermes does that well.

But Hermes’s learning loop is trust-flat: one user, learn -> store -> reuse, no gate on what’s learned or applied. It can be, because the trust boundary is a single person.

Ours can’t be. Port that loop naively into a multi-user, gated system and you manufacture two escalation paths:

  • Learn-once, replay-forever. Our approval gate is per-invocation. A loop that turns an approved trajectory into an auto-runnable skill quietly makes it per-first-invocation. Approve a mutating action once, replay it forever, ungated. That’s a privilege-escalation bug wearing a feature’s clothes.
  • Memory as an injection surface. Multi-user Slack means untrusted messages become memory. Feed memory back into the agent and a non-trusted user can plant instructions that later execute in a trusted context. A single-user agent never meets an adversarial memory writer. We meet one every day.

So our principle is simple: learning inherits the authz spine. It doesn’t bypass the gate; it changes when the gate fires. Capture every approved trajectory with its provenance. Promote a trajectory into a reusable skill only by a trusted user. On reuse, check the skill’s recorded envelope against the current user/channel, and if it still contains mutating steps, it still gets gated. The invariant we refuse to give up: no learned skill runs a mutating action outside the authority it was learned under.

The nice part is we already own the primitive. voitta-yolt classifies read-only vs mutating commands — the same classifier tags trajectories at learn-time and gates skills at apply-time, and it’s what lets us treat trusted-authored skills as executable and untrusted memory as reference-only. Self-improvement gets the same treatment: a skill that rewrites itself proposes a new version, and a trusted human disposes.

The actual reason we’re staying

Strip it down and the moat isn’t the model, the transport, or even the skills — it’s the authorization. Hermes has no per-channel, per-user gate, because it was never built to need one. That gate is exactly what makes shmobster useful in a shared team channel, and it’s precisely the thing a trust-flat learning loop would erode.

So we borrow the mechanisms — FTS5 memory, extraction prompts, the skill-from-experience pattern — and we keep our spine. The learning loop we’re building is just our authz model extended to the time axis.

We could be wrong. Hermes is moving fast and the gap in polish is real. If the day comes that its trust model grows up, or our constraints change, we’ll switch without ceremony. Until then: same path, eyes open, ticket filed.

Shmobster: a Slack agent built bottom-up

Gregory Golberg holding forth, voitta , , , , , ,

We open-sourced Shmobster today (MIT). It’s a Slack agent: you @mention it in a channel and it reads threads, runs commands, opens PRs, and answers. There are a dozen of those. Here’s why we built another one, and the two ideas that shaped it enough to be worth stealing.

Two features forced it to exist

Most of what an agent does — hold a conversation, call tools, read a repo — is a solved problem you can rent. We only started writing code because two requirements had no off-the-shelf answer that fit:

  1. A multi-vendor API waterfall. Any single LLM vendor rate-limits you at the worst possible moment. We wanted an ordered list of vendors — a primary, then fallbacks — with a cooldown so a throttled vendor is skipped for a window instead of hammered every call. When the primary 429s, slide to the next one without the user ever noticing.

  2. Multi-user Slack authz. The moment more than one person can talk to an agent that runs shell commands, “what is this allowed to do?” stops being a constant and becomes a function of who is asking and where.

Everything else, we refused to write.

Own, rent, delegate

The organizing question for every capability was: own it, rent it, or delegate it?

  • Own the things that are the actual reason the project exists: the orchestration loop, the Slack door and its socket reliability, and authorization.
  • Rent the things that are somebody else’s core competency: LiteLLM for the multi-vendor waterfall, voitta-yolt as a library for classifying whether a shell command mutates state.
  • Delegate the things another agent already does better: browser work goes to claude -p, which has a browser tool. We don’t reimplement it — and we don’t waterfall it.
  • Transplant the parts that are runtime-agnostic: the agent’s “spine” is a set of markdown files (SOUL, RUNBOOKS, and friends) adapted from OpenClaw (MIT). The loop boots by reading them.

The waterfall — the first forcing function — turned out to be almost entirely rented. It’s a LiteLLM Router built from a config list:

"waterfall": [
  { "name": "anthropic",  "model": "anthropic/claude-sonnet-5", "api_key": "..." },
  { "name": "openrouter", "model": "openrouter/openai/gpt-4o",  "api_key": "..." },
  { "name": "nvidia",     "model": "nvidia_nim/meta/llama-3.1-405b-instruct",
    "api_key": "...", "api_base": "https://integrate.api.nvidia.com/v1" }
]

First entry is primary; the rest are fallbacks; a 429 slides down the list; a broken vendor cools down for sixty seconds. That’s the whole feature. We own maybe forty lines around it.

0, 1, 2, 3, many

Here’s the idea I’d steal for your next project. Every axis of a system sits at its own cardinality, and you should architect each one to its actual number — not to infinity, and not advance until the count really increments.

Axis Count Consequence
vendors many rented (LiteLLM, a config list)
owners 2 a hardcoded pair
collaborators 0 -> 1 binary owner / non-owner, no RBAC
workspace / tenant 1 hardcoded, no multi-tenant
channels many per-channel policy in config

“Many” earns a real abstraction. “Two” gets hardcoded. “One” gets a constant. “Zero heading to one” gets a boolean, not a role system. The mistake most systems make is treating every axis as “many” on day one — building RBAC for two users, multi-tenancy for one tenant, a plugin system for one plugin. You generalize an axis only when someone else actually wants in. Not before.

Authz = f(user, channel)

The second forcing function. Authorization here isn’t a global setting; it’s a function of two arguments.

  • What a command may do is answered by YOLT: is this command read-only or mutating? Read-only just runs. Mutating pauses for approval. You don’t want to approve git status; you do want to approve git push.
  • Where the agent is answers scope: each channel carries a policy — a working directory, an allow-list of GitHub repos, an AWS profile, a set of extra tools. A channel scoped to one project points at that project’s directory and can’t wander out of it.

Compose them and the agent that helps you in a locked-down production channel is the same binary as the one running free in your scratch channel — only the policy differs.

There’s a third argument creeping in — who — but at cardinality 0->1 it’s still just a trusted_users list: a handful of Slack IDs allowed to widen a channel’s restrictions by chat. The list itself is file-only. The agent can’t add to it, so there’s no privilege escalation to design against yet. When collaborators become “many,” that boolean grows into something real. Not before.

Built bottom-up, in public

Shmobster was built one issue at a time — skeleton, then the exec gate, then per-channel policy, then waterfall hardening — each a small PR against a design argued out in the issue tracker. That history is now public too.

If you run agents in Slack and have felt any of these edges — the rate-limit cliff, or the “wait, what can this thing actually do” question — the code is at github.com/voitta-ai/shmobster, MIT-licensed. Steal the cardinality table at minimum.