agents

Eventual Consistency Is the New Agile

Gregory Golberg method-and-apparatus, voitta , , , , ,

Let me get the scary demo out of the way, because it’s the wrong lesson.

Last week I had three AI agents working the same repository at the same time. They shared one working tree — one HEAD, one index — and they raced on it. One agent quietly re-based another’s work onto the wrong commit. No error, no warning, just wrong. I caught it reading the git reflog, not by watching the agents.

The obvious takeaway is “don’t do that,” and it’s correct. But that’s the outlier — three workers fighting over one branch is the degenerate case, interesting precisely because it’s rare. The real pattern is the opposite shape, and it’s where the whole story is.

The pattern is many, not one

Normal multi-agent work isn’t a knife fight over one branch. It’s many agents across many repos and many branches, each building toward something, each reaching intermediate states that are still workable on their own. Think teams running microservices: nobody sane ships a microservices org by making every service commit to one trunk in lockstep. Each moves on its own clock. The contracts at the seams are what you actually manage.

And here’s the thing that took me a minute to see: the same fleet looks like a different system depending on where you stand.

  • From the engineer’s chair, it’s microservices — disjoint work, interfaces at the boundaries.
  • From the PM’s chair, it’s shards to integrate — separate pieces, assembled.
  • From the CEO’s chair, it’s one thing. Every agent is a partial replica of a single value — the product, the vision — converging on “is this what we said it would be.” That is eventual consistency, literally: the databases’ bargain where workers act independently and you converge later, as long as there’s a target to converge toward and someone who owns the merge when two of them disagree.

Same fleet. Three truths, by altitude. None of them wrong.

The table of rivals — and who actually belongs at it

Which brings me to the objection I keep hearing: this CEO can’t even pick the right React library, how can he lead engineering?

He doesn’t need to. Kennedy didn’t know rocket metallurgy — rocketry is one field, metallurgy another, and the whole trick is that they have to meet somewhere he’d never stand. His job was to stand up to Khrushchev, name the audacious goal, and know he commanded the resources to chase it. Picking the framework was never the work — and now the fleet picks the framework anyway.

What the CEO does need is a table of rivals: competing agents, competing approaches, and — critically — the expert who says no. Roger Boisjoly is the engineer who warned Morton Thiokol that the Challenger’s O-rings would fail in the cold. He was overruled. Seven people died. His seat at that table is the entire reason the table exists. The vision-setter marshals resources; the rival with evidence holds a veto on cannot. Heed your Boisjoly and you get the moon. Overrule him and you get Challenger. Same table, two outcomes, and history has the receipts both ways.

So what dies here isn’t expertise — it’s nitpicking. “You’re not a real technical leader unless you know X” was gatekeeping wearing competence’s clothes. Consequential dissent, the O-ring warning, matters more than ever. Taste-policing the framework choice matters not at all. The fleet ate that job.

The hard part — the one thing that got harder, not cheaper — is telling them apart. At 11pm before launch, Boisjoly looked like a nitpicker to the managers who overruled him. Consequential dissent and bikeshedding wear the same face in the moment. Distinguishing them is now the leader’s whole job, because everything around it got automated and that didn’t.

What actually collapsed

Here’s the narrow, true claim under all of this: the cost of building incrementally — and showing value at every stage — collapsed. In software.

Not in rocketry. A rocket at 50% gets you 0% of the way to the moon; a half-built bridge carries nobody. Physical moonshots need the giant up-front commitment precisely because they can’t demo their way there. Software is the opposite, and always was — that was the whole bet of the Agile Manifesto: working software over comprehensive plans, respond to change over follow the plan, ship increments.

We mostly couldn’t. The increments were too expensive to actually produce, so the values curdled into ceremony — standups, story points, a poster on the wall. Agent fleets attack that cost directly. Spinning up a unit of work is free. Pivoting discards cheap work instead of sunk careers. So “maybe not the moon at all, maybe something else entirely — but at every stage, something real to show” stops being an aspiration and becomes the default operating point.

The Manifesto wasn’t wrong. It was rate-limited, in the one domain it was written for. Now it runs.

What’s left

Almost everything got cheap. One thing didn’t: the merge. Execution collapsed, pivots collapsed, but integration is still where all the residual difficulty concentrates — and with more parallel units, it may even rise. Brooks’s Law doesn’t die; it relocates. Owning the merge — designing the seams, resolving the conflicts, deciding what converges and what gets discarded — is the job now.

Which is exactly why the three agents at the top of this post are a cautionary tale and not a triumph. A shared working tree with no isolation and no reconcile is what “always something to show” looks like when you skip the discipline. Do the discipline, and the same fleet is a moonshot. Skip it, and it’s the knife fight we opened with.

This is the flip side of something I wrote earlier — that our tools still grade the work as if it happened inside one process boundary. The work lives across boundaries. So does the value. So does the job.

herdr and cmux: two shapes of the same agent multiplexer

Gregory Golberg voitta , , , , , , ,

André Lindenberg’s post about herdr came across my feed, and the pitch landed on something I have been living in for months:

You already run tmux to keep agents alive when you close the laptop. herdr goes further: through its socket API an agent splits a sibling pane, starts another agent, and blocks on its settled state before continuing.

My terminal is cmux — I wrote up my setup a while back, including the fight to make agent panes come back on their real conversations after a reboot. Same problem, two tools, so: an honest comparison, and an actual decision at the end rather than a shrug about a thousand flowers.

The structural difference

herdr is a daemon plus a TUI client that runs inside the terminal you already have. The daemon owns the PTYs; clients attach and detach. ctrl+b q detaches, herdr reattaches, including over SSH. macOS, Linux, Windows (beta). Rust, Apache-2.0, v0.7.5. The repo was created in late March 2026 and is at 20.8k stars — four months, from zero. That pace is not an accident and it is not a toy.

cmux is a native macOS app that embeds Ghostty as its renderer. It is the terminal, not a program running inside one. Swift, GPU rendering, vertical workspace tabs, browser panes, a notification center. GPL-3.0-or-later with a commercial option, v0.64.20, 25.1k stars.

Almost everything below follows from that one choice.

Which seat is each one optimizing?

This is the whole comparison, so I will put it before the evidence rather than after.

herdr optimizes the seat the agent sits in. agent wait --until done is a primitive for a program coordinating other programs. Occupant pinning, fused prompt-and-wait, the HERDR_ENV gate — those are the concerns of a caller that is not a person.

cmux optimizes the seat I sit in. The approval feed, notifications, browser panes, hook-recorded native session restore — those matter when a human is the scheduler and the agents are the ones asking permission.

So “which is better” resolves to “who does the scheduling in your workflow.” In mine, today, it is still me: I fan agents out, they come back with questions, I unblock them. That is a human-in-the-loop shape and cmux is built for it. The day the dominant pattern becomes agent spawns agent and blocks on it, herdr’s design is the right one.

What herdr does better

1. The wait verb. This is the real content of André’s post and the thing I would take today:

herdr agent wait w1:p1 --until done
herdr agent wait w1:p1 --until blocked

Server-owned, event-driven rather than polled, and it pins the resolved pane occupant so a replacement agent cannot satisfy the wait. agent.prompt also accepts an optional wait object, so submit-and-wait is one request with no race between the calls.

That is a genuine orchestration primitive. “Start the sibling, hand it work, block until it settles” becomes three lines of shell instead of a bespoke state machine.

cmux has the state — its hook integrations record running / idle / needsInput / unknown — and it has a durable event stream. It just does not expose a verb that joins them.

2. Detach is a real concept. cmux’s session lives in the app; herdr’s lives in a daemon you attach to. That difference is why my reboot post needed an appendix. To be fair: a power cycle kills the herdr daemon too, and nothing resurrects a dead PTY. But “close the laptop, reattach from another terminal, reattach over SSH” is a first-class flow there and a workaround-shaped thing in a GUI app.

3. It runs where the work runs. Linux boxes, remote hosts, Windows beta. cmux is macOS-only by construction. If your agents live on a build server, that is not a preference, it is a constraint.

4. Plugins are shipped surface. A herdr-plugin.toml declares startup hooks, actions, event hooks, and pane entrypoints; plugins launch as processes with HERDR_* context injected. There is a marketplace and a visible third-party ecosystem — review sidebars, file viewers, phone clients, remote mirrors. cmux’s ExtensionKit sidebars are younger and have been through at least one revert.

What cmux does better

1. It is a terminal, so it does not have to borrow one. No nested-multiplexer key contention, no arguing over ctrl+b, no “which layer ate my mouse event.” Real tabs, real drag-and-drop, GPU rendering.

2. Panes are not only PTYs. Surfaces can be terminals, browsers, markdown viewers, or file previews — and the browser is scriptable from the same CLI (cmux browser navigate|click|wait|download). An agent can be handed a rendered page and a doc alongside its shell. In herdr everything is a character grid; the nearest analog is experimental Kitty-protocol pane graphics.

3. Agent state is told, not inferred. cmux hooks setup installs session hooks for 14 agents — Claude Code, Codex, Grok, OpenCode, Pi, Amp, Cursor, Gemini, Kiro, Rovo Dev, Copilot, CodeBuddy, Factory, Qoder — and stores each one’s native resume command (claude --resume <id>, codex resume <id>, amp threads continue <id>, …), so a relaunch continues the real conversation. herdr detects state by evaluating manifests against a terminal snapshot. Detection is clever; being told is sturdier.

4. The human is in the protocol. The Feed collects permission requests and questions from every agent into one approval queue. Notifications, sidebar status pills, progress bars, and log lines are all CLI-writable by the agents themselves. herdr’s notification.show is a toast; six running agents need one blocked-list, not six toasts.

5. Remote and cloud are features, not an absence. cmux ssh creates remote workspaces with a bundled daemon and persisted PTY sessions you can list, attach, and clean up; cmux vm manages cloud VMs. Different shape from detach/reattach, but the “my agents are on another machine” case is covered.

The decision

Not “let a thousand flowers bloom.” That is what you say when you do not want to choose, and two multiplexers on one machine means two keymaps, two session stores, and two places to look for the agent that is blocked.

cmux stays the cockpit on macOS. Not because it wins on paper — on the agent-facing API it does not — but because switching cockpits costs everything built around the human loop: the approval feed, the notification wiring, hooks for 14 agents, workspace layouts, muscle memory. herdr would have to be better by a lot to clear that, and on the axis I actually sit on it is not better, it is differently good.

herdr gets adopted where cmux structurally cannot go: Linux boxes, remote hosts, SSH-first work. That is not hedging, because cmux is not competing there. It is a division of territory, not a bake-off.

No dual-running on the same Mac. If I catch myself doing it, that is evidence this split is wrong and I should re-run the comparison rather than live in both.

The tripwire, stated in advance so it is falsifiable. I switch outright if either becomes true:

  1. Primary development moves off macOS. Then cmux’s best feature — being an excellent native Mac terminal — is simply unavailable, and the rest is a wash.
  2. Agent-to-agent orchestration becomes the dominant mode — agents spawning and blocking on agents rather than me fanning out and unblocking — and cmux still has no wait verb. At that point I would be hand-rolling in event-stream shell what herdr ships as one command, which is the definition of using the wrong tool politely.

Neither is true today. Both are plausible within a year, and #2 is the one I would bet on. Review date: January 2027. A decision with no review date is just a preference.

Where a thousand flowers genuinely help is at the ecosystem level, not on my desk: herdr existing is the best argument cmux will ever get for shipping a wait verb, and cmux’s hook-based session capture is the best argument herdr will get for taking state from hooks instead of the screen. Each is holding up a mirror the other needs. I would rather have both projects than a merged one — and still pick one per machine.

What I asked of cmux

Filed, because I use it daily and can answer the follow-ups:

  • #8950 — a wait verb: cmux wait --surface <id> --until idle|needs-input --timeout <ms>, occupant-pinned, plus send --wait-until to close the submit-then-wait race. Today the closest thing is cmux events --name agent.hook.Stop --limit 1, which matches one agent’s hook vocabulary rather than semantic state and pins nothing.
  • #8951 — publish agent lifecycle as an event (agent.state.changed). The running / idle / needsInput state exists but lives in ~/.cmuxterm/<agent>-hook-sessions.json and the hibernation subsystem; it is absent from the public event catalog that already carries window.*, workspace.*, surface.*, and feed.*. It is the substrate the wait verb should be built on.

Two more that need no issue: resume bindings should stay PATH-relative rather than storing a resolved absolute path at pane creation (#6572, already fixed by #6582), and reboot restore should be a stated contract — my panes started coming back on 0.64.15 while the flag I expected to gate it was false for every pane (#5802, still open). Getting the right answer for a reason you cannot name is not a fixed bug, it is a deferred one.

What I would ask of herdr, and why it stays here

Two things would move herdr from “right tool for the remote boxes” to “candidate for the cockpit”:

Take state from hooks, not from the screen. The scaffolding exists — pane.report_agent accepts exactly that shape, pane.report_agent_session stores native session references, integration.install is there. The gap is breadth: cover agents first-party the way cmux hooks setup covers 14 of them, and let screen detection be the fallback rather than the primary path.

Give the human a queue. When six agents are running, what I need is not six toasts, it is one list of what is blocked. herdr already has the ingredients — semantic blocked state, agent.view.set projections, an agent sidebar — so this may be more assembly than invention. Both of these are things a plugin could prototype without touching the core.

I am deliberately not filing either as an issue. I have read herdr’s docs closely and have not run it in anger, and a feature request from a non-user is a maintainer tax: they have to reconstruct my context before they can even judge whether I found a real gap or just did not finish the manual. The cmux asks went to its tracker precisely because I use it daily. These stay at blog volume, where someone who actually runs herdr can correct me cheaply — and I would rather be corrected here than spend a maintainer’s triage.

One note on addressing, since the post that started this was not from the maintainer: herdr is Can Celik’s. Thanks to André for putting it in front of me — the framing in that post is what made me go read the socket API instead of skimming another launch.

Footnote: the fork nobody was maintaining

While writing this I checked our own voitta-ai/cmux fork. It was 0 commits ahead and 3,171 behind upstream — a June snapshot with no patches on it. That is not a fork, it is a stale bookmark that quietly implies we carry local changes.

We do not, and that is correct: the cmux problems I actually hit went upstream as issues, and one is already fixed there by a maintainer. Filing beats forking whenever the maintainer is responsive — a fork you do not rebase is a liability with a nice URL. Resynced while writing this; it is identical to upstream again.

It also mattered for the two issues above. Our checkout was seven weeks stale, so I checked both proposals against upstream main before filing — “open an issue for a feature that shipped last month” is a real way to waste someone’s afternoon. Still missing on current main: wait-for remains the tmux-compat named synchronization point, and the event catalog still has nothing for agent state.

Recursive self-improvement, you said?

Gregory Golberg voitta , , , , , , ,

Spawning a fleet of coding agents is a solved problem. You write a for loop, you call the Agent tool N times, you go get coffee. The unsolved problem is everything wrapped around the spawn: deciding what can actually run in parallel, stopping the agent that wrote the code from also grading (or eating) its own homework, and — the part nobody ships — recording how the run went so the next one isn’t the same run with the same mistakes.

I wish I didn’t remember this anymore but this used to be called a “retrospective” in that sect I was once a member of.

I’ve been dogfooding a small orchestration skill (agent-team-orchestration, open in voitta-ai/skillz) that treats those as the actual work. Three runs in. This is the first write-up, warts very much included — the warts are the only part with information in them.

The shape, and the one non-negotiable rule

Start with a conversation, not a spawn. Before any developer agent exists, an architect reads the open issues (gh issue list, then actually gh issue view each one) and the repo, and produces the one deliverable that’s genuinely hard: the parallel set. Independent work (different modules, no shared schema, PRs that won’t collide on merge) fans out; everything else serializes (shared files, a migration that has to land first, B’s acceptance depends on A). Get that wrong and you don’t get parallelism, you get merge conflicts with extra steps.

Then each issue in the wave gets a squad, roles deliberately split so no agent both writes and blesses the same diff:

  • developer — its own git worktree, opens the PR;
  • adversarial reviewer — a different agent, briefed to break the diff, not rubber-stamp it;
  • SDET — drives the change like a user;
  • productivity engineer — a meta-role that watches the process: every stall, every human approval, every bit of rework, written down.

The dev/reviewer split is load-bearing. The instant the context that wrote the code also reviews it, the review is theater.

And the telemetry is free, which is the best price. Every Claude Code session is a complete JSONL transcript at ~/.claude/projects/<slug>/<uuid>.jsonl — every tool call, every AskUserQuestion, every answer you gave. (We’ll gate the privacy policy to not log every breath you take).

TFW that retrospective is not a wishful thinking, it’s actionable.

Three runs, in ascending order of interesting

Run 1 — shipped clean, screwed up in a way I didn’t catch until I read the log. Two bug fixes on a production Next.js + Prisma app (two-branch staging/prod). Both merged, deployed, SDET-verified green. Then I read the transcript: the two bugs already had open PRs from a prior run. The architect never looked. We’d built and squash-merged duplicates, closed the issues, and orphaned two perfectly good PRs.

That’s not an agent being dumb. It’s a hole in the recipe. “Choose the parallel set” reasoned about file overlap and ordering and never asked the first question a human lead asks — is anyone already on this? — which is one gh pr list away. Second tell, same run: asked “where’s the evidence the reviewer approved these?”, the answer was nowhere. The verdicts lived in the agents’ context and never touched the PR. An approval that leaves no durable artifact didn’t happen. (Worse, squash-merge later buried even the merge-commit note, but I’m getting ahead of myself.)

Run 2 — the loop closed, and I have receipts. New work — a homepage redesign across seven sub-issues — same skill. At startup the agent did something I didn’t tell it to: it ran gh issue view 122 on the prior run’s recorded retro and read the engagement log. Then it did exactly the things Run 1 botched. It pre-flighted existing PRs. Every merge carried an adversarial verdict with specifics; the reviewer caught a dead query param (?q= where the target route reads ?search=) and sent it back with REQUEST_CHANGES.

Then it got interesting. A staging route started returning 500. The team traced it to schema drift, and went to fix the deploy pipeline by adding prisma db push. The safe version (no --accept-data-loss) did the right thing and aborted:

⚠️ There might be data loss when applying the changes:
  • drop column `negotiableTerms` on `Property` (1 non-null value)
Error: Use the --accept-data-loss flag to ignore the data loss warnings

It refused to drop a column with live data, surfaced it for a human call, took a one-time --accept-data-loss against staging only, reconciled, and reverted — production never saw the flag. The redesign isn’t the headline. The headline is that the run improved because it had read how the last run went. Best current read: that’s the flywheel, showing up unprompted.

Run 3 — we pointed it at itself, which is geekily elegant, and scientifically noble I scraped every point across Runs 1–2 where an agent stopped to ask a human to approve something — fifteen gates — dumped them into one issue, and ran the skill on that issue. The architect grouped the fifteen by type, correctly separated the gates worth keeping (destructive DB ops — yes, always ask) from the avoidable friction (re-asking a runtime question it already answered two turns ago), and — the good part — ran two of the fixes on its own execution before they were written into the skill. It pre-flighted with gh pr list and caught two pre-existing issues that overlapped the work, exactly the Run-1 bug, fixed live by the thing being fixed.

What’s actually carrying the weight

  • The parallel-set call is real architecture. Run 3 ran two repos in parallel but serialized five edits that all touched one SKILL.md into a single PR — instead of four agents racing to conflict on the same file.
  • Build/attack/verify pays rent. The reviewer caught a bug the developer was happy with. Once is enough to justify the second agent.
  • Worktree-per-issue keeps the squads from knifing each other.
  • The flight recorder is the product. Every stall is a candidate fix — a default, a permission, a pre-flight, a sharper brief.

Where it falls down (best current read)

  • The headline feature has never once fired. The skill leads with “every agent is a watchable terminal tab you can steer mid-run.” That needs the root session launched through the cmux claude-teams wrapper, which prepends a tmux shim to PATH (CLAUDE_CODE_EXPERIMENTAL_AGENT_TEAMS=1 alone is a red herring — diagnose with which tmux + echo $TMUX). Three runs, three fallbacks to background agents, because the session wasn’t started that one specific way. A feature nobody reaches isn’t a feature, it’s a positioning bug.
  • The same two process bugs recur every run until baked in: a setup question asked at spawn time instead of as a step-0 precondition, and re-asking a decision already made. Prose doesn’t self-correct — the executor re-litigates your opinions until you encode them as defaults.
  • N=3 and confounded. Run 2’s wins rode on memory carried from Run 1, so I can’t yet split skill-value from memory-value. The compounding loop is a strong signal, not a proof. The honest next experiment is one run on a clean, never-seen repo, launched under cmux, with no carried memory, measured by a typed telemetry schema — which doesn’t exist yet, so I’m building that before I build anything else.

The actual thesis

Spawning is commodity; the moat is the operating doctrine plus the telemetry loop — the thing that makes human-interventions-per-issue trend down run over run. Build the instrument first, defer the spaceship. YAGNI applies to strategy, too.

Skill’s open in voitta-ai/skillz. Run it on your backlog and tell me where it stalls. The stalls are the entire point.