agentic

cmux, eight weeks later: the two-hop PATH trap

Gregory Golberg voitta , , , ,

A June post covered getting cmux set up so agent panes survive a reboot. Since then the setup has been running continuously, and three things went wrong that were worth filing (#1, #2, #3). Here is what held up and what did not.

How teammate tabs actually work

cmux claude-teams gives each teammate its own watchable cmux tab by impersonating tmux. It drops a shim at ~/.cmuxterm/claude-teams-bin/tmux:

#!/usr/bin/env bash
exec "${CMUX_CLAUDE_TEAMS_CMUX_BIN:-cmux}" __tmux-compat "$@"

and sets $TMUX to a synthetic socket path. No tmux server exists anywhere. Claude Code thinks it is talking to tmux; cmux answers, and each new-window becomes a tab.

It works well. tmux -V returns tmux 3.4, and tmux list-windows enumerates your cmux workspaces:

0 m&a
1 voitta
2 agent-teams
3 debedb

The catch is that the whole thing rests on PATH resolution, in two places.

Hop one: which tmux wins

If the launching process resolves tmux to real Homebrew tmux instead of the shim, real tmux tries to connect to a socket that was never a socket, and teammate spawn breaks outright.

On this machine Homebrew sat at PATH position 11 and the shim at 26. Both sources pushing it there were self-inflicted: my own cmux.json command re-prepended /opt/homebrew/bin, and so did .bash_profile. The fix is to prepend the shim directory in the workspace command.

Worth noting for anyone copying the June post: on cmux 0.64.16 cmux claude-teams now puts the shim at position 1 itself. The cmux.json prepend is belt-and-braces today, not the load-bearing fix. Which matters, because it is not what was actually broken.

Hop two: the shim’s own cmux

Look at the shim again. It execs bare cmux. And the cmux binary lives at /Applications/cmux.app/Contents/Resources/bin/cmux, which is not on a normal login shell’s PATH:

$ env -i HOME=$HOME /bin/bash -lc 'command -v cmux'
$

So winning hop one buys you nothing if hop two loses:

$ env -i HOME=$HOME /bin/bash -lc \
    'export PATH="$HOME/.cmuxterm/claude-teams-bin:$PATH"; tmux -V'
/Users/gregory/.cmuxterm/claude-teams-bin/tmux: line 3: exec: cmux: not found

One symlink fixes it:

ln -s /Applications/cmux.app/Contents/Resources/bin/cmux ~/.local/bin/cmux
$ env -i HOME=$HOME /bin/bash -lc \
    'export PATH="$HOME/.cmuxterm/claude-teams-bin:$PATH"; tmux -V'
tmux 3.4

I had filed that symlink as an ergonomics nit — scripts otherwise need CMUX="${CMUX_BUNDLED_CLI_PATH:-/Applications/…/cmux}". It was not a nit. A shim on PATH whose own dependency is off PATH fails in a way that reads as a tmux problem.

The diagnostic that saved the most time

A queued teammate and a shadowed shim look identical from outside: the pane sits there, nothing tabs. They are distinguished by one check.

If ps shows no __tmux-compat process ever appeared, the spawn never reached tmux, so PATH is not your problem. In my case the launcher pane was in manual mode on and the teammate sat at ◯ indefinitely, having called nothing at all. I would have spent the evening on PATH.

The other habit worth keeping: read the live process environment, not the shell’s.

ps -Eww -o command= -p <pid> | tr ' ' '\n' | grep -E '^(TMUX|PATH)='

A pane’s shell will happily report a PATH the long-running agent inside it never saw.

What about reboot survival?

That part held. Across a real reboot on 0.64.16, all 7 panes came back on their exact resume commands, none silently dropped — including a teams-named pane, which now carries an agent-hook resume binding it did not have before.

One caveat the June post did not mention: identity is not preserved. Comparing session-com.cmuxterm.app-previous.json against the live session file, workspaceId and panel id share zero values across a restart. Restore recreates everything by replaying resume bindings. That is the mechanism behind rescued tabs landing on Claude Code’s “is this a project you trust?” prompt — folder trust was scoped to a workspace id that no longer exists.

If you go diffing those files yourself, the identity key is workspaceId, not id. Key on id and every workspace collapses to None, which reads as “ids are stable” — precisely backwards. I made that mistake on the first pass.

Net

Two config lines, both one-time:

ln -s /Applications/cmux.app/Contents/Resources/bin/cmux ~/.local/bin/cmux
# and in cmux.json's teams command:
export PATH="$HOME/.cmuxterm/claude-teams-bin:$PATH"

The setup from June is still the setup. What eroded was the environment around it — which is the recurring theme with long-lived agent sessions: nothing breaks, things merely get reordered underneath you.

herdr and cmux: two shapes of the same agent multiplexer

Gregory Golberg voitta , , , , , , ,

André Lindenberg’s post about herdr came across my feed, and the pitch landed on something I have been living in for months:

You already run tmux to keep agents alive when you close the laptop. herdr goes further: through its socket API an agent splits a sibling pane, starts another agent, and blocks on its settled state before continuing.

My terminal is cmux — I wrote up my setup a while back, including the fight to make agent panes come back on their real conversations after a reboot. Same problem, two tools, so: an honest comparison, and an actual decision at the end rather than a shrug about a thousand flowers.

The structural difference

herdr is a daemon plus a TUI client that runs inside the terminal you already have. The daemon owns the PTYs; clients attach and detach. ctrl+b q detaches, herdr reattaches, including over SSH. macOS, Linux, Windows (beta). Rust, Apache-2.0, v0.7.5. The repo was created in late March 2026 and is at 20.8k stars — four months, from zero. That pace is not an accident and it is not a toy.

cmux is a native macOS app that embeds Ghostty as its renderer. It is the terminal, not a program running inside one. Swift, GPU rendering, vertical workspace tabs, browser panes, a notification center. GPL-3.0-or-later with a commercial option, v0.64.20, 25.1k stars.

Almost everything below follows from that one choice.

Which seat is each one optimizing?

This is the whole comparison, so I will put it before the evidence rather than after.

herdr optimizes the seat the agent sits in. agent wait --until done is a primitive for a program coordinating other programs. Occupant pinning, fused prompt-and-wait, the HERDR_ENV gate — those are the concerns of a caller that is not a person.

cmux optimizes the seat I sit in. The approval feed, notifications, browser panes, hook-recorded native session restore — those matter when a human is the scheduler and the agents are the ones asking permission.

So “which is better” resolves to “who does the scheduling in your workflow.” In mine, today, it is still me: I fan agents out, they come back with questions, I unblock them. That is a human-in-the-loop shape and cmux is built for it. The day the dominant pattern becomes agent spawns agent and blocks on it, herdr’s design is the right one.

What herdr does better

1. The wait verb. This is the real content of André’s post and the thing I would take today:

herdr agent wait w1:p1 --until done
herdr agent wait w1:p1 --until blocked

Server-owned, event-driven rather than polled, and it pins the resolved pane occupant so a replacement agent cannot satisfy the wait. agent.prompt also accepts an optional wait object, so submit-and-wait is one request with no race between the calls.

That is a genuine orchestration primitive. “Start the sibling, hand it work, block until it settles” becomes three lines of shell instead of a bespoke state machine.

cmux has the state — its hook integrations record running / idle / needsInput / unknown — and it has a durable event stream. It just does not expose a verb that joins them.

2. Detach is a real concept. cmux’s session lives in the app; herdr’s lives in a daemon you attach to. That difference is why my reboot post needed an appendix. To be fair: a power cycle kills the herdr daemon too, and nothing resurrects a dead PTY. But “close the laptop, reattach from another terminal, reattach over SSH” is a first-class flow there and a workaround-shaped thing in a GUI app.

3. It runs where the work runs. Linux boxes, remote hosts, Windows beta. cmux is macOS-only by construction. If your agents live on a build server, that is not a preference, it is a constraint.

4. Plugins are shipped surface. A herdr-plugin.toml declares startup hooks, actions, event hooks, and pane entrypoints; plugins launch as processes with HERDR_* context injected. There is a marketplace and a visible third-party ecosystem — review sidebars, file viewers, phone clients, remote mirrors. cmux’s ExtensionKit sidebars are younger and have been through at least one revert.

What cmux does better

1. It is a terminal, so it does not have to borrow one. No nested-multiplexer key contention, no arguing over ctrl+b, no “which layer ate my mouse event.” Real tabs, real drag-and-drop, GPU rendering.

2. Panes are not only PTYs. Surfaces can be terminals, browsers, markdown viewers, or file previews — and the browser is scriptable from the same CLI (cmux browser navigate|click|wait|download). An agent can be handed a rendered page and a doc alongside its shell. In herdr everything is a character grid; the nearest analog is experimental Kitty-protocol pane graphics.

3. Agent state is told, not inferred. cmux hooks setup installs session hooks for 14 agents — Claude Code, Codex, Grok, OpenCode, Pi, Amp, Cursor, Gemini, Kiro, Rovo Dev, Copilot, CodeBuddy, Factory, Qoder — and stores each one’s native resume command (claude --resume <id>, codex resume <id>, amp threads continue <id>, …), so a relaunch continues the real conversation. herdr detects state by evaluating manifests against a terminal snapshot. Detection is clever; being told is sturdier.

4. The human is in the protocol. The Feed collects permission requests and questions from every agent into one approval queue. Notifications, sidebar status pills, progress bars, and log lines are all CLI-writable by the agents themselves. herdr’s notification.show is a toast; six running agents need one blocked-list, not six toasts.

5. Remote and cloud are features, not an absence. cmux ssh creates remote workspaces with a bundled daemon and persisted PTY sessions you can list, attach, and clean up; cmux vm manages cloud VMs. Different shape from detach/reattach, but the “my agents are on another machine” case is covered.

The decision

Not “let a thousand flowers bloom.” That is what you say when you do not want to choose, and two multiplexers on one machine means two keymaps, two session stores, and two places to look for the agent that is blocked.

cmux stays the cockpit on macOS. Not because it wins on paper — on the agent-facing API it does not — but because switching cockpits costs everything built around the human loop: the approval feed, the notification wiring, hooks for 14 agents, workspace layouts, muscle memory. herdr would have to be better by a lot to clear that, and on the axis I actually sit on it is not better, it is differently good.

herdr gets adopted where cmux structurally cannot go: Linux boxes, remote hosts, SSH-first work. That is not hedging, because cmux is not competing there. It is a division of territory, not a bake-off.

No dual-running on the same Mac. If I catch myself doing it, that is evidence this split is wrong and I should re-run the comparison rather than live in both.

The tripwire, stated in advance so it is falsifiable. I switch outright if either becomes true:

  1. Primary development moves off macOS. Then cmux’s best feature — being an excellent native Mac terminal — is simply unavailable, and the rest is a wash.
  2. Agent-to-agent orchestration becomes the dominant mode — agents spawning and blocking on agents rather than me fanning out and unblocking — and cmux still has no wait verb. At that point I would be hand-rolling in event-stream shell what herdr ships as one command, which is the definition of using the wrong tool politely.

Neither is true today. Both are plausible within a year, and #2 is the one I would bet on. Review date: January 2027. A decision with no review date is just a preference.

Where a thousand flowers genuinely help is at the ecosystem level, not on my desk: herdr existing is the best argument cmux will ever get for shipping a wait verb, and cmux’s hook-based session capture is the best argument herdr will get for taking state from hooks instead of the screen. Each is holding up a mirror the other needs. I would rather have both projects than a merged one — and still pick one per machine.

What I asked of cmux

Filed, because I use it daily and can answer the follow-ups:

  • #8950 — a wait verb: cmux wait --surface <id> --until idle|needs-input --timeout <ms>, occupant-pinned, plus send --wait-until to close the submit-then-wait race. Today the closest thing is cmux events --name agent.hook.Stop --limit 1, which matches one agent’s hook vocabulary rather than semantic state and pins nothing.
  • #8951 — publish agent lifecycle as an event (agent.state.changed). The running / idle / needsInput state exists but lives in ~/.cmuxterm/<agent>-hook-sessions.json and the hibernation subsystem; it is absent from the public event catalog that already carries window.*, workspace.*, surface.*, and feed.*. It is the substrate the wait verb should be built on.

Two more that need no issue: resume bindings should stay PATH-relative rather than storing a resolved absolute path at pane creation (#6572, already fixed by #6582), and reboot restore should be a stated contract — my panes started coming back on 0.64.15 while the flag I expected to gate it was false for every pane (#5802, still open). Getting the right answer for a reason you cannot name is not a fixed bug, it is a deferred one.

What I would ask of herdr, and why it stays here

Two things would move herdr from “right tool for the remote boxes” to “candidate for the cockpit”:

Take state from hooks, not from the screen. The scaffolding exists — pane.report_agent accepts exactly that shape, pane.report_agent_session stores native session references, integration.install is there. The gap is breadth: cover agents first-party the way cmux hooks setup covers 14 of them, and let screen detection be the fallback rather than the primary path.

Give the human a queue. When six agents are running, what I need is not six toasts, it is one list of what is blocked. herdr already has the ingredients — semantic blocked state, agent.view.set projections, an agent sidebar — so this may be more assembly than invention. Both of these are things a plugin could prototype without touching the core.

I am deliberately not filing either as an issue. I have read herdr’s docs closely and have not run it in anger, and a feature request from a non-user is a maintainer tax: they have to reconstruct my context before they can even judge whether I found a real gap or just did not finish the manual. The cmux asks went to its tracker precisely because I use it daily. These stay at blog volume, where someone who actually runs herdr can correct me cheaply — and I would rather be corrected here than spend a maintainer’s triage.

One note on addressing, since the post that started this was not from the maintainer: herdr is Can Celik’s. Thanks to André for putting it in front of me — the framing in that post is what made me go read the socket API instead of skimming another launch.

Footnote: the fork nobody was maintaining

While writing this I checked our own voitta-ai/cmux fork. It was 0 commits ahead and 3,171 behind upstream — a June snapshot with no patches on it. That is not a fork, it is a stale bookmark that quietly implies we carry local changes.

We do not, and that is correct: the cmux problems I actually hit went upstream as issues, and one is already fixed there by a maintainer. Filing beats forking whenever the maintainer is responsive — a fork you do not rebase is a liability with a nice URL. Resynced while writing this; it is identical to upstream again.

It also mattered for the two issues above. Our checkout was seven weeks stale, so I checked both proposals against upstream main before filing — “open an issue for a feature that shipped last month” is a real way to waste someone’s afternoon. Still missing on current main: wait-for remains the tmux-compat named synchronization point, and the event catalog still has nothing for agent state.

The metric that graded my orchestration a C, and what it was actually measuring

Gregory Golberg holding forth, voitta , , , , , , , ,

I let an AI proficiency scanner read thirty days of my Claude Code transcripts. It handed back a 349 out of 1000 — a C — and, more interestingly, two flat zeros: Planning: 0. Customization: 0. Agent dispatches: 5.

I spend most of my time running multi-agent squads and writing skills. So either I’m worse at my own workflow than I thought, or the instrument is measuring the wrong thing. It turns out the source is right there — Elastic-licensed, readable — so I did the thing the score didn’t credit me for, and read it.

What it actually measures

The scanner is a transcript-shape heuristic. Volume it gets right: 168 sessions, 11.3k messages, 524M tokens — all there. The trouble is the three dimensions meant to capture sophistication, each pinned to a single tool-name signal:

  • Orchestration is the literal tool named Agent — and “parallel agents” only counts when two or more Agent blocks fire in one assistant turn.
  • Planning is the ExitPlanMode tool. That’s the whole definition.
  • Customization is a Write or Edit whose path ends in SKILL.md / CLAUDE.md / .mcp.json.

Three proxies, each standing in for a competence, each satisfiable by clicking the corresponding button.

Why my real work scored zero

Regular readers know the shape of my orchestration skill: an architect derives the parallel set, then each issue gets a developer in its own git worktree, an adversarial reviewer, an SDET, a productivity engineer. That is the opposite of “two Agent calls in one turn.” The squads run as background agents and separate worktree sessions, each producing its own top-level transcript. From the scanner’s vantage, that isn’t orchestration — it’s a pile of independent user sessions. The one line that decides this, is_main = "subagents" not in path, doesn’t just miss cross-process orchestration; it actively penalizes it, because every coordinated sub-session inflates the denominator as a plain session.

Planning is worse, and funnier. My whole doctrine is start with a conversation, not a spawn — the hard, valuable artifact is the parallel set an architect reasons out after reading the issues. None of that touches ExitPlanMode. So a planning discipline more deliberate than the feature scores lower than one press of the Plan button. The map graded me on whether I visited a specific city, not on whether I arrived.

And Customization: I author SKILL.md files as routine work, but from worktree sub-sessions and through git and PRs — not via the Write tool under ~/.claude/skills/ in a session it happens to be watching. The scanner even records prs_opened; it just doesn’t spend it on the dimension that would have caught me.

The actual defect

This is a construct-validity failure, not a bug. Every proxy assumes competence lives inside one process boundary and expresses itself through a named UI feature. Mine lives across boundaries — worktrees, sibling sessions, gh — and in git history the instrument never opens. It measures feature adoption and calls it proficiency. Goodhart is right there: I could raise the number tomorrow by making my workflow worse — two Agent calls per turn, a ceremonial ExitPlanMode, a Write I don’t need — and the score would thank me for the regression.

What I’d change

Same thesis I keep landing on: the score is the spaceship; the missing instrument is telemetry that reads across boundaries. Concretely — correlate sibling worktree sessions plus gh pr create into a single orchestration event; score customization from committed SKILL.md diffs, not in-transcript writes; credit the gh issue view fan-out and the long analysis turn that precede a spawn as the planning they are. I’ve written it up as an issue against the plugin, and I’ll send the scanner half as a PR. The server-side scoring isn’t in the open repo, so I can hand them a better signal but not a better weight — which is its own small lesson about what “open source” buys you.

None of this makes the tool useless. A cheap proxy over free telemetry is a reasonable place to start; I’d have started there too. But a proxy has to know it’s a proxy, and this one reports a C with the confidence of a measurement. The gap between what it counted and what I did is, as usual, the only part with information in it.

Recursive self-improvement, you said?

Gregory Golberg voitta , , , , , , ,

Spawning a fleet of coding agents is a solved problem. You write a for loop, you call the Agent tool N times, you go get coffee. The unsolved problem is everything wrapped around the spawn: deciding what can actually run in parallel, stopping the agent that wrote the code from also grading (or eating) its own homework, and — the part nobody ships — recording how the run went so the next one isn’t the same run with the same mistakes.

I wish I didn’t remember this anymore but this used to be called a “retrospective” in that sect I was once a member of.

I’ve been dogfooding a small orchestration skill (agent-team-orchestration, open in voitta-ai/skillz) that treats those as the actual work. Three runs in. This is the first write-up, warts very much included — the warts are the only part with information in them.

The shape, and the one non-negotiable rule

Start with a conversation, not a spawn. Before any developer agent exists, an architect reads the open issues (gh issue list, then actually gh issue view each one) and the repo, and produces the one deliverable that’s genuinely hard: the parallel set. Independent work (different modules, no shared schema, PRs that won’t collide on merge) fans out; everything else serializes (shared files, a migration that has to land first, B’s acceptance depends on A). Get that wrong and you don’t get parallelism, you get merge conflicts with extra steps.

Then each issue in the wave gets a squad, roles deliberately split so no agent both writes and blesses the same diff:

  • developer — its own git worktree, opens the PR;
  • adversarial reviewer — a different agent, briefed to break the diff, not rubber-stamp it;
  • SDET — drives the change like a user;
  • productivity engineer — a meta-role that watches the process: every stall, every human approval, every bit of rework, written down.

The dev/reviewer split is load-bearing. The instant the context that wrote the code also reviews it, the review is theater.

And the telemetry is free, which is the best price. Every Claude Code session is a complete JSONL transcript at ~/.claude/projects/<slug>/<uuid>.jsonl — every tool call, every AskUserQuestion, every answer you gave. (We’ll gate the privacy policy to not log every breath you take).

TFW that retrospective is not a wishful thinking, it’s actionable.

Three runs, in ascending order of interesting

Run 1 — shipped clean, screwed up in a way I didn’t catch until I read the log. Two bug fixes on a production Next.js + Prisma app (two-branch staging/prod). Both merged, deployed, SDET-verified green. Then I read the transcript: the two bugs already had open PRs from a prior run. The architect never looked. We’d built and squash-merged duplicates, closed the issues, and orphaned two perfectly good PRs.

That’s not an agent being dumb. It’s a hole in the recipe. “Choose the parallel set” reasoned about file overlap and ordering and never asked the first question a human lead asks — is anyone already on this? — which is one gh pr list away. Second tell, same run: asked “where’s the evidence the reviewer approved these?”, the answer was nowhere. The verdicts lived in the agents’ context and never touched the PR. An approval that leaves no durable artifact didn’t happen. (Worse, squash-merge later buried even the merge-commit note, but I’m getting ahead of myself.)

Run 2 — the loop closed, and I have receipts. New work — a homepage redesign across seven sub-issues — same skill. At startup the agent did something I didn’t tell it to: it ran gh issue view 122 on the prior run’s recorded retro and read the engagement log. Then it did exactly the things Run 1 botched. It pre-flighted existing PRs. Every merge carried an adversarial verdict with specifics; the reviewer caught a dead query param (?q= where the target route reads ?search=) and sent it back with REQUEST_CHANGES.

Then it got interesting. A staging route started returning 500. The team traced it to schema drift, and went to fix the deploy pipeline by adding prisma db push. The safe version (no --accept-data-loss) did the right thing and aborted:

⚠️ There might be data loss when applying the changes:
  • drop column `negotiableTerms` on `Property` (1 non-null value)
Error: Use the --accept-data-loss flag to ignore the data loss warnings

It refused to drop a column with live data, surfaced it for a human call, took a one-time --accept-data-loss against staging only, reconciled, and reverted — production never saw the flag. The redesign isn’t the headline. The headline is that the run improved because it had read how the last run went. Best current read: that’s the flywheel, showing up unprompted.

Run 3 — we pointed it at itself, which is geekily elegant, and scientifically noble I scraped every point across Runs 1–2 where an agent stopped to ask a human to approve something — fifteen gates — dumped them into one issue, and ran the skill on that issue. The architect grouped the fifteen by type, correctly separated the gates worth keeping (destructive DB ops — yes, always ask) from the avoidable friction (re-asking a runtime question it already answered two turns ago), and — the good part — ran two of the fixes on its own execution before they were written into the skill. It pre-flighted with gh pr list and caught two pre-existing issues that overlapped the work, exactly the Run-1 bug, fixed live by the thing being fixed.

What’s actually carrying the weight

  • The parallel-set call is real architecture. Run 3 ran two repos in parallel but serialized five edits that all touched one SKILL.md into a single PR — instead of four agents racing to conflict on the same file.
  • Build/attack/verify pays rent. The reviewer caught a bug the developer was happy with. Once is enough to justify the second agent.
  • Worktree-per-issue keeps the squads from knifing each other.
  • The flight recorder is the product. Every stall is a candidate fix — a default, a permission, a pre-flight, a sharper brief.

Where it falls down (best current read)

  • The headline feature has never once fired. The skill leads with “every agent is a watchable terminal tab you can steer mid-run.” That needs the root session launched through the cmux claude-teams wrapper, which prepends a tmux shim to PATH (CLAUDE_CODE_EXPERIMENTAL_AGENT_TEAMS=1 alone is a red herring — diagnose with which tmux + echo $TMUX). Three runs, three fallbacks to background agents, because the session wasn’t started that one specific way. A feature nobody reaches isn’t a feature, it’s a positioning bug.
  • The same two process bugs recur every run until baked in: a setup question asked at spawn time instead of as a step-0 precondition, and re-asking a decision already made. Prose doesn’t self-correct — the executor re-litigates your opinions until you encode them as defaults.
  • N=3 and confounded. Run 2’s wins rode on memory carried from Run 1, so I can’t yet split skill-value from memory-value. The compounding loop is a strong signal, not a proof. The honest next experiment is one run on a clean, never-seen repo, launched under cmux, with no carried memory, measured by a typed telemetry schema — which doesn’t exist yet, so I’m building that before I build anything else.

The actual thesis

Spawning is commodity; the moat is the operating doctrine plus the telemetry loop — the thing that makes human-interventions-per-issue trend down run over run. Build the instrument first, defer the spaceship. YAGNI applies to strategy, too.

Skill’s open in voitta-ai/skillz. Run it on your backlog and tell me where it stalls. The stalls are the entire point.