Recursive self-improvement, you said?

Spawning a fleet of coding agents is a solved problem. You write a for loop, you call the Agent tool N times, you go get coffee. The unsolved problem is everything wrapped around the spawn: deciding what can actually run in parallel, stopping the agent that wrote the code from also grading (or eating) its own homework, and — the part nobody ships — recording how the run went so the next one isn’t the same run with the same mistakes.

I wish I didn’t remember this anymore but this used to be called a “retrospective” in that sect I was once a member of.

I’ve been dogfooding a small orchestration skill (agent-team-orchestration, open in voitta-ai/skillz) that treats those as the actual work. Three runs in. This is the first write-up, warts very much included — the warts are the only part with information in them.

The shape, and the one non-negotiable rule

Start with a conversation, not a spawn. Before any developer agent exists, an architect reads the open issues (gh issue list, then actually gh issue view each one) and the repo, and produces the one deliverable that’s genuinely hard: the parallel set. Independent work (different modules, no shared schema, PRs that won’t collide on merge) fans out; everything else serializes (shared files, a migration that has to land first, B’s acceptance depends on A). Get that wrong and you don’t get parallelism, you get merge conflicts with extra steps.

Then each issue in the wave gets a squad, roles deliberately split so no agent both writes and blesses the same diff:

  • developer — its own git worktree, opens the PR;
  • adversarial reviewer — a different agent, briefed to break the diff, not rubber-stamp it;
  • SDET — drives the change like a user;
  • productivity engineer — a meta-role that watches the process: every stall, every human approval, every bit of rework, written down.

The dev/reviewer split is load-bearing. The instant the context that wrote the code also reviews it, the review is theater.

And the telemetry is free, which is the best price. Every Claude Code session is a complete JSONL transcript at ~/.claude/projects/<slug>/<uuid>.jsonl — every tool call, every AskUserQuestion, every answer you gave. (We’ll gate the privacy policy to not log every breath you take).

TFW that retrospective is not a wishful thinking, it’s actionable.

Three runs, in ascending order of interesting

Run 1 — shipped clean, screwed up in a way I didn’t catch until I read the log. Two bug fixes on a production Next.js + Prisma app (two-branch staging/prod). Both merged, deployed, SDET-verified green. Then I read the transcript: the two bugs already had open PRs from a prior run. The architect never looked. We’d built and squash-merged duplicates, closed the issues, and orphaned two perfectly good PRs.

That’s not an agent being dumb. It’s a hole in the recipe. “Choose the parallel set” reasoned about file overlap and ordering and never asked the first question a human lead asks — is anyone already on this? — which is one gh pr list away. Second tell, same run: asked “where’s the evidence the reviewer approved these?”, the answer was nowhere. The verdicts lived in the agents’ context and never touched the PR. An approval that leaves no durable artifact didn’t happen. (Worse, squash-merge later buried even the merge-commit note, but I’m getting ahead of myself.)

Run 2 — the loop closed, and I have receipts. New work — a homepage redesign across seven sub-issues — same skill. At startup the agent did something I didn’t tell it to: it ran gh issue view 122 on the prior run’s recorded retro and read the engagement log. Then it did exactly the things Run 1 botched. It pre-flighted existing PRs. Every merge carried an adversarial verdict with specifics; the reviewer caught a dead query param (?q= where the target route reads ?search=) and sent it back with REQUEST_CHANGES.

Then it got interesting. A staging route started returning 500. The team traced it to schema drift, and went to fix the deploy pipeline by adding prisma db push. The safe version (no --accept-data-loss) did the right thing and aborted:

⚠️ There might be data loss when applying the changes:
• drop column `negotiableTerms` on `Property` (1 non-null value)
Error: Use the --accept-data-loss flag to ignore the data loss warnings

It refused to drop a column with live data, surfaced it for a human call, took a one-time --accept-data-loss against staging only, reconciled, and reverted — production never saw the flag. The redesign isn’t the headline. The headline is that the run improved because it had read how the last run went. Best current read: that’s the flywheel, showing up unprompted.

Run 3 — we pointed it at itself, which is geekily elegant, and scientifically noble I scraped every point across Runs 1–2 where an agent stopped to ask a human to approve something — fifteen gates — dumped them into one issue, and ran the skill on that issue. The architect grouped the fifteen by type, correctly separated the gates worth keeping (destructive DB ops — yes, always ask) from the avoidable friction (re-asking a runtime question it already answered two turns ago), and — the good part — ran two of the fixes on its own execution before they were written into the skill. It pre-flighted with gh pr list and caught two pre-existing issues that overlapped the work, exactly the Run-1 bug, fixed live by the thing being fixed.

What’s actually carrying the weight

  • The parallel-set call is real architecture. Run 3 ran two repos in parallel but serialized five edits that all touched one SKILL.md into a single PR — instead of four agents racing to conflict on the same file.
  • Build/attack/verify pays rent. The reviewer caught a bug the developer was happy with. Once is enough to justify the second agent.
  • Worktree-per-issue keeps the squads from knifing each other.
  • The flight recorder is the product. Every stall is a candidate fix — a default, a permission, a pre-flight, a sharper brief.

Where it falls down (best current read)

  • The headline feature has never once fired. The skill leads with “every agent is a watchable terminal tab you can steer mid-run.” That needs the root session launched through the cmux claude-teams wrapper, which prepends a tmux shim to PATH (CLAUDE_CODE_EXPERIMENTAL_AGENT_TEAMS=1 alone is a red herring — diagnose with which tmux + echo $TMUX). Three runs, three fallbacks to background agents, because the session wasn’t started that one specific way. A feature nobody reaches isn’t a feature, it’s a positioning bug.
  • The same two process bugs recur every run until baked in: a setup question asked at spawn time instead of as a step-0 precondition, and re-asking a decision already made. Prose doesn’t self-correct — the executor re-litigates your opinions until you encode them as defaults.
  • N=3 and confounded. Run 2’s wins rode on memory carried from Run 1, so I can’t yet split skill-value from memory-value. The compounding loop is a strong signal, not a proof. The honest next experiment is one run on a clean, never-seen repo, launched under cmux, with no carried memory, measured by a typed telemetry schema — which doesn’t exist yet, so I’m building that before I build anything else.

The actual thesis

Spawning is commodity; the moat is the operating doctrine plus the telemetry loop — the thing that makes human-interventions-per-issue trend down run over run. Build the instrument first, defer the spaceship. YAGNI applies to strategy, too.

Skill’s open in voitta-ai/skillz. Run it on your backlog and tell me where it stalls. The stalls are the entire point.

cmux setup

My terminal of choice is now cmux. Here are my setup notes – how I run agent teams, and how I get my Claude/Codex sessions to come back after a reboot.

Contents

Agent Teams

My model: a workspace per team, either Claude Teams or OMX.

Here’s my cmux config (~/.config/cmux/cmux.json):

{
  "$schema": "https://raw.githubusercontent.com/manaflow-ai/cmux/main/web/data/cmux.schema.json",
  "schemaVersion": 1,

  "terminal": {
    "autoResumeAgentSessions": true
  },

  "actions": {
    "agents.openOMX": {
      "type": "workspaceCommand",
      "title": "Open OMX",
      "subtitle": "Start Oh My Codex in its own workspace",
      "commandName": "OMX"
    },

    "agents.openClaudeTeams": {
      "type": "workspaceCommand",
      "title": "Open Claude Teams",
      "subtitle": "Start Claude Teams in its own workspace",
      "commandName": "Claude Teams"
    }
  },

  "commands": [
    {
      "name": "OMX",
      "description": "Start interactive Oh My Codex in its own cmux workspace",
      "keywords": ["omx", "codex", "agents"],
      "restart": "ignore",
      "workspace": {
        "name": "OMX",
        "cwd": ".",
        "layout": {
          "pane": {
            "surfaces": [
              {
                "type": "terminal",
                "name": "OMX",
                "command": "bash -lc 'export PATH=\"/opt/homebrew/bin:/usr/local/bin:$PATH\"; exec cmux omx'",
                "focus": true
              }
            ]
          }
        }
      }
    },

    {
      "name": "Claude Teams",
      "description": "Start Claude Code Teams in its own cmux workspace",
      "keywords": ["claude", "teams", "agents"],
      "restart": "ignore",
      "workspace": {
        "name": "Claude Teams",
        "cwd": ".",
        "layout": {
          "pane": {
            "surfaces": [
              {
                "type": "terminal",
                "name": "Claude Teams",
                "command": "bash -lc 'export PATH=\"/opt/homebrew/bin:/usr/local/bin:$PATH\"; exec cmux claude-teams'",
                "focus": true
              }
            ]
          }
        }
      }
    }
  ]
}

After saving, run (right inside cmux):

cmux reload-config

Then open the Command Palette with Cmd+Shift+P and you can launch OMX or Claude Teams.

Surviving a reboot

The goal: after a restart, the agent panes come back on their existing conversations, not as fresh sessions.

This is the setup that currently does that for me (cmux 0.64.15, macOS 15, Apple Silicon):

  1. autoResumeAgentSessions – already true in the config above. It tells cmux to re-run each pane’s saved resume command when cmux reopens. (It is not a boot daemon; it only acts once cmux is running again.)
  2. Run cmux hooks setup once.
  3. Add cmux as a Login Item so it relaunches at login: System Settings > General > Login Items & Extensions > Open at Login > + > /Applications/cmux.app.
  4. Keep launches argument-free. cmux only restores on a no-argument launch (a Login Item / Spotlight / Dock launch all qualify). Don’t set CMUX_DISABLE_SESSION_RESTORE=1 – and watch out if you sync env vars into launchctl.
  5. Be on cmux 0.64.15 or newer – that’s the version where reboot resume started working for me.

You do not need macOS’s “Reopen windows when logging back in” – mine was off (the box was unchecked) during the reboot where everything resumed.

With all of that, after a reboot my agent panes came back resumed – 15 of 15 in my last test, each on its real prior conversation.

An honest caveat. I have not fully isolated which single piece is load-bearing, and there is an oddity: cmux records a per-pane wasAgentRunning flag that was false for every pane in the snapshot, yet the sessions still resumed. Best current read: the resume comes from cmux’s own restore in 0.64.15 plus the Login Item relaunch – not from any macOS window-reopen feature (that was off). I have since confirmed this on a later reboot: with that option off, no ordinary app restored its windows – only my auto-launch items (cmux/Slack/Texty) came back – yet 14 of 14 agent panes resumed. Full trail in the appendix.

Fallback: reopen everything yourself

A forced/hard reboot skips macOS state restoration, and you might leave the box unchecked. For those cases the conversation ids are still saved in cmux’s session snapshot, so you can re-open them yourself. This script reads the snapshot live (no hard-coded ids, so it is safe to share) and opens each saved agent in its own workspace:

#!/usr/bin/env bash
# resume-cmux-agents.sh [list|all|N]
set -u
CMUX="${CMUX_BUNDLED_CLI_PATH:-/Applications/cmux.app/Contents/Resources/bin/cmux}"
mapfile_cmds() {
  python3 - <<'PY'
import json, os
d=json.load(open(os.path.expanduser('~/Library/Application Support/cmux/session-com.cmuxterm.app.json')))
for w in d.get('windows',[]):
  for ws in w.get('tabManager',{}).get('workspaces',[]):
    for pn in ws.get('panels',[]):
      rb=(pn.get('terminal') or {}).get('resumeBinding')
      if rb and rb.get('command'):
        print((pn.get('customTitle') or 'agent') + '\t' + (rb.get('cwd') or '.') + '\t' + rb['command'])
PY
}
case "${1:-list}" in
  list) mapfile_cmds | cut -f1 | nl ;;
  all)
    win=""; [ -z "${CMUX_WORKSPACE_ID:-}" ] && win="--window $("$CMUX" current-window 2>/dev/null | head -1)"
    mapfile_cmds | while IFS=$'\t' read -r name cwd cmd; do
      "$CMUX" new-workspace --name "$name" --cwd "$cwd" --command "$cmd" $win --focus true >/dev/null 2>&1 \
        || echo "failed: $name" >&2
    done ;;
  *) mapfile_cmds | sed -n "${1}p" | cut -f3 | bash ;;
esac

To make it launchable from Spotlight, wrap it in a tiny .app bundle (Spotlight indexes .apps, not bare .sh files):

APP="$HOME/Applications/Resume cmux agents.app"
mkdir -p "$APP/Contents/MacOS"
cat > "$APP/Contents/Info.plist" <<'PLIST'
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0"><dict>
  <key>CFBundleName</key><string>Resume cmux agents</string>
  <key>CFBundleIdentifier</key><string>local.resume-cmux-agents</string>
  <key>CFBundleExecutable</key><string>resume</string>
  <key>CFBundlePackageType</key><string>APPL</string>
  <key>CFBundleVersion</key><string>1.0</string>
  <key>LSUIElement</key><true/>
</dict></plist>
PLIST
cat > "$APP/Contents/MacOS/resume" <<'SH'
#!/bin/bash
exec /bin/bash "$HOME/.config/cmux/resume-cmux-agents.sh" all
SH
chmod +x "$APP/Contents/MacOS/resume"
/System/Library/Frameworks/CoreServices.framework/Frameworks/LaunchServices.framework/Support/lsregister -f "$APP"
mdimport "$APP"

Locally-built bundles carry no quarantine, so they launch without Gatekeeper friction. Type “Resume cmux agents” in Spotlight to fire it.

If the in-app updater refuses to run

If you update cmux via its built-in (Sparkle) updater and hit SUSparkleErrorDomain 4005 “remote port connection was invalidated” with an underlying “Failed to create installation cache directory”, the usual cause is a stale com.apple.quarantine attribute on the app bundle (it was downloaded with a browser). The cache dir and code signing are red herrings. Fix:

xattr -dr com.apple.quarantine /Applications/cmux.app
rm -rf "$HOME/Library/Caches/com.cmuxterm.app/org.sparkle-project.Sparkle/PersistentDownloads/"* \
       "$HOME/Library/Caches/com.cmuxterm.app/org.sparkle-project.Sparkle/Installation/"*

Then quit cmux (Cmd-Q) and relaunch it from /Applications – then Check for Updates. The relaunch is the part people miss: if the running cmux was launched while still quarantined, stripping the attribute and clearing caches is not enough on its own – the installer keeps failing 4005 on same-process Retry until the un-quarantined bundle is relaunched. (A graceful Cmd-Q then reopen resumes your panes; only the reboot path needs the update.)

If a pane fails with “No such file or directory”

cmux saves each pane’s resume command with the absolute path to the agent binary as it was when the pane was created (e.g. ~/.nvm/versions/node/v24.2.0/bin/claude). If that binary later moves or is removed, resume execs a dead path and the pane shows No such file or directory – even though claude still works on your PATH (cmux injects a CLI shim).

The common trigger right now: Claude Code migrating from the npm/nvm global install to the native installer. The new claude lives at ~/.local/bin/claude (-> ~/.local/share/claude/versions/<v>) and the old nvm copy is deleted, so every pre-migration binding is stale. A node version upgrade/removal does the same.

Fix – reinstall so the agent is back on PATH, then relaunch each affected pane’s agent once (cmux re-records the binding with the current path; until then a stale pane fails again on the next reboot):

curl -fsSL https://claude.ai/install.sh | bash    # Claude Code (native installer)

Find the stale ones:

python3 - <<'PY'
import json, os, re
d=json.load(open(os.path.expanduser('~/Library/Application Support/cmux/session-com.cmuxterm.app.json')))
for w in d.get('windows',[]):
  for ws in w.get('tabManager',{}).get('workspaces',[]):
    for pn in ws.get('panels',[]):
      rb=(pn.get('terminal') or {}).get('resumeBinding')
      if rb:
        m=re.search(r'(/\S+/(?:claude|codex))', rb.get('command') or '')
        if m and not os.path.exists(m.group(1)):
          print('STALE:', pn.get('customTitle'), '->', m.group(1))
PY

Fixed upstream in manaflow-ai/cmux#6582 (it canonicalizes PATH-managed absolute claude/codex paths back to the bare name at restore, repairing existing stale snapshots) – merged to main, ships in the first cmux release after 0.64.16. On a build with that fix the stale bindings self-repair and the steps below are unnecessary.

P.S.

remote-control is your friend!

Appendix: how I arrived at the reboot setup

This is the investigation trail behind the Surviving a reboot section – kept separate because it is the “why”, not the “do this”.

Symptom. With autoResumeAgentSessions: true, agents came back fine after a normal quit-and-reopen, but after a macOS reboot the panes were fresh – new sessions, lost conversations.

Two distinct problems.

  1. cmux was not relaunching at all. The setting only runs when cmux reopens; it is not a boot daemon. Adding cmux as a Login Item fixed the relaunch. A trap while diagnosing this: don’t compare a process’s start time to kern.boottime – the machine can sit at the login screen for hours, and Login Items fire at login, not at kernel boot. Compare to the login time instead:

    for p in loginwindow Finder Dock; do pid=$(pgrep -x "$p"|head -1); \
      [ -n "$pid" ] && echo "$p: $(ps -o lstart= -p $pid)"; done
    ps -o lstart= -p "$(pgrep -x cmux | head -1)"
  2. Even once it relaunched, the cold-start restore came back fresh. cmux’s own session restore is gated by a per-pane wasAgentRunning flag. An interactive agent sitting idle at its prompt is recorded as wasAgentRunning=false, and the restore then skips auto-resume and starts it fresh. (Upstream: manaflow-ai/cmux#4269 added that gate to avoid resuming agents you had explicitly exited; it also catches idle-but-alive agents. I reported the reboot impact in #5802.) The fingerprint of a “came back fresh” boot – every pane wasAgentRunning=false, no process on --resume:

    ps -axo command | grep -E '/(claude|codex)' | grep -v grep \
      | grep -oE -- '--resume [0-9a-f-]+|--session-id [0-9a-f-]+' | sort | uniq -c
    
    python3 - <<'PY'
    import json, os
    d=json.load(open(os.path.expanduser('~/Library/Application Support/cmux/session-com.cmuxterm.app.json')))
    tot=run=0
    for w in d.get('windows',[]):
      for ws in w.get('tabManager',{}).get('workspaces',[]):
        for pn in ws.get('panels',[]):
          t=pn.get('terminal') or {}
          if t.get('resumeBinding'):
            tot+=1; run+= 1 if t.get('wasAgentRunning') else 0
    print(f"agent panes={tot} wasAgentRunning_true={run}")
    PY

    On cmux 0.64.10 I saw this on both a graceful reboot (0 of 12 resumed) and a forced reboot (0 of 15) – wasAgentRunning was 0 every time.

What changed. I updated cmux 0.64.10 -> 0.64.15 and kept the Login Item (and, as it turns out, “Reopen windows when logging back in” was off). After the next reboot, all 15 panes resumed:

python3 - <<'PY'
import json, os, subprocess, re
d=json.load(open(os.path.expanduser('~/Library/Application Support/cmux/session-com.cmuxterm.app.json')))
saved=set()
for w in d.get('windows',[]):
  for ws in w.get('tabManager',{}).get('workspaces',[]):
    for pn in ws.get('panels',[]):
      rb=(pn.get('terminal') or {}).get('resumeBinding')
      if rb:
        m=re.search(r'(?:--resume|resume)\s+([0-9a-f-]{36})', rb.get('command') or '')
        if m: saved.add(m.group(1))
out=subprocess.run("ps -axo command", shell=True, capture_output=True, text=True).stdout
running=set(re.findall(r'--session-id ([0-9a-f-]{36})', out)) | set(re.findall(r'(?:--resume|resume) ([0-9a-f-]{36})', out))
print(f"resumed {len(saved & running)} / {len(saved)}")
PY

reported resumed 15 / 15.

The open question. In that same snapshot wasAgentRunning was still 0 for every pane – so the resume did not come through the gate I expected; something else brought them back. I first guessed macOS “Reopen windows when logging back in” – but the next time I hit the restart dialog that box was unchecked, and I had never touched it (macOS remembers the last state), so it was almost certainly off during the successful reboot too. That effectively rules it out as the mechanism. Best current explanation: cmux 0.64.15’s own session restore, plus the Login Item relaunch, bring the agents back. I later ran that reboot: with “Reopen windows” off, only auto-launch items came back yet 14/14 panes resumed – so macOS window-reopen is not involved. On 0.64.15 the per-pane wasAgentRunning flag is null rather than false, which the #4269 gate treats as resumable – the likely reason the version bump fixed it. Upstream thread: #5802.

Upstream: https://github.com/manaflow-ai/cmux/issues/5802