blog-series-process

The LinkedIn Checklist Is the Part to Delete

Gregory Golberg voitta , , , , ,

A LinkedIn post called a fifteen-minute Boris Cherny video mandatory viewing and listed the stack to copy: plan mode, CLAUDE.md, subagents, hooks, worktrees. We had already built every item on it. In the talk itself, Boris says “maybe don’t listen to the LinkedIn influencers“, and his actual advice is the opposite of a checklist: “every six months, delete your [CLAUDE.md], delete your skills, delete your hooks.” So we measured what ours was worth before deleting it.

What our setup bought

Three arms, same model (claude-sonnet-5), same tasks: our full setup; claude --bare, which strips nearly everything; and --safe-mode, which keeps Anthropic’s default prompt and drops only ours. Our setup added about 28,800 tokens to every request: 53,300 against the default’s 24,500.

On five code questions, scored out of 12 by a judge that resolves every citation, three runs each:

SetupMean scoreCost per answerFabricated citations, 15 answers
Ours, full11.00$0.336
Anthropic default10.87$0.1913
Bare10.47$0.1923

Ours and the default are the same within noise; the 95% interval on the difference runs from −0.7 to +1.0. Ours costs 1.7 times as much. On six replays of real pull requests, every arm passed the repository’s three checks in all 18 runs, and ours cost $0.97 a run against $0.61. So: neutral on quality, 1.6 to 1.7 times on cost.

And one hook did worse than nothing. Our command-safety hook blocked the repository’s own validation script in every one of our replay runs, 23 refusals across six runs, so the agent “verified by hand”. That is the false positive part 4 fixed.

The bare arm is the useful contrast. It cost the same as the default and fabricated nearly twice as many citations. Anthropic’s prompt earns its size; ours, on these tasks, did not.

What we deleted, and what we kept

We merged two global instruction files into one and cut it from 18.7 KB to 3.2 KB: dead pointers, rules the harness already enforces, style mandates. What stayed is what an experiment defended:

  • the independent reviewer, a model from another provider, which found defects in every diff across our experiments while the agent’s own tests passed (part 1);
  • the rule that the agent does not write tests unless asked, kept on that same evidence;
  • the credential rules;
  • task, guardrails and done-when, instead of step lists (part 3).

Not everything went. The hook that injects a “mandatory skill evaluation” into every prompt still fires; it fired on every message while this post was written. Deleting is a practice, not an event.

Someone else’s checklist

The same arithmetic applies to the popular bundles. mattpocock/skills is a good one, with 38 skills and some 292,000 stars, and adopting it whole would be exactly the packing we just measured. Our rule now is to bring in a skill only against a stumble we have seen, never as a bundle.

Read skill by skill, it agrees with us more than it disagrees. Its test-driven-development skill names the “tautological” test and says “expected values must come from an independent source of truth”, which is part 1’s conclusion in one line. Its code review runs two reviewers in parallel sub-agents of the same model. Separate context is not an independent model; we would add it to cross-provider review, not swap one for the other. We have not tested same-model review, so that is a position, not a finding. Its spec skill asks for “a LONG, numbered list of user stories”, which leans against what part 3 found: length did not help, saying what done looks like did.

What a star counts

About those stars: they are not a scoreboard, and ours are in single digits. A GitHub star is one click, by a logged-in person, on the repository’s page. It costs nothing, requires no use, and is almost never taken back. It counts people who reached the page and felt like clicking, which mostly measures the author’s audience and who linked to it. Borrowed eyeballs open the door.

Our skills are installed through the plugin marketplace, which fetches the repository without anyone visiting the page where stars are given. The install path we built bypasses the place the metric is collected. The number that looks like usage instead, git clones, counts machines: plugin updates, our own sessions, CI. Pulls, not people.

That is the whole series in one move. Before reading a number, say what produced it.

Lightly held

Strong opinions, weakly held, or in our case opinionated but lightly held. What would change our minds: a task type where the full setup beats the default beyond noise, or a model generation where the bare arm stops fabricating. We will rerun the ablation when the next model ships, not on a calendar.

Limits: three to five runs per cell, claude-sonnet-5, a handful of task types, two repositories. The cost ratio is solid; “neutral on quality” means we could not see a difference, not that there is none.

Part 5, the last of a series on what our process experiments found. Part 1: the agent’s tests agreed with the agent. Part 2: our eval read the answer key. Part 3: say what done looks like. Part 4: widen a filter by layering, never by editing.

Widen a Filter by Layering, Never by Editing

Gregory Golberg voitta , , , , ,

Twice in one week we had to make a filter accept more than it did, and twice (yet again, come) the first attempt let through things the old one had stopped. One was a secret redactor, the other a safety classifier. The fix was the same both times: keep the old filter running exactly as it was, and make the new part unable to weaken its verdict.

The redactor: eight review rounds

A hook in voitta-rag saves session transcripts to a searchable memory store and redacts credentials on the way. It missed three common shapes: sk-proj- keys, API_TOKEN= assignments and Authorization: Bearer values. The task was to widen it.

The first five attempts edited its patterns in place. Each one passed our own fixture. After each one, an adversarial reviewer (Codex, read-only) found new defects, and most of them were regressions against the version already shipped: one pattern split another’s token, or an assignment pattern ate the delimiter of a private key block. Per round, the reviewer found:

Round12345678
Defects found77535112

Rounds 1 to 5 never converged, because every fix moved the ordering problem somewhere else. Between rounds 5 and 6 we changed the structure instead of the patterns:

  1. The shipped redactor runs first, unchanged. Every later layer can only add markers, so nothing it removed can reappear.
  2. Each new detector reports spans, and the union of those spans is redacted. No detector sees another’s output, so ordering cannot split a token.
  3. The passes repeat until nothing changes, because one pattern’s matches cannot overlap and a value can run straight into the next key.

Rounds 6 to 8 found only glued-token variants that also leaked on the old version. The check that matters is a property, not a fixture: across 50,000 random mixes of credentials, labels and text, nothing the old redactor removed survived the new one. The fixture went from 12 to 15 of 15 shapes.

The classifier: the same mistake, the other way up

Three days later the filter was yolt, which decides whether an agent’s command needs a human. Its Python analyzer refused any subprocess.run, so it refused a repository’s own read-only check, git rev-parse included. In 4 of 4 replay runs the agent gave up on that check and verified by hand, which removed verification rather than adding a prompt.

The first version of the fix said: if the shell rules do not call this argv unsafe, it is safe. That is the redactor’s mistake in another form. It accepted strictly more than before, and adversarial probes went straight through it: rebinding subprocess.run to os.system, git told to run a program through its environment, *extra spliced into git reset --hard, shell=True, a pager set with git -c. Each was refused before and passed after.

The merged version caps the new path at the old one. A literal argv that the shell rules do not refuse becomes “the host decides”, never “safe”; anything the argv cannot be trusted to mean keeps the old refusal. The gate is a direct comparison against the previous release on 16 probe scripts: zero unsafe-to-safe transitions.

We also nearly made it wider a second way. We proposed an allow-list that returned “safe” for read-only git, then measured what the shell path returns for git rev-parse: “unknown”. A Python “safe” would have been wider than the reference path. We withdrew it.

The rule

When a filter has to catch or allow more, do not edit it. Layer on it:

  • run the old filter verbatim, first;
  • let the new part only move the verdict in the safe direction (add redactions; downgrade a refusal to “ask”, never to “allow”);
  • gate the change on a property against the old version (nothing it caught gets through, no unsafe becomes safe), not on a fixture you wrote alongside the change.

The property is the point. A fixture checks the cases its author thought of, as part 3 found with step lists; a comparison with the previous version also checks the cases nobody wrote down.

Limits: two filters, two repositories, one week. The review-round counts are from one reviewer on one change. The layering rule says nothing about whether a filter’s coverage is good, only that widening it cannot make it worse.

Part 4 of a series on what our process experiments found. Part 1: the agent’s tests agreed with the agent. Part 2: our eval read the answer key. Part 3: say what done looks like. Next: the LinkedIn checklist is the part to delete.

Say What Done Looks Like

Gregory Golberg voitta , , , , , ,

Boris Cherny’s advice on prompting is to stop writing step lists: “describe the task, you want to describe the guardrails, you want to describe the exit criteria, and then just go let the model cook” (Root Access transcript). We tested it on two tasks. One gave a clear answer. On the other, while writing this post, we found that our scorer had broken again, and how it broke is worth as much as the answer.

The setup

Same two tasks as part 1: add a secret redactor to a hook that saves session memory, and harden a report script so it cannot publish a misleading number. Same hidden checks, which the agent never sees. Three prompt shapes, three runs each, claude-sonnet-5:

  • Goal: the task, the guardrails, and a “done when” line.
  • Steps: a plausible numbered list, written without knowledge of the hidden checks.
  • Plan: the task with no done-when line, plus “write a plan to PLAN.md first, then execute it.”

The redactor: saying what done looks like wins

The hidden check has 15 credential shapes the redactor must catch, and 8 lookalikes it must leave alone.

PromptShapes caught, of 15Cost per run
Goal14, 13, 14$0.29
Plan10, 11, 11$0.18
Steps7, 7, 7$0.06

No run, in any shape, touched a lookalike.

The step list is the instructive one. It said: add the function; add regexes for GitHub tokens, AWS access keys and private key blocks; call it before the POST. The agent did exactly that, in four turns and about thirteen seconds, and stopped. The eight shapes it missed were, in all three runs, the families the list did not name: GitLab, Slack, OpenAI, bearer tokens, API-key headers, token= assignments. A step list caps the result at its author’s foresight.

The goal prompt was not shorter. Its done-when line named kinds of outcome, not patterns: “every credential format a developer is likely to paste (cloud keys, VCS and chat tokens, API keys, bearer tokens, private keys, credentials in URLs and in assignments) is removed, and ordinary code and prose come through unchanged.” So this does not show that short prompts beat long ones. It shows that saying what done looks like beats saying what to do. Plan-first sat in between, at about two thirds of the goal prompt’s cost.

The report script: our control was not clean

Our first summary of this task said the goal prompt refused 3.7 of 5 bad inputs and the step list refused none. Tracing that number for this post, we found that four runs scored a perfect 5 of 5 by refusing every input, including the clean control.

Their refusal messages said why. Two refused records missing the ran_at or scored_at timestamp. Two refused records whose total did not equal the sum of their subscores. Both complaints were correct. The real pipeline always writes those timestamps and always computes the total as the sum. Our hand-built fixture did neither: every record carried subscores of 3, 3, 3 and 3, with totals of 12, 9, 10, 6, 8 and 7. Every defect fixture was built from that same control, so those four runs never reached the defect we had planted. They stopped at one we had not meant to plant.

So the report task gives no verdict on prompt shape. The scorer could not tell a script that refuses everything from one that enforces the real pipeline’s invariants, and the second is arguably what the goal prompt asked for: “where it cannot be sure, it says why and fails instead of printing.” What survives: the three step-list runs added no input validation that refused anything, and accepted the broken control without comment.

We checked the earlier experiment that used the same fixtures. None of its runs refused the control, so the numbers in part 1 stand.

Никогда такого не было, и вот опять

“It never happened before, and here it is again.” (Viktor Chernomyrdin as (Yogi Berra “really didn’t say everything I said” (which he said (or didn’t (see I Am a Strange Loop))))).

This is the second post running where the instrument, not the agent, produced the headline. Part 2 was a replay that leaked the answer; this one is a control that was not a valid input. Counting the broken control group and the citation judge from our earlier benchmark, it is the fourth. The fix for this one is general: build fixtures with the code that writes real records, or validate them against it, instead of typing them by hand.

What caught it is the point of the post. The tracking issue for this article was written in the shape it describes: a task, guardrails, a done-when line, and a verification line that reads “each claim traced to its source.” Tracing the 3.7 back to the run logs is what turned it up. A step list for the same article would have said “write the post, add the table, publish.”

What we changed

Task issues and handoffs here are now written as task, guardrails, done when and verification, and our global agent instructions say so. The redactor result is why. The report result is why the verification line stays.

Limits: three runs per cell, two tasks in one repository, one model. The redactor gap is large and consistent across runs; the report task needs a valid control before it says anything.

Part 3 of a series on what our process experiments found. Part 1: the agent’s tests agreed with the agent. Part 2: our eval read the answer key. Next: widen a filter by layering, never by editing.

Our Eval Read the Answer Key

Gregory Golberg voitta , , , , , ,

We had a clean result. Our full agent configuration (the instructions, skills and hooks we have piled up) made the agent worse at real work. Replaying merged pull requests, the stripped-down setups matched the merged answer four times out of four. Our full setup managed two.

It even had a mechanism. We read the transcripts. In one failed cell, the agent with our config compared its work against origin/master, decided the change had already been merged, and reverted its own work. It finished with an empty diff. Our config tells it to check origin/master. Case closed: too much instruction, the agent overthinks.

It was wrong. The agent had read the answer key.

How a replay leaks

The experiment (part 1 has the setup) replayed real pull requests: start the agent at the commit before the PR, give it the task, compare what it produces with what actually merged.

We started each replay in a git worktree. A worktree is a second checkout of the same repository, and it shares the repository’s object store and refs. So from inside a replay at an old commit, the agent could still see everything after it: origin/master, the merge commit, and the landing branch that held the exact answer. Nothing hid them. A plain git log would not show them, but git log --all would, and git show would open any of them.

We went back and classified every counted replay cell by whether any command it ran touched a commit from the future. Eight of twelve did, spread across all three setups. The “four of four versus two of four” comparison was void.

And the cell with the mechanism, the agent that reverted its own work? It looked at origin/master, saw the real merged PR sitting there, and correctly concluded the work was already done. It was the most rational agent in the experiment. We had built it a world where the future was on disk.

Clever Hans was the horse who could do arithmetic until someone noticed he was reading his questioner’s face. The horse was not cheating. The setup was answering for him.

The fix

Each replay now gets a clone that holds only the parent commit and its ancestors:

git branch -f isolate <parent>
git clone --no-local --single-branch --no-tags --branch isolate <repo> <dest>
git -C <dest> remote remove origin
git -C <dest> reflog expire --expire=now --all

No remote, no tags, one branch, no reflog. We checked it: the merge commit is unreachable, and git log --all ends at the parent. The --no-local matters: by default, git clones a local path by hard-linking the source’s whole object store, future commits included.

What changed when we reran

With the leak closed, the full setup no longer lost on correctness: it matched the merged answer in 4 of 6 cells, the same as Anthropic’s default setup. What survived was cost: the same answers at roughly 1.6 to 1.7 times the price, and a hook of ours that blocked the repository’s own validator in every full-setup cell. The verdict went from “our config makes the agent wrong” to “our config makes the agent expensive.” Those lead to different decisions, and we had very nearly acted on the first one.

The same leak had voided the first version of another task in the same week. It was caught by the same audit and rerun the same way.

What did not need rerunning: the question-and-answer half of the experiment, because that repository’s history holds no answers to our questions. Knowing why a result is unaffected is part of the audit, not a footnote to it.

The pattern

It is not the first time an audit of our own instrument changed a result. Our control group was broken and it cost us 4.2 points; a citation judge resolved paths wrong and scored real citations as fabricated. Each time the bad number came with a plausible story. Each time the story was what made it dangerous.

So the rule we follow: before comparing the arms, audit what each arm could see. For replays of version-controlled work, that means asking whether the answer is reachable from the starting state, because if it is, some agent will find it, and it will be right to.

We wrote the technique up as a public skill, agent-eval-replay-history-leak.

Part 2 of a series on what our process experiments found. Part 1: the agent’s tests agreed with the agent. Next: say what done looks like.

The Agent’s Tests Agreed With the Agent

Gregory Golberg voitta , , , , , ,

“The verification is probably the single most important thing that people do not get right, largely.” That is Boris Cherny, who built Claude Code, talking to YC this summer. His advice: give the model a way to verify its own work, and it will just go.

Our global instructions to every agent say the opposite, in capitals: “NEVER include tests unless explicitly requested.” So we tested it. We gave the same agent the same real tasks twice. Once with our rule. Once with “before implementing, write an executable check for this task’s exit criteria, run it, and keep working until it passes.”

The rule stays. But not for the reason we wrote it, and that is the interesting part.

The setup

Two real changes from one of our repositories, replayed from the commit before anyone reviewed them:

  • Harden a benchmark report script that our reviewer had later caught printing wrong numbers.
  • Add a secret redactor to a hook that saves session memory.

The task text described the goal and the stakes, not the known defects. Then three things scored each run, none of which the agent could see:

  • Hidden fixtures. Five broken inputs for the report script (an errored cell, an unscored one, an empty answer that still carries a score, a duplicate, a missing question), and fifteen credential shapes plus eight lookalikes that must pass through untouched for the redactor.
  • A fresh adversarial review of the diff by a different model from a different provider.
  • Cost.

Five runs per variant, per task, on one model.

The result

no testswrite a check first
report script: broken inputs refused, of 50.40.2
report script: high-severity review findings4.44.4
redactor: credential shapes caught, of 1511.610.8
redactor: high-severity review findings5.84.0
cost per run, report / redactor$0.42 / $0.15$0.62 / $0.20

No measurable difference in the defects that got through. The redactor’s review count leans toward the check-first runs, but the confidence interval includes zero. The cost is 33 to 48 percent higher.

For scale: after our reviewer had been through it, the real report script refuses four of the five broken inputs. Neither variant got close.

Why

The check-first runs did write tests. The tests passed. They covered exactly the cases the agent had already thought of, which is to say they checked the agent’s reading of the task against the agent’s reading of the task. The defects that mattered lived outside that reading: credential shapes nobody listed, errored rows treated as a warning instead of a failure.

Quis custodiet ipsos custodes: who guards the guards. Juvenal meant something else, but it fits. A test written by the code’s author is a second opinion from the same person.

Now reread Boris’s examples. The Bun runtime was rewritten from Zig to Rust against test suites the agent did not write: “There’s a big test suite in Bun, there’s a big test suite in Node.js. So it’s easy to know if you did the right thing.” The agent porting a desktop app to Swift screenshots the original app and compares the two “pixel by pixel”. In both, the check is something the agent did not write and cannot talk into agreeing. That is not self-verification. It is an independent oracle that happens to be automated.

Ours is a reviewer from another provider, run adversarially against every diff. In this experiment it was the only thing that found the defects: four to six high-severity ones per diff on average, in both variants.

The same thing, in production

After the experiment, we shipped the fix for the redactor gaps it had turned up (voitta-rag#65, released as v0.1.5). Eight review rounds. Findings per round: 7, 7, 5, 3, 5, 1, 1, 2.

Before each of the nine pushes, the agent’s own regression suite was green: the fixture, every earlier round’s reproductions, and later a fuzz property. It was not useless. Twice it caught a real problem before the push. What it never did was anticipate the next round. Each time, the reviewer found shapes the suite did not contain.

The count did not come down because the tests got better. It came down when the design changed so that it could not regress against the shipped version by construction. That is part 4 of this series.

What we kept

The no-tests rule stays, on evidence this time: asking the agent to write its own check cost more and caught nothing extra. Independent review stays, and gets leaned on harder. Boris is right that verification is the thing. The word doing the work is the one he did not have to say: independent.

Limits, plainly: five runs per cell, two tasks, one model, one repository. This is a result about our setup, not a law.

Part 1 of a series on what our process experiments found. Next: the eval that read the answer key.