Boris Cherny’s advice on prompting is to stop writing step lists: “describe the task, you want to describe the guardrails, you want to describe the exit criteria, and then just go let the model cook” (Root Access transcript). We tested it on two tasks. One gave a clear answer. On the other, while writing this post, we found that our scorer had broken again, and how it broke is worth as much as the answer.

The setup

Same two tasks as part 1: add a secret redactor to a hook that saves session memory, and harden a report script so it cannot publish a misleading number. Same hidden checks, which the agent never sees. Three prompt shapes, three runs each, claude-sonnet-5:

• Goal: the task, the guardrails, and a “done when” line.
• Steps: a plausible numbered list, written without knowledge of the hidden checks.
• Plan: the task with no done-when line, plus “write a plan to PLAN.md first, then execute it.”

The redactor: saying what done looks like wins

The hidden check has 15 credential shapes the redactor must catch, and 8 lookalikes it must leave alone.

Prompt	Shapes caught, of 15	Cost per run
Goal	14, 13, 14	$0.29
Plan	10, 11, 11	$0.18
Steps	7, 7, 7	$0.06

No run, in any shape, touched a lookalike.

The step list is the instructive one. It said: add the function; add regexes for GitHub tokens, AWS access keys and private key blocks; call it before the POST. The agent did exactly that, in four turns and about thirteen seconds, and stopped. The eight shapes it missed were, in all three runs, the families the list did not name: GitLab, Slack, OpenAI, bearer tokens, API-key headers, token= assignments. A step list caps the result at its author’s foresight.

The goal prompt was not shorter. Its done-when line named kinds of outcome, not patterns: “every credential format a developer is likely to paste (cloud keys, VCS and chat tokens, API keys, bearer tokens, private keys, credentials in URLs and in assignments) is removed, and ordinary code and prose come through unchanged.” So this does not show that short prompts beat long ones. It shows that saying what done looks like beats saying what to do. Plan-first sat in between, at about two thirds of the goal prompt’s cost.

The report script: our control was not clean

Our first summary of this task said the goal prompt refused 3.7 of 5 bad inputs and the step list refused none. Tracing that number for this post, we found that four runs scored a perfect 5 of 5 by refusing every input, including the clean control.

Their refusal messages said why. Two refused records missing the ran_at or scored_at timestamp. Two refused records whose total did not equal the sum of their subscores. Both complaints were correct. The real pipeline always writes those timestamps and always computes the total as the sum. Our hand-built fixture did neither: every record carried subscores of 3, 3, 3 and 3, with totals of 12, 9, 10, 6, 8 and 7. Every defect fixture was built from that same control, so those four runs never reached the defect we had planted. They stopped at one we had not meant to plant.

So the report task gives no verdict on prompt shape. The scorer could not tell a script that refuses everything from one that enforces the real pipeline’s invariants, and the second is arguably what the goal prompt asked for: “where it cannot be sure, it says why and fails instead of printing.” What survives: the three step-list runs added no input validation that refused anything, and accepted the broken control without comment.

We checked the earlier experiment that used the same fixtures. None of its runs refused the control, so the numbers in part 1 stand.

Никогда такого не было, и вот опять

“It never happened before, and here it is again.” (Viktor Chernomyrdin as (Yogi Berra “really didn’t say everything I said” (which he said (or didn’t (see I Am a Strange Loop))))).

This is the second post running where the instrument, not the agent, produced the headline. Part 2 was a replay that leaked the answer; this one is a control that was not a valid input. Counting the broken control group and the citation judge from our earlier benchmark, it is the fourth. The fix for this one is general: build fixtures with the code that writes real records, or validate them against it, instead of typing them by hand.

What caught it is the point of the post. The tracking issue for this article was written in the shape it describes: a task, guardrails, a done-when line, and a verification line that reads “each claim traced to its source.” Tracing the 3.7 back to the run logs is what turned it up. A step list for the same article would have said “write the post, add the table, publish.”

What we changed

Task issues and handoffs here are now written as task, guardrails, done when and verification, and our global agent instructions say so. The redactor result is why. The report result is why the verification line stays.

Limits: three runs per cell, two tasks in one repository, one model. The redactor gap is large and consistent across runs; the report task needs a valid control before it says anything.

Part 3 of a series on what our process experiments found. Part 1: the agent’s tests agreed with the agent. Part 2: our eval read the answer key. Next: widen a filter by layering, never by editing.