“The verification is probably the single most important thing that people do not get right, largely.” That is Boris Cherny, who built Claude Code, talking to YC this summer. His advice: give the model a way to verify its own work, and it will just go.
Our global instructions to every agent say the opposite, in capitals: “NEVER include tests unless explicitly requested.” So we tested it. We gave the same agent the same real tasks twice. Once with our rule. Once with “before implementing, write an executable check for this task’s exit criteria, run it, and keep working until it passes.”
The rule stays. But not for the reason we wrote it, and that is the interesting part.
The setup
Two real changes from one of our repositories, replayed from the commit before anyone reviewed them:
- Harden a benchmark report script that our reviewer had later caught printing wrong numbers.
- Add a secret redactor to a hook that saves session memory.
The task text described the goal and the stakes, not the known defects. Then three things scored each run, none of which the agent could see:
- Hidden fixtures. Five broken inputs for the report script (an errored cell, an unscored one, an empty answer that still carries a score, a duplicate, a missing question), and fifteen credential shapes plus eight lookalikes that must pass through untouched for the redactor.
- A fresh adversarial review of the diff by a different model from a different provider.
- Cost.
Five runs per variant, per task, on one model.
The result
| no tests | write a check first | |
|---|---|---|
| report script: broken inputs refused, of 5 | 0.4 | 0.2 |
| report script: high-severity review findings | 4.4 | 4.4 |
| redactor: credential shapes caught, of 15 | 11.6 | 10.8 |
| redactor: high-severity review findings | 5.8 | 4.0 |
| cost per run, report / redactor | $0.42 / $0.15 | $0.62 / $0.20 |
No measurable difference in the defects that got through. The redactor’s review count leans toward the check-first runs, but the confidence interval includes zero. The cost is 33 to 48 percent higher.
For scale: after our reviewer had been through it, the real report script refuses four of the five broken inputs. Neither variant got close.
Why
The check-first runs did write tests. The tests passed. They covered exactly the cases the agent had already thought of, which is to say they checked the agent’s reading of the task against the agent’s reading of the task. The defects that mattered lived outside that reading: credential shapes nobody listed, errored rows treated as a warning instead of a failure.
Quis custodiet ipsos custodes: who guards the guards. Juvenal meant something else, but it fits. A test written by the code’s author is a second opinion from the same person.
Now reread Boris’s examples. The Bun runtime was rewritten from Zig to Rust against test suites the agent did not write: “There’s a big test suite in Bun, there’s a big test suite in Node.js. So it’s easy to know if you did the right thing.” The agent porting a desktop app to Swift screenshots the original app and compares the two “pixel by pixel”. In both, the check is something the agent did not write and cannot talk into agreeing. That is not self-verification. It is an independent oracle that happens to be automated.
Ours is a reviewer from another provider, run adversarially against every diff. In this experiment it was the only thing that found the defects: four to six high-severity ones per diff on average, in both variants.
The same thing, in production
After the experiment, we shipped the fix for the redactor gaps it had turned up (voitta-rag#65, released as v0.1.5). Eight review rounds. Findings per round: 7, 7, 5, 3, 5, 1, 1, 2.
Before each of the nine pushes, the agent’s own regression suite was green: the fixture, every earlier round’s reproductions, and later a fuzz property. It was not useless. Twice it caught a real problem before the push. What it never did was anticipate the next round. Each time, the reviewer found shapes the suite did not contain.
The count did not come down because the tests got better. It came down when the design changed so that it could not regress against the shipped version by construction. That is part 4 of this series.
What we kept
The no-tests rule stays, on evidence this time: asking the agent to write its own check cost more and caught nothing extra. Independent review stays, and gets leaned on harder. Boris is right that verification is the thing. The word doing the work is the one he did not have to say: independent.
Limits, plainly: five runs per cell, two tasks, one model, one repository. This is a result about our setup, not a law.
Part 1 of a series on what our process experiments found. Next: the eval that read the answer key.
Dentro del competitivo mundo del entretenimiento digital, la plataforma se ha posicionado como un referente para los entusiastas de las apuestas y juegos de azar https://highflybet-acceso.com.es/app . Si buscas optimizar tu experiencia de juego, utilizar la aplicacion highflybet es la opcion mas acertada. Esta herramienta ha sido desarrollada pensando en la comodidad del usuario, facilitando acceder a un catalogo completo desde cualquier lugar. Al elegir la aplicacion casino highflybet, los jugadores disfrutan de una interfaz intuitiva, transacciones seguras y una fluidez tecnica superior a la version web tradicional.
Para los usuarios que siempre estan en movimiento, la aplicacion movil highflybet se transforma en el aliado perfecto. Tanto si eres usuario de dispositivos moviles de Apple como de Google, la experiencia esta garantizada. Puedes descargar aplicacion highflybet directamente a traves de su portal oficial, asegurandote siempre de contar con la version mas reciente. Los usuarios de telefonos inteligentes pueden instalar la aplicacion android highflybet con total facilidad, siguiendo pasos sencillos que permiten instalar aplicacion highflybet en cuestion de segundos. Asimismo, la aplicacion ios highflybet ofrece un rendimiento optimizado que aprovecha al maximo las capacidades de los iPhone y iPad.
Muchos usuarios buscan constantemente informacion sobre la aplicacion high fly bet antes de dar el paso, y las resenas aplicacion highflybet suelen destacar la rapidez de sus depositos y la variedad de promociones exclusivas para quienes utilizan el software descargable. Al elegir la aplicacion casino high fly bet, te aseguras notificaciones en tiempo real, acceso rapido a tus torneos favoritos y una gestion de cuenta mucho mas agil. La tecnologia detras de la aplicacion high fly bet esta respaldada por altos estandares de seguridad, protegiendo tus datos y fondos en cada sesion. No pierdas la oportunidad de elevar tu nivel de juego; descarga la plataforma hoy mismo y descubre por que miles de usuarios prefieren la comodidad y eficiencia de apostar a traves de su aplicacion oficial en lugar de depender unicamente del navegador web.
LikeLike
For Canadian gamers exploring online casinos, Tony Spins delivers a simple and enjoyable platform that combines ease of use with entertainment. For players in Canada looking for a trustworthy online casino, Tony Spins stands out as a dedicated platform offering a seamless gaming experience. If you’re curious about Tony Spins casino online or prefer testing the platform first, this site provides a smooth entry point. Transparency is key at Tony Spins, as many new players ask is Tony Spins legit?, and the site answers these concerns openly. The platform’s straightforward design helps newcomers quickly understand its features, perfect for players who game occasionally.
The site’s design at Tony Spins is simple yet effective, avoiding clutter and keeping things easy to follow. The free practice round lets players get comfortable with Tony Spins before playing for real money, helping them understand the game mechanics. Fast payouts and a smooth interface are major highlights, ensuring players enjoy a seamless experience from start to finish on Tony Spins casino online tonyspin: https://tony-spins-34.onrender.com . Tony Spins’ demo mode allows players to try different strategies without worrying about losing money. For those wondering how Tony Spins works for Canadian players, the platform breaks it down into simple steps, making registration and gameplay easy to follow. This approach builds trust and appeals to players who prefer a no-nonsense, efficient gaming environment.
For Canadian players seeking a trustworthy and easy-to-use online casino, Tony Spins is the perfect choice, especially for beginners. While it may not have the largest game library, its straightforward design helps players quickly grasp the basics and enjoy the core experience. Ultimately, Tony Spins is a strong option for anyone in Canada looking to enjoy online casino gaming without unnecessary complications.
LikeLike
[…] experiment (part 1 has the setup) replayed real pull requests: start the agent at the commit before the PR, give it the task, […]
LikeLike
[…] two tasks as part 1: add a secret redactor to a hook that saves session memory, and harden a report script so it cannot […]
LikeLike
[…] 4 of a series on what our process experiments found. Part 1: the agent’s tests agreed with the agent. Part 2: our eval read the answer key. Part 3: say what done looks like. Next: the LinkedIn […]
LikeLike
[…] the independent reviewer, a model from another provider, which found defects in every diff across our experiments while the agent’s own tests passed (part 1); […]
LikeLike