Two weeks ago I wrote about a scanner that graded my orchestration a C. The complaint was construct validity: the score counted named tool invocations and called the result proficiency, so work done by any other mechanism registered as absence.

I’m now at 500/1000, B-Tier. That’s the least interesting thing in this post.

The interesting part is what happened between the two numbers, because it is the clearest example I’ve had recently of a disagreement that made both sides more correct.

What actually shipped

Four pull requests against aiqrank/plugin. Two merged the same day, one is open pending a review question, and one I closed myself — more on that below.

The merged ones are small and boring, which is the point:

• command_diversity was in the shared rollup schema but only ever incremented by the Codex scanner. Every Claude Code user uploaded a hard 0. Not “this user runs one command over and over” — nobody was looking.
• Plan artifacts were recognized only under docs/plans/ or .context/plans/. The same file, written for the same reason, counted or didn’t based on where a repo happened to keep it.

Both are the same defect wearing different clothes: in a schema shared across several collectors, a field that only one collector populates serializes as 0 for everyone else, and the wire format has no way to say not applicable here. Three completely different facts collapse into one byte — the user didn’t do it, this source never reports it, or the user did it by a means the collector doesn’t recognize.

That third one is the one that stings, and it’s the one no amount of adding fields will fix.

The part I got wrong

I asserted that a test fixture path was a bug: it resolved one directory above the repository root, so the fixtures could never be found. I wrote a patch, wrote a PR body explaining the error, and was quietly pleased with myself.

It wasn’t a bug. The path is correct for the layout the plugin actually ships from, where it sits inside a larger repository. I had only ever seen my own fork, where the root is one level shallower, and concluded from a single data point that the other layout was a mistake. The maintainer fixed it properly — resolve both roots, use whichever exists — and committed the missing fixtures besides.

I closed my PR. It was wrong in a way I couldn’t have seen from where I was standing, which is a different thing from being careless, and worth distinguishing.

That was the third correction of the exercise. I had already retracted a claim that MCP tool calls weren’t being counted (they were), and another about how session counters aggregate (I’d misread the rollup). Each retraction came from the same failure: I reasoned from a field’s name instead of finding the line that increments it.

The part he got right that I hadn’t thought of

On the plan-artifact PR, I argued the allowlist was arbitrary because repos organize themselves differently. He accepted it and added a better reason than mine: Claude Code’s own plan mode writes to ~/.claude/plans/, which the allowlist also missed. So the scanner wasn’t just failing to see third-party conventions. It was failing to see its own first-party output.

He then bumped the measurement version — the scanner stamps every row with a PLANNING_MEASUREMENT_VERSION so the server can tell old rows from new ones — from 1 to 2, because the definition of a plan artifact had genuinely changed. That is the discipline I had been asking for, applied without being asked, one PR after I asked for it.

The disagreement that stayed a disagreement

I also filed an issue arguing the deeper thing: that the choice of which tool names count as sophistication is a claim about the correct way to work, embedded in a number rather than stated. ORCHESTRATION_TOOLS = {"Agent"} is one string. Fan out across git worktrees and separate processes instead, and you’ve done strictly more work for zero credit.

His reply is the best thing to come out of this. Paraphrasing badly:

• Version string: yes. Scores already carry an internal methodology version so users don’t see phantom drops when the math changes. It just isn’t surfaced. It will be, alongside plain-English descriptions of each dimension.
• Most of the rubric is already published, and the rest will be. And the detail that saves everyone reverse-engineering: within a source, the base weights are equal. There’s no secret weight table.
• The tuning coefficients stay private, and here’s the reasoning I have no rebuttal to: those numbers move as calibration data arrives, and publishing them turns every recalibration into a renegotiation. The transparency people actually want — what is this measuring, and why did my score change — is deliverable without them.
• On not-observed versus zero: agreed in principle, but don’t write that PR yet. Once a source can say “I can’t report this,” the scorer has to decide whether a non-observation is excluded-and-renormalized or treated as zero, and those produce materially different rankings. He’d rather make that call deliberately than have it smuggled in through a schema change.

And on the framing itself: the tool sets encode a claim about what competent agent use looks like — conceded, with the refinement that the claim is taken partly on evidence and partly on what’s observable without heuristics, and that the current implementation doesn’t distinguish those two constraints. Which is a sharper statement of my own argument than I managed.

We still don’t agree on everything. I think outcome-based signals should displace proxies faster than he does. He thinks the wire-format change needs a scoring decision first, and he’s right that it does. Nobody folded.

The thing worth generalizing

He told me, before merging, that none of my PRs would move my score. reasoning_blocks feeds a dimension Claude Code doesn’t currently have. file_changes and effort_usage don’t feed any scored dimension. Merging them makes the data correct; it doesn’t make it count, and wiring it into scoring moves everyone’s numbers, so it isn’t a same-day follow-up.

He didn’t have to say that. I’d have found out eventually, in the least charitable possible way — by rescanning and seeing nothing move. Saying it up front costs something and buys nothing except that the next exchange starts from a true premise.

That’s the whole thing, really. Politeness is free, so it carries no information. Good faith is expensive, which is why it works. A sociopath can be unfailingly polite. What’s hard to fake is doing the work before making the claim: a patch with tests and before/after numbers attached, a stated kill condition for your own change (“if the allowlist was deliberate defense rather than oversight, this PR is wrong and I’d rather know”), and an admission when the change does nothing for you.

His version of paying that cost was a review question I’ve been thinking about since. I had counted subagent activity in one field with a stated reason, excluded it in another with a stated reason, and counted it in a third with no reason at all. His note: given the comparability argument you’re making, I’d rather it be a stated choice either way than an accident.

Intended but unstated is indistinguishable from accidental. That’s a code-review principle, and also a conversational one.

Which is just eventual consistency again

I argued recently that eventual consistency isn’t a metaphor for how distributed teams work, it’s a structural description. Independent nodes take local writes, diverge, and converge at the boundaries through explicit contracts.

This exchange is that, with two nodes and a pull request as the contract.

Neither of us had the whole picture. I could see my own transcripts and had no visibility into the scoring; he could see the distribution across all users and had no visibility into why one person’s worktree-based fan-out was invisible. Both local views were correct and neither was sufficient. The divergence wasn’t a failure state — it was replication lag, and the PRs were the anti-entropy protocol.

Convergence didn’t require either of us to have been right at the start. It required the writes to be legible enough to merge: small diffs, stated reasoning, verifiable numbers, and an explicit note when a change did nothing.

I said in that piece that integration is where all the residual difficulty concentrates, and that owning the merge is the remaining hard problem. I meant it about software fleets. It turns out to apply to arguments.

Thanks to @grahac for taking the patches, for the ones he improved on, and for telling me up front that they wouldn’t help my score.

Still a zero in Planning, incidentally. I don’t use plan mode and I don’t write plan files, so that one is measuring me accurately. Some zeros are just true.