# Supervising coding agents without lying to yourself

> Nine guardrails, the failures that produced them, and where they sit in a durable multi-agent supervision loop.

This document is generated from `README.md`, the canonical source for the nine practices, and the supervision claims shared with the human-readable page and structured JSON.

If the builder believes a false completion report, unfinished work ships.

## A day at the helm

**31 August 2026, Toolroll fleet.** 11 work records closed: 6 merged pull requests, 3 review or design records, and 2 local corrections. 2 distinct captain replies were recorded.

One reply routed to two tasks and is counted once. This is an attention count, not a speed or quality score; fewer interruptions do not prove fewer mistakes.

The keyboard could not reach a snippet a user asked to paste. A worker proposed one message for everyone.

**Question that stopped the work.** Should a customer who has already paid be told to pin the snippet, when pinning does nothing for them and implies their purchase is not working?

**Builder's answer.** No. A paying customer gets a plain statement that the snippet cannot be reached right now. The pin remedy is offered only to someone it would actually help.

That distinction decided what a person was told about something they bought. No agent should make it.

Source: Firstmate durable backlog and done archive: dated closures, merged pull-request records, and captain-hold resolutions.

## The supervision loop

Work moves through dispatch, work, review, validation, landing, and teardown. Authority does not move with it.

## 01 — Every worker gets a separate world.

Tasks begin in separate copies of the repository. Worker state is durable on disk, so a crash, restart, or closed laptop does not erase what happened.

- Separate files. Separate branches.
- Restart from disk, not from memory.
- Reconcile what exists before acting.

## 02 — Questions go up. Work waits.

A worker can act inside its authority. A question above that boundary becomes a durable stopping state routed to the supervisor.

- The worker cannot answer itself.
- The exact open decision must be resolved.
- When context is unknowable, say so.

### 3. The worker never answers its own escalation

**Where in the flow.** Worker authority boundary

**The rule.** When an agent hits a question above its authority, it stops and asks.
It does not decide, and it does not decide *and then mention it*.

**What happened.** A review gate asked whether a test seam was acceptable. The worker
escalated rather than approving its own work — and was right to, because the answer
changed the design. An agent that resolves its own blockers produces work nobody
reviewed at the moment review mattered.

### 8. Prefer explanation to inference when the model can't know

**Where in the flow.** Feature and interface design

**The rule.** When a feature cannot behave correctly in a given context, say so
plainly rather than guessing at the user's intent.

**What happened.** Snippets could contain fill-in placeholders that the keyboard
replaces on insertion. Copying one in the app pasted raw `{ask:Time}` at whoever
received it.

The tempting fix was to teach Copy to resolve placeholders. But a fill-in needs a
cursor to position, and Copy has none — there was no correct value to substitute.

The right fix was to relabel the action **Copy Exact Text** and explain what it does.
The button now makes an accurate promise to someone who reads nothing else.

## 03 — Models check each other.

Firstmate can run Claude, Codex, Cursor, Grok, OpenCode, Pi, Kimi, and Muse. The useful part is not the list. Review goes to a different model family, and work routes across providers with remaining quota.

- Different family from the implementer.
- Remaining quota routes the next job.
- Product review leaves the diff behind.

**One real day.** Implementation ran on Codex, adversarial review on Cursor running Grok, and supervision on Claude. Firstmate can also route work through OpenCode, Pi, Kimi, and Muse.

### 4. Two rounds is diligence, three is a design problem

**Where in the flow.** Repeated review findings

**The rule.** Set a stop condition before you need it. If review raises a third
finding in the same theme, stop patching and look at the model.

**What happened.** Three separate defects, each fixed individually:

1. Setting a pin to the value it already held cleared a snippet's ownership.
2. Appending an empty string cleared it too.
3. Saving a form without changing anything retired a first-run state.

Three fixes, one cause: **each consumer decided for itself whether something had
changed, instead of being told by the mutation that performed it.** Patch one caller
and the next repeats the mistake.

The stop condition caught it on the third round. Without it there would have been a
fourth, a fifth, and a codebase full of guards.

**The fix was smaller than the patches.** One answer — the mutation reports what it
actually did — replaced three independent guesses.

### 5. Review from a different model family

**Where in the flow.** Review assignment

**The rule.** Have work reviewed by a model that did not write it and does not share
its training. Independence matters more than capability.

**What happened.** Three same-family code reviews passed over an app that had **no way
to create a record inside it.** Every one checked features against their own briefs.
None opened the product cold.

A cross-family review found it in one pass — along with a privacy policy describing a
keyboard the app did not ship.

### 6. Walk the product like a stranger

**Where in the flow.** Product-level review

**The rule.** Periodically use the thing as someone who has never seen it. Not as a
test: open it, and try to do the obvious first thing.

**What happened.** See above. The missing create button was found by the person who
owns the product simply opening it — after three reviews and weeks of work by agents
that had read every file.

Reviewing a diff answers "is this change correct." It cannot answer "is this product
sensible."

## 04 — Green is a request for proof.

The validation pipeline can refuse a merge. A test has to demonstrate that it can fail, and a merge is never reported as landed without evidence from the repository.

- Red proves the test can see the seam.
- Green checks are not a merge argument.
- Release identity only moves forward.

### 1. A test must fail before you trust it passing

**Where in the flow.** Validation gate

**The rule.** Before accepting any new test, confirm it fails against the broken
behaviour it claims to cover. If it passes both ways, it is decoration.

**What happened.** This project shipped tests that pass whether or not their subject
exists — three separate times before anyone noticed the pattern.

The clearest case: a test meant to prove that cancelling a compose screen saves
nothing. It swapped one draft for another and compared storage. It never pressed
Cancel. A reviewer put it exactly right: *"A regression where those actions call
create or editBody would still pass."* The test was testing that a test did nothing.

**Why it recurs.** A passing test feels like evidence. Writing one that fails first
takes a deliberate extra step, and nothing in the tooling asks for it.

### 2. Green checks are not an argument for merging

**Where in the flow.** Merge decision and landing proof

**The rule.** CI proves the code compiles and existing tests pass. It cannot prove
the change does what its description says. Read the diff.

**What happened.** A status reader reported a task as `done · PR merged/closed`. No
pull request existed — the pipeline had skipped creating one because a credential had
expired, and only a branch had been pushed. Trusting that reading would have meant
reporting shipped work that was still sitting on a branch.

**The corollary.** Never report work as landed without proof it landed. A tool that
refuses to confirm a merge it cannot verify is worth more than one that assumes.

### 9. Version identity is functional, not cosmetic

**Where in the flow.** Release preparation

**The rule.** Never reuse a build identifier.

**What happened.** A build installed with the same version and build number as its
predecessor caused iOS to serve **stale App Intents metadata**. Siri did not learn the
app's actions at all until the phone was restarted. An hour went into diagnosing what
looked like a broken feature and was a recycled number.

## 05 — Teardown gets one chance to be right.

Cleanup refuses to proceed when saved work or current ownership cannot be proved. A safety refusal is a result, not an invitation to keep retrying.

- Unsaved work stops cleanup.
- Current ownership is checked again.
- Nothing is discarded on inference.

### 7. Never discard work you cannot prove is saved

**Where in the flow.** Worker teardown

**The rule.** Refuse teardown, force, reset, and stash by default. A refusal citing a
safety reason is a stop-and-investigate result, not an obstacle to retry past.

**What happened.** A worktree was returned to a pool and immediately reassigned. A
retried teardown then killed a *different, live* worker occupying the same slot.
Nothing was lost that time, by luck.

## Limits

These guardrails close failures that have already happened. They do not make autonomous work infallible, and they do not turn a passing pipeline into product judgment.

A refresh-timing gap was deliberately accepted and documented rather than disguised as finished. Its details are not included because this repository does not contain a public source for them.

Written by Clayton Johnson while building [Toolroll](https://toolroll.clayj.app), a snippet vault for iPhone and iPad. Every failure recorded here happened there. [clay@clayj.app](mailto:clay@clayj.app)