The Refutation Swarm: 842 Subagents That Default to Disbelieving Each Other

Forty-three. Out of 239 candidate defects my audit swarm raised against the yt-shorts-detector codebase, 43 survived to the final list. Among the refutation verdicts that actually completed, the split was 43 confirmed to 39 refuted. Nearly half of what the auditors reported, each with an exact line number and a confident rationale, was fake. I never saw any of the fake ones. A second layer of agents killed them before they reached me.

That kill rate is the whole story of this post. The reflexive way to audit a codebase with agents is to fan out one reviewer per file and trust what comes back. I have run that topology, and what comes back is a flood of plausible-but-wrong findings: real file, real line, fluent explanation, no bug. Reading 239 of those myself would have taken a full day and I would have believed too many of them. So on June 4th I wired the opposite loop and let it run for 82 minutes. My session logs show 842 subagent transcripts on disk when it finished: 601 tagged FileAuditAgent, 239 tagged RefutationAgent.

The design fits in one sentence, which I lifted straight from my own orchestration prompt: fan out one audit agent per file, have a separate agent try to refute each finding against the code, and loop until a clean pass turns up no new issues.

The inversion

Everything interesting lives in the refuter's opening line. This is the verbatim start of the prompt every RefutationAgent received (from agent-a6ed55cd675bfc170.jsonl in my session logs):

You are RefutationAgent. DISPROVE a claimed defect via adversarial review
of real code. Default to REFUTED unless the issue is undeniably real.

Default to REFUTED. Not "verify this finding," not "double-check this." The agent's success condition is destruction. A reviewer whose prior is "this is fake" filters harder than one whose prior is "looks right," because the second one is grading its own homework genre. Language models are eager to agree with fluent claims, especially claims produced by other language models in the same house style. The only reliable counterweight I have found is to make disagreement the job.

And the refuter is not allowed to disagree lazily either. Eyeballing the cited line and shrugging does not count. The same prompt walks it through a numbered break-the-claim protocol: read the cited file and confirm the exact line actually contains the claimed code; trace control and data flow to see whether the bad path is reachable with realistic inputs, or whether an upstream guard, early return, or coercion prevents it; check imports and call sites project-wide for a real caller that triggers the bug; check configuration, defaults, and constants that change whether it fires; and finally throw out anything duplicate, purely stylistic, or not pinned to an exact line.

The confirm branch is deliberately narrow (same transcript):

CONFIRM ONLY if the defect is real, reachable on a realistic input/call
path, and tied to a specific line. If the defect is real but the cited
line was slightly off, CONFIRM and set corrected_line to the true line.

That corrected_line field matters. Without it, refuters kill real bugs on a technicality when the auditor's line number drifts by one. With it, the adversary fixes the claim instead of discarding it.

The topology

The claim-generating side is just as constrained, in the opposite direction. Each FileAuditAgent gets exactly one file and a mandate to read all of it before judging anything (from the paired transcript agent-aeaaa3722e0a32196.jsonl):

You are FileAuditAgent. Audit EXACTLY ONE file for real defects.

TARGET FILE (read the ENTIRE file first via the Read tool so you see
true 1-based line numbers):

One file per agent keeps context small and line numbers honest. The auditor may read neighboring files only to validate imports, types, and call sites; it audits only its target. And its output bar is strict: report only defects grounded in specific code with an exact 1-based line number pointing at the offending code. No style notes, no "consider refactoring." Every claim that clears that bar becomes a structured record: file, line, category, severity, description, rationale, cited code. That record is the handoff. The refuter never sees the auditor's reasoning process, only the claim itself, which means it cannot be charmed by the argument. It has to go read the code.

The parent session described the model split in one line: FileAuditAgent on sonnet, casting wide; per-candidate RefutationAgent on opus as the adversarial gate. Cheap breadth for claims, expensive skepticism for verdicts. Confirmed findings merge into a canonical set keyed by file_path:line_number:issue_fingerprint, and the orchestrator's termination rule is blunt: do not terminate after finding issues; terminate only when a complete pass over all files produces zero new confirmed issues.

One defect's lifecycle

Here is a real candidate walking the gauntlet, from a later run of the same protocol (session 0abf386f, June 15). An auditor flagged filters.py:1238, severity high: the previous-short OCR fallback reads f.get("frame_num", -1), but runtime frames store their index under the key "frame", so fnum is always -1 and the range comparison segments[k2]['start'] <= -1 <= segments[k2]['end'] can never hold. Dead path, weakened consolidation guard. Plausible, specific, scary.

The refuter assigned to that claim did not evaluate the prose. It read filters.py around line 1238 and confirmed the cited code. Then it grepped frame_history.py to check what key runtime frames actually carry. Then it grepped debug_json_exporter.py and api.py to establish the full data contract, and found that the JSON-export path builds frame dicts with both "frame" and "frame_num" keys, with an explicit comment that phase 2 reads frame_num first, while the live path differs. Three files deep before rendering any judgment. That transcript is the protocol working exactly as written: the verdict, whichever way it lands, is grounded in traced data flow, not in whether the claim sounded right.

The honest numbers

I want to report this run the way it actually went, because the failure modes are as instructive as the design. The 842 transcripts break down as 779 from the workflow that finished plus 63 from a first launch I aborted after a diagnostic scare that turned out to be a false positive in my own workflow script. The finished run's self-reported tally: 778 agents, 29.5M tokens, roughly 77 minutes, 43 confirmed issues. The mtime span across all 842 transcripts is 81.6 minutes. The 601 auditor transcripts are not 62 files times three passes; they include a 4-attempt retry-with-backoff that respawned agents lost to API throttling, plus a re-pass for any file that got skipped.

The attrition math, stated carefully: 239 candidates entered refutation, 43 made the final list. Not every non-survivor died to a formal REFUTED verdict; late-run throttling washed out 157 refuters before they filed structured output. But among the 82 refuters that completed and returned a verdict through the StructuredOutput schema, the split was 43 confirmed to 39 refuted. Nearly half of the auditors' output, filtered agent by agent, claim by claim, was noise I never had to read.

Refutation versus consensus

Post 02 in this series covered my other trust mechanism: a three-agent unanimous consensus gate, where independent agents each evaluate the same artifact and the change ships only if all three vote yes. Consensus and refutation sound like siblings. They are opposites in one load-bearing way: where the burden of proof sits.

Consensus voting asks each agent, "is this right?" The default answer for a language model reading fluent output is yes, so unanimity mostly protects you when at least one agent independently stumbles onto the flaw. Refutation asks a dedicated adversary, "prove this wrong," and treats failure to disprove as the only path to confirmation. The claim is guilty until the code exonerates it. Consensus scales trust by adding agreeable voices; refutation scales trust by making one disagreeable voice do mandatory legwork: confirm the line, trace the flow, find the guard, find the caller.

For generative work, consensus is the right shape, since there is no single claim to attack. For verification work, where the swarm's output is a pile of factual claims about code, refutation filters harder for the same spend. My numbers say the auditors alone would have handed me a list that was roughly half wrong. The refutation layer absorbed that half for the cost of one opus agent per claim.

“When agents produce claims at scale, do not add more reviewers asked to agree. Invert the burden of proof — give every claim its own adversary.”

The transferable lesson is small enough to carry anywhere: when agents produce claims at scale, do not add more reviewers who are asked to agree. Invert the burden of proof. Give every claim its own adversary, tell that adversary its default verdict is REFUTED, and force the confirm branch through evidence the prompt spells out. It is the cheapest change I have made to a swarm, and it is the difference between output I skim and output I trust.