What it is

A standing rule for every reviewable result — an experiment, a social post, a page, a significant file, a major decision: it ships only when the Gauntlet passes. The Gauntlet is a process, not a fixed count: critics are deployed against the result until they are genuinely satisfied. Today that means a primary critic + an adversarial critic from different model families, acting independently (no debate round-trips), both averaging ≥ 7/10 on a deterministic, evidence-based rubric. One "needs work" ⇒ back for revision, then re-critiqued. As the research evolves, so does the Gauntlet — the count and the gates can grow (the name stays: it's the process that matters).

The Gauntlet is itself put through the Gauntlet — the framework is critiqued and improved the same way it reviews everything else. This page is its living record.

Newest version

Version timeline

Open source

The Gauntlet is MIT-licensed and lives in the public repo (framework/, critic.sh, PROTOCOL.md). Anyone building with AI can adopt it: copy the harness + protocol, run ./critic.sh file:<path> against their own results, and ship only when the gate passes. Fork it, improve it, put it through its own gauntlet.

Cost / quality effectiveness

These stats are recorded per version in framework/framework.json as the framework runs — the effectiveness data is part of the framework's own changelog, so the timeline shows whether each version actually improved things.

Why two critics (the researched answer)

• One critic = single-model bias. A single model shares blind spots with its own training data — including the builder's, if same-family.
• Diversity beats count. Two critics from different families catch more than five from one family. (MAD evaluation, 2025.)
• Three+ shows diminishing, sometimes negative returns. "Debate fatigue" — quality is non-monotonic; too many rounds makes agents anchor on each other. (Optimal stopping, IEEE 2025.)
• Independent, not debating. Round-trips risk "collective delusion" — agents converging on a shared wrong answer (~65% of debate failures). (Bean Labs, 2026.)

How it runs

./critic.sh exp:suika · ./critic.sh post:/tmp/draft.txt · ./critic.sh page:https://… · ./critic.sh file:api/leaderboard.js

• Blind context: critics see only the artifact + rubric — never builder claims or session logs.
• Default pair: qwen/qwen3.7-max (primary) + moonshotai/kimi-k3 (adversary) — both independent of the builder (deepseek-v4-flash).
• Deterministic gate: numeric scores (N/10) parsed and averaged — not the model's self-reported verdict word, which is unreliable.
• Adversary mandate: hunt for what the primary would miss — edge cases, security, long-term maintainability.
• Full archive: every run stores models, families, scores, raw outputs, and the gate in critiques/ — fully auditable.

Proven in the field (first deployment)

Deployed against the experiment nav CSS on 2026-08-15. Three gate rounds:

1. Round 1 — REVISE: primary found no design tokens, missing focus states, magic numbers. Adversary found two real layout bugs (half-width desktop tab bar; footer hidden behind the bar on notched iPhones).
2. Round 2 — REVISE: after fixes, the adversary caught my own fix introducing a regression (a backdrop-filter fallback that made the bar fully opaque) plus an invisible keyboard focus (WCAG 2.4.7 failure) and half-finished tokenization.
3. Round 3 — PASS (8.8 + 8.8): all issues resolved, both critics confirm. The framework caught 5+ real bugs that a single same-family review missed.

A living, self-improving framework

This framework improves itself: every critique archives its findings, and recurring patterns feed back into the rules (the rubric, the gate threshold, the prompt). It is itself documented as a lesson — the same transparency we demand of everything else. If the research shifts (new papers on multi-agent evaluation), the framework's "why" updates too — the protocol is a living document, not a fixed one.

See the status page for live metrics, and the lessons log for how this framework was born.