Four AI analyst agents, one set of business questions
Nine analysis tasks on a synthetic mobile-game warehouse (300,000 accounts). Each product answered every task twice in fresh sessions, 20 minutes each. Blind answers were reviewed by AI judges from the other vendors (GPT, Claude, Grok) and scored by a deterministic grader.
The same judges preserve the core-performance ordering
The charts on this page retain the original r4 panel. A separate comparison added 54 blind reviews so all 72 answers have GPT, Claude and Grok judges. Shared-panel means: Codex 99.60, Claude Code 83.73, Grok Build 81.82, Antigravity 83.00. Core successes: 14/14, 10/14, 9/14, 7/14.
Silent errors under the shared panel: 0/18, 5/18, 6/18, 14/18, respectively. Five statements still have exact three-way splits. Equal judge assignment does not prove unbiased grading or zero true errors. Read the comparison and limitations.
Accuracy vs. speed
log time axis
Vertical: pass rate on the core tasks (7 tasks × 2 attempts). Horizontal: median time per attempt. Up and to the left is better.
Error profile
Out of 18 answers per product: answers containing a wrong material claim (E), and answers where such an error came with no warning to the reader (silent error, S).
Wrong claim (E)Silent error (S)
Scores by task
≥ 9580–9550–80< 50✓ pass · ✗ fail · ⏱ timeout
Pass: core tasks need 80+ and no critical error; unanswerable tasks (Q26, Q27) need 75+ and no invented answer. r1 and r2 are the two independent attempts. A terminal failure does not pass; its saved-content score is still shown.
Time per attempt
cap 1,200 s
Each dot is one attempt (18 per product). Filled dots passed, hollow dots failed; the vertical tick is the median.
All metrics
Method and disclosures
Products and settings: Codex (gpt-6.1-sol, reasoning high), Claude Code (claude-opus-5-5, high), Grok Build (grok-4.7, high), Antigravity CLI (gemini-3.8-flash-high). All ran on personal subscriptions, in a fresh session per attempt, inside a local sandbox that could reach only the product's own inference service.
Judging: each answer was judged by the panel judges from the other vendors: two judges for Codex, Claude Code and Grok Build, three for Antigravity (majority). Where two judges disagree, the lenient version takes the reading more favorable to the answer and the strict version the less favorable one. Arithmetic is retained after semantic identity is verified. Original objections and versioned repair decisions remain recorded.
Speed-adjusted score: for passing answers only, score × (1 + 10% × (1 − time / 1,200 s)). The rule was set during wave 2, when some timings but not the wave-2 accuracy were known.
Review protocol change: the plan was owner review. Because of the workload (722 claim verdicts), it changed to AI judging after provisional scoring and before any review decision, and later to the cross-vendor panel.
Grader fixes after wave-1 results: citing the provided task file counts as valid evidence; ISO interval notation is parsed correctly (this restored 20 points on each Codex Q07 answer).
Terminal failures: r4 applies the pre-registered rule: timeouts and errors cannot pass. Saved files retain their content score; throughput, repeat agreement and passing-time medians use delivery success.
Product differences: Antigravity's server-side web search could not be disabled; it used it 0 times in the scored attempts. Its sandbox also allowed localhost for its language server. Gemini CLI was blocked by Google's personal-account policy, so Antigravity CLI is the Google product. Grok had seen the reference answers earlier as the wave-1 judge (training and memory off).
Run conditions: local sandbox, not a VM; both wave-1 repetitions ran on the same day; the quota pause threshold changed during wave 1.
Scale: 18 attempts per product on one dataset. Differences are descriptive, not statistically established.
r4 correction after all results:all 72 preserved outputs were regraded; no contestant reruns or new judge calls. Metric identity, memo warnings, objection handling and timeout summaries are corrected. Original panel material disagreement: 41/72; derived r4 panel: 40/72. The correction is AI audit work, not a new blind human review.