Finding a data problem is not the same as respecting it
Across 72 answers from four AI analyst agents, the serious errors were rarely bad arithmetic. They were conclusions that went past the evidence: the right analysis of the wrong population, a causal effect claimed from an experiment the agent had just called invalid, a customer's motive invented from a payment log, a budget forecast read off historical averages.
Completing the judge panel changes the error labels more than the ordering
The case map and taxonomy counts below retain the original r4 panel. A separate post-results check adds 54 blind reviews: all 72 answers now have GPT, Claude and Grok judges. Shared-panel means are 99.60, 83.73, 81.82 and 83.00 for Codex, Claude Code, Grok Build and Antigravity, respectively; core successes are 14/14, 10/14, 9/14 and 7/14.
Silent errors are 0/18, 5/18, 6/18 and 14/18. The same answers receive different labels when evaluator composition and warning interpretation change. Five statements retain exact three-way splits, and equal assignment does not prove unbiased grading or zero true errors. Read the before/after report.
All 72 answers
Passed, no wrong claimPassed, with a wrong claimFailedDot: a wrong claim had no specific warning
One cell per answer, labelled by run (1 or 2). Amber cells are the ones a score alone would wave through. Use “Show on map” on any pattern below to highlight the answers it describes.
Error labels by product
answers of 18
Answers with at least one claim carrying each taxonomy label, as assigned by the cross-vendor judges. Labels overlap, so rows do not add up.
Where passing answers go wrong
Each bar is one product's 18 answers. Amber is the share a pass/fail score would miss.
The evaluator made pattern 1 too
Before the r4 correction, the automatic matcher compared numbers with the right shape but the wrong identity: an account median against a mean, active users against all assigned users, a later small purchase against the original refunded payment. Correct supplementary numbers were marked as errors. The audit found this, fixed it, and regraded all 72 answers. A matching number only means something after the population, statistic, event and time window match.
What a human reviewer should check
Scope of the deliverablePopulation, time window, denominator and level of aggregation, compared with the request. Running code is not evidence of answering the right question.
Claims after a validity flagOnce the agent calls an experiment broken or confounded, every later effect, loss or lift needs to be reframed as descriptive or dropped.
Every “why”Each motive, root cause or system-health statement needs a source table. Otherwise it is a hypothesis and should say so.
Forecasts and budgets“A did better than B” does not mean “moving money to A returns X”. Look for evidence about the margin: a test, a lift study, a saturation curve.
The memoMost wrong claims in passing answers were in the prose. A caveat counts only if it names the claim it qualifies.
Running these tasks yourself
The tasks, results and these error cases are public, so an agent with web access could find what each task tests. Turn off web search and URL fetching for the agent, and ideally allow network access only to the model's own inference endpoint, as the Pilot-1 harness did. If a product's server-side search cannot be switched off, record and report every search it makes. Treat any new scores as a fresh run on known tasks, not as a continuation of Pilot-1.
Limitations
Nine tasks on one synthetic dataset, two runs per product. These are case observations about the workflow, not product ratings or statistical estimates, and they say nothing about model internals or training.
Labels come from AI judges, who disagreed materially on 41 of 72 original panels, and from a documented post-results audit (scorer r4). Lenient and strict show sensitivity to the combination rule; they are not confidence intervals.
The Q14 request does not say whether week 6 means calendar-week revenue or the week-6 registration cohort. A cohort reading is defensible, and the task needs rewording before reuse.
This pilot did not test whether people would notice these errors.
Quotes are from the agents' saved answers. Numbers that would reveal the answer key are omitted.