Benchmark explainer
What is SimpleQA Verified?
LLM short-form factuality on questions with single, indisputable answers (de-noised, de-duplicated, topically balanced from OpenAI's original SimpleQA).
SimpleQA Verified as reported in Epoch AI's Capabilities Index CSV.
How it's scored
- Metric
- normalized performance in Epoch CSV, rendered as accuracy (%)
- Score ceiling
- 100
- Construction
- 1,000 prompts; multi-stage filtering of OpenAI's 4,326-question SimpleQA: de-duplication, topic balancing, source reconciliation; reference answers verified by independent AI trainers.
- Human baseline
- Reference answers set by two independent AI trainers per question (paper §3); F1 metric tracks precision/recall against ground truth.
How trustworthy
Discrimination
100/100Does it still separate models?
Saturation headroom
98.1/100How far from ceiling / clustered at the top?
Contamination resistance
17.3/100Public vs held-out; training-leak risk.
Harness comparability
100/100Is it apples-to-apples, or mixed / vendor-optimized?
Freshness
97.8/100How old is the benchmark?
Contamination history: Released 2025-09; builds on OpenAI's public SimpleQA — answers likely appear in any web-scraped training corpus.
Limitations: Drawn from OpenAI's 2024 SimpleQA — questions span public web knowledge, contamination risk acknowledged; English-only.
Who leads SimpleQA Verified
| Model | Score | Evidence |
|---|---|---|
| Gemini 4 Argon | 76 | unverified |
| GPT-6 Astra | 75.6 | unverified |
| GPT-6.1 Sol | 73.9 | unverified |
| Gemini 3.1 Pro | 73.5 | unverified |
| Claude Opus 5.5 | 72.2 | unverified |
Compare the top SimpleQA Verified scorers
Frequently asked questions
SimpleQA Verified: SimpleQA Verified as reported in Epoch AI's Capabilities Index CSV.
Gemini 4 Argon leads SimpleQA Verified at 76. The full leaderboard above lists every recorded measurement, not just the headline number.
81 models have recorded scores on SimpleQA Verified, spanning a score spread of 70.
tensor.news grades SimpleQA Verified A for integrity (score 87/100), ranking #28 of 65 benchmarks we assess across discrimination, saturation, contamination resistance, harness comparability, and freshness.
No — SimpleQA Verified still has headroom and continues to discriminate between models rather than bunching them at the ceiling.
Its test set is unknown, so contamination risk is on the table: a high score may partly reflect training-data overlap rather than capability.
Scores are reported under a consistent harness, so comparisons on SimpleQA Verified are reasonably apples-to-apples.
Every SimpleQA Verified measurement is source-backed and tagged reproduced, self-reported, or unverified; the underlying record cites arxiv.org/abs/2509.07968.