Meta AI·released 2024-07-23
Scored on 14 benchmarks, leading 2. 21% of its scores are independently reproduced — the rest are self-reported or unverified.
Leads on: ARC AI2, Winogrande
Every score, ranked, colored by evidence status.
| Benchmark | Score | Evidence status | Measured |
|---|---|---|---|
| ARC AI2 | 93.73accuracy (%) | self-reported· optimized | 2024-07-23 |
| HellaSwag | 85.6accuracy (%) | self-reported· optimized | 2024-07-23 |
| TriviaQA | 82.7accuracy (%) | self-reported· optimized | 2024-07-23 |
| MMLU | 79.33accuracy (%) | unverified· optimized | 2024-07-23 |
| Winogrande | 78.4accuracy (%) | unverified· optimized | 2024-07-23 |
| BBH | 77.2accuracy (%) | self-reported· optimized | 2024-07-23 |
| PIQA | 71.8accuracy (%) | self-reported· optimized | 2024-07-23 |
| MATH level 5 | 49.77accuracy (%) | reproduced· optimized | 2024-07-23 |
| GPQA diamond | 34.55accuracy (%) | reproduced· optimized | 2024-07-23 |
| WeirdML | 21.38accuracy (%) | unverified | 2024-07-23 |
| OTIS Mock AIME 2024-2025 | 9.63accuracy (%) | reproduced· optimized | 2024-07-23 |
| SimpleBench | 7.6accuracy (%) | unverified | 2024-07-23 |
| Cybench | 7.5accuracy (%) | unverified· optimized | 2024-07-23 |
| The Agent Company | 7.4accuracy (%) | unverified | 2024-07-23 |