Google DeepMind·released 2024-09-241 source
Gemini 1.5 Pro (Sept 2024) benchmark scores: 12 benchmarks tracked. 25% of its scores are independently reproduced — the rest are self-reported or unverified.
Leads on 0 of 12 benchmarks·best result 82.53 on MMLU (unverified)·3 of 12 independently reproduced
| Benchmark | Score | Evidence status | Measured |
|---|---|---|---|
| MMLU | 82.53accuracy (%) | unverified· optimizedT1 | 2024-09-24Stanford CRFM Leaderboard |
| MATH level 5 | 70.39accuracy (%) | reproduced· optimizedT1 | 2024-09-24Epoch AI |
| GPQA diamond | 42.97accuracy (%) | reproduced· optimizedT1 | 2024-09-24Epoch AI |
| CadEval | 34accuracy (%) | unverifiedT1 | 2024-09-24CadEval Dashboard |
| DTBench | 31.63accuracy (%) | unverifiedT1 | 2024-09-24https://conceptualreasoning.ai/dtbench |
| OTIS Mock AIME 2024-2025 | 22.98accuracy (%) | reproduced· optimizedT1 | 2024-09-24Epoch AI |
| WeirdML | 22.2accuracy (%) | unverifiedT1 | 2024-09-24https://htihle.github.io/weirdml.html |
| Balrog | 21accuracy (%) | unverifiedT1 | 2024-09-24https://balrogai.com/ |
| SimpleBench | 12.52accuracy (%) | unverifiedT1 | 2024-09-24SimpleBench Leaderboard |
| The Agent Company | 3.4accuracy (%) | unverifiedT1 | 2024-09-24TheAgentCompany experiment results github |
| ARC-AGI-2 | 0.8accuracy (%) | unverifiedT2 | 2024-09-24 |
| HLE | 0accuracy (%) | unverifiedT2 | 2024-09-24 |
Claims drawn from cited facts, not live model generation.
This model was developed by Google DeepMind, placing its origin within a leading research organization. It was released in 2024, making it a recent vintage among contemporary models.
2 cited facts
This model is scored on 12 tracked benchmarks, and it currently holds no top score on any of them. Its record can be taken at face value, since the results have been independently reproduced and are backed up by outside evaluation rather than resting on vendor claims alone.
3 cited facts
On average this model trails the best reported score by 51.21 points, a wide gap under the disclosed harness that reads as well behind the leaders rather than merely short of the front of the pack. Consistent with that, it holds the top score on 0 benchmarks, meaning it has no category leadership anywhere in the reported suite.
2 cited facts
1 facts cross-checked across data sources: 1 single-source.
Source facts, citations, and refresh stamp for this record.
Sources: epoch_benchmarks
Gemini 1.5 Pro (Sept 2024) is an AI model developed by Google DeepMind, released 2024-09-24. Its benchmark record below tags every score as independently reproduced, self-reported, or unverified.
Gemini 1.5 Pro (Sept 2024) has recorded scores on 12 benchmarks, each shown with its evidence status.
Gemini 1.5 Pro (Sept 2024) has recorded scores on 12 benchmarks — GPQA diamond, MATH level 5, OTIS Mock AIME 2024-2025, DTBench, WeirdML, CadEval, and 6 more. The full table above shows each score with its evidence status.
3 of 12 recorded scores (25%) are independently reproduced rather than self-reported by the lab.
Gemini 1.5 Pro (Sept 2024) has 12 tracked claims: 3 independently reproduced, 9 unverified.