Claude 3.5 Sonnet vs Gemini 1.5 Pro (Sept 2024)

Verdict

Gemini 1.5 Pro (Sept 2024) leads 4–3 across 7 shared benchmarks.

Claude 3.5 Sonnet 3 · Gemini 1.5 Pro (Sept 2024) 4 · higher isn't always better — check the caveats below.

Per-benchmark head-to-head

Reported scores — protocols may differ. How we compare

Sort

Claude 3.5 SonnetGemini 1.5 Pro (Sept 2024)

46.28
ADTBench
31.63
38.72
BGPQA diamond
42.97
51.68
BMATH level 5
70.39
82
CMMLU
82.53
6.43
BOTIS Mock AIME 2024-2025
22.98
13
ASimpleBench
12.52
30.97
AWeirdML
22.2

Capabilities

Math

Math: Gemini 1.5 Pro (Sept 2024) leads 2–0 across 2 shared math benchmarks (largest gap: MATH level 5, 70.39 vs 51.68 — single evaluator).

4 reproduced

Reasoning

Reasoning: Evenly split 1–1 across 2 shared reasoning benchmarks (largest gap: GPQA diamond, 42.97 vs 38.72 — single evaluator).

2 reproduced2 unverified

A higher score does not always mean a better model. Each number is task performance under a disclosed harness. Of the 14 measurements across 7 shared benchmarks, 14 single-evaluator or undisclosed-protocol reports; 43% are independently reproduced. Check any benchmark's integrity grade before reading a win as decisive.

Frequently asked questions

Gemini 1.5 Pro (Sept 2024) leads 4–3 across 7 shared benchmarks. Claude 3.5 Sonnet leads 3 benchmarks and Gemini 1.5 Pro (Sept 2024) leads 4. Higher isn't always better — see the integrity caveats on each benchmark.

Claude 3.5 Sonnet and Gemini 1.5 Pro (Sept 2024) have 7 benchmarks in common in our data — those are the only rows where a direct, apples-to-apples comparison is drawn.

Gemini 1.5 Pro (Sept 2024) leads math 2–0.

Follow the record

These models

Same tier