Claude 3.5 Sonnet (October 2024) vs Gemini 1.5 Pro (Sept 2024)

Verdict

Claude 3.5 Sonnet (October 2024) leads 7–3 across 11 shared benchmarks.

Claude 3.5 Sonnet (October 2024) 7 · Gemini 1.5 Pro (Sept 2024) 3 · 1 tied · higher isn't always better — check the caveats below.

Per-benchmark head-to-head

Reported scores — protocols may differ. How we compare

Sort

Claude 3.5 Sonnet (October 2024)Gemini 1.5 Pro (Sept 2024)

32.6
ABalrog
21
48
ACadEval
34
46.4
ADTBench
31.63
40.4
BGPQA diamond
42.97
0
AHLE
0
56.95
BMATH level 5
70.39
83.07
CMMLU
82.53
8.38
BOTIS Mock AIME 2024-2025
22.98
29.68
ASimpleBench
12.52
24
AThe Agent Company
3.4
39.97
AWeirdML
22.2

Capabilities

Reasoning

Reasoning: Evenly split 1–1 across 3 shared reasoning benchmarks (largest gap: SimpleBench, 29.68 vs 12.52 — single evaluator).

2 reproduced4 unverified

Math

Math: Gemini 1.5 Pro (Sept 2024) leads 2–0 across 2 shared math benchmarks (largest gap: OTIS Mock AIME 2024-2025, 22.98 vs 8.38 — single evaluator).

4 reproduced

A higher score does not always mean a better model. Each number is task performance under a disclosed harness. Of the 22 measurements across 11 shared benchmarks, 22 single-evaluator or undisclosed-protocol reports; 27% are independently reproduced. Check any benchmark's integrity grade before reading a win as decisive.

Frequently asked questions

Claude 3.5 Sonnet (October 2024) leads 7–3 across 11 shared benchmarks. Claude 3.5 Sonnet (October 2024) leads 7 benchmarks and Gemini 1.5 Pro (Sept 2024) leads 3, with 1 tied. Higher isn't always better — see the integrity caveats on each benchmark.

Claude 3.5 Sonnet (October 2024) and Gemini 1.5 Pro (Sept 2024) have 11 benchmarks in common in our data — those are the only rows where a direct, apples-to-apples comparison is drawn.

Gemini 1.5 Pro (Sept 2024) leads math 2–0.

Follow the record

These models

Same tier