Gemini 1.5 Pro (Sept 2024) vs GPT-4o (Nov 2024)

Verdict

Gemini 1.5 Pro (Sept 2024) leads 4–3 across 8 shared benchmarks.

Gemini 1.5 Pro (Sept 2024) 4 · GPT-4o (Nov 2024) 3 · 1 tied · higher isn't always better — check the caveats below.

Per-benchmark head-to-head

Reported scores — protocols may differ. How we compare

Sort

Gemini 1.5 Pro (Sept 2024)GPT-4o (Nov 2024)

0.8
AARC-AGI-2
0
42.97
BGPQA diamond
30.51
0
AHLE
0
70.39
BMATH level 5
49.77
82.53
CMMLU
84.13
22.98
AOTIS Mock AIME 2024-2025
6.16
3.4
AThe Agent Company
8.6
22.2
AWeirdML
25.12

Capabilities

Math

Math: Gemini 1.5 Pro (Sept 2024) leads 2–0 across 2 shared math benchmarks (largest gap: MATH level 5, 70.39 vs 49.77 — single evaluator).

4 reproduced

Reasoning

Reasoning: Gemini 1.5 Pro (Sept 2024) leads 1–0 across 2 shared reasoning benchmarks (largest gap: GPQA diamond, 42.97 vs 30.51 — single evaluator).

2 reproduced2 unverified

A higher score does not always mean a better model. Each number is task performance under a disclosed harness. Of the 16 measurements across 8 shared benchmarks, 16 single-evaluator or undisclosed-protocol reports; 38% are independently reproduced. Check any benchmark's integrity grade before reading a win as decisive.

Frequently asked questions

Gemini 1.5 Pro (Sept 2024) leads 4–3 across 8 shared benchmarks. Gemini 1.5 Pro (Sept 2024) leads 4 benchmarks and GPT-4o (Nov 2024) leads 3, with 1 tied. Higher isn't always better — see the integrity caveats on each benchmark.

Gemini 1.5 Pro (Sept 2024) and GPT-4o (Nov 2024) have 8 benchmarks in common in our data — those are the only rows where a direct, apples-to-apples comparison is drawn.

Gemini 1.5 Pro (Sept 2024) leads math 2–0. Gemini 1.5 Pro (Sept 2024) leads reasoning 1–0.

Follow the record

These models

Same tier