Gemini 1.5 Flash (Sep 2024) vs GPT-4o mini

Verdict

Gemini 1.5 Flash (Sep 2024) leads 5–3 across 8 shared benchmarks.

Gemini 1.5 Flash (Sep 2024) 5 · GPT-4o mini 3 · higher isn't always better — check the caveats below.

Per-benchmark head-to-head

Reported scores — protocols may differ. How we compare

Sort

Gemini 1.5 Flash (Sep 2024)GPT-4o mini

14.6
ABalrog
17.4
76
AGeoBench
64
29.76
BGPQA diamond
16.96
61.87
BMATH level 5
52.63
65.2
CMMLU
75.73
16.17
AOTIS Mock AIME 2024-2025
6.85
75
CPIQA
77.4
24.87
AWeirdML
11.76

Capabilities

Math

Math: Gemini 1.5 Flash (Sep 2024) leads 2–0 across 2 shared math benchmarks (largest gap: OTIS Mock AIME 2024-2025, 16.17 vs 6.85 — single evaluator).

4 reproduced

A higher score does not always mean a better model. Each number is task performance under a disclosed harness. Of the 16 measurements across 8 shared benchmarks, 16 single-evaluator or undisclosed-protocol reports; 38% are independently reproduced. Check any benchmark's integrity grade before reading a win as decisive.

Frequently asked questions

Gemini 1.5 Flash (Sep 2024) leads 5–3 across 8 shared benchmarks. Gemini 1.5 Flash (Sep 2024) leads 5 benchmarks and GPT-4o mini leads 3. Higher isn't always better — see the integrity caveats on each benchmark.

Gemini 1.5 Flash (Sep 2024) and GPT-4o mini have 8 benchmarks in common in our data — those are the only rows where a direct, apples-to-apples comparison is drawn.

Gemini 1.5 Flash (Sep 2024) leads math 2–0.

Follow the record

These models

Same tier

Frequently asked questions

Gemini 1.5 Flash (Sep 2024) and GPT-4o mini have 8 benchmarks in common in our data — those are the only rows where a direct, apples-to-apples comparison is drawn.

Gemini 1.5 Flash (Sep 2024) leads math 2–0.