Gemini 1.5 Pro (May 2024) vs phi-3-medium 14B

Verdict

Gemini 1.5 Pro (May 2024) leads 4–0 across 4 shared benchmarks.

Gemini 1.5 Pro (May 2024) 4 · phi-3-medium 14B 0 · higher isn't always better — check the caveats below.

Per-benchmark head-to-head

Reported scores — protocols may differ. How we compare

Sort

Gemini 1.5 Pro (May 2024)phi-3-medium 14B

85.6
BBBH
75.2
27.82
BGPQA diamond
3.45
40.75
BMATH level 5
17.56
81.2
CMMLU
70.67

Capabilities

Reasoning

Reasoning: Gemini 1.5 Pro (May 2024) leads 2–0 across 2 shared reasoning benchmarks (largest gap: GPQA diamond, 27.82 vs 3.45 — single evaluator).

2 reproduced1 self-reported1 unverified

A higher score does not always mean a better model. Each number is task performance under a disclosed harness. Of the 8 measurements across 4 shared benchmarks, 8 single-evaluator or undisclosed-protocol reports; 50% are independently reproduced. Check any benchmark's integrity grade before reading a win as decisive.

Frequently asked questions

Gemini 1.5 Pro (May 2024) leads 4–0 across 4 shared benchmarks. Gemini 1.5 Pro (May 2024) leads 4 benchmarks and phi-3-medium 14B leads 0. Higher isn't always better — see the integrity caveats on each benchmark.

Gemini 1.5 Pro (May 2024) and phi-3-medium 14B have 4 benchmarks in common in our data — those are the only rows where a direct, apples-to-apples comparison is drawn.

Gemini 1.5 Pro (May 2024) leads reasoning 2–0.

Follow the record

These models

Same tier

Frequently asked questions

Gemini 1.5 Pro (May 2024) and phi-3-medium 14B have 4 benchmarks in common in our data — those are the only rows where a direct, apples-to-apples comparison is drawn.

Gemini 1.5 Pro (May 2024) leads reasoning 2–0.