Massive Multitask Language Understanding
Integrity rank #57 of 65 · 102 models scored · top score 84.13 · GPT-4o (Nov 2024)
Knowledge breadth across 57 subjects via MC
Strongest on discrimination, weakest on contamination resistance. Caveat: scores come from an inconsistent mix of harnesses, so head-to-head comparisons are not apples-to-apples.
Reported scores — protocols may differ. How we compare
| # | Model | Score | Evidence | Measured |
|---|---|---|---|---|
| 1 | GPT-4o (Nov 2024) | 84.13 | self-reported· optimizedT1 | 2024-11-20 |
| 2 | Claude 3.5 Sonnet (October 2024) | 83.07 | unverified· optimizedT1 | 2024-10-22 |
| 3 | DeepSeek-V3 | 82.93 | unverified· optimizedT1 | 2024-12-26 |
| 4 | Gemini 1.5 Pro (Sept 2024) | 82.53 | unverified· optimizedT1 | 2024-09-24 |
| 5 | Claude 3.5 Sonnet | 82 | unverified· optimizedT1 | 2024-06-20 |
| 6 | GPT-4 (Mar 2023) | 81.87 | self-reported· optimizedT1 | 2023-03-14 |
| 7 | Llama 3.3 70B | 81.73 | unverified· optimizedT1 | 2024-12-06 |
| 8 | Gemini 1.5 Pro (May 2024) | 81.2 | unverified· optimizedT1 | 2024-05-14 |
| 9 | Qwen2.5-72B | 80.4 | self-reported· optimizedT1 | 2024-09-19 |
| 10 | Phi-4 | 79.73 | self-reported· optimizedT1 | 2024-12-12 |
| 11 | Claude 3 Opus | 79.47 | unverified· optimizedT1 | 2024-02-29 |
| 12 | Llama 3.1-405B | 79.33 | unverified· optimizedT1 | 2024-07-23 |
| 13 | GPT-4o (Aug 2024) | 79.07 | unverified· optimizedT1 | 2024-08-06 |
| 14 | GPT-4o (May 2024) | 78.93 | unverified· optimizedT1 | 2024-05-13 |
| 15 | Qwen2-72B | 76.53 | unverified· optimizedT1 | 2024-06-07 |
| 16 | GPT-4 (Jun 2023) | 76.53 | unverified· optimizedT1 | 2023-06-13 |
| 17 | Amazon Nova Pro | 76 | unverified· optimizedT1 | 2024-12-03 |
| 18 | GPT-4o mini | 75.73 | self-reported· optimizedT1 | 2024-07-18 |
| 19 | GPT-4 Turbo (Apr 2024) | 75.07 | unverified· optimizedT1 | 2024-04-09 |
| 20 | Llama 3.2 90B | 73.73 | unverified· optimizedT1 | 2024-09-24 |
| 21 | Llama 3.1-70B | 73.47 | unverified· optimizedT1 | 2024-07-23 |
| 22 | Mistral Large 2 (Jul 2024) | 73.33 | unverified· optimizedT1 | 2024-07-24 |
| 23 | Gemini 2.0 Flash (Dec 2024) | 72.93 | unverified· optimizedT1 | 2024-12-11 |
| 24 | GPT-4 Turbo (Nov 2023) | 72.8 | unverified· optimizedT1 | 2023-11-06 |
| 25 | Llama 3-70B | 72.4 | unverified· optimizedT1 | 2024-04-18 |
| 26 | Qwen2.5-Coder-32B | 72.13 | self-reported· optimizedT1 | 2024-09-18 |
| 27 | Claude 2 | 71.33 | self-reported· optimizedT1 | 2023-07-11 |
| 28 | DeepSeek-V2 (MoE-236B, May 2024) | 71.2 | self-reported· optimizedT1 | 2024-05-07 |
| 29 | phi-3-medium 14B | 70.67 | unverified· optimizedT1 | 2024-04-23 |
| 30 | Gemini 1.5 Flash (May 2024) | 70.53 | unverified· optimizedT1 | 2024-05-23 |
| 31 | Mixtral 8x22B | 70.4 | unverified· optimizedT1 | 2024-04-17 |
| 32 | Yi-34B | 68.4 | unverified· optimizedT1 | 2023-11-02 |
| 33 | Claude 3 Sonnet | 67.87 | unverified· optimizedT1 | 2024-02-29 |
| 34 | Gemma 2 27B | 67.6 | unverified· optimizedT1 | 2024-06-24 |
| 35 | phi-3-small 7.4B | 67.6 | self-reported· optimizedT1 | 2024-04-23 |
| 36 | Qwen2.5-Coder-14B | 66.93 | self-reported· optimizedT1 | 2024-09-18 |
| 37 | Claude 3.5 Haiku | 65.73 | unverified· optimizedT1 | 2024-10-22 |
| 38 | Gemini 1.5 Flash (Sep 2024) | 65.2 | unverified· optimizedT1 | 2024-09-24 |
| 39 | Claude 3 Haiku | 65.07 | unverified· optimizedT1 | 2024-03-07 |
| 40 | Claude 2.1 | 64.67 | unverified· optimizedT1 | 2023-11-21 |
| 41 | Claude Instant | 64.53 | self-reported· optimizedT1 | 2023-08-09 |
| 42 | Qwen2.5-7B | 63.87 | unverifiedT1 | 2024-09-19 |
| 43 | Gemma 2 9B | 62.8 | unverified· optimizedT1 | 2024-06-24 |
| 44 | GPT-3.5 Turbo (Nov 2023) | 61.87 | self-reported· optimizedT1 | 2023-11-06 |
| 45 | Mixtral 8x7B | 60.8 | unverified· optimizedT1 | 2023-12-11 |
| 46 | Falcon-180B | 60.8 | unverified· optimizedT1 | 2023-09-06 |
| 47 | Gemini 1.0 Pro | 60 | unverified· optimizedT1 | 2023-12-13 |
| 48 | Llama 2-70B | 59.87 | unverified· optimizedT1 | 2023-07-18 |
| 49 | Command R+ | 59.2 | unverifiedT1 | 2024-08-30 |
| 50 | GPT-3.5 Turbo (Jun 2023) | 58.53 | unverified· optimizedT1 | 2023-06-13 |
| 51 | Llama 3-8B | 58.4 | unverified· optimizedT1 | 2024-04-18 |
| 52 | Mistral Large | 58.4 | unverified· optimizedT1 | 2024-02-26 |
| 53 | phi-3-mini 3.8B | 58.4 | self-reported· optimizedT1 | 2024-04-23 |
| 54 | Stable Beluga 2 | 58.13 | self-reported· optimizedT1 | 2023-07-20 |
| 55 | Yi-9B | 57.87 | unverified· optimizedT1 | 2024-03-01 |
| 56 | Qwen2.5-Coder (7B) | 57.33 | self-reported· optimizedT1 | 2024-09-18 |
| 57 | GPT-3.5 Turbo (Jan 2024) | 56.4 | unverified· optimizedT1 | 2024-01-25 |
| 58 | Qwen-14B | 55.07 | self-reported· optimizedT1 | 2023-09-24 |
| 59 | Gemma 7B | 54.8 | unverified· optimizedT1 | 2024-02-21 |
| 60 | StarCoder 2 15B | 52.13 | self-reported· optimizedT1 | 2024-02-20 |
| 61 | Yi 6B | 52 | unverified· optimizedT1 | 2023-11-22 |
| 62 | LLaMA-65B | 51.2 | unverified· optimizedT1 | 2023-02-24 |
| 63 | Llama 2-34B | 50.13 | unverified· optimizedT1 | 2023-07-18 |
| 64 | Mistral 7B v0.1 | 50 | unverified· optimizedT1 | 2023-09-27 |
| 65 | DeepSeek-Coder-V2-Lite-Base | 47.33 | self-reported· optimizedT1 | 2024-06-13 |
| 66 | Mistral 7B v0.3 | 46.53 | unverifiedT1 | 2024-05-27 |
| 67 | Baichuan2-13B | 45.6 | unverified· optimizedT1 | 2023-09-06 |
| 68 | LLaMA-33B | 44.93 | unverified· optimizedT1 | 2023-02-24 |
| 69 | Nemotron-4 15B | 44.93 | self-reported· optimizedT1 | 2024-02-26 |
| 70 | Phi-2 | 44.53 | unverified· optimizedT1 | 2023-12-12 |
| 71 | Falcon 2 11B | 44.53 | self-reported· optimizedT1 | 2024-05-09 |
| 72 | Falcon-40B | 42.53 | self-reported· optimizedT1 | 2023-03-15 |
| 73 | Llama 3.1-8B | 41.47 | unverified· optimizedT1 | 2024-07-23 |
| 74 | Llama 2-13B | 40.8 | unverified· optimizedT1 | 2023-07-18 |
| 75 | Baichuan 2-7B | 38.88 | unverified· optimizedT1 | 2023-09-20 |
| 76 | Qwen2.5-Coder (1.5B) | 38.13 | self-reported· optimizedT1 | 2024-09-18 |
| 77 | internlm-7b | 34.67 | self-reported· optimizedT1 | 2023-07-05 |
| 78 | INTELLECT-1 | 33.19 | self-reported· optimizedT1 | 2024-11-29 |
| 79 | MPT-30B | 30.53 | self-reported· optimizedT1 | 2023-06-22 |
| 80 | chatglm2-6b | 30.53 | self-reported· optimizedT1 | 2023-06-24 |
| 81 | LLaMA-13B | 30.27 | unverified· optimizedT1 | 2023-02-24 |
| 82 | Llama 2-7B | 27.73 | unverified· optimizedT1 | 2023-07-18 |
| 83 | Qwen-7B | 26.67 | self-reported· optimizedT1 | 2023-09-28 |
| 84 | Gemma 2B | 23.07 | unverified· optimizedT1 | 2024-02-21 |
| 85 | Baichuan1-7B | 23.07 | unverified· optimizedT1 | 2023-06-01 |
| 86 | Qwen2.5-Coder-0.5B | 22.67 | self-reported· optimizedT1 | 2024-09-18 |
| 87 | CodeQwen1.5-7B | 20.67 | self-reported· optimizedT1 | 2024-04-15 |
| 88 | DeepSeek Coder 33B | 19.2 | self-reported· optimizedT1 | 2023-11-02 |
| 89 | StarCoder 2 7B | 18.4 | self-reported· optimizedT1 | 2024-02-20 |
| 90 | Phi-1.5 | 16.8 | self-reported· optimizedT1 | 2023-09-11 |
| 91 | StarCoder 2 3B | 15.47 | self-reported· optimizedT1 | 2024-02-22 |
| 92 | DeepSeek Coder 6.7B | 15.2 | self-reported· optimizedT1 | 2023-11-02 |
| 93 | XGen-7B | 15.07 | self-reported· optimizedT1 | 2023-06-27 |
| 94 | LLaMA-7B | 14.13 | unverified· optimizedT1 | 2023-02-24 |
| 95 | Falcon-7B | 13.33 | self-reported· optimizedT1 | 2023-04-24 |
| 96 | MPT-7B | 7.73 | self-reported· optimizedT1 | 2023-05-05 |
| 97 | open_llama_7b | 6.53 | self-reported· optimizedT1 | 2023-06-07 |
| 98 | Qwen-1_8B | 4.27 | self-reported· optimizedT1 | 2023-11-30 |
| 99 | RedPajama-INCITE-7B-Base | 1.73 | self-reported· optimizedT1 | 2023-05-04 |
| 100 | Cerebras-GPT-13B | 1.6 | self-reported· optimizedT1 | 2023-03-20 |
| 101 | Dolly 2.0-12b | 1.6 | self-reported· optimizedT1 | 2023-04-11 |
| 102 | DeepSeek Coder 1.3B | 1.07 | self-reported· optimizedT1 | 2023-11-02 |
Continue from this leaderboard into the tightest head-to-head reads.
Weighted composite. Each dimension opens to the source-backed sub-signals it is computed from.
Claims drawn from cited facts, not live model generation.
This benchmark scores 102 models, with a spread of 83.06 points between the best and worst scores. Such a wide spread supports real separation between models, whereas a narrower spread would mean close scores may not reflect a genuine capability gap.
3 cited facts
The Benchmark Integrity Index is 67, grade C, ranking 54th of 62 benchmarks, reflecting the benchmark's health as a discriminator of frontier models under a disclosed harness rather than any model's capability. The weakest component is contamination, which narrows how much the top scores can be trusted.
5 cited facts
This benchmark is not saturated: the best result stands at 84.13 points, leaving 15.87 points of headroom before the ceiling is reached. Four models cluster within a couple of points of the top, so the leading group is still tightly bunched rather than decided. Because the benchmark is not saturated, there remains genuine room for the field to separate at the top, and differences among the leaders can still reflect real capability gaps rather than noise.
5 cited facts
Scores for this benchmark are not directly comparable across setups, because the harness is mixed and evaluation details such as few-shot configuration, answer extraction, and output formatting can move results by multiple points; any reported score should therefore be read as task performance under one disclosed harness rather than as a measure of deployed capability. The test set is public, so strong results warrant more scrutiny for possible training-set contamination than a held-out set would; this benchmark has an extensive contamination history, having been fully public and widely absorbed into training data, which reinforces that concern.
4 cited facts
This benchmark is designed to gauge knowledge breadth, testing a model across a wide range of subjects using multiple-choice questions. Its test set was assembled by having students scrape questions from exams and textbooks, so the items reflect exam-style and textbook-style material rather than freshly authored probes. The design rests on the assumption that performance on fixed-option multiple-choice questions spanning many subjects is a reasonable proxy for general knowledge. The sharpest caveat is that a meaningful share of items are mislabeled or ambiguous, the questions reward pure recall, and the material is heavily contaminated in pretraining data — so scores are capped below a perfect ceiling and cannot be read as proof of broad understanding, and the human comparison point comes from top specialists rather than average people.
5 cited facts
MMLU (Massive Multitask Language Understanding): Massive Multitask Language Understanding — a 57-subject multiple-choice knowledge and reasoning benchmark; scored as accuracy (%). Widely regarded as saturated for frontier models.
GPT-4o (Nov 2024) leads MMLU at 84.13 — self-reported by the lab. The full leaderboard above lists every recorded measurement, not just the headline number.
102 models have recorded scores on MMLU, spanning a score spread of 83.06.
tensor.news grades MMLU C for integrity (score 67/100), ranking #57 of 65 benchmarks we assess across discrimination, saturation, contamination resistance, harness comparability, and freshness.
No — MMLU still has headroom and continues to discriminate between models rather than bunching them at the ceiling.
Its test set is public, so contamination risk is on the table: a high score may partly reflect training-data overlap rather than capability.
Only partly — scores come from a mix of harnesses and protocols, so a head-to-head on MMLU is not fully apples-to-apples.
Every MMLU measurement is source-backed and tagged reproduced, self-reported, or unverified; the underlying record cites arxiv.org/abs/2009.03300.