ARC AI2
Integrity rank #55 of 61 · 66 models scored · top score 93.73 · DeepSeek-V3
unknown
Strongest on discrimination, weakest on contamination resistance. Caveat: scores come from an inconsistent mix of harnesses, so head-to-head comparisons are not apples-to-apples.
Reported scores — protocols may differ. How we compare
| # | Model | Score | Evidence | Measured |
|---|---|---|---|---|
| 1 | DeepSeek-V3 | 93.73 | self-reported· optimizedT1 | 2024-12-24 |
| 2 | Llama 3.1-405B | 93.73 | self-reported· optimizedT1 | 2024-07-23 |
| 3 | Qwen2.5-72B | 92.67 | self-reported· optimizedT1 | 2024-09-19 |
| 4 | DeepSeek-V2 (MoE-236B, May 2024) | 89.6 | self-reported· optimizedT1 | 2024-05-07 |
| 5 | phi-3-medium 14B | 88.8 | self-reported· optimizedT1 | 2024-04-23 |
| 6 | phi-3-small 7.4B | 87.6 | self-reported· optimizedT1 | 2024-04-23 |
| 7 | GPT-3.5 Turbo (Nov 2023) | 83.2 | self-reported· optimizedT1 | 2023-06-13 |
| 8 | Mixtral 8x7B | 83.07 | unverified· optimizedT1 | 2023-12-11 |
| 9 | Claude Instant | 81.73 | unverified· optimizedT1 | 2023-08-09 |
| 10 | Stable Beluga 2 | 81.47 | self-reported· optimizedT1 | 2023-07-20 |
| 11 | phi-3-mini 3.8B | 79.87 | self-reported· optimizedT1 | 2024-04-23 |
| 12 | Qwen-14B | 79.2 | self-reported· optimizedT1 | 2023-09-24 |
| 13 | Llama 3-8B | 77.07 | self-reported· optimizedT1 | 2024-04-18 |
| 14 | internlm-20b | 75.6 | self-reported· optimizedT1 | 2023-09-18 |
| 15 | Mistral 7B v0.1 | 71.47 | self-reported· optimizedT1 | 2023-10-10 |
| 16 | Gemma 7B | 71.07 | self-reported· optimizedT1 | 2024-02-21 |
| 17 | Llama 2-70B | 71.07 | unverified· optimizedT1 | 2023-07-18 |
| 18 | Phi-2 | 67.87 | self-reported· optimizedT1 | 2023-12-12 |
| 19 | Qwen-7B | 67.07 | self-reported· optimizedT1 | 2023-09-28 |
| 20 | Qwen2.5-Coder-32B | 60.67 | self-reported· optimizedT1 | 2024-09-18 |
| 21 | LLaMA-65B | 59.33 | self-reported· optimizedT1 | 2023-02-24 |
| 22 | internlm-7b | 59.33 | self-reported· optimizedT1 | 2023-07-05 |
| 23 | PaLM 2-L | 58.93 | unverified· optimizedT1 | 2023-05-17 |
| 24 | Falcon-180B | 57.07 | unverified· optimizedT1 | 2023-09-06 |
| 25 | LLaMA-33B | 56.67 | self-reported· optimizedT1 | 2023-02-27 |
| 26 | Qwen2.5-Coder-14B | 54.67 | self-reported· optimizedT1 | 2024-09-18 |
| 27 | PaLM 2-M | 53.2 | unverified· optimizedT1 | 2023-05-17 |
| 28 | Falcon-40B | 49.15 | unverified· optimizedT1 | 2023-03-15 |
| 29 | chatglm2-6b | 48 | self-reported· optimizedT1 | 2023-06-24 |
| 30 | Qwen2.5-Coder (7B) | 47.87 | self-reported· optimizedT1 | 2024-09-18 |
| 31 | Llama 2-13B | 47.07 | unverified· optimizedT1 | 2023-07-18 |
| 32 | PaLM 2-S | 46.13 | unverified· optimizedT1 | 2023-05-17 |
| 33 | DeepSeek-Coder-V2-Lite-Base | 43.07 | self-reported· optimizedT1 | 2024-06-13 |
| 34 | Yi-9B | 40.8 | unverified· optimizedT1 | 2024-03-01 |
| 35 | Nemotron-4 15B | 40.67 | self-reported· optimizedT1 | 2024-02-27 |
| 36 | INTELLECT-1 | 39.36 | self-reported· optimizedT1 | 2024-11-29 |
| 37 | Llama 2-34B | 39.33 | unverified· optimizedT1 | 2023-07-18 |
| 38 | Qwen-1_8B | 37.6 | self-reported· optimizedT1 | 2023-11-30 |
| 39 | Qwen2.5-Coder-3B | 37.2 | self-reported· optimizedT1 | 2024-09-18 |
| 40 | LLaMA-13B | 36.93 | self-reported· optimizedT1 | 2023-02-27 |
| 41 | MPT-30B | 34.13 | unverified· optimizedT1 | 2023-06-22 |
| 42 | Yi 6B | 33.73 | unverified· optimizedT1 | 2023-11-02 |
| 43 | Falcon-7B | 30.48 | unverified· optimizedT1 | 2023-04-24 |
| 44 | LLaMA-7B | 30.13 | self-reported· optimizedT1 | 2023-02-24 |
| 45 | StarCoder 2 15B | 29.6 | self-reported· optimizedT1 | 2024-02-29 |
| 46 | Llama 2-7B | 27.87 | unverified· optimizedT1 | 2023-07-18 |
| 47 | Qwen2.5-Coder (1.5B) | 26.93 | self-reported· optimizedT1 | 2024-09-18 |
| 48 | Phi-1.5 | 25.87 | self-reported· optimizedT1 | 2023-09-11 |
| 49 | vicuna-13b-v1.1 | 24.27 | self-reported· optimizedT1 | 2023-04-12 |
| 50 | MPT-7B | 23.47 | self-reported· optimizedT1 | 2023-05-05 |
| 51 | DeepSeek Coder 33B | 22.93 | self-reported· optimizedT1 | 2024-01-25 |
| 52 | Gemma 2B | 22.8 | unverified· optimizedT1 | 2024-02-21 |
| 53 | XGen-7B | 21.6 | self-reported· optimizedT1 | 2023-09-07 |
| 54 | Dolly 2.0-12b | 19.47 | self-reported· optimizedT1 | 2023-04-12 |
| 55 | RedPajama-INCITE-7B-Base | 18.8 | self-reported· optimizedT1 | 2023-05-04 |
| 56 | StarCoder 2 7B | 18.27 | self-reported· optimizedT1 | 2024-02-29 |
| 57 | open_llama_7b | 18.27 | self-reported· optimizedT1 | 2023-06-07 |
| 58 | Baichuan2-13B | 17.33 | self-reported· optimizedT1 | 2023-09-06 |
| 59 | DeepSeek Coder 6.7B | 15.2 | self-reported· optimizedT1 | 2024-01-25 |
| 60 | CodeQwen1.5-7B | 14.27 | self-reported· optimizedT1 | 2024-04-15 |
| 61 | Qwen2.5-Coder-0.5B | 12.53 | self-reported· optimizedT1 | 2024-09-18 |
| 62 | StarCoder 2 3B | 12.27 | self-reported· optimizedT1 | 2024-02-29 |
| 63 | Baichuan 2-7B | 10 | self-reported· optimizedT1 | 2023-09-20 |
| 64 | Cerebras-GPT-13B | 9.87 | self-reported· optimizedT1 | 2023-04-06 |
| 65 | stablelm-tuned-alpha-7b | 2.67 | self-reported· optimizedT1 | 2023-04-19 |
| 66 | DeepSeek Coder 1.3B | 0.53 | self-reported· optimizedT1 | 2024-01-25 |
Continue from this leaderboard into the tightest head-to-head reads.
Weighted composite. Each dimension opens to the source-backed sub-signals it is computed from.
Claims drawn from cited facts, not live model generation.
Scores from this benchmark are labeled as low harness comparability, meaning they come from a mixed setup and should not be compared directly with scores from other harnesses. The test set privacy for this benchmark is unknown, so it is unclear whether the test set is public or held out, but public status would warrant extra scrutiny for possible contamination. The contamination history is unknown and the contamination window ratio spans the full dataset, so the degree of potential contamination cannot be estimated from the available information.
3 cited facts
This benchmark evaluates a specific capability. Its design specifics are not documented, leaving the exact test construction unspecified. Because its limitations are not documented, the results cannot be extrapolated beyond the test.
3 cited facts
ARC AI2: ARC AI2 as reported in Epoch AI's Capabilities Index CSV.
DeepSeek-V3 leads ARC AI2 at 93.73 — self-reported by the lab. The full leaderboard above lists every recorded measurement, not just the headline number.
66 models have recorded scores on ARC AI2, spanning a score spread of 93.2.
tensor.news grades ARC AI2 C for integrity (score 66/100), ranking #55 of 61 benchmarks we assess across discrimination, saturation, contamination resistance, harness comparability, and freshness.
No — ARC AI2 still has headroom and continues to discriminate between models rather than bunching them at the ceiling.
Its test set is unknown, so contamination risk is on the table: a high score may partly reflect training-data overlap rather than capability.
Only partly — scores come from a mix of harnesses and protocols, so a head-to-head on ARC AI2 is not fully apples-to-apples.
Every ARC AI2 measurement is source-backed and tagged reproduced, self-reported, or unverified; the underlying record cites epoch.ai/data/eci_benchmarks.csv.