BIG-Bench Hard
Rank #40 of 51 · 41 models scored · top score 85.6
23 hardest BIG-Bench tasks (logic/arithmetic/symbolic)
Strongest on discrimination, weakest on contamination resistance. Caveat: scores come from mixed harnesses, so a head-to-head is not apples-to-apples.
Weighted composite. Each signal is computed from the source-backed leaderboard below.
Does it still separate models?
How far from ceiling / clustered at the top?
Public vs held-out; training-leak risk.
Apples-to-apples, or mixed / vendor-optimized?
How old is the benchmark?
| # | Model | Score | Evidence | Measured |
|---|---|---|---|---|
| 1 | Gemini 1.5 Pro (May 2024) | 85.6 | unverified· optimized | 2024-05-14 |
| 2 | DeepSeek-V3 | 83.33 | self-reported· optimized | 2024-12-24 |
| 3 | Llama 3.1-405B | 77.2 | self-reported· optimized | 2024-07-23 |
| 4 | phi-3-medium 14B | 75.2 | self-reported· optimized | 2024-04-23 |
| 5 | Qwen2.5-72B | 73.07 | self-reported· optimized | 2024-09-19 |
| 6 | phi-3-small 7.4B | 72.13 | self-reported· optimized | 2024-04-23 |
| 7 | DeepSeek-V2 (MoE-236B, May 2024) | 71.73 | self-reported· optimized | 2024-05-07 |
| 8 | GPT-4 (Jun 2023) | 66.83 | unverified· optimized | 2023-06-13 |
| 9 | Yi-34B | 62.27 | unverified· optimized | 2023-11-02 |
| 10 | phi-3-mini 3.8B | 62.27 | self-reported· optimized | 2024-04-23 |
| 11 | Stable Beluga 2 | 59.07 | self-reported· optimized | 2023-07-20 |
| 12 | Llama 2-70B | 53.2 | self-reported· optimized | 2023-07-18 |
| 13 | GPT-3.5 Turbo (Jun 2023) | 48.79 | unverified· optimized | 2023-06-13 |
| 14 | Phi-2 | 45.87 | self-reported· optimized | 2023-12-12 |
| 15 | Nemotron-4 15B | 44.93 | self-reported· optimized | 2024-02-27 |
| 16 | LLaMA-65B | 44.53 | self-reported· optimized | 2023-02-24 |
| 17 | Llama 2-13B | 44.27 | unverified· optimized | 2023-07-18 |
| 18 | Mistral 7B v0.1 | 41.47 | unverified· optimized | 2023-10-10 |
| 19 | Gemma 7B | 40.13 | unverified· optimized | 2024-02-21 |
| 20 | Qwen-14B | 40 | self-reported· optimized | 2023-09-24 |
| 21 | internlm-20b | 36.67 | self-reported· optimized | 2023-09-18 |
| 22 | LLaMA-33B | 33.33 | self-reported· optimized | 2023-02-27 |
| 23 | Baichuan2-13B | 32 | self-reported· optimized | 2023-09-06 |
| 24 | Yi 6B | 29.6 | unverified· optimized | 2023-11-02 |
| 25 | Qwen-7B | 26.67 | self-reported· optimized | 2023-09-28 |
| 26 | Llama 2-34B | 25.47 | self-reported· optimized | 2023-07-18 |
| 27 | vicuna-13b-v1.1 | 24.05 | unverified· optimized | 2023-04-12 |
| 28 | Baichuan 2-7B | 22.13 | unverified· optimized | 2023-09-20 |
| 29 | Llama 2-7B | 18.88 | unverified· optimized | 2023-07-18 |
| 30 | MPT-30B | 17.33 | self-reported· optimized | 2023-06-22 |
| 31 | LLaMA-13B | 17.2 | unverified· optimized | 2023-02-27 |
| 32 | Falcon-40B | 16.13 | self-reported· optimized | 2023-03-15 |
| 33 | internlm-7b | 16 | self-reported· optimized | 2023-07-05 |
| 34 | MPT-7B | 14.13 | unverified· optimized | 2023-05-05 |
| 35 | Gemma 2B | 13.6 | unverified· optimized | 2024-02-21 |
| 36 | INTELLECT-1 | 13.13 | self-reported· optimized | 2024-11-29 |
| 37 | chatglm2-6b | 11.6 | unverified· optimized | 2023-06-24 |
| 38 | LLaMA-7B | 11.33 | unverified· optimized | 2023-02-24 |
| 39 | Baichuan1-7B | 9.97 | unverified· optimized | 2023-06-01 |
| 40 | Falcon-7B | 5.03 | unverified· optimized | 2023-04-24 |
| 41 | Qwen-1_8B | 4.27 | self-reported· optimized | 2023-11-30 |