BIG-Bench Hard
Integrity rank #50 of 61 · 41 models scored · top score 85.6 · Gemini 1.5 Pro (May 2024)
23 hardest BIG-Bench tasks (logic/arithmetic/symbolic)
Strongest on discrimination, weakest on contamination resistance. Caveat: scores come from an inconsistent mix of harnesses, so head-to-head comparisons are not apples-to-apples.
Reported scores — protocols may differ. How we compare
| # | Model | Score | Evidence | Measured |
|---|---|---|---|---|
| 1 | Gemini 1.5 Pro (May 2024) | 85.6 | unverified· optimizedT1 | 2024-05-14 |
| 2 | DeepSeek-V3 | 83.33 | self-reported· optimizedT1 | 2024-12-24 |
| 3 | Llama 3.1-405B | 77.2 | self-reported· optimizedT1 | 2024-07-23 |
| 4 | phi-3-medium 14B | 75.2 | self-reported· optimizedT1 | 2024-04-23 |
| 5 | Qwen2.5-72B | 73.07 | self-reported· optimizedT1 | 2024-09-19 |
| 6 | phi-3-small 7.4B | 72.13 | self-reported· optimizedT1 | 2024-04-23 |
| 7 | DeepSeek-V2 (MoE-236B, May 2024) | 71.73 | self-reported· optimizedT1 | 2024-05-07 |
| 8 | GPT-4 (Jun 2023) | 66.83 | unverified· optimizedT1 | 2023-06-13 |
| 9 | Yi-34B | 62.27 | unverified· optimizedT1 | 2023-11-02 |
| 10 | phi-3-mini 3.8B | 62.27 | self-reported· optimizedT1 | 2024-04-23 |
| 11 | Stable Beluga 2 | 59.07 | self-reported· optimizedT1 | 2023-07-20 |
| 12 | Llama 2-70B | 53.2 | self-reported· optimizedT1 | 2023-07-18 |
| 13 | GPT-3.5 Turbo (Jun 2023) | 48.79 | unverified· optimizedT1 | 2023-06-13 |
| 14 | Phi-2 | 45.87 | self-reported· optimizedT1 | 2023-12-12 |
| 15 | Nemotron-4 15B | 44.93 | self-reported· optimizedT1 | 2024-02-27 |
| 16 | LLaMA-65B | 44.53 | self-reported· optimizedT1 | 2023-02-24 |
| 17 | Llama 2-13B | 44.27 | unverified· optimizedT1 | 2023-07-18 |
| 18 | Mistral 7B v0.1 | 41.47 | unverified· optimizedT1 | 2023-10-10 |
| 19 | Gemma 7B | 40.13 | unverified· optimizedT1 | 2024-02-21 |
| 20 | Qwen-14B | 40 | self-reported· optimizedT1 | 2023-09-24 |
| 21 | internlm-20b | 36.67 | self-reported· optimizedT1 | 2023-09-18 |
| 22 | LLaMA-33B | 33.33 | self-reported· optimizedT1 | 2023-02-27 |
| 23 | Baichuan2-13B | 32 | self-reported· optimizedT1 | 2023-09-06 |
| 24 | Yi 6B | 29.6 | unverified· optimizedT1 | 2023-11-02 |
| 25 | Qwen-7B | 26.67 | self-reported· optimizedT1 | 2023-09-28 |
| 26 | Llama 2-34B | 25.47 | self-reported· optimizedT1 | 2023-07-18 |
| 27 | vicuna-13b-v1.1 | 24.05 | unverified· optimizedT1 | 2023-04-12 |
| 28 | Baichuan 2-7B | 22.13 | unverified· optimizedT1 | 2023-09-20 |
| 29 | Llama 2-7B | 18.88 | unverified· optimizedT1 | 2023-07-18 |
| 30 | MPT-30B | 17.33 | self-reported· optimizedT1 | 2023-06-22 |
| 31 | LLaMA-13B | 17.2 | unverified· optimizedT1 | 2023-02-27 |
| 32 | Falcon-40B | 16.13 | self-reported· optimizedT1 | 2023-03-15 |
| 33 | internlm-7b | 16 | self-reported· optimizedT1 | 2023-07-05 |
| 34 | MPT-7B | 14.13 | unverified· optimizedT1 | 2023-05-05 |
| 35 | Gemma 2B | 13.6 | unverified· optimizedT1 | 2024-02-21 |
| 36 | INTELLECT-1 | 13.13 | self-reported· optimizedT1 | 2024-11-29 |
| 37 | chatglm2-6b | 11.6 | unverified· optimizedT1 | 2023-06-24 |
| 38 | LLaMA-7B | 11.33 | unverified· optimizedT1 | 2023-02-24 |
| 39 | Baichuan1-7B | 9.97 | unverified· optimizedT1 | 2023-06-01 |
| 40 | Falcon-7B | 5.03 | unverified· optimizedT1 | 2023-04-24 |
| 41 | Qwen-1_8B | 4.27 | self-reported· optimizedT1 | 2023-11-30 |
Continue from this leaderboard into the tightest head-to-head reads.
Weighted composite. Each dimension opens to the source-backed sub-signals it is computed from.
Claims drawn from cited facts, not live model generation.
Best to worst span 81.33 points across the 41 models scored, which is wide separation: when scores differ here, the difference is usually real. The wide spread indicates that rank differences between models are likely meaningful, as such a gap suggests genuine capability distinctions rather than noise.
3 cited facts
The benchmark's Benchmark Integrity Index is 71, corresponding to a B grade, and ranks 40 out of 51 benchmarks in the universe. Saturation is the single weakest component, clustering top scores closely together and narrowing the margin by which frontier models can be discriminated. The health of this benchmark as a discriminator under the disclosed harness is indicated by its moderate integrity score of 71.
3 cited facts
This benchmark still discriminates: room remains above today's leaders, so a change in the ordering at the top would register as measured improvement, not ceiling noise. The top score of 85.6 has remaining headroom of 14.4 points, reflecting genuine separation potential. A single cluster near the top suggests current models are not widely diverging in performance.
4 cited facts
This benchmark measures performance on 23 of the hardest BIG-Bench tasks, which involve logic, arithmetic, and symbolic reasoning. It was built from 6511 examples where models trailed the average human, using a templated and procedural construction. Tasks were selected because chain-of-thought reasoning unlocked gains on them; that selection is itself an assumption about isolating multi-step reasoning, and if models find shortcuts, the score measures less than intended. The sharpest caveat is that this benchmark is now saturated, its templated design makes it memorizable, and it has been superseded by a 2025 successor, so its scores can no longer reliably discriminate new model capabilities.
4 cited facts
BBH (BIG-Bench Hard): BIG-Bench Hard — 23 hardest BIG-Bench tasks (logic/arithmetic/symbolic); scored as Exact-match w/ CoT, task-averaged.
Gemini 1.5 Pro (May 2024) leads BBH at 85.6. The full leaderboard above lists every recorded measurement, not just the headline number.
41 models have recorded scores on BBH, spanning a score spread of 81.33.
tensor.news grades BBH B for integrity (score 71/100), ranking #50 of 61 benchmarks we assess across discrimination, saturation, contamination resistance, harness comparability, and freshness.
No — BBH still has headroom and continues to discriminate between models rather than bunching them at the ceiling.
Its test set is public, so contamination risk is on the table: a high score may partly reflect training-data overlap rather than capability.
Only partly — scores come from a mix of harnesses and protocols, so a head-to-head on BBH is not fully apples-to-apples.
Every BBH measurement is source-backed and tagged reproduced, self-reported, or unverified; the underlying record cites arxiv.org/abs/2210.09261.