Benchmark explainer
What is BBH?
23 hardest BIG-Bench tasks (logic/arithmetic/symbolic)
BIG-Bench Hard — 23 hardest BIG-Bench tasks (logic/arithmetic/symbolic); scored as Exact-match w/ CoT, task-averaged.
How it's scored
- Metric
- Exact-match w/ CoT, task-averaged
- Score ceiling
- 100
- Construction
- 6511 examples where models trailed avg human; templated/procedural
- Human baseline
- BIG-Bench avg/best human rater baselines
How trustworthy
Discrimination
100/100Does it still separate models?
Saturation headroom
97.9/100How far from ceiling / clustered at the top?
Contamination resistance
0/100Public vs held-out; training-leak risk.
Harness comparability
50/100Is it apples-to-apples, or mixed / vendor-optimized?
Freshness
39.9/100How old is the benchmark?
Contamination history: Public since 2022, contaminated/saturated (motivated BBEH)
Limitations: Saturated; templated -> memorizable; superseded by BBEH (2025)
Who leads BBH
| Model | Score | Evidence |
|---|---|---|
| Gemini 1.5 Pro (May 2024) | 85.6 | unverified |
| DeepSeek-V3 | 83.33 | self-reported |
| Llama 3.1-405B | 77.2 | self-reported |
| phi-3-medium 14B | 75.2 | self-reported |
| Qwen2.5-72B | 73.07 | self-reported |
Compare the top BBH scorers
Frequently asked questions
BBH (BIG-Bench Hard): BIG-Bench Hard — 23 hardest BIG-Bench tasks (logic/arithmetic/symbolic); scored as Exact-match w/ CoT, task-averaged.
Gemini 1.5 Pro (May 2024) leads BBH at 85.6. The full leaderboard above lists every recorded measurement, not just the headline number.
41 models have recorded scores on BBH, spanning a score spread of 81.33.
tensor.news grades BBH B for integrity (score 71/100), ranking #53 of 65 benchmarks we assess across discrimination, saturation, contamination resistance, harness comparability, and freshness.
No — BBH still has headroom and continues to discriminate between models rather than bunching them at the ceiling.
Its test set is public, so contamination risk is on the table: a high score may partly reflect training-data overlap rather than capability.
Only partly — scores come from a mix of harnesses and protocols, so a head-to-head on BBH is not fully apples-to-apples.
Every BBH measurement is source-backed and tagged reproduced, self-reported, or unverified; the underlying record cites arxiv.org/abs/2210.09261.