IBM Research·released 2024-08-14
Scored on 11 benchmarks, leading 9. 0% of its scores are independently reproduced — the rest are self-reported or unverified.
Leads on: ARC, BoolQ, GSM8k (5 shot), Hellaswag, MBPP, MMLU (5 shot), PIQA, humaneval, math (4 shot)
Every score, ranked, colored by evidence status.
| Benchmark | Score | Evidence status | Measured |
|---|---|---|---|
| PIQA | 79.1accuracy-norm | unverified | — |
| Hellaswag | 71.5accuracy-norm | unverified | — |
| BoolQ | 65accuracy | unverified | — |
| Winogrande | 65accuracy-norm | unverified | — |
| ARC | 58.1accuracy-norm | unverified | — |
| MMLU (5 shot) | 42.8accuracy | unverified | — |
| OpenBookQA | 41accuracy-norm | unverified | — |
| MBPP | 32.4pass@1 | unverified | — |
| GSM8k (5 shot) | 25.9accuracy | unverified | — |
| humaneval | 20.1pass@1 | unverified | — |
| math (4 shot) | 14.8accuracy | unverified | — |