The Benchmark Integrity Index grades 63 benchmarks on five axes. Add up what each axis withholds and one of them dominates: contamination takes 508.7 of the 1,006.5 points the index gives away, from a weight of only 15%.
That matters if you pick benchmarks to track frontier models on. The loss is not spread evenly across old, washed-up sets. It concentrates in 16 named benchmarks, and the four largest of them — MMLU, ARC-AGI, MATH level 5, and ARC AI2 — still carry 322 model scores between them.
| Axis | Weight | Points lost | Share of its own budget | Share of all loss |
|---|---|---|---|---|
| Discrimination | 30% | 141.4 | 7.5% | 14.0% |
| Saturation | 30% | 96.3 | 5.1% | 9.6% |
| Contamination | 15% | 508.7 | 53.8% | 50.5% |
| Harness comparability | 15% | 97.5 | 10.3% | 9.7% |
| Age | 10% | 162.6 | 25.8% | 16.2% |
| Total | 100% | 1,006.5 | 16.0% | 100% |
The index gives away 16.0% of its possible 6,300 points. Contamination supplies half of all that loss on a 15% weight — the same weight as harness comparability, and one and a half times the age axis's 10%. Saturation, the axis that makes leaderboards look crowded, withholds 96.3 points — under a fifth of what contamination withholds.
A benchmark scores 0 on contamination when every dated model score in the record comes from a model released after the benchmark's own release date. Sixteen benchmarks score 0. Between them they hold 734 of the 2,994 model scores in the record — 24.5% of every model score on the site.
Their grades are B (6), C (9), and D (1). Not one is an A. All 36 A-grade benchmarks score above 0 on contamination. Thirteen of the sixteen are more than three years old, and those thirteen alone carry 662 model scores.
Fifty-nine of the 63 benchmarks carry a release date. Their median age is 601 days, against a freshness window of 365 days before any penalty starts. The newest is SpireClimb, at 32 days. The oldest is LAMBADA, at 3,800 days.
Freshness tracks grades. The 21 benchmarks under a year old hold 1,034 model scores and produce 17 A grades. The 18 benchmarks two years old or older hold 998 model scores and produce one — SimpleBench, at 745 days.
Four benchmarks carry no release date at all: AIME, LiveCodeBench, Codeforces rating, and MATH-500. With no date the record cannot measure the contamination window and invents no penalty, so each takes full credit on two axes at once — contamination 100 and age 100. That is 25 free points per benchmark. Had the record scored both axes 0 for them instead, AIME would drop from 97 to 72, from an A at rank 12 to a C at rank 48; LiveCodeBench would fall from 98 to 73, from rank 10 to 47; MATH-500 from 76 to 51, and from rank 45 to last of 63.
Contamination here measures exposure, not an observed leak. A score of 0 means every dated score in the record comes from a model that postdates the benchmark's release, so every score had the opportunity to train on a public test set. It does not mean any model did. The axis grades the risk a benchmark carries; it does not grade the honesty of the scores recorded on it.