Skip to content

Methodology

A benchmark score is a function of the model, the harness, and the protocol, not of the model alone. Run the same model through a different scaffold and the number moves. And even a clean score on a held-out benchmark doesn't tell you how a model performs on an open-ended, real-world, agentic task. So we don't only grade models. We grade the evaluations.

Score scale

Every Epoch-sourced score is normalized to a 0–100 scale — raw performance × 100, rounded to two decimals. The underlying metric still differs by benchmark (accuracy, pass@1, percent of issues resolved, exact-match), so normalizing doesn't make two benchmarks measure the same thing; it puts them on one comparable axis for a leaderboard or a fingerprint chart. It never implies a MATH benchmark score and a SWE-bench score are the same kind of claim — see comparability, below.

The Integrity score

Every benchmark on tensor.news carries an Integrity score from 0 to 100, built from five weighted signals:

Integrity = 0.30 · discrimination + 0.30 · saturation + 0.15 · contamination + 0.15 · harness + 0.10 · age

Discrimination (30%)

Whether the benchmark still separates models. When every model clusters within a point or two of each other, the benchmark has stopped measuring anything useful. We score this from the spread of results across the full field, not just the top few entries.

Saturation (30%)

How close the leading models sit to the ceiling, and how tightly they're bunched there. A benchmark near saturation can no longer tell a genuinely better model apart from one that's merely tied for first.

Contamination (15%)

Whether the test set is public or held out, and whether a model's release date postdates the benchmark's publication. A public test set paired with a model trained after it was released is a training-leak risk, not a clean measurement.

Harness comparability (15%)

Whether scores come from one comparable harness, or a mix of vendor-optimized and independently-run harnesses. We lean on Epoch's optimized flag, plus whether a score is independently reproduced or self-reported, to catch mixed harnesses before they get averaged into a single leaderboard.

Age (10%)

How long the benchmark has been in circulation. Older benchmarks accumulate more contamination surface and are more likely to have been optimized against directly, whether deliberately or not.

Grade bands

The composite score maps to a letter grade:

The Model Claim Ledger

Every benchmark score attached to a model carries a status: independently reproduced, self-reported, contradicted, or unverified. Independently-reproduced scores come from Epoch's own evaluation runs; self-reported scores come from a vendor's technical report. We keep the two separate rather than blending them into one number.

Comparability: tiers and the display rule

A score is a function of the model, the harness, and the protocol, not the model alone. Two scores on the same benchmark are a clean comparison only when they were measured the same way. Every score carries an additive, versioned comparability class computed from benchmark identity, metric, whether the harness was vendor-optimized, and — when disclosed — the eval harness, its version, and six protocol dimensions (shot count, chain-of-thought, pass@k, temperature, sampling, prompt variant). A class is only non-null when EVERY one of those dimensions is actually disclosed for that score; an undisclosed dimension never silently matches another undisclosed one.

Every score then falls into one of three tiers:

A fourth case — different metrics entirely — is never combined at all, not even into a shared range. As of this write, full protocol disclosure (exact shot count, sampling, decoding settings) is still rare industry-wide, so most scores on this site sit at T1 — that's the honest state of the underlying evidence, not a gap in how we read it. It's also why comparative pages (leaderboards, head-to-heads, the pulse) carry a “reported scores — protocols may differ” note by default.

Where more than one measurement exists for the same model and benchmark, the site picks ONE to headline using a fixed rule, in order: prefer the comparability class the most models share; within that, prefer a corroborated consensus, then an independently-reproduced score, then a non-self-reported one, then a non-optimized one, then the highest-priority data source, then the most recent measurement, then a deterministic tiebreak. It never picks whichever number happens to flatter the model.

Ranking models

The leaderboard at /models ranks by a single frontier-quality score, not by raw benchmark-lead count — a model that tops nine tiny, low-integrity benchmarks isn't a frontier model; a model that leads two large, A-grade benchmarks is. The score sums four terms:

That total is then multiplied by an evidence-status factor: independently-reproduced scores count in full, self-reported 0.75×, unverified 0.5×, contradicted 0.4×. A model whose only leads are self-reported or unverified never outranks one with the same raw signal, independently confirmed.

Pricing: the 3:1 blended rate

/models/pricing ranks by a blended cost — (3 × input price + 1 × output price) ÷ 4, in USD per million tokens. Most real traffic is input-heavy (long context, short completions), so input price carries three times the weight of output price. Published prices come from LiteLLM's or models.dev's own numbers; OpenRouter's aggregated pricing is checked as a cross-reference only — a listed price disagreeing with the published number flags the row disputed — and is never republished directly, per OpenRouter's terms of service.

What a score is not

Every number on this site is task performance under a disclosed harness, never a statement about deployed, real-world, or agentic capability. Where a benchmark's integrity is low, or its harness is mixed, we say so next to the score rather than leaving it to be read as a verdict.

Provenance

Every fact behind these scores links to a source record: which evaluator ran it, when it was measured, and where it was published. Nothing here is asserted without an evidence trail you can follow.