The record tracks two benchmarks with the same name.
WeirdML, released 2025-01-16, carries 116 scored models ranging from 1.73 to 93.57. WeirdML v3, released 2026-09-18, carries 26 scored models ranging from 4.87 to 60.05.
They are different task sets. And until this week, nothing on either page said so — a reader
arriving at /benchmarks/weirdml-v3 had no path to /benchmarks/weirdml and no reason to
suspect they were different things.
The two leaderboards do share 11 models — and that is what makes the version gap measurable rather than merely asserted. Eleven models were scored on both versions, and their scores moved.
The clearest sign that these are not the same scale is the top of each board:
| WeirdML | WeirdML v3 | |
|---|---|---|
| Released | 2025-01-16 | 2026-09-18 |
| Scored models | 116 | 26 |
| Lowest score | 1.73 | 4.87 |
| Highest score | 93.57 | 60.05 |
| Grade | A (85/100), rank 36 of 63 | A (92/100), rank 22 of 63 |
| Contamination axis | 20.7 | 50.0 |
A model that scores 93.57 on WeirdML would be extraordinary on WeirdML v3, where the best model in the record scores 60.05. Whether that means v3 is harder, or that fewer models were run against it, or that the task sets differ in kind, the record does not say — but the numbers cannot be read against each other.
The two also grade differently for reasons that have nothing to do with difficulty. WeirdML's contamination axis scores 20.7 because 79.3% of its dated rows belong to models released after it went public. WeirdML v3 scores 50.0 on the same axis, because it was released eight months later and fewer of its 26 models postdate it. v3 is the cleaner benchmark by that measure, and it has the lower ceiling.
Until this week, WeirdML v3's own description in the record read, in full:
WeirdML v3 as reported in Epoch AI's Capabilities Index CSV.
Its what_it_measures, construction, task_type, human_baseline and limitations fields
were all unknown. WeirdML v1, by contrast, had a full definition: it tests whether a model
can understand an unusual dataset, design an appropriate ML architecture, and write PyTorch
code that trains and predicts, iterating over feedback — with six public example tasks and
about thirteen undisclosed hidden tasks in v2, all hand-crafted novel datasets by the author.
The site knows what WeirdML v3 is (it is scored, graded, ranked) and did not say what WeirdML v3 measures. Twenty of the 63 tracked benchmarks still carried a description in that same placeholder form — among them ARC-AGI, ARC AI2, Aider polyglot, EBR-bench, ExploitBench, FrontierCode, both dated FrontierMath snapshots, FrontierSWE, Furniture Assembly, GeoBench, MirrorCode, Mystery Game Puzzles, OSWorld, OSWorld 2.0, Remote Labor Index, SpireClimb, Surface Evolver Bench, The Agent Company and WeirdML v3 itself.
That gap was metadata, not data: the scores were real, and the benchmark's own author documents it. All twenty have now been written from their own primary sources — the project site, the paper, or the official leaderboard — and the record states what each one measures.
WeirdML v3's own definition, from the author's page:
WeirdML (v3) is an agentic benchmark featuring 11 complex hand-made tasks made to challenge the model to explore and understand unfamiliar data, develop machine learning and data analysis pipelines and produce appropriate results despite limited data, unspecified goals and/or very limited feedback.
One thing the new entry makes sharper, and this piece had only guessed at before: v3's official score is not raw accuracy. It is 80% normalized area under the model's best-so-far effective-score curve on a log token axis, plus 20% of its final value, with effective scores carrying normalisation and hint penalties. That is a different measurement from v1's held-out accuracy, and it is the reason the two boards cannot be read against each other.
The eleven models appear on both leaderboards, and every one of them scored lower on v3. That is the sharpest evidence in the record that the two versions are not the same scale: if v3 were a re-run of v1 with a newer model set, a model scoring 91.94 on v1 would not land at 60 on v3.
| Model | WeirdML | WeirdML v3 | Drop |
|---|---|---|---|
| Inkling | 32.29 | 4.87 | 27.42 |
| GPT-5.6 Luna | 60.86 | 10.38 | 50.48 |
| Claude Opus 4.5 | 63.74 | 7.92 | 55.82 |
| GPT-5.2 | 72.19 | 10.80 | 61.39 |
| Kimi K3 | 82.57 | 17.12 | 65.45 |
| Gemini 3.8 Flash | 84.81 | 16.19 | 68.62 |
| GPT-5.6 Sol | 89.43 | 23.83 | 65.60 |
| Claude Opus 5 | 91.78 | 36.15 | 55.63 |
| Claude Fable 5 | 91.94 | 30.96 | 60.98 |
| Claude Fable 5.1 | 92.90 | 48.36 | 44.54 |
| GPT-6 Astra | 93.57 | 60.05 | 33.52 |
Every one of the eleven dropped. The drops run from 27.42 points (Inkling) to 68.62 (Gemini 3.8 Flash), and the median shared model falls from 84.81 to 17.12.
The record holds both numbers for each of them. What it cannot hold is a conversion between the scales, because the benchmark's own metric changed — v1 states a held-out accuracy, v3 reports normalized performance in Epoch CSV, rendered as accuracy (%).
WeirdML's 116 rows come from two publisher feeds — 101 from the author's own page and 15 from a second leaderboard — and the two sit 4.91 points apart at the median with zero shared model versions. WeirdML v3's 26 rows all come from the author's page alone.
So neither version of WeirdML has a second feed to cross-check it against. The v1 two-feed gap and the v1-versus-v3 version gap are two separate weaknesses, and the version gap is the larger of the two.
Each of the two benchmark pages now carries a panel naming the other version, with its release date and how many models it carries, and it says the thing that matters plainly:
They are different task sets on different scales — a score on one is not comparable with a score on another.
The versions are found by NAME, not by a maintained list. A benchmark whose name carries a
version suffix — v3, 2.0 — is grouped with its unversioned sibling, and a qualifier that
names a different variant (Tier-4, Tiers-1-3) keeps it apart. Nothing about the grouping
is hand-maintained, so it cannot rot when the record changes.
The same rule found a third family: three FrontierMath variants and a separate FrontierMath-2025-02-28-Private, all named for the same program but measuring different things.
The record still cannot tell you why v3's ceiling is 33 points lower than v1's. Three explanations are consistent with what it holds: v3 is harder, v3 covers a narrower kind of task, or the 26 models run against it are a different population from the 116 run against v1.
The most likely answer is in v3's own metric field — normalized performance in Epoch CSV, rendered as accuracy (%) — against v1's, which is a stated held-out accuracy. Two different metrics, two different scales. But that is the record's metadata saying so, not a measurement this site made, and the site does not claim otherwise.
What the record can say, and now does say, is the part a reader needs before comparing anything: these are two benchmarks, their scores do not line up, and here is the evidence.
Every one of the 142 rows across the two WeirdML leaderboards names an evaluator, and not one is marked independently reproduced or self-reported. On v1, 101 rows name the author's page and 15 name the second leaderboard; on v3, all 26 name the author's page. Every score sits in the unverified tier.
That is true of WeirdML as a whole and it is worth stating in the same breath as the version gap. The record's grade for both versions is still meaningful — an integrity grade is about the benchmark, not about the provenance of any particular row — but the scores themselves are vendor-published numbers that nobody has independently reproduced. The two-feed gap on v1, the version gap between v1 and v3, and the fact that neither has been reproduced by anyone else, are three different weaknesses, and a reader comparing a v1 score to a v3 score is walking over all three at once.