WeirdML scores practical machine-learning engineering: reading a data set, choosing an architecture, writing code that runs, and debugging it under a constrained compute budget. The tensor.news record carries 116 scored models on it. It grades the benchmark A, 85 out of 100, 36th of 63 tracked benchmarks, with 6.43 points of headroom and a 91.84-point spread across the board.
Two things about that record are worth a buyer's attention, and neither is visible on the leaderboard.
The record's human-baseline field for WeirdML reads: unknown (not established; author notes a task can take a human several hours, human eval attributed to METR but no score reported).
That is a real gap, not a stub. 29 of the 63 tracked benchmarks have no usable human baseline either, so WeirdML is not an outlier — but it matters here more than it does elsewhere, because WeirdML's own author flagged the missing number as the reason the benchmark's headline claim was withdrawn. The benchmark's page now strikes through the assertion that everyday human reasoning still surpasses frontier models. What survives on the record is the score itself, with no comparison point.
The benchmark's task set is small and hand-built: six public example tasks plus about thirteen undisclosed hidden tasks in v2, all novel datasets written by the author. The metric is held-out test-set accuracy, where each run takes the maximum accuracy over five code submissions and averages across runs and tasks.
Two properties of that metric matter for a reader comparing scores. The first is that it is a maximum over five submissions, not a single attempt: a model that solves a task on its third try scores the same as one that solves it first try. That rewards retry behaviour, which is the benchmark's design intent — it is testing iterative debugging — but it also means the number describes a best-of-five process rather than first-try competence. The second is that the score is an average over both runs and tasks, so a model strong on data-understanding tasks and weak on debugging tasks lands in the same place as the reverse.
The record's own limitations field for WeirdML names the cost of the harness: high run-to-run variance, a single-GPU plus timeout constraint that caps how large a task can be, and a note that expensive models get fewer and noisier runs. A row that says 93.57 and a row that says 1.73 are not measuring the same amount of compute, and the record does not publish a per-model run count.
The author also excluded small-context models released before mid-2023 from the leaderboard. That is a defensible scope choice, and it also means the 116 rows are not a complete census of models that could attempt the tasks — they are the models the leaderboard chose to include.
The 116 rows come from two publisher feeds. 101 come from the author's own page, htihle.github.io/weirdml.html. 15 come from a second source labelled WeirdML Leaderboard. The two feeds share zero model versions.
Their scores do not sit in the same range. The author's feed has a median of 45.86 and runs from 1.73 to 93.57. The second feed has a median of 40.95 and runs from 15.97 to 66.07 — its best model, Claude Sonnet 4.6 at 66.07, sits at overall rank 22. The median gap is 4.91 points.
Because no model appears in both feeds, the record cannot test whether the two scales agree. It can only observe that they do not overlap much, and that the second feed's ceiling is lower by a wide margin. Any model added from one feed is placed on a leaderboard whose comparability to the other feed is unverified.
The record grades WeirdML's harness comparability a perfect 100 and calls it a "consistent harness". That is a statement about the benchmark's own sandbox — a single 12GB TITAN V GPU, Docker, a per-run timeout — and it is correct as far as it goes. It does not cover the fact that two publisher feeds with different model sets are being merged into one ranked table.
WeirdML is one of the better-covered benchmarks in the price table: 55 of 116 rows carry a published price, and the first priced row is at rank 3. But the price-per-point picture is unusual.
The most efficient priced model is GPT-5 nano, at 38.06 for $0.40 per million output tokens — 95.15 points per $1M. It sits at rank 82 of 116. The least efficient is GPT-5.4 Pro, at 57.44 for $180 per million output tokens — 0.319 points per $1M. Between them the spread is 298 times.
Read the leaderboard top-down and the ordering tracks capability. Read it by points per dollar and GPT-5 nano is 81 places higher than its rank suggests, while the top of the board is 4 times more expensive per point than a mid-table model. A reader choosing on rank and a reader choosing on cost are looking at two different orderings.
Two prices are worth flagging when quoting them. The GPT-5 nano price is disputed, and DeepSeek-R1 — second on price-per-point at 86.88 points per $1M — is disputed as well. The most expensive end of the range, GPT-5.4 Pro at $180, is corroborated.
WeirdML's discrimination axis scores 100, on a 91.84-point spread. Its age axis scores 83.4 at 668 days, measured from the benchmark's 2025-01-16 release. Saturation scores 96.8, with the saturation flag off.
The contamination axis is the weakest: 20.7 out of 100. The test set is semi-private, and the contamination window ratio is 0.793 — 92 of the 116 dated rows belong to models released after the benchmark went public. A further 24 rows carry dates before the benchmark's release, because those models predate it; their evaluations necessarily happened later, so the true exposure is higher than the ratio alone suggests.
None of the 116 rows is marked independently reproduced, and none is marked self-reported. The evaluator field on every row names a publisher feed rather than a person or lab — the author's own page on 101 rows, WeirdML Leaderboard on the other 15 — so the provenance of a WeirdML score is the feed it arrived in, and nothing more.
What the number supports: a model's accuracy on held-out ML-engineering tasks, averaged over five submissions per run. What it does not support: any claim about how that compares to a human, because no human score exists, and any claim that a row from one feed is directly comparable to a row from the other.