Benchmark explainer
What is WeirdML?
Practical iterative ML-engineering skill: data understanding, architecture choice, working-code generation, and debugging under limited compute
Tests an LLM's ability to understand an unusual dataset, design an appropriate ML architecture, and write PyTorch code that trains and predicts, iterating over feedback. It targets creative ML-engineering rather than memorized patterns.
How it's scored
- Metric
- Held-out test-set accuracy; per run = max accuracy over 5 code submissions, averaged across runs and across all tasks
- Score ceiling
- 100
- Construction
- 6 public example tasks (e.g. Shapes, Image Patch Shuffling, Chess Outcome, Unsupervised Digits) plus ~13 undisclosed hidden tasks in v2; all hand-crafted novel ML datasets by the author
- Human baseline
- unknown (not established; author notes a task can take a human several hours, human eval attributed to METR but no score reported)
How trustworthy
Discrimination
100/100Does it still separate models?
Saturation headroom
96.9/100How far from ceiling / clustered at the top?
Contamination resistance
21.2/100Public vs held-out; training-leak risk.
Harness comparability
100/100Is it apples-to-apples, or mixed / vendor-optimized?
Freshness
86.5/100How old is the benchmark?
Contamination history: Hidden tasks specifically designed to be unseen (primary defense); 6 example tasks public since 2025-01; no known contamination incident
Limitations: High run-to-run variance; constrained compute (single GPU + timeout) caps task scope; expensive models get fewer, noisier runs; excludes small-context pre-mid-2023 models
Who leads WeirdML
| Model | Score | Evidence |
|---|---|---|
| Claude Fable 5.1 | 92.9 | unverified |
| GPT-6 Astra | 92.87 | unverified |
| Claude Fable 5 | 91.94 | unverified |
| Claude Opus 5 | 91.78 | unverified |
| GPT-5.6 Sol | 89.43 | unverified |
Compare the top WeirdML scorers
Frequently asked questions
WeirdML (WeirdML (currently WeirdML v2)): Tests an LLM's ability to understand an unusual dataset, design an appropriate ML architecture, and write PyTorch code that trains and predicts, iterating over feedback. It targets creative ML-engineering rather than memorized patterns.
Claude Fable 5.1 leads WeirdML at 92.9. The full leaderboard above lists every recorded measurement, not just the headline number.
118 models have recorded scores on WeirdML, spanning a score spread of 91.17.
tensor.news grades WeirdML A for integrity (score 86/100), ranking #31 of 62 benchmarks we assess across discrimination, saturation, contamination resistance, harness comparability, and freshness.
No — WeirdML still has headroom and continues to discriminate between models rather than bunching them at the ceiling.
Its test set is semi-private, which lowers — but does not eliminate — the risk that training data leaked into the questions.
Scores are reported under a consistent harness, so comparisons on WeirdML are reasonably apples-to-apples.
Every WeirdML measurement is source-backed and tagged reproduced, self-reported, or unverified; the underlying record cites htihle.github.io/weirdml.html.