WeirdML (currently WeirdML v2)
Integrity rank #32 of 61 · 108 models scored · top score 91.94 · Claude Fable 5
Practical iterative ML-engineering skill: data understanding, architecture choice, working-code generation, and debugging under limited compute
Strongest on discrimination, weakest on contamination resistance.
Reported scores — protocols may differ. How we compare
| # | Model | Score | Evidence | Measured |
|---|---|---|---|---|
| 1 | Claude Fable 5 | 91.94 | unverifiedT1 | 2026-06-09 |
| 2 | Claude Opus 5 | 91.78 | unverifiedT1 | 2026-07-24 |
| 3 | GPT-5.6 Sol | 89.43 | unverifiedT1 | 2026-07-09 |
| 4 | GPT-5.5 | 84.91 | unverifiedT1 | 2026-04-23 |
| 5 | Claude Opus 4.8 | 82.89 | unverifiedT1 | 2026-05-28 |
| 6 | Kimi K3 | 82.57 | unverifiedT1 | 2026-07-16 |
| 7 | GPT-5.3 Codex | 79.3 | unverifiedT1 | 2026-02-05 |
| 8 | GPT-5.6 Terra | 78.27 | unverifiedT1 | 2026-07-09 |
| 9 | Claude Opus 4.6 | 77.95 | unverifiedT1 | 2026-02-05 |
| 10 | GPT-5.4 | 77.7 | unverifiedT1 | 2026-03-05 |
| 11 | Claude Opus 4.7 | 76.4 | unverifiedT1 | 2026-04-16 |
| 12 | GPT-5.2 | 72.2 | unverifiedT1 | 2025-12-11 |
| 13 | Gemini 3.1 Pro | 72.1 | unverifiedT1 | 2026-02-19 |
| 14 | GLM-5.2 | 70.12 | unverifiedT1 | 2026-06-16 |
| 15 | Gemini 3 Pro | 69.93 | unverifiedT1 | 2025-11-18 |
| 16 | Claude Sonnet 5 | 68.78 | unverifiedT1 | 2026-06-30 |
| 17 | Grok 4.6 | 67.29 | unverifiedT1 | 2026-08-12 |
| 18 | Claude Sonnet 4.6 | 66.07 | unverifiedT1 | 2026-02-17 |
| 19 | Claude Opus 4.5 | 63.7 | unverifiedT1 | 2025-11-24 |
| 20 | DeepSeek V4 Flash 0731 | 62.96 | unverifiedT1 | 2026-07-31 |
| 21 | Gemini 3.5 Flash | 62.64 | unverifiedT1 | 2026-05-19 |
| 22 | Gemini 3 Flash | 61.6 | unverifiedT1 | 2025-12-17 |
| 23 | GPT-5.6 Luna | 60.86 | unverifiedT1 | 2026-07-09 |
| 24 | GPT-5.1 | 60.77 | unverifiedT1 | 2025-11-13 |
| 25 | GPT-5 | 60.7 | unverifiedT1 | 2025-08-07 |
| 26 | GPT-5 Pro | 60.39 | unverifiedT1 | 2025-10-07 |
| 27 | GPT-5.4 Mini | 60.3 | unverifiedT1 | 2026-03-17 |
| 28 | o3-pro | 58.21 | unverifiedT1 | 2025-06-10 |
| 29 | GPT-5.4 Pro | 57.44 | unverifiedT1 | 2026-03-05 |
| 30 | GLM-5.1 | 57.1 | unverifiedT1 | 2026-04-07 |
| 31 | Gemini 3.6 Flash | 56.1 | unverifiedT1 | 2026-07-21 |
| 32 | Kimi K2.6 | 55.86 | unverifiedT1 | 2026-04-20 |
| 33 | Kimi K2.7 Code | 54.12 | unverifiedT1 | 2026-06-12 |
| 34 | Gemini 2.5 Pro (Jun 2025) | 54.03 | unverifiedT1 | 2025-06-05 |
| 35 | GPT-5 mini | 52.67 | unverifiedT1 | 2025-08-07 |
| 36 | o4-mini | 52.56 | unverifiedT1 | 2025-04-16 |
| 37 | o3 | 52.42 | unverifiedT1 | 2024-12-20 |
| 38 | Gemma 4 31B IT | 52.26 | unverifiedT1 | 2026-04-02 |
| 39 | Grok 4.20 | 52.26 | unverifiedT1 | 2026-02-17 |
| 40 | Gemini 3.1 Flash-Lite | 52.19 | unverifiedT1 | 2026-03-03 |
| 41 | Grok 4.3 Beta | 49.89 | unverifiedT1 | 2026-04-17 |
| 42 | GPT-5.4 Nano | 49.23 | unverifiedT1 | 2026-03-17 |
| 43 | DeepSeek-V4-Pro | 48.9 | unverifiedT1 | 2026-04-24 |
| 44 | gpt-oss-120b | 48.17 | unverifiedT1 | 2025-08-05 |
| 45 | GLM-5 | 48.17 | unverifiedT1 | 2026-02-11 |
| 46 | Claude Sonnet 4.5 | 47.71 | unverifiedT1 | 2025-09-29 |
| 47 | o1-preview | 47.56 | unverifiedT1 | 2024-09-12 |
| 48 | Grok 4.5 | 46.43 | unverifiedT1 | 2026-07-08 |
| 49 | Claude Sonnet 4 | 46.11 | unverifiedT1 | 2025-05-22 |
| 50 | Grok 4 | 45.73 | unverifiedT1 | 2025-07-09 |
| 51 | Kimi K2.5 | 45.6 | unverifiedT1 | 2026-02-02 |
| 52 | Claude Haiku 4.5 | 45.4 | unverifiedT1 | 2025-10-15 |
| 53 | o1 | 43.82 | unverifiedT1 | 2024-12-05 |
| 54 | Mistral Medium 3.5 | 43.71 | unverifiedT1 | 2026-04-29 |
| 55 | o3-mini | 43.7 | unverifiedT1 | 2025-01-31 |
| 56 | Nemotron 3 Ultra | 43.45 | unverifiedT1 | 2026-06-04 |
| 57 | Claude Opus 4 | 43.4 | unverifiedT1 | 2025-05-22 |
| 58 | Grok 4 Fast | 42.86 | unverifiedT1 | 2025-09-19 |
| 59 | Kimi K2 Thinking | 42.79 | unverifiedT1 | 2025-11-06 |
| 60 | Claude Opus 4.1 | 42.76 | unverifiedT1 | 2025-08-05 |
| 61 | Grok-3 mini | 42.58 | unverifiedT1 | 2025-02-19 |
| 62 | Gemini 2.5 Flash (Sep 2025) | 41.91 | unverifiedT1 | 2025-09-25 |
| 63 | DeepSeek-R1 (May 2025) | 41.63 | unverifiedT1 | 2025-05-28 |
| 64 | Qwen3-235B-A22B-Thinking (Jul 2025) | 41.04 | unverifiedT1 | 2025-07-25 |
| 65 | Gemini 2.5 Flash (Apr 2025) | 40.95 | unverifiedT1 | 2025-04-17 |
| 66 | Gemini 2.5 Flash (May 2025) | 40.95 | unverifiedT1 | 2025-05-20 |
| 67 | gpt-oss-20b | 40.93 | unverifiedT1 | 2025-08-05 |
| 68 | Claude 3.5 Sonnet (October 2024) | 39.97 | unverifiedT1 | 2024-10-22 |
| 69 | DeepSeek-V3.2-Exp | 39.48 | unverifiedT1 | 2025-09-29 |
| 70 | GPT-4.5 | 39.37 | unverifiedT1 | 2025-02-27 |
| 71 | Kimi K2 (Jul 2025) | 39.36 | unverifiedT1 | 2025-07-11 |
| 72 | GPT-4.1 | 39.04 | unverifiedT1 | 2025-04-14 |
| 73 | Gemini 3.5 Flash-Lite | 39 | unverifiedT1 | 2026-07-21 |
| 74 | Qwen3-235B-A22B-Instruct (Jul 2025) | 38.7 | unverifiedT1 | 2025-07-25 |
| 75 | GPT-5 nano | 38.06 | unverifiedT1 | 2025-08-07 |
| 76 | GPT-4.1 mini | 37.61 | unverifiedT1 | 2025-04-14 |
| 77 | Qwen3-235B-A22B | 37.28 | unverifiedT1 | 2025-04-28 |
| 78 | Grok 3 | 37.24 | unverifiedT1 | 2025-02-17 |
| 79 | MiniMax-M2.7 | 36.95 | unverifiedT1 | 2026-03-18 |
| 80 | DeepSeek-R1 | 36.49 | unverifiedT1 | 2025-01-20 |
| 81 | o1-mini | 36.32 | unverifiedT1 | 2024-09-12 |
| 82 | DeepSeek-V3 (Mar 2025) | 36.08 | unverifiedT1 | 2025-03-24 |
| 83 | Qwen 3.6 35B-A3B | 34.49 | unverifiedT1 | 2026-04-14 |
| 84 | Inkling | 32.29 | unverifiedT1 | 2026-07-15 |
| 85 | Claude 3.5 Sonnet | 30.97 | unverifiedT1 | 2024-06-20 |
| 86 | Claude 3.5 Haiku | 30.73 | unverifiedT1 | 2024-10-22 |
| 87 | Gemini 2.0 Flash (Feb 2025) | 25.77 | unverifiedT1 | 2024-12-11 |
| 88 | GPT-4o (Nov 2024) | 25.12 | unverifiedT1 | 2024-05-13 |
| 89 | Gemini 1.5 Flash (Sep 2024) | 24.87 | unverifiedT1 | 2024-05-10 |
| 90 | Llama 4 Maverick | 24.47 | unverifiedT1 | 2025-04-05 |
| 91 | Claude 3 Opus | 23.18 | unverifiedT1 | 2024-03-04 |
| 92 | Grok-2 (Dec 2024) | 22.24 | unverifiedT1 | 2024-08-13 |
| 93 | Gemini 1.5 Pro (Sept 2024) | 22.2 | unverifiedT1 | 2024-09-24 |
| 94 | Llama 3.1-405B | 21.38 | unverifiedT1 | 2024-07-23 |
| 95 | GPT-4.1 nano | 18.98 | unverifiedT1 | 2025-04-14 |
| 96 | GPT-4 Turbo (Apr 2024) | 18.01 | unverifiedT1 | 2024-04-09 |
| 97 | Qwen2.5-72B | 15.97 | unverifiedT1 | 2024-09-19 |
| 98 | Llama 3.3 70B | 14.44 | unverifiedT1 | 2024-12-06 |
| 99 | GPT-4 (Jun 2023) | 12.44 | unverifiedT1 | 2023-06-13 |
| 100 | GPT-4o mini | 11.76 | unverifiedT1 | 2024-07-18 |
| 101 | Qwen2-72B | 11.3 | unverifiedT1 | 2024-06-07 |
| 102 | Claude 3 Sonnet | 10.16 | unverifiedT1 | 2024-03-04 |
| 103 | Claude 3 Haiku | 9.84 | unverifiedT1 | 2024-03-04 |
| 104 | Llama 3.1-70B | 8.97 | unverifiedT1 | 2024-07-23 |
| 105 | Claude 2.1 | 7.06 | unverifiedT1 | 2023-11-21 |
| 106 | GPT-3.5 Turbo (Jan 2024) | 3.48 | unverifiedT1 | 2023-06-13 |
| 107 | Mixtral 8x22B | 3.17 | unverifiedT1 | 2024-04-17 |
| 108 | Llama 3.1-8B | 1.73 | unverifiedT1 | 2024-07-23 |
Continue from this leaderboard into the tightest head-to-head reads.
Weighted composite. Each dimension opens to the source-backed sub-signals it is computed from.
Claims drawn from cited facts, not live model generation.
This benchmark scores 108 models, with a 90.21-point spread between the best and worst results. Such a wide spread indicates that observed score differences likely reflect genuine capability separation rather than noise.
2 cited facts
The benchmark's integrity score is 87, corresponding to an integrity grade of A and an integrity rank of 32 among the 61 benchmarks in the universe. This integrity score gauges the benchmark's health as a discriminator of frontier models under a disclosed harness, and its weakest integrity breakdown component is contamination, which narrows how much to trust top scores.
5 cited facts
This benchmark is not saturated, so leading models can still be separated by genuine capability differences. The current top score is 91.94, leaving 8.06 points of headroom before the ceiling. Only a small cluster of models sits within a couple of points of the top, so the field is not yet compressed into noise-level ties.
4 cited facts
Scores on this benchmark are directly comparable under a consistent harness, but because the harness is bespoke and version-sensitive, they should be read as task performance under that disclosed harness rather than deployed capability. The test set is semi-private, with hidden tasks as the primary defense and no known contamination incident, although a small set of example tasks has been public for some time, so high scores still warrant some scrutiny for possible contamination. Because comparability is limited to a consistent harness, results from different harnesses should not be compared head-to-head, and the semi-private test set means high scores merit more caution than they would with a fully held-out set.
6 cited facts
This benchmark evaluates practical iterative ML-engineering skill by presenting hand-crafted novel datasets and requiring data understanding, architecture choice, working-code generation, and debugging under limited compute. Its design uses a small set of public example tasks plus a larger set of undisclosed hidden tasks, all novel, to reduce the chance they appear in training data. It assumes held-out accuracy on these novel tasks reflects genuine ML competence and that a fixed single-GPU sandbox is a fair comparable environment. The sharpest caveat is that the benchmark's high run-to-run variance means a single accuracy result can mislead, constrained single-GPU compute caps task scope, and a human baseline is not documented.
7 cited facts
WeirdML (WeirdML (currently WeirdML v2)): Tests an LLM's ability to understand an unusual dataset, design an appropriate ML architecture, and write PyTorch code that trains and predicts, iterating over feedback. It targets creative ML-engineering rather than memorized patterns.
Claude Fable 5 leads WeirdML at 91.94. The full leaderboard above lists every recorded measurement, not just the headline number.
108 models have recorded scores on WeirdML, spanning a score spread of 90.21.
tensor.news grades WeirdML A for integrity (score 87/100), ranking #32 of 61 benchmarks we assess across discrimination, saturation, contamination resistance, harness comparability, and freshness.
No — WeirdML still has headroom and continues to discriminate between models rather than bunching them at the ceiling.
Its test set is semi-private, which lowers — but does not eliminate — the risk that training data leaked into the questions.
Scores are reported under a consistent harness, so comparisons on WeirdML are reasonably apples-to-apples.
Every WeirdML measurement is source-backed and tagged reproduced, self-reported, or unverified; the underlying record cites htihle.github.io/weirdml.html.