LLM Creative Story-Writing Benchmark
Integrity rank #46 of 65 · 43 models scored · top score 86 · GPT-5
Creative short-fiction quality with hard constraint satisfaction
Strongest on harness comparability, weakest on contamination resistance. Caveat: the public test set does not rule out training-data contamination.
Reported scores — protocols may differ. How we compare
| # | Model | Score | Evidence | Measured |
|---|---|---|---|---|
| 1 | GPT-5 | 86 | unverifiedT1 | 2025-08-07 |
| 2 | Kimi K2 (Jul 2025) | 85.6 | unverifiedT1 | 2025-07-12 |
| 3 | Claude Opus 4.1 | 84.7 | unverifiedT1 | 2025-08-05 |
| 4 | o3-pro | 84.4 | unverifiedT1 | 2025-06-10 |
| 5 | o3 | 83.9 | unverifiedT1 | 2025-04-16 |
| 6 | Gemini 2.5 Pro (Jun 2025) | 83.8 | unverifiedT1 | 2025-06-05 |
| 7 | Claude Opus 4 | 83.6 | unverifiedT1 | 2025-05-22 |
| 8 | GPT-5 mini | 83.1 | unverifiedT1 | 2025-08-07 |
| 9 | DeepSeek-R1 | 83 | unverifiedT1 | 2025-01-20 |
| 10 | Qwen3-235B-A22B | 83 | unverifiedT1 | 2025-04-28 |
| 11 | Qwen3-235B-A22B-Thinking (Jul 2025) | 82.4 | unverifiedT1 | 2025-07-25 |
| 12 | DeepSeek-R1 (May 2025) | 81.9 | unverifiedT1 | 2025-05-28 |
| 13 | GPT-4o (Nov 2024) | 81.8 | unverifiedT1 | 2024-11-20 |
| 14 | Claude Sonnet 4 | 81.4 | unverifiedT1 | 2025-05-22 |
| 15 | Claude 3.7 Sonnet | 81.1 | unverifiedT1 | 2025-02-24 |
| 16 | Gemini 2.5 Pro (May 2025) | 80.9 | unverifiedT1 | 2025-05-06 |
| 17 | Gemini 2.5 Pro (Mar 2025) | 80.5 | unverifiedT1 | 2025-03-25 |
| 18 | Claude 3.5 Sonnet (October 2024) | 80.3 | unverifiedT1 | 2024-10-22 |
| 19 | QwQ-32B | 80.2 | unverifiedT1 | 2025-03-05 |
| 20 | Gemma 3 27B | 79.9 | unverifiedT1 | 2025-03-12 |
| 21 | Mistral Medium 3 | 77.3 | unverifiedT1 | 2025-05-07 |
| 22 | gpt-oss-120b | 77.1 | unverifiedT1 | 2025-08-05 |
| 23 | DeepSeek-V3 (Mar 2025) | 77 | unverifiedT1 | 2025-03-24 |
| 24 | Grok 4 | 76.9 | unverifiedT1 | 2025-07-09 |
| 25 | Gemini 2.5 Flash (Apr 2025) | 76.5 | unverifiedT1 | 2025-04-17 |
| 26 | Grok 3 | 76.4 | unverifiedT1 | 2025-04-09 |
| 27 | GPT-4.5 | 75.6 | unverifiedT1 | 2025-02-27 |
| 28 | Qwen3-30B-A3B | 75.3 | unverifiedT1 | 2025-04-28 |
| 29 | o4-mini | 75 | unverifiedT1 | 2025-04-16 |
| 30 | Gemini 2.0 Flash Thinking (Jan 2025) | 73.8 | unverifiedT1 | 2025-01-21 |
| 31 | Claude 3.5 Haiku | 73.5 | unverifiedT1 | 2024-10-22 |
| 32 | Grok-3 mini | 73.5 | unverifiedT1 | 2025-04-09 |
| 33 | Qwen2.5-Max | 72.9 | unverifiedT1 | 2025-01-25 |
| 34 | Gemini 2.0 Flash (Dec 2024) | 71.5 | unverifiedT1 | 2024-12-11 |
| 35 | o1 | 70.2 | unverifiedT1 | 2024-12-17 |
| 36 | Mistral Large 2 (Jul 2024) | 69 | unverifiedT1 | 2024-07-24 |
| 37 | GPT-4o mini | 67.2 | unverifiedT1 | 2024-07-18 |
| 38 | o1-mini | 64.9 | unverifiedT1 | 2024-09-12 |
| 39 | Grok-2 (Dec 2024) | 63.6 | unverifiedT1 | 2024-12-12 |
| 40 | Phi-4 | 62.6 | unverifiedT1 | 2024-12-12 |
| 41 | Llama 4 Maverick | 62 | unverifiedT1 | 2025-04-05 |
| 42 | o3-mini | 61.7 | unverifiedT1 | 2025-01-31 |
| 43 | Amazon Nova Pro | 60.5 | unverifiedT1 | 2024-12-03 |
Continue from this leaderboard into the tightest head-to-head reads.
Weighted composite. Each dimension opens to the source-backed sub-signals it is computed from.
Claims drawn from cited facts, not live model generation.
This benchmark scores 43 models, and the spread between its best and worst scores is 25.5 points. A spread of that size supports real separation between models, so rank differences on this benchmark are more likely to reflect genuine capability gaps; were the spread narrow, closely clustered scores could not be trusted to indicate a true difference.
3 cited facts
Its Benchmark Integrity Index score is 76, with grade B and rank 46 out of 65 tracked benchmarks.
4 cited facts
This benchmark is not saturated: the top score sits at 86.0, leaving 14.0 points of headroom to the ceiling. Four models cluster within a couple of points of the top, so the leading pack is tight. Because the benchmark is not saturated, there is still genuine room to separate the field at the top, and differences among the leaders can reflect real capability gaps rather than noise.
5 cited facts
Under the offered harness comparability label of a consistent harness, scores for this benchmark are directly comparable within that disclosed setup, and if comparability were low, scores from different harnesses should not be compared head-to-head. Under the offered test set privacy label of public, a public test set means high scores deserve more scrutiny for possible contamination than a held-out test set would. Qualitatively, contamination history includes public prompts and reference stories on GitHub, creating train-on-test exposure while the large combinatorial element space limits memorization payoff and no formal contamination study exists; the harness is sensitive in that it uses a rotating LLM-as-judge panel with pairwise order swaps, side-bias correction, and bootstrapping, so scores should be read as task performance under a disclosed harness rather than deployed capability.
4 cited facts
This benchmark measures creative short-fiction quality with hard constraint satisfaction. Its test was built by requiring each story to weave in a set of required elements spanning character, object, concept, attribute, action, method, setting, timeframe, motivation, and tone, with element sets auto-generated per seed by proposer/rater LLMs, and prompts and generated stories published in the repo. The benchmark assumes an LLM judge panel (or pairwise LLM judges) proxies human literary judgment, that multi-grader aggregation plus bias correction yields a real quality signal, and that auto-generated random element sets produce comparable-difficulty briefs. The sharpest caveat is that there is no human validation or baseline — the human baseline is not documented — leaving LLM-judge risks such as self-preference, style-over-substance, position bias, and grader leniency drift, plus rubric-era cross-story repetition not being penalized and snapshot-relative normalized scores, so the quality signal may not reflect human literary judgment.
5 cited facts
Lech Mazur Writing (LLM Creative Story-Writing Benchmark): LLM Creative Story-Writing Benchmark: models write short stories that must organically incorporate 10 randomly-assigned required elements within a target word range, and other LLMs judge the outputs. The current canonical method (2026) is pairwise head-to-head comparison (relative Thurstone score); the widely-cited earlier method was a 0-10 LLM-grader rubric panel.
GPT-5 leads Lech Mazur Writing at 86. The full leaderboard above lists every recorded measurement, not just the headline number.
43 models have recorded scores on Lech Mazur Writing, spanning a score spread of 25.5.
tensor.news grades Lech Mazur Writing B for integrity (score 76/100), ranking #46 of 65 benchmarks we assess across discrimination, saturation, contamination resistance, harness comparability, and freshness.
No — Lech Mazur Writing still has headroom and continues to discriminate between models rather than bunching them at the ceiling.
Its test set is public, so contamination risk is on the table: a high score may partly reflect training-data overlap rather than capability.
Scores are reported under a consistent harness, so comparisons on Lech Mazur Writing are reasonably apples-to-apples.
Every Lech Mazur Writing measurement is source-backed and tagged reproduced, self-reported, or unverified; the underlying record cites github.com/lechmazur/writing.