Benchmark explainer
What is Lech Mazur Writing?
Creative short-fiction quality with hard constraint satisfaction
LLM Creative Story-Writing Benchmark: models write short stories that must organically incorporate 10 randomly-assigned required elements within a target word range, and other LLMs judge the outputs. The current canonical method (2026) is pairwise head-to-head comparison (relative Thurstone score); the widely-cited earlier method was a 0-10 LLM-grader rubric panel.
How it's scored
- Metric
- Native metric is version-dependent (current: relative pairwise Thurstone score with no fixed ceiling; archived rubric era: 0-10 grader-panel mean). Scores in this index are Epoch's normalized 0-100 rendering, not the native scale.
- Score ceiling
- 100
- Construction
- Each story must weave in 10 required elements (character, object, concept, attribute, action, method, setting, timeframe, motivation, tone); ~400 prompts (current), 600-800 word target; element sets auto-generated per seed by proposer/rater LLMs; prompts and generated stories are published in the repo
- Human baseline
- unknown (no human score published in any version)
How trustworthy
Discrimination
63.7/100Does it still separate models?
Saturation headroom
92/100How far from ceiling / clustered at the top?
Contamination resistance
34.9/100Public vs held-out; training-leak risk.
Harness comparability
100/100Is it apples-to-apples, or mixed / vendor-optimized?
Freshness
85.7/100How old is the benchmark?
Contamination history: All prompts and reference stories are public on GitHub, creating real train-on-test exposure; large combinatorial element space limits memorization payoff; no formal contamination study
Limitations: No human validation or baseline; LLM-judge risks (self-preference, style-over-substance, position bias, grader leniency drift); rubric era graded stories individually so cross-story repetition was not penalized; native pairwise scores have no fixed ceiling, so normalized values are snapshot-relative
Who leads Lech Mazur Writing
| Model | Score | Evidence |
|---|---|---|
| GPT-5 | 86 | unverified |
| Kimi K2 (Jul 2025) | 85.6 | unverified |
| Claude Opus 4.1 | 84.7 | unverified |
| o3-pro | 84.4 | unverified |
| o3 | 83.9 | unverified |
Compare the top Lech Mazur Writing scorers
Frequently asked questions
Lech Mazur Writing (LLM Creative Story-Writing Benchmark): LLM Creative Story-Writing Benchmark: models write short stories that must organically incorporate 10 randomly-assigned required elements within a target word range, and other LLMs judge the outputs. The current canonical method (2026) is pairwise head-to-head comparison (relative Thurstone score); the widely-cited earlier method was a 0-10 LLM-grader rubric panel.
GPT-5 leads Lech Mazur Writing at 86. The full leaderboard above lists every recorded measurement, not just the headline number.
43 models have recorded scores on Lech Mazur Writing, spanning a score spread of 25.5.
tensor.news grades Lech Mazur Writing B for integrity (score 76/100), ranking #46 of 65 benchmarks we assess across discrimination, saturation, contamination resistance, harness comparability, and freshness.
No — Lech Mazur Writing still has headroom and continues to discriminate between models rather than bunching them at the ceiling.
Its test set is public, so contamination risk is on the table: a high score may partly reflect training-data overlap rather than capability.
Scores are reported under a consistent harness, so comparisons on Lech Mazur Writing are reasonably apples-to-apples.
Every Lech Mazur Writing measurement is source-backed and tagged reproduced, self-reported, or unverified; the underlying record cites github.com/lechmazur/writing.