Fiction.liveBench
Integrity rank #33 of 65 · 50 models scored · top score 97.2 · o3-pro
Deep reading comprehension of long fiction across increasing context lengths
Strongest on discrimination, weakest on contamination resistance. Caveat: the public test set does not rule out training-data contamination.
Reported scores — protocols may differ. How we compare
| # | Model | Score | Evidence | Measured |
|---|---|---|---|---|
| 1 | o3-pro | 97.2 | unverifiedT1 | 2025-06-10 |
| 2 | GPT-5 | 97.2 | unverifiedT1 | 2025-08-07 |
| 3 | Grok 4 | 94.4 | unverifiedT1 | 2025-07-09 |
| 4 | Grok 4 Fast | 94.4 | unverifiedT1 | 2025-09-19 |
| 5 | Gemini 2.5 Pro (Jun 2025) | 91.7 | unverifiedT1 | 2025-06-05 |
| 6 | o3 | 88.9 | unverifiedT1 | 2025-04-16 |
| 7 | Kimi K2.5 | 86.1 | unverifiedT1 | 2026-01-27 |
| 8 | Claude 3.7 Sonnet | 83.3 | unverifiedT1 | 2025-02-24 |
| 9 | QwQ-32B | 83.3 | unverifiedT1 | 2025-03-05 |
| 10 | o1 | 83.3 | unverifiedT1 | 2024-12-17 |
| 11 | DeepSeek-V3.2-Exp | 83.3 | unverifiedT1 | 2025-09-29 |
| 12 | Gemini 2.5 Flash (May 2025) | 77.8 | unverifiedT1 | 2025-05-20 |
| 13 | o4-mini | 77.8 | unverifiedT1 | 2025-04-16 |
| 14 | Qwen3-235B-A22B-Thinking (Jul 2025) | 75 | unverifiedT1 | 2025-07-25 |
| 15 | DeepSeek-R1 (May 2025) | 75 | unverifiedT1 | 2025-05-28 |
| 16 | Qwen3-32B | 74.2 | unverifiedT1 | 2025-04-28 |
| 17 | DeepSeek-R1 | 69.4 | unverifiedT1 | 2025-01-20 |
| 18 | GPT-5 mini | 69.4 | unverifiedT1 | 2025-08-07 |
| 19 | Qwen3-235B-A22B | 67.7 | unverifiedT1 | 2025-04-28 |
| 20 | Gemini 2.5 Pro (Mar 2025) | 66.7 | unverifiedT1 | 2025-03-31 |
| 21 | Gemini 2.5 Pro (May 2025) | 66.7 | unverifiedT1 | 2025-05-06 |
| 22 | Grok-3 mini | 66.7 | unverifiedT1 | 2025-04-09 |
| 23 | Qwen2.5-Max | 66.7 | unverifiedT1 | 2025-01-25 |
| 24 | Qwen3-Max | 66.7 | unverifiedT1 | 2025-09-24 |
| 25 | GPT-4.1 | 63.9 | unverifiedT1 | 2025-04-14 |
| 26 | GPT-4.5 | 63.9 | unverifiedT1 | 2025-02-27 |
| 27 | Qwen3-14B | 62.5 | unverifiedT1 | 2025-04-28 |
| 28 | Qwen3-8B | 62.1 | unverifiedT1 | 2025-04-28 |
| 29 | Gemini 2.0 Flash (Feb 2025) | 61.1 | unverifiedT1 | 2025-02-05 |
| 30 | Kimi K2 (Jul 2025) | 61.1 | unverifiedT1 | 2025-07-12 |
| 31 | Claude Opus 4 | 61.1 | unverifiedT1 | 2025-05-22 |
| 32 | Grok 3 | 58.3 | unverifiedT1 | 2025-04-09 |
| 33 | Qwen3-235B-A22B-Instruct (Jul 2025) | 52.9 | unverifiedT1 | 2025-07-25 |
| 34 | Gemini 2.0 Flash Thinking (Jan 2025) | 52.8 | unverifiedT1 | 2025-01-21 |
| 35 | DeepSeek-V3.1 | 52.8 | unverifiedT1 | 2025-08-21 |
| 36 | o3-mini | 50 | unverifiedT1 | 2025-01-31 |
| 37 | DeepSeek-V3 (Mar 2025) | 50 | unverifiedT1 | 2025-03-24 |
| 38 | Gemini 2.5 Flash (Apr 2025) | 47.2 | unverifiedT1 | 2025-04-17 |
| 39 | Gemini 2.5 Flash-Lite (Jun 2025) | 47.2 | unverifiedT1 | 2025-06-17 |
| 40 | Claude Sonnet 4 | 46.9 | unverifiedT1 | 2025-05-22 |
| 41 | Llama 4 Maverick | 46.2 | unverifiedT1 | 2025-04-06 |
| 42 | gpt-oss-120b | 44.4 | unverifiedT1 | 2025-08-05 |
| 43 | GPT-4.1 mini | 44.4 | unverifiedT1 | 2025-04-14 |
| 44 | GPT-5 nano | 44.4 | unverifiedT1 | 2025-08-07 |
| 45 | Gemini 2.0 Pro | 41.7 | unverifiedT1 | 2025-02-05 |
| 46 | Qwen3-30B-A3B | 40.6 | unverifiedT1 | 2025-04-28 |
| 47 | Llama 4 Scout | 36 | unverifiedT1 | 2025-04-05 |
| 48 | Llama 3.3 70B | 33.3 | unverifiedT1 | 2024-12-06 |
| 49 | Gemma 3 27B | 33.3 | unverifiedT1 | 2025-03-11 |
| 50 | GPT-4.1 nano | 25 | unverifiedT1 | 2025-04-14 |
Continue from this leaderboard into the tightest head-to-head reads.
Weighted composite. Each dimension opens to the source-backed sub-signals it is computed from.
Claims drawn from cited facts, not live model generation.
This benchmark scores 50 models, with a spread of 72.2 points between the best and worst performers. A spread that wide indicates the models are genuinely separated in performance, so rank differences among them are more likely to reflect real capability gaps than scoring noise.
3 cited facts
The Benchmark Integrity Index for this benchmark is 85 with a grade of A, ranking 35 of 62 benchmarks, reflecting its health as a discriminator of frontier models under a disclosed harness. The weakest component of the integrity breakdown is contamination, which narrows how much readers should trust top scores. This index describes the benchmark's discrimination quality, not any model's capability.
6 cited facts
The benchmark is not saturated, with a top score of 97.2 and a remaining headroom of 2.8 points to the ceiling, while two models cluster near the top within a couple of points of one another. Saturated would mean today's ranking among the top models is close to noise, so small differences there should not be read as real capability gaps; because this benchmark is not saturated, there remains genuine room to separate the field at the top.
5 cited facts
Because scores are reported under a consistent harness, they are directly comparable across models only when the harness settings are held fixed, and depth in particular must be pinned since reported scores vary with it. The test set is public rather than held out, so strong results deserve more scrutiny for possible contamination than a held-out set would, consistent with the standing leakage risk created by openly posted material, although no specific model has been documented as contaminated. Read these figures as task performance under a disclosed harness rather than deployed capability, keeping in mind that models are harness-sensitive and that reasoning models tend to hold up better at long depths while others degrade.
6 cited facts
This benchmark measures deep reading comprehension of long fiction across increasing context lengths. It was constructed from questions over stories, with multiple shortened versions of each story that preserve answer-relevant detail, and models are tested separately at each length. The design assumes the questions require genuine subtext comprehension rather than retrieval, and that shortened versions preserve the answer-relevant information so accuracy differences across lengths isolate a long-context effect. The sharpest caveat is that the small question set gives coarse, noisy per-bucket estimates, public data invites contamination and overfitting, the domain is fiction-specific, and the shortening method could introduce artifacts; human baseline is not documented.
5 cited facts
Fiction.LiveBench (Fiction.liveBench): A long-context comprehension benchmark built from creative-fiction stories on Fiction.live; each question requires theory-of-mind, event chronology, and inference from implicit information rather than needle-in-a-haystack retrieval, tested at increasing context lengths.
o3-pro leads Fiction.LiveBench at 97.2. The full leaderboard above lists every recorded measurement, not just the headline number.
50 models have recorded scores on Fiction.LiveBench, spanning a score spread of 72.2.
tensor.news grades Fiction.LiveBench A for integrity (score 85/100), ranking #33 of 65 benchmarks we assess across discrimination, saturation, contamination resistance, harness comparability, and freshness.
No — Fiction.LiveBench still has headroom and continues to discriminate between models rather than bunching them at the ceiling.
Its test set is public, so contamination risk is on the table: a high score may partly reflect training-data overlap rather than capability.
Scores are reported under a consistent harness, so comparisons on Fiction.LiveBench are reasonably apples-to-apples.
Every Fiction.LiveBench measurement is source-backed and tagged reproduced, self-reported, or unverified; the underlying record cites fiction.live/stories/Fiction-liveBench-Feb-21-2025/oQdzQvKHw8JyXbN87.