Fiction.liveBench
Integrity rank #33 of 61 · 43 models scored · top score 97.2 · o3-pro
Deep reading comprehension of long fiction across increasing context lengths
Strongest on discrimination, weakest on contamination resistance. Caveat: the public test set does not rule out training-data contamination.
Reported scores — protocols may differ. How we compare
| # | Model | Score | Evidence | Measured |
|---|---|---|---|---|
| 1 | o3-pro | 97.2 | unverifiedT1 | 2025-06-10 |
| 2 | GPT-5 | 97.2 | unverifiedT1 | 2025-08-07 |
| 3 | Grok 4 | 94.4 | unverifiedT1 | 2025-07-09 |
| 4 | Grok 4 Fast | 94.4 | unverifiedT1 | 2025-09-19 |
| 5 | Gemini 2.5 Pro (Jun 2025) | 91.7 | unverifiedT1 | 2025-06-05 |
| 6 | o3 | 88.9 | unverifiedT1 | 2024-12-20 |
| 7 | Kimi K2.5 | 86.1 | unverifiedT1 | 2026-02-02 |
| 8 | Claude 3.7 Sonnet | 83.3 | unverifiedT1 | 2025-02-24 |
| 9 | o1 | 83.3 | unverifiedT1 | 2024-12-05 |
| 10 | DeepSeek-V3.2-Exp | 83.3 | unverifiedT1 | 2025-09-29 |
| 11 | Gemini 2.5 Flash (May 2025) | 77.8 | unverifiedT1 | 2025-05-20 |
| 12 | o4-mini | 77.8 | unverifiedT1 | 2025-04-16 |
| 13 | Qwen3-235B-A22B-Thinking (Jul 2025) | 75 | unverifiedT1 | 2025-07-25 |
| 14 | DeepSeek-R1 (May 2025) | 75 | unverifiedT1 | 2025-05-28 |
| 15 | DeepSeek-R1 | 69.4 | unverifiedT1 | 2025-01-20 |
| 16 | GPT-5 mini | 69.4 | unverifiedT1 | 2025-08-07 |
| 17 | Qwen3-235B-A22B | 67.7 | unverifiedT1 | 2025-04-28 |
| 18 | Gemini 2.5 Pro (Mar 2025) | 66.7 | unverifiedT1 | 2025-03-25 |
| 19 | Gemini 2.5 Pro (May 2025) | 66.7 | unverifiedT1 | 2025-05-06 |
| 20 | Grok-3 mini | 66.7 | unverifiedT1 | 2025-02-19 |
| 21 | Qwen2.5-Max | 66.7 | unverifiedT1 | 2025-01-25 |
| 22 | Qwen3-Max | 66.7 | unverifiedT1 | 2025-09-05 |
| 23 | GPT-4.1 | 63.9 | unverifiedT1 | 2025-04-14 |
| 24 | GPT-4.5 | 63.9 | unverifiedT1 | 2025-02-27 |
| 25 | Kimi K2 (Jul 2025) | 61.1 | unverifiedT1 | 2025-07-11 |
| 26 | Claude Opus 4 | 61.1 | unverifiedT1 | 2025-05-22 |
| 27 | Gemini 2.0 Flash (Feb 2025) | 61.1 | unverifiedT1 | 2024-12-11 |
| 28 | Grok 3 | 58.3 | unverifiedT1 | 2025-02-17 |
| 29 | Qwen3-235B-A22B-Instruct (Jul 2025) | 52.9 | unverifiedT1 | 2025-07-25 |
| 30 | Gemini 2.0 Flash Thinking (Jan 2025) | 52.8 | unverifiedT1 | 2025-01-21 |
| 31 | o3-mini | 50 | unverifiedT1 | 2025-01-31 |
| 32 | DeepSeek-V3 (Mar 2025) | 50 | unverifiedT1 | 2025-03-24 |
| 33 | Gemini 2.5 Flash (Apr 2025) | 47.2 | unverifiedT1 | 2025-04-17 |
| 34 | Claude Sonnet 4 | 46.9 | unverifiedT1 | 2025-05-22 |
| 35 | Llama 4 Maverick | 46.2 | unverifiedT1 | 2025-04-05 |
| 36 | gpt-oss-120b | 44.4 | unverifiedT1 | 2025-08-05 |
| 37 | GPT-4.1 mini | 44.4 | unverifiedT1 | 2025-04-14 |
| 38 | GPT-5 nano | 44.4 | unverifiedT1 | 2025-08-07 |
| 39 | Gemini 2.0 Pro | 41.7 | unverifiedT1 | 2024-12-11 |
| 40 | Llama 4 Scout | 36 | unverifiedT1 | 2025-04-05 |
| 41 | Llama 3.3 70B | 33.3 | unverifiedT1 | 2024-12-06 |
| 42 | Gemma 3 27B | 33.3 | unverifiedT1 | 2025-03-12 |
| 43 | GPT-4.1 nano | 25 | unverifiedT1 | 2025-04-14 |
Continue from this leaderboard into the tightest head-to-head reads.
Weighted composite. Each dimension opens to the source-backed sub-signals it is computed from.
Claims drawn from cited facts, not live model generation.
Across 43 models the scores fan out over 72.2 points, separation wide enough that position on this table tracks real capability differences. This wide spread indicates that the scores reliably differentiate model capabilities.
3 cited facts
The benchmark's integrity score of 87 (grade A) ranks 26th out of 53, reflecting its health as a discriminator of frontier models under a disclosed harness, with contamination as its weakest component, which narrows how much to trust top scores.
5 cited facts
Formally unsaturated, but barely — the 97.2 top score leaves only 2.8 points of ceiling. At this altitude further separation among leaders gets hard to measure, so treat the flag with more caution than usual. Two models cluster near the top, all within a couple of points of each other. Because it is not saturated, there is still genuine room to separate the field at the top, so small differences in scores should be considered meaningful capability gaps.
5 cited facts
Scores for this benchmark are reported under a consistent known harness, meaning direct comparison is appropriate across models, and the test set is publicly available, which necessitates scrutiny of high scores for possible contamination. While the test set has been public since early 2025, creating a standing leakage risk, no specific model has been documented as contaminated. Moreover, because scores are reported per depth, cross-model comparisons require fixing the context depth; reasoning models tend to maintain performance better at longer depths.
4 cited facts
This benchmark measures deep reading comprehension of long fiction by testing models at progressively longer context lengths. Shortening each story into multiple versions that preserve the answer-relevant details is what isolates length as the variable, so per-length scores read as a controlled comparison rather than an artifact of different content. A core assumption is that the questions require genuine subtext comprehension rather than surface-level retrieval, and that the shortened versions isolate the effect of context length on accuracy. A sharp caveat is that the small number of questions and stories yields coarse, noisy per-length estimates, and the public data risks contamination and artifacts from the shortening method.
4 cited facts
Fiction.LiveBench (Fiction.liveBench): A long-context comprehension benchmark built from creative-fiction stories on Fiction.live; each question requires theory-of-mind, event chronology, and inference from implicit information rather than needle-in-a-haystack retrieval, tested at increasing context lengths.
o3-pro leads Fiction.LiveBench at 97.2. The full leaderboard above lists every recorded measurement, not just the headline number.
43 models have recorded scores on Fiction.LiveBench, spanning a score spread of 72.2.
tensor.news grades Fiction.LiveBench A for integrity (score 86/100), ranking #33 of 61 benchmarks we assess across discrimination, saturation, contamination resistance, harness comparability, and freshness.
No — Fiction.LiveBench still has headroom and continues to discriminate between models rather than bunching them at the ceiling.
Its test set is public, so contamination risk is on the table: a high score may partly reflect training-data overlap rather than capability.
Scores are reported under a consistent harness, so comparisons on Fiction.LiveBench are reasonably apples-to-apples.
Every Fiction.LiveBench measurement is source-backed and tagged reproduced, self-reported, or unverified; the underlying record cites fiction.live/stories/Fiction-liveBench-Feb-21-2025/oQdzQvKHw8JyXbN87.