Benchmarking Agentic LLM and VLM Reasoning On Games
Integrity rank #22 of 65 · 32 models scored · top score 68.3 · GPT-6 Astra
Long-horizon agentic planning, spatial reasoning, and sequential decision-making in games
Strongest on discrimination, weakest on contamination resistance. Caveat: the public test set does not rule out training-data contamination.
Reported scores — protocols may differ. How we compare
| # | Model | Score | Evidence | Measured |
|---|---|---|---|---|
| 1 | GPT-6 Astra | 68.3 | unverifiedT1 | 2026-09-03 |
| 2 | Claude Opus 5 | 63.4 | unverifiedT1 | 2026-07-24 |
| 3 | GPT-5.6 Sol | 60 | unverifiedT1 | 2026-07-09 |
| 4 | Gemini 3 Pro | 58.1 | unverifiedT1 | 2025-11-18 |
| 5 | Gemini 3.1 Pro | 57 | unverifiedT1 | 2026-02-19 |
| 6 | GPT-5.6 Terra | 53.2 | unverifiedT1 | 2026-07-09 |
| 7 | Gemini 3 Flash | 48.1 | unverifiedT1 | 2025-12-17 |
| 8 | GPT-5.6 Luna | 45.6 | unverifiedT1 | 2026-07-09 |
| 9 | Grok 4 | 43.6 | unverifiedT1 | 2025-07-09 |
| 10 | Claude Opus 4.5 | 43.5 | unverifiedT1 | 2025-11-24 |
| 11 | Gemini 2.5 Pro (Mar 2025) | 43.3 | unverifiedT1 | 2025-03-25 |
| 12 | DeepSeek-R1 | 34.9 | unverifiedT1 | 2025-01-20 |
| 13 | Gemini 2.5 Flash (Jun 2025) | 33.5 | unverifiedT1 | 2025-06-17 |
| 14 | GPT-5 | 32.8 | unverifiedT1 | 2025-08-07 |
| 15 | Claude 3.5 Sonnet (October 2024) | 32.6 | unverifiedT1 | 2024-10-22 |
| 16 | GPT-4o (May 2024) | 32.3 | unverifiedT1 | 2024-05-13 |
| 17 | Claude Haiku 4.5 | 31.2 | unverifiedT1 | 2025-10-15 |
| 18 | Grok 3 | 29.5 | unverifiedT1 | 2025-04-09 |
| 19 | Llama 3.1-70B | 27.9 | unverifiedT1 | 2024-07-23 |
| 20 | Llama 3.2 90B | 27.3 | unverifiedT1 | 2024-09-24 |
| 21 | Llama 3.3 70B | 23 | unverifiedT1 | 2024-12-06 |
| 22 | Gemini 1.5 Pro (Sept 2024) | 21 | unverifiedT1 | 2024-09-24 |
| 23 | DeepSeek-R1-Distill-Qwen-32B | 19.5 | unverifiedT1 | 2025-01-20 |
| 24 | Claude 3.5 Haiku | 19.3 | unverifiedT1 | 2024-10-22 |
| 25 | Mistral NeMo | 17.6 | unverifiedT1 | 2024-07-18 |
| 26 | GPT-4o mini | 17.4 | unverifiedT1 | 2024-07-18 |
| 27 | Qwen2.5-72B | 16.2 | unverifiedT1 | 2024-09-19 |
| 28 | Llama 3.1-8B | 15.1 | unverifiedT1 | 2024-07-23 |
| 29 | Gemini 1.5 Flash (Sep 2024) | 14.6 | unverifiedT1 | 2024-09-24 |
| 30 | Phi-4 | 11.6 | unverifiedT1 | 2024-12-12 |
| 31 | Qwen2.5-7B | 7.8 | unverifiedT1 | 2024-09-19 |
| 32 | Llama 3.2 1B | 6.6 | unverifiedT1 | 2024-09-24 |
Continue from this leaderboard into the tightest head-to-head reads.
Weighted composite. Each dimension opens to the source-backed sub-signals it is computed from.
Claims drawn from cited facts, not live model generation.
Across the models scored on this benchmark, the count stands at 32, and the gap between the best and worst scores is 61.7 points. A spread that wide indicates substantial separation between the strongest and weakest models, so rank differences at the top and bottom of the field are likely to reflect genuine capability gaps rather than minor scoring noise.
3 cited facts
This benchmark-level health measure—not a model capability score—has an integrity score of 89 with an integrity grade of A, ranking 20th among 62 benchmarks as a discriminator of frontier models under a disclosed harness. The weakest integrity component is contamination, which narrows how much top scores can be trusted as evidence of genuine discrimination.
5 cited facts
This benchmark is not saturated: the top score is 68.3 and the remaining headroom to the ceiling is 31.7 points. Only one model clusters near the top, within a couple of points, meaning there is still genuine room to separate the field at the top rather than treating small differences as noise.
5 cited facts
Scores for this benchmark are produced under a consistent harness, so they are internally meaningful but should not be compared head-to-head against results from other harnesses; the test set is public, meaning strong results deserve more scrutiny for possible contamination than would a held-out set. The harness notes that prompting and scaffold choices materially affect scores, while the contamination history indicates low memorization risk because environments are procedurally generated and no contamination incident is known. Reported values should therefore be read as task performance under that disclosed harness rather than as evidence of deployed capability, and any harness sensitivity should be treated as a caveat on cross-scaffold interpretation.
6 cited facts
This benchmark measures long-horizon agentic planning, spatial reasoning, and sequential decision-making in games. It is built by wrapping multiple existing procedurally generated reinforcement-learning environments and generating instances per run, rather than using a fixed hand-written item set. The design assumes that progress in these games proxies general agentic ability — planning, credit assignment, and spatial or exploratory reasoning — and that natural-language action output faithfully reflects the model's decisions; no numeric human baseline is documented. The sharpest caveat is that weak vision-based decisions, a zero-shot-only evaluation, and uneven per-environment difficulty mean a score may reflect environment-specific quirks or one dominant hard environment rather than transferable agentic ability.
5 cited facts
Balrog (Benchmarking Agentic LLM and VLM Reasoning On Games): Benchmarking Agentic LLM and VLM Reasoning On Games (BALROG) drops models into six procedurally-generated reinforcement-learning game environments (NetHack, MiniHack, BabyAI, Crafter, TextWorld, Baba Is AI) and measures how far they progress via natural-language actions. Far from saturated as of 2026.
GPT-6 Astra leads Balrog at 68.3. The full leaderboard above lists every recorded measurement, not just the headline number.
32 models have recorded scores on Balrog, spanning a score spread of 61.7.
tensor.news grades Balrog A for integrity (score 89/100), ranking #22 of 65 benchmarks we assess across discrimination, saturation, contamination resistance, harness comparability, and freshness.
No — Balrog still has headroom and continues to discriminate between models rather than bunching them at the ceiling.
Its test set is public, so contamination risk is on the table: a high score may partly reflect training-data overlap rather than capability.
Scores are reported under a consistent harness, so comparisons on Balrog are reasonably apples-to-apples.
Every Balrog measurement is source-backed and tagged reproduced, self-reported, or unverified; the underlying record cites balrogai.com.