SWE-bench Verified
Integrity rank #40 of 65 · 32 models scored · top score 83.47 · Claude Opus 4.7
Resolve real GitHub issues so hidden repo tests pass
Strongest on discrimination, weakest on contamination resistance. Caveat: the public test set does not rule out training-data contamination.
Reported scores — protocols may differ. How we compare
| # | Model | Score | Evidence | Measured |
|---|---|---|---|---|
| 1 | Claude Opus 4.7 | 83.47 | reproduced· optimizedT1 | 2026-04-16 |
| 2 | GPT-5.5 | 80.58 | reproduced· optimizedT1 | 2026-04-23 |
| 3 | Gemini 3.5 Flash | 79.34 | reproduced· optimizedT1 | 2026-05-19 |
| 4 | Claude Opus 4.6 | 78.72 | reproduced· optimizedT1 | 2026-02-05 |
| 5 | GLM-5.2 | 78.7 | reproduced· optimizedT1 | 2026-06-16 |
| 6 | DeepSeek-V4-Pro | 77.64 | reproduced· optimizedT1 | 2026-04-24 |
| 7 | Qwen3.7-Max | 77.27 | reproduced· optimizedT1 | 2026-05-19 |
| 8 | GPT-5.4 | 76.86 | reproduced· optimizedT1 | 2026-03-05 |
| 9 | Kimi K2.6 | 76.65 | reproduced· optimizedT1 | 2026-04-20 |
| 10 | Qwen 3.6 Max (Preview) | 76.65 | reproduced· optimizedT1 | 2026-04-20 |
| 11 | Claude Opus 4.5 | 76.65 | reproduced· optimizedT1 | 2025-11-24 |
| 12 | Gemini 3.1 Pro | 75.62 | reproduced· optimizedT1 | 2026-02-19 |
| 13 | Gemini 3 Flash | 75.41 | reproduced· optimizedT1 | 2025-12-17 |
| 14 | Claude Sonnet 4.6 | 75.21 | reproduced· optimizedT1 | 2026-02-17 |
| 15 | GPT-5.3 Codex | 74.79 | reproduced· optimizedT1 | 2026-02-05 |
| 16 | GLM-5.1 | 74.17 | reproduced· optimizedT1 | 2026-04-07 |
| 17 | Kimi K2.5 | 73.76 | reproduced· optimizedT1 | 2026-01-27 |
| 18 | GPT-5.2 | 73.76 | reproduced· optimizedT1 | 2025-12-11 |
| 19 | GPT-5 | 73.55 | reproduced· optimizedT1 | 2025-08-07 |
| 20 | Claude Opus 4.1 | 73.35 | reproduced· optimizedT1 | 2025-08-05 |
| 21 | Gemini 3 Pro | 72.93 | reproduced· optimizedT1 | 2025-11-18 |
| 22 | GLM-5 | 72.08 | reproduced· optimizedT1 | 2026-02-11 |
| 23 | Claude Sonnet 4.5 | 71.28 | reproduced· optimizedT1 | 2025-09-29 |
| 24 | Claude Opus 4 | 70.66 | reproduced· optimizedT1 | 2025-05-22 |
| 25 | GPT-5.1 | 67.98 | reproduced· optimizedT1 | 2025-11-13 |
| 26 | GPT-5 mini | 64.67 | reproduced· optimizedT1 | 2025-08-07 |
| 27 | o3 | 62.32 | reproduced· optimizedT1 | 2025-04-16 |
| 28 | Claude 3.7 Sonnet | 60.95 | reproduced· optimizedT1 | 2025-02-24 |
| 29 | Qwen 3.6 Plus | 57.85 | reproduced· optimizedT1 | 2026-03-31 |
| 30 | Gemini 2.5 Pro (Jun 2025) | 57.56 | reproduced· optimizedT1 | 2025-06-05 |
| 31 | GPT-4.1 | 48.54 | reproduced· optimizedT1 | 2025-04-14 |
| 32 | GPT-4o (Nov 2024) | 30.99 | reproduced· optimizedT1 | 2024-11-20 |
Continue from this leaderboard into the tightest head-to-head reads.
Weighted composite. Each dimension opens to the source-backed sub-signals it is computed from.
Claims drawn from cited facts, not live model generation.
This benchmark is scored on 32 models, with a 52.48-point gap between the best and worst scores. That wide spread indicates meaningful separation between model scores, whereas a narrow spread would imply that close scores might not reflect a genuine capability difference.
3 cited facts
The benchmark's Benchmark Integrity Index is 83, corresponding to grade B, and it ranks 37th out of 61 benchmarks. This score reflects the benchmark's health as a discriminator of frontier models under its disclosed harness. The weakest integrity component is contamination, and that narrows how much to trust top scores.
6 cited facts
This benchmark is not saturated: the leading score is eighty-three point four seven, leaving a headroom of sixteen point five three, and only one model sits within a couple of points of the top. That means the current top ranking still has genuine room to separate the field, so small differences among the best models should not be read as noise.
4 cited facts
Under this benchmark's disclosed harness, harness comparability is consistent, so task performance scores are directly comparable within this harness; however, harness sensitivity is significant — the same model's scores swing widely across scaffolds — so cross-harness head-to-head comparisons still merit caution. The test set is public, meaning high scores deserve more scrutiny for possible contamination than a held-out set would. Contamination history includes public PRs predating cutoffs and documented 'cheating' by reading future commit state, reinforcing that scrutiny.
4 cited facts
This benchmark measures whether a model can resolve real GitHub issues so that hidden repository tests pass, using tasks that were human-validated from a larger item pool spanning multiple repositories. Its central assumption is that passing hidden tests equates to correctly fixing the issue, treating that equivalence as sufficient for judging correctness. Documented limitations are that the benchmark is gameable with overfit patches, is Python-only, and that scores are dominated by the agent scaffold rather than the model; the human baseline is not documented as a numeric figure, though a real merged fix counts as perfect by construction. The sharpest caveat is that high performance may reflect patch overfitting or scaffold effects more than genuine issue-solving ability, so the benchmark can mislead about the underlying model's capability.
6 cited facts
SWE-Bench verified (SWE-bench Verified): SWE-bench Verified — a human-validated subset of SWE-bench in which a model must resolve real GitHub issues from Python repositories end-to-end; scored as the percent of issues resolved.
Claude Opus 4.7 leads SWE-Bench verified at 83.47 — independently reproduced. The full leaderboard above lists every recorded measurement, not just the headline number.
32 models have recorded scores on SWE-Bench verified, spanning a score spread of 52.48.
tensor.news grades SWE-Bench verified B for integrity (score 82/100), ranking #40 of 65 benchmarks we assess across discrimination, saturation, contamination resistance, harness comparability, and freshness.
No — SWE-Bench verified still has headroom and continues to discriminate between models rather than bunching them at the ceiling.
Its test set is public, so contamination risk is on the table: a high score may partly reflect training-data overlap rather than capability.
Scores are reported under a consistent harness, so comparisons on SWE-Bench verified are reasonably apples-to-apples.
Every SWE-Bench verified measurement is source-backed and tagged reproduced, self-reported, or unverified; the underlying record cites openai.com/index/introducing-swe-bench-verified.