OpenAI · coding
February 23, 2026 · Read the original
OpenAI published an analysis stating that SWE-bench Verified is increasingly contaminated and is no longer suitable for measuring progress on autonomous software engineering capabilities. The analysis found that at least 59.4% of problems in an audited 27.6% subset have flawed test cases that reject functionally correct submissions, and that all frontier models tested could reproduce the gold patch or verbatim problem statement specifics for certain tasks, indicating exposure during training. OpenAI states it has stopped reporting SWE-bench Verified scores and recommends other model developers do so too, and it recommends SWE-bench Pro.
Written from the article itself at capture time and checked against it before publication: every number and every name in it comes from the source. It is our summary, not the lab’s wording — the original is linked above.
The announcement says what the lab claims. The record says what was measured, by whom, and whether anyone reproduced it. Read both before believing either.