OpenAI · coding
August 13, 2024 · Read the original
OpenAI released SWE-bench Verified, a human-validated subset of SWE-bench that it says more reliably evaluates AI models' ability to solve real-world software issues. OpenAI states its testing found some SWE-bench tasks that may be hard or impossible to solve, leading the benchmark to systematically underestimate models' autonomous software engineering capabilities. OpenAI says it collaborated with the authors of SWE-bench on a new release of the benchmark intended to provide more accurate evaluations.
Written from the article itself at capture time and checked against it before publication: every number and every name in it comes from the source. It is our summary, not the lab’s wording — the original is linked above.
The announcement says what the lab claims. The record says what was measured, by whom, and whether anyone reproduced it. Read both before believing either.