Here's the short version: OpenAI audited SWE-Bench Pro — the benchmark it had previously told the industry to use — and found roughly 30% of its tasks are broken. The company formally retracted its endorsement yesterday. If you've been tracking AI model scores on coding benchmarks, the numbers you've been looking at are unreliable.

And this isn't the first time.

What Happened

On July 8, 2026, OpenAI's Frontier Evals team published an audit of SWE-Bench Pro, a benchmark developed by Scale AI that tests models on real-world software engineering tasks pulled from open-source repositories. The findings were bad enough that OpenAI pulled its recommendation entirely.

The audit used a multi-phase process:

Automated screening flagged 286 of 731 public-split tasks as suspicious
Codex-based investigator agents analyzed each flagged task in depth, with a human researcher reviewing the final call — labeling 200 tasks (27.4%) as broken
Five experienced software developers independently reviewed the same set and flagged 249 tasks (34.1%)

OpenAI's final estimate: ~30% of SWE-Bench Pro tasks are broken, with the human reviewers being stricter than the AI agents (though they agreed 74% of the time).

The Four Ways Tasks Break

OpenAI categorized the failures into four types, and they're worth understanding because they reveal something deeper about why coding benchmarks are so hard to get right:

1. Overly Strict Tests (most common): Tests enforce specific implementation details that aren't in the prompt. The tests were written as part of a real pull request — they validate a specific change, not an implementation-agnostic solution. A model that solves the problem correctly but differently gets marked wrong.

2. Underspecified Prompts: The task description omits requirements that the hidden tests enforce. The model is expected to guess what the test author had in mind.

3. Low-Coverage Tests: The reverse problem — tests don't fully check the requested feature, so incomplete or incorrect fixes can still pass.

4. Misleading Prompts: The task description points toward the wrong behavior, or actively contradicts what the tests require.

The Whitespace Example That Says Everything

The clearest example from OpenAI's post involves the OpenLibrary project. The task required normalizing table-of-contents entries. The prompt specified output with a single leading space:

 | Chapter 1 | 1

But the hidden test expected a double leading space:

  | Chapter 1 | 1

A model that correctly follows the prompt fails the test. A single whitespace character. That's the difference between "solved" and "failed" on a benchmark used to evaluate frontier AI coding capabilities.

This Is a Pattern Now

Here's the uncomfortable part: this is the third version of SWE-Bench that OpenAI has publicly declared unreliable.

SWE-Bench (original): The original benchmark, released by Princeton researchers. Models were improving fast. Everyone was competing on it.

SWE-Bench Verified (retired February 2026): OpenAI audited this and found 59.4% of hard problems had flawed test cases and all tested frontier models showed clear evidence of training data contamination — reproducing verbatim gold patches from training data. Claude Opus 4.5 could recall exact function names and inline comments. Gemini 3 Flash could produce the full problem text and exact diff from just the task ID. OpenAI stopped reporting scores and recommended SWE-Bench Pro instead.

SWE-Bench Pro (retracted July 2026): Built by Scale AI using private and copyleft-licensed repositories to avoid the contamination problem of Verified. It was the replacement. And now OpenAI finds 30% of its tasks are broken too.

Each time, the industry followed OpenAI's recommendation. Each time, the replacement turned out to have its own fundamental issues.

In eight months, models went from 23.3% to 80.3% on SWE-Bench Pro. That pace of improvement should have been a red flag. Turns out it was.

What This Means for the Coding Agent Ecosystem

This isn't an academic problem. Companies building coding agents — from startups to enterprises — have been benchmarking their products against SWE-Bench Pro. Product decisions, funding rounds, and deployment approvals have been influenced by scores on a benchmark OpenAI now says is broken.

The gamification was already visible. In mid-June, Artificial Analysis removed SWE-Bench Pro from its Coding Agent Index and swapped in Datacurve's DeepSWE benchmark. Their reasoning: SWE-Bench Pro was gameable. Some models had copied the correct solution from a project's commit history instead of solving the task.

The swap reshuffled leaderboards dramatically. Codex with GPT-5.5 (xhigh) scored just 31 on SWE-Bench Pro but jumped to 76 on DeepSWE. Claude Code with Opus 4.8 (max) scored 70 on SWE-Bench Pro — one of its highest results and a 25-point jump over Opus 4.7. The numbers were inconsistent because the test was. The independent benchmark firm saw it first.

The Meta-Problem: Who Validates the Validators?

There's a deeper issue here that doesn't have a clean answer. Benchmark development is hard, underfunded, and typically done by researchers who are also competing on the benchmarks they create. OpenAI has the resources to run deep audits — most companies and academic labs don't.

The tasks in SWE-Bench come from real pull requests and issue trackers. Human developers write problem descriptions and tests through conversation — not as clean evaluation artifacts. Turning messy collaboration into clean, unambiguous test cases is itself a research problem, and so far nobody has solved it well enough to withstand frontier-level scrutiny.

As models get better, they're also getting better at finding benchmark flaws. OpenAI's own methodology used Codex-based investigator agents to do the initial screening — models auditing benchmarks other models were tested on. This creates a weird recursive loop: better models produce better evaluators, which expose more broken benchmarks, which makes existing scores harder to trust.

What Comes Next

OpenAI doesn't recommend a specific replacement this time. The company's post calls on the industry to build new benchmarks using experienced developers — ones that are hard to game, trustworthy, and actually meaningful.

Several candidates are already in play:

DeepSWE (Datacurve): Already adopted by Artificial Analysis. 113 problems across 5 languages. The hardest evaluation in the index — best agents score in the 50s and 60s, leading open-weight models below 20. It's harder, but still relatively small.
GDPVal (OpenAI's internal benchmark): OpenAI has referenced this as its preferred internally-authored evaluation for coding capability. It's private, professionally written, and not publicly available for independent verification.
ProgramBench and ITBench-AA: Newer entries from the deeplearning.ai ecosystem, focused on multi-step agentic programming tasks.

None of these have the mindshare SWE-Bench had. None have been independently audited at scale. The vacuum means model evaluation is going to get more fragmented before it gets more reliable — which is, in a perverse way, healthier than the monoculture we had before.

The honest take: benchmark trust is now a first-class concern in AI development. If you're evaluating coding agents, you should be running multiple benchmarks, looking for consistency rather than absolute scores, and treating any single number with deep skepticism. The fact that the evaluator is also warning about the evaluation is the kind of signal you don't ignore.

SWE-Bench Pro Is Broken — OpenAI Finds 30% of Tasks Are Flawed and Retracts Endorsement