SWE-RPG: Coding Agents Fail at Implicit Requirements, Not Code

Existing coding-agent benchmarks score only the final patch: did the tests pass or not? SWE-RPG, a new stage-aware benchmark from a Singapore Management University–NUS team, asks the question that outcome-only evaluation can't: where do agents actually fail? The answer reshuffles the priorities for anyone building or prompting coding agents.

The Benchmark

SWE-RPG evaluates the full chain — requirement clarification → implementation planning → code generation → artifact submission — with validated ground-truth references for the first two stages. That's the key design decision: instead of guessing why a failed patch failed, the benchmark can align a trajectory against gold clarifications and gold plans.

The construction pipeline is ruthlessly selective. From more than 2,000 eligible PR–issue pairs, successive checks of task relevance, environment reproducibility, test stability, and ground-truth validity reduce to 163 quality-controlled tasks from 31 Python and Java repositories (113 bug fixes, 50 feature additions). These aren't toy repos: codebases average 272.5K lines of code, tasks carry an average of 2,248 test cases, and each task has an average of 4.53 implicit, implementation-critical clarification points the issue description never states.

Evaluation runs a 3×6 matrix — three coding-agent frameworks (Claude Code, Codex, OpenCode) paired with six LLM backends including Claude-Sonnet-5, DeepSeek-V4-Pro, GLM-5.2, GPT-5.6-Terra, and Kimi-K3.

The headline is sobering: agents resolve only 31.5% on average, and the best configuration (OpenCode + MoonshotAI's Kimi-K3) still leaves half the tasks unresolved at 49.7%.

The Finding: It's the Requirements, Not the Code

46.7% of agent runs fail during requirement clarification or implementation planning — before the code generation that everyone benchmarks. Digging in:

• Implicit requirement recovery is the single biggest bottleneck, accounting for 24.5–46.0% of failed runs
• Code-generation failures follow at 7.4–37.4%, planning at 5.5–17.8%

The task's unwritten requirements — the constraints any experienced engineer would ask about — are what trips agents up. SWE-RPG's clarification ground truth spans six practitioner-informed dimensions: functional intent, business semantics, technical context, interface/protocol, structure/conventions, and data-structure semantics. All agents cover intent and high-level scope reliably, but collapse on the implementation-facing categories. Claude Code's lowest clarification score is code structure (54.2%). Codex drops to 42.0% on structure and 41.9% on data semantics.

The type of missing detail matters too. A feature that needs a business-semantics assumption (should this action respect existing permission checks?) fails differently from one needing interface detail (which protocol version do we target?). Prompting for both kinds — not just "be thorough" — is what the data rewards.

The Six Clarification Dimensions

SWE-RPG's ground truth is organized around the questions an experienced engineer would actually ask before implementing. Each dimension is a distinct class of missing requirement, and agents fail them unevenly:

1. Functional intent — what behavior is actually being asked for (the issue's why, not its literal wording)
2. Business semantics — which rules from the domain apply: permission checks, authorization boundaries, existing invariants ("should this respect the org's audit trail?")
3. Technical context — environment constraints: supported Python versions, dependency policy, platform targets
4. Interface / protocol — the contracts the change must honor: API shapes, schema versions, wire formats
5. Structure / conventions — where code should live and how it should be organized to match the repo's existing patterns
6. Data semantics — invariants on the data itself: what counts as valid, what must never be null, which fields are identity keys

A concrete illustration: an issue asking to "improve download reliability" might never state which retry policy exists, whether failures should be retried at the HTTP layer or the application layer, or what "reliable" means operationally. An agent that doesn't recover those assumptions will produce a plausible patch that fails the maintainer's real intent. That's the 4.53-average gap, and it's why clarifying-question training is the benchmark's core recommendation.

All agents cover functional intent and high-level scope well; the collapse happens on the implementation-facing categories — the very dimensions an issue title never encodes.

The Planning Cascade

There's also a consistent pattern in how planning degrades. For every framework, coverage cascades from where to edit, to how to implement, to constraints and validation:

Agents reliably find where to edit but rarely specify how they'll implement and verify the change — boundary conditions, compatibility requirements, and validation obligations go unstated. Notice the cascade: even the strongest framework starts strong on location and bleeds ~38 points by the time it reaches constraints.

Why the Best Config Wins

The per-configuration numbers explain why resolve rates differ more than the raw score suggests:

Two configs at similar resolve rates can have completely different failure profiles — one failing on clarification, another on code generation. A single-number leaderboard hides the diagnostic signal.

What This Means for Your Prompts

The findings translate into concrete prompting practice:

1. Ask clarifying questions before coding. Implicit requirement recovery is the dominant failure mode — an average of 4.53 missing requirement points per task. A prompt that instructs the agent to enumerate missing constraints — interface contracts, repository conventions, data invariants, edge cases — attacks the exact bottleneck this benchmark identifies. Explicit acceptance criteria in the prompt force the agent to address each dimension before generating code.
2. Demand a plan that covers constraints and validation, not just a location. When an agent proposes a fix, the plan should state boundary conditions, compatibility requirements, and how it will verify the change. The cascade shows planning stops at "where to edit" unless you push further.
3. Test stage-by-stage, not just pass/fail. If a task resolves as a failure, SWE-RPG shows the useful question is which stage — requirement, planning, or code. Retrospective diagnosis beats a binary score, and it tells you whether to fix your prompt, your plan format, or your test setup.

The takeaway: modern coding agents can write code. What they can't reliably do yet is recover what you didn't say — and that's where 24.5–46% of their failures live. Prompting for explicit clarification is the cheapest fix available today. The same theme runs through Harness-IF: the gap isn't in generating output, it's in recovering and honoring constraints that were never stated.