SWE-Touch: Coding Agents Break When Users Touch the Code

Most coding-agent benchmarks evaluate agents working alone, in a clean repository. Real developers don't work that way — they edit code while the agent works. SWE-Touch's own analysis of SWE-chat data found 59% of sessions contain user-authored repository changes, so this isn't a corner case; it's the default. SWE-Touch tests exactly this: what happens when a user modifies task-relevant code mid-task? The answer is uncomfortable: agents lose ground, often badly, and they don't even notice the change.

The Setup

SWE-Touch mines task-critical regions from repair trajectories, then injects Counter-Edits — small, plausible user edits that conflict with the task — at the moment the agent reaches the relevant code, accompanied by a contextual user message ("I've made a quick update to the core logic; please use this version"). Think: the user changes a function signature the agent is about to fix, or edits the exact file the agent is patching.

Each edit is validated three ways before it counts: it must not solve the task on its own, the reference patch must still solve the task on the original codebase, and the reference patch combined with the user's edit must fail the tests. That last check is what makes it a genuine obstacle rather than a trivial distraction.

Across nine models on SWE-bench Verified, the Counter-Edit condition lowers the mean resolve rate by 7.7 points, with individual models losing anywhere from 1.3 to 16.5 points. The degradation persists on longer-horizon benchmarks (SWE-Bench Pro, DeepSWE), and rankings reshuffle — shared-workspace robustness is a distinct evaluation dimension, not a proxy for autonomous skill.

The Model Breakdown

The heterogeneity is the story. Strong autonomous models aren't necessarily the ones that survive contact with a user:

The two strongest autonomous agents (Claude Opus 4.8, GPT 5.5) retain over 95% of their Vanilla-solved tasks. But a model at 76.5% Vanilla (MiniMax M2.7) drops to 62.7% — worse than DeepSeek V4 Pro's post-edit 63.8%. Autonomous resolve rate alone does not predict interactive robustness.

The Failure Mode: No Workspace Awareness

Trajectory analysis shows what goes wrong. 63.3% of failed runs retain the user's conflicting code — the agent either misses the external modification entirely, defers to the user's code without verifying it, or revises it without re-checking the affected behavior. The remaining failures are instructive too: incorrect replacement (the agent notices the problem but introduces a new error) at 13.9%, incomplete reconciliation (fixes the local conflict but breaks dependent code) at 11.6%, and off-target implementation at 5.5%. Even when agents challenge the user's edit, they fail roughly a third of the time at the reconcile-and-validate step.

Two controls make the diagnosis precise:

• Message-only (no edit): sending a user message without changing the repository has limited and inconsistent impact (−2.0 to +3.0 points). Words alone don't break agents.
• Silent edit (no message): the code edit alone causes a consistent decline for every model (−1.0 to −9.5 points), because the agent must detect the conflict from the code itself.
• Co-Edit control: a small, non-conflicting workspace edit changes average resolve rate by only −0.1 points across seven models. It's not "any external change" that breaks agents — it's specifically conflicting state they must reconcile.

And notably, adding a message on top of the edit doesn't reliably help — even when users explicitly announce their edits, most models don't use the message to locate and reconcile the conflicting code.

Open-Source Models Suffer Most

The gap between autonomous and interactive performance is especially pronounced for open-source models that score competitively on static leaderboards. Models optimized for autonomous SWE-bench performance degrade substantially when the workspace is a moving target — a warning about treating leaderboard scores as production readiness. The two most stable models under intervention are both closed, frontier-tier APIs; the two worst-performing are open-weight.

What to Measure on Your Own Agents

SWE-Touch's value is that its method transfers to your stack. If you run agents in repos where other people commit, add an interactive stress test to your eval suite:

1. Simulate the shared workspace. Pick 3–5 tasks from your backlog. Mid-run, introduce a small conflicting edit to a task-critical file — change the signature the agent is about to touch, or rewrite the exact block it's patching. Announce it in the conversation, the way SWE-Touch does.
2. Grade the reconcile stage, not just the outcome. Did the agent detect the edit? Did it verify the affected behavior with a test? A patch that "passes" by silently reverting the user's change is a failure under interactive conditions, and it's the most common one (63.3% of SWE-Touch failures retain conflicting code).
3. Compare against your autonomous baseline. The metric isn't the interactive score in isolation — it's the delta from your clean-workspace runs. A model that loses 10+ points under intervention is a model that fights its users, no matter how good its leaderboard number looks.
4. Re-test after prompt changes. A pre-completion re-inspection instruction ("re-read the files you changed, diff against any external edits, run the affected tests") targets the exact failing stage. Whether it works is measurable with the same two-run protocol.

What This Means

1. Agents don't re-inspect the workspace. The core deficit is state awareness: agents rarely re-read the repository after an external change and rarely validate the affected behavior with targeted tests.
2. For users: if you edit code during an agent run, expect it to fight you — and consider making the change explicit in the conversation and verifying the agent re-checked the file. Announce the edit, but don't assume the announcement is enough.
3. For prompters: instruct the agent to re-inspect task-critical files before finalizing and to validate the affected behavior with targeted tests. A pre-completion re-inspection step — "re-read the files you changed, diff against the user's edits, and run the affected tests" — attacks the exact stage where the benchmark shows failure lives. The failure is at the reconcile-and-validate step, not at the code-writing step.
4. For builders: training objectives that reward detecting workspace changes, reconciling conflicting edits, and re-validating the shared codebase are the roadmap — and the reward signal most models currently lack.

Strong autonomous performance does not predict robust shared-workspace collaboration. If your agents work on code other people touch, SWE-Touch's 7.7-point cliff is the number to keep in mind — and for teams running open-weight models in shared repos, the interactive penalty is the cost of the privacy win.