"Better Models, Worse Tools" — Armin Ronacher Exposes a Tool-Calling Regression in Modern Claude

Here's a sentence that should make every developer building agent pipelines pay attention.

Armin Ronacher — creator of Flask, Jinja2, Click, and Rye, and one of the most respected tool-builders in open source — published a post today documenting a tool-calling regression in Claude that he found genuinely bizarre to debug.

The headline: Claude Opus 4.8 and Sonnet 5 are measurably worse at correctly calling a specific tool schema than their older sibling, Opus 4.5. Smarter models, worse tool use. And Ronacher's hypothesis for why this is happening should unsettle anyone building agentic tools outside of Anthropic's walled garden.

The Bug That Started It

Ronacher was debugging an issue in Pi, an increasingly popular open-source coding agent. Pi has an edit tool that accepts a nested edits[] array — each edit is an object with oldText and newText fields. Standard JSON schema. Dead simple.

Or it should be. Here's what he found:

• Opus 4.5 calls the tool correctly, every time.
• Opus 4.8 and Sonnet 5 call it correctly in terms of content — the actual oldText and newText values are byte-perfect — but they invent extra, undefined keys inside the nested edits[] array.

The zoo of hallucinated keys he catalogued: type, id, kind, unique, requireUnique, matchCase, in_file, forceMatchCount, children, notes, cost, oldText2, newText2, oldText_2, newText_2, and even the strikingly meta event.0.additionalProperties.

Pi rejects these calls. The model retries. Sometimes it works on the second try. Sometimes it doesn't. In a long agentic session with a 50+ message history, this failure mode compounds into a grinding, expensive, maddening experience.

And the weirdest part? He couldn't reproduce it in a single-turn prompt. It only emerges after extended agentic history — reading files, composing edits, building context. The regression is hiding in the tail, not the head.

The "Slop Harness" Hypothesis

Ronacher's best explanation for the regression is worth quoting at length:

"If reinforcement learning happens in a harness like that... then slightly malformed tool calls can still complete the task and receive reward... There is little gradient against inventing an alias, adding a stray field or using a nearby parameter name."

The argument is this: Claude Code is Anthropic's primary RL training harness. Claude Code internally:

• Has retry paths for malformed tool calls
• Silently filters out unknown keys
• Supports parameter aliases (path for file_path, old_str for old_string)
• Performs Unicode and escape-sequence repairs

Claude Code's tool schema is flat — file_path, old_string, new_string, replace_all. When a model trained on this flat schema encounters Pi's nested edits[] array, it tries to map the familiar flat structure onto it. Since there's no slot in Pi's schema for these extra fields, the model samples a random plausible key name each time it generates one.

The result: a zoo of hallucinated keys that changes with every retry.

Ronacher calls this the "Moore's Law for Adapters" — the more post-training happens inside one dominant harness, the more every other harness inherits its quirks. If you want the best model performance, your tools must increasingly mimic Claude Code's internal environment.

This is not a small claim. If true, it means the open-source tool ecosystem is fighting an uphill battle against an invisible RL prior embedded in the models themselves.

The Evidence That Makes It Convincing

Several details in Ronacher's post push this from "anecdotal complaint" to "actionable signal":

Strict mode fixes it. Enabling Anthropic's strict parameter — which constrains the sampler to the exact JSON schema — eliminates the problem entirely. The 4.8 and Sonnet 5 models call the tool correctly when strict mode is on. The catch: strict mode has complexity limits that prevent Claude Code itself from using it, so the models don't get RL training on strict-mode interactions either.

Removing thinking blocks reduces failures. Ronacher found that stripping thinking blocks from the conversation history roughly halved the failure rate. The thinking process appears to nudge the model toward more creative (and more incorrect) schema generation. This is a concrete, testable claim — anyone with access to the API can verify it.

Context dependency is a fingerprint. The fact that this only manifests in long agentic sessions (not single-turn prompts) is consistent with a schema-drifting behavior that accumulates over time. The model starts accurate and gradually introduces more spurious keys as the conversation lengthens.

OpenAI's Codex models don't show this. Ronacher noted that the Codex lineup uses a different format ("Harmony") with explicit grammar constraints (<|constrain|>json). These models didn't have the same regression. This points to a training-side cause, not a fundamental limitation of LLM tool calling.

What This Means for Developers

If you're building agent tools or agent pipelines, there are three takeaways you can act on today:

1. Test your tool schemas at length, not just at start. Ronacher's bug would pass any single-turn evaluation. It only emerges after 50+ messages. If you're not running long-context regression tests on your tool schemas, you're flying blind.

2. Understand your model's training bias. Every model has a "preferred" tool shape based on what it saw during RL. Flat schemas with optional fields and forgiving parameter names (Claude Code-style) will get better results than deeply nested arrays, regardless of what your schema technically supports. You can fight this with strict mode, but you need to check whether the model you're using supports it within your complexity limits.

3. Grammar-constrained sampling is no longer optional. Ronacher says this bug shifted his priors towards strictness. If the model cannot be trusted to faithfully emit the schema during complex tasks, the harness must enforce it. LARK grammars, strict mode, JSON-mode constraints — pick your poison, but don't skip it. The "the model will figure it out" assumption is costing you reliability.

The Bigger Pattern

This single bug report is a data point in a much larger story about 2026. The narrative that "models are getting worse" has been building all year — AMD's senior AI director published a detailed regression analysis in April, Claude Code users have been filing escalating GitHub issues about tool-use degradation, and Ronacher himself wrote two weeks ago about seeing worse code from modern harnesses than what we were producing last autumn.

The usual explanation is that Anthropic is optimizing for different axes — safety, instruction-following, resistance to prompt injection — and tool-fidelity loses out. Ronacher's post adds a more specific mechanism: it's not just trade-offs, it's Claude Code's own dominance shaping the model's behavior in a way that penalizes every other tool.

This is worth watching closely. If the RL training harness uniformly shapes model behavior toward a single agent's tool conventions, then every developer running a non-Claude-Code agent is implicitly paying a tool-calling tax. And that tax grows every time Anthropic ships a smarter model.

The models get better. The tools get worse. That's not a bug report anymore. It's an industry condition.

---

Sources: lucumr.pocoo.org, Pi issue #6278, Anthropic strict mode docs, Claude Code regression issue #70327, VentureBeat on Claude regression narrative