CodeAgents + Structure: Why Enforcing JSON Format on Code-Generating Agents Actually Works

The Three Generations of Agent Action Execution

Every agent, regardless of framework, follows the same loop: think → act → observe → repeat. The question is how the agent expresses its actions. That representation choice — JSON blob, free-form code, or something in between — has consequences for reliability, flexibility, and parsing stability that cascade through the entire system.

HuggingFace's smolagents library has quietly settled this debate with an elegant solution that combines the best of both approaches. The findings, published in their May 2025 blog post, show that forcing code agents to produce structured JSON output can significantly outperform both traditional JSON-based tool calling and free-form code generation.

This isn't a marginal improvement. On the SmolBench suite (GAIA, MATH, SimpleQA, Frames), the structured variant consistently beats both alternatives across every benchmark, with the gap widening on harder tasks.

Let's walk through why.

Generation 1: The JSON Agent (ToolCallingAgent)

The earliest and still most common approach to agent action execution is JSON-based function calling. The LLM receives a tool schema and emits structured JSON:

{
  "action": "web_search",
  "action_input": {
    "query": "latest HuggingFace model releases 2026"
  }
}

The system parses this JSON, dispatches the tool, captures the result, and feeds it back into the next iteration.

What works: This is the simplest possible action representation. Parsing is deterministic — if the JSON is valid, you know exactly what tool to call and with what arguments. The constrained action space prevents the LLM from doing anything unexpected.

What breaks: Three failure modes dominate:

• Malformed JSON. Small models especially struggle with JSON formatting. A missing comma, an unescaped quote, or a trailing comma can break the entire step. Analysis of 15,724 agent traces by Steve Kinney showed that first-call JSON parsing errors dropped success rates from 51.3% to 42.3%.

• Hallucinated tool names. The LLM might call web_search when the actual tool is search_web. These subtle naming mismatches are hard to catch at parse time and produce silent failures.

• Rigid composition. JSON actions can't compose tools. If you need to call search_web("Paris hotels"), then translate(result, "French")` based on the first result, you need two separate tool-call → observation cycles. The LLM can't express "take the first search result and pass it to the translate function" in a single action.

Generation 2: The Code Agent (CodeAgent)

The CodeAgent approach, grounded in the CodeAct research (Wang et al., ICML 2024), replaces JSON blobs with executable Python code:

search_results = web_search("latest HuggingFace model releases 2026")
first_result = search_results[0]
analysis = summarize(first_result)
print(analysis)

What works: Code is dramatically more expressive than JSON. The agent can compose tools, use control flow (loops, conditionals), and leverage the entire Python standard library. Multiple research papers have shown that code-based actions consistently outperform JSON-based actions because Python was literally designed to express computational actions.

What breaks: Code is harder for LLMs to generate reliably than JSON:

• Syntax errors. An unmatched parenthesis, a misnamed variable, or an incorrect import crashes the execution. The local Python interpreter in smolagents catches these, but every crash costs a step in the agent loop.

• Over-generation. Free-form code encourages the LLM to write more than necessary. Extra imports, helper functions, and comments inflate token usage and introduce more potential failure points.

• Inconsistent structure. A CodeAgent might write a function in one step and a script in the next. There's no enforced schema for what "good code" looks like, which makes post-processing and debugging harder.

• The "structure tax" on small models. Smaller LLMs struggle to simultaneously handle Python syntax, tool API design, and the actual problem-solving logic. The cognitive overhead of free-form code generation can overwhelm models that would otherwise perform reasonably well with simpler markdown-based approaches.

Generation 3: The Structured CodeAgent

The structured variant in smolagents (use_structured_outputs_internally=True) takes the middle path: it generates JSON that contains both thoughts and code as named fields, then executes the code portion.

The LLM is prompted to produce output matching this JSON schema:

{
  "thoughts": "I need to find the distance across the Golden Gate Bridge, then calculate how long a cheetah takes to run it.",
  "code": "import googlemaps\nfrom datetime import datetime\ngmaps = googlemaps.Client(os.getenv('GMAPS_API_KEY'))\nresult = gmaps.distance_matrix(\n    origins='Golden Gate Bridge, San Francisco',\n    destinations='Golden Gate Bridge, San Francisco (other end)',\n    mode='driving',\n    departure_time=datetime.now()\n)\ndistance_meters = result['rows'][0]['elements'][0]['distance']['value']\ncheetah_speed_mps = 29.0\ntime_seconds = distance_meters / cheetah_speed_mps\nfinal_answer(time_seconds)"
}

Why This Works

100% parsing reliability. Because the LLM outputs JSON (not raw Python), the system can use constrained decoding or JSON schema enforcement to guarantee valid structure. The thoughts and code fields are always present and parseable. Steve Kinney's trace analysis — which showed first-call parsing errors dropping success rates by 9 percentage points with JSON agents — doesn't apply here because parse errors go to zero.

Code still executes natively. The extracted code field runs in smolagents' local Python interpreter, just like a regular CodeAgent. You get all the composition and flexibility benefits of code (control flow, variable passing, multi-tool chaining) with none of the parsing overhead.

Separation of concerns. Splitting thoughts from code forces the LLM to articulate its reasoning separately from its actions. This makes the trace more interpretable for debugging and gives the system cleaner context to feed back into the next iteration.

Smaller models benefit disproportionately. The "structure tax" that hurts small models with free-form code actually flips to an advantage with structured output. The explicit JSON scaffolding guides generation, reducing the cognitive load of simultaneously managing syntax and logic. On the SmolBench results, the gain from adding structure is larger for smaller models than for frontier models that already generate clean code.

How to Enable It

The API surface is minimal — one boolean parameter:

from smolagents import CodeAgent, InferenceClientModel, GoogleSearchTool

agent = CodeAgent(
    tools=[GoogleSearchTool(provider="serper")],
    model=InferenceClientModel("Qwen/Qwen3-235B-A22B", provider="nebius"),
    use_structured_outputs_internally=True
)

result = agent.run("Calculate the time for a cheetah to run across the Golden Gate Bridge")

You can also combine it with the planning interval for multi-step reasoning:

agent = CodeAgent(
    tools=[WebSearchTool()],
    model=LiteLLMModel(model_id="openrouter/anthropic/claude-sonnet-4"),
    use_structured_outputs_internally=True,
    planning_interval=1
)

What Goes On Under the Hood

When use_structured_outputs_internally is True, smolagents does three things differently:

1. System prompt injection. The agent's system prompt is augmented with a JSON schema specification telling the LLM to output an object with thoughts (string) and code (string) fields.

2. Schema-based decoding. For models that support it (via InferenceClientModel with structured output), the schema is passed directly to the inference provider for guided/constrained generation. This guarantees structural validity even for models that struggle with JSON formatting.

3. Two-phase execution. The raw LLM output is parsed as JSON. The thoughts field is appended to the agent's memory for context. The code field is passed to smolagents' local Python interpreter for execution.

The smolagents source code handles this in the CodeAgent base class — the structured output path shares the same code execution engine as the free-form path, just with a different prompt template and output parser.

Benchmarks: What the Numbers Say

The SmolBench suite tests agents across four dimensions:

Across all four, the Structured CodeAgent outperforms both the standard CodeAgent and the ToolCallingAgent. The gap is most pronounced on GAIA (multi-step reasoning) and Frames (multi-modal), where the parsing reliability of structured output prevents cascading failures in long chains.

The key insight from the error analysis: structured output doesn't just fix parse errors — it changes the model's behavior. The explicit separation of thoughts from code produces more focused, concise code. Models produce fewer extraneous imports, fewer redundant tool calls, and cleaner variable handling when they're forced to articulate their reasoning first.

Where This Fits in the Agent Landscape

The structured code agent occupies a specific niche in the agent architecture spectrum:

The structured code agent isn't always the right choice. For simple single-tool calls, function calling via the provider API is simpler and faster. For maximum expressiveness on frontier models, free-form code agents still offer the most flexibility.

But for the common case — multi-step agents running on mid-size models where you need both reliability and expressiveness — the structured code agent is the best option available today.

Pitfalls

Structured output requires model support. The use_structured_outputs_internally=True flag works best with models that support guided/constrained JSON generation via the inference API. With models that don't (some local models, some LiteLLM routes), the system falls back to prompt-based format enforcement, which is still better than free-form code but doesn't eliminate parsing errors entirely.

The JSON envelope adds token overhead. Each step now includes a thoughts field plus JSON framing characters. In practice this is negligible (50-100 extra tokens per step), but it adds up over long agent runs.

Structured code agents can't do everything. Actions that require dynamic state mutation (e.g., launching a subprocess, modifying the filesystem) are better served by dedicated tool calls than code generation with side effects. The structured code agent is optimized for the information-processing part of the agent loop, not for system-level operations.

Not all inference providers handle structured output the same way. The HuggingFace Inference API, OpenAI, and Anthropic each have different mechanisms for constrained JSON generation. The abstraction layer in smolagents handles most of this transparently, but the reliability ceiling is set by the provider's implementation, not the library.

The Bottom Line

The structured code agent is a rare case where adding constraints actually improves flexibility. By wrapping Python code execution in a guaranteed-parseable JSON envelope, smolagents delivers the best of both worlds: the composability and expressiveness of code-based actions with the reliability and debuggability of structured output.

If you're building agents with smolagents today, use_structured_outputs_internally=True should be your default. Turn it off only when you have a specific reason — smaller model that doesn't support guided decoding, a frontier model where free-form code produces measurably better results, or a single-tool chain where the overhead isn't worth it.

For everyone else, this is the closest thing we have to a free lunch in agent architecture.