There's a pattern I've been watching all year. The teams that ship real agent systems keep landing on the same conclusion, independently, from different starting points.

Codex and Claude Code found it building coding agents. OpenAI found it with Codex CLI's multi-agent architecture. And now Allen AI's Skylight team has published their postmortem on building Shippy — a maritime domain awareness agent that serves 300+ partners across 70 countries — and it reads like a corroboration of the same hard-won thesis:

The model is not the hard part. The system around it is.

"the real work wasn't the model. It was building a system we could trust to be correct, to stay within its limits, and to hold up across a wide range of tasks." — Allen AI Skylight Team

Let me walk through what Shippy's architecture looks like, what broke, and what the rest of us should steal.

Source: Ai2 Blog - What building Shippy taught us about building agents | Hugging Face Blog

What Shippy Actually Does

Shippy is an AI agent for real-time maritime domain awareness. Maritime analysts use it to query fishing activity, track vessels, understand maritime boundaries, and plan patrols. The stakes are high — a wrong answer could send a patrol vessel miles off course, wasting resources and potentially endangering personnel.

This isn't a chatbot. It's an agent wired into live APIs, running code in sandboxed environments, producing answers that human operators act on. That context is important because it forces a level of reliability engineering most agent demos never have to think about.

The Architecture: Soul, Skills, Config

Shippy's architecture decomposes the agent into three independently versioned components. This is worth paying attention to because it's the same decomposition pattern we're seeing across every serious production agent system.

Soul — the system prompt, explicitly

The soul is the system prompt, but treated as a first-class versioned artifact. It defines Shippy's persona and, more importantly, its behavioral boundaries. The team calls them "guardrails" — explicit rules like "won't make legal determinations" or "won't speculate beyond available data."

Key design choice: these boundaries are in the prompt, not implicit in fine-tuning. That makes them auditable and easy to revise. A domain expert can read the guardrails and say "no, we should also refuse X" without needing ML engineering. A subtle but powerful organizational decision.

Skills — plain markdown files, shared spec

Shippy's skills follow the agent-skills spec — the same spec used by Claude Code and Codex. Plain markdown files with structured frontmatter that encode entire workflows.

A single question from a maritime analyst might hook into multiple skills simultaneously: querying Skylight API data, pulling marine protected area boundaries from ProtectedSeas, and interpreting vessel track patterns. The skills chain in a single dialogue turn, which means the agent isn't doing sequential tool calls — it's composing multiple capabilities in parallel.

Config — everything else is a config change

The harness (OpenClaw), the model (Claude Opus 4.6), the runtime settings. All of it is config. API keys are injected at runtime. Swapping the model means changing a config field, not rebuilding the agent.

This is the same pattern Codex CLI uses with its MODULE.md and CLI.md config separation — version your prompts and tool definitions independently from your infra and model selection.

The Core Reliability Pattern: Deterministic Tools

This is where Shippy's engineering gets genuinely instructive.

The team initially had the agent call the Skylight API directly. The result: "a steady stream of subtle bugs." Malformed pagination dropping results. Geometry encoding errors. Queries that looked correct but returned wrong data.

Their solution was to build a purpose-built Skylight CLI — a deterministic wrapper around the complex API. Now the agent issues a single command (skylight events search with typed filter flags) instead of constructing raw API calls.

"Agents are nondeterministic. You can't control what the model decides to do, but you can make the tools it reaches for predictable."

The layering strategy is clean:

Typed API → Deterministic CLI → Agent Skills (referencing CLI commands) → Agent

Each layer narrows what the next layer can get wrong. Every layer is independently testable.

This is the same principle behind why Claude Code uses explicit curl commands in its ecosystem rather than abstract tool interfaces, and why Codex's MCP architecture validates tool shapes at the boundary. The industry is converging on a pattern: make the agent's tools deterministic so the agent's nondeterminism is tolerable.

Three specific CLI design choices worth noting:

Output always goes to local JSON files — avoids pipe buffer limits and shell quoting issues. The file path is predictable, so downstream skills can read it with jq or a script without guessing.
Extensive --help text and error messages — the CLI documents itself for the agent. When the agent makes a mistake, the error message gives it enough context to self-correct on the next attempt.
Handles all the bookkeeping — auth, pagination, geometry encoding, structured output. The agent never thinks about any of it.

Mothership: Isolation as a Service

Multitenancy in agent systems is an unsolved problem for most teams. Shippy's approach is aggressive: each user gets an ephemeral, isolated Kubernetes deployment per session.

The user's Skylight JWT is injected at provision time, so every API call is automatically scoped to that user's data. Files, conversation history, and executed code exist only within that sandbox. The sandbox has network access restricted to only necessary services. When the session ends, the deployment is destroyed.

Inside the sandbox, the agent has full freedom — write and run code, install dependencies, pull datasets. The isolation, not constraint, is what makes this safe. It's the same philosophy as browser sandboxing or container-based CI runners: give the code freedom, isolate the environment.

Mothership is designed to generalize beyond Shippy. The team is already applying it to EarthRanger (wildlife conservation) and OlmoEarth (Earth observation tools). If you're building an agent platform that needs to serve multiple customers with data isolation, this is the architecture to study.

Harbor: Evaluating the Whole Agent

Shippy's evaluation pipeline is worth a deep read on its own. The team uses the Harbor open evaluation framework with a custom plugin, and the approach addresses a fundamental gap in how we evaluate agents today.

The problem: Static benchmarks test a model on canned questions. They don't capture how an agent behaves once wired into a real workflow — how it selects tools, queries live data, acts on results, and knows where to stop.

Shippy's eval pipeline:

Subject-matter experts write scenarios and weighted rubrics (data accuracy weighted highest for fishing queries, response style weighted lower)
Experts annotate individual responses as correct/incorrect
An LLM judge grades each criterion 0–1 with written reasoning
Weighted aggregate score checked against a fixed pass threshold
Pipeline runs against live, real-time data in an actual sandbox session

"a version of Shippy that regresses on our eval criteria doesn't reach end users."

The suite auto-runs whenever skills, the model, or underlying data change. Regressions block deployment. This is the bar every production agent should clear.

Real Failures the Eval Caught

These are the most valuable part of the post because they're concrete:

Guardrails held — Shippy correctly refused military intelligence requests, maintained data isolation across users, and attributed sources accurately. The soul boundary design worked.
Overstepping into action — Patrol-planning tasks where Shippy started giving tactical recommendations instead of decision support. The prompt boundaries held on data scope but didn't sufficiently constrain the depth of analysis. Fix: tighter phrasing in the soul.
Geometry simplification lost events — Boundary simplification in geometry queries caused missed fishing events at the edges. The CLI was encoding geometries with too much aggressive simplification. Fix: CLI change, not a prompt change.
Hallucinated CLI commands — Shippy invented a non-existent --date-range flag on one occasion. The CLI's error messages gave it enough context to self-correct on the next attempt, but this still indicates that the skill definitions needed more explicit examples.

These failures are exactly the kind of thing a static benchmark would never catch. They only surface when an agent is operating against live data in a real execution environment.

What This Means for Anyone Building Agents

I've been tracking production agent architectures across the industry for months, and Shippy's post reinforces a pattern I'm increasingly confident about:

The agent engineering playbook that works:

Version your prompts and tool specs independently. Soul, Skills, Config as separate versioned artifacts is the right decomposition. It lets you change each without retraining or rebuilding.
Build deterministic tool wrappers, don't let agents touch raw APIs. The Skylight CLI pattern — a typed, self-documenting wrapper that handles all bookkeeping — eliminates an entire class of failure modes. If your agent is constructing API calls directly, you're accumulating technical debt.
Evaluate the whole system against live data. A benchmark score on a static test set tells you nothing about whether your agent will hallucinate a CLI flag or overstep into tactical recommendations. You need scenarios written by domain experts, weighted rubrics, and automated eval pipelines running against production data.
Isolate by default, not as an afterthought. Per-user ephemeral sandboxes are the right bar. If you can't do Kubernetes-level isolation, at least run each agent session in a container with scoped credentials. Shared-state agents are a security incident waiting to happen.
Use the shared spec for skills. The fact that Allen AI, Claude Code, and Codex are all converging on the same agent-skills markdown spec is not a coincidence. It's a standard forming in real time. Standardizing on it now means your skills are portable across agent harnesses.

The Bottom Line

Shippy is not a research project. It's a production system serving real users in high-stakes environments, and its architecture reflects that reality. The team's willingness to publish their failures alongside their wins makes this post unusually valuable.

The article also reveals future directions worth watching: investigative multi-turn interactions (following a vessel across an ocean), satellite tasking to cover data gaps, and OLMoEther — an open-source baseline version of Shippy that the team plans to release. If they follow through on that, it could be the closest thing we have to a reference implementation for production agent architecture.

The model is not the hard part. The system around it is. That's the lesson from Shippy, and it's the same lesson every production agent team seems to be learning independently. The teams that internalize it earliest will be the ones shipping agents that actually work.

What Building Shippy Taught Allen AI About Agent Engineering — Lessons from Production