July 2, 2026 — There's a growing gap between what AI agents demo at conferences and what they actually do in production. A model calls one tool, answers one question, and we call it an "agent." But real enterprise workflows chain five decisions across three systems while following compliance rules written in natural language — and benchmarks have mostly sidestepped that complexity.

IBM Research just released VAKRA — and it's designed to catch that gap.

VAKRA (eValuating API and Knowledge Retrieval Agents) is not another leaderboard-topping multiple-choice test. It's an executable benchmark where agents actually call APIs, retrieve documents, follow policy rules, and get scored on whether their entire trajectory — not just their final answer — is correct. The results are the most honest picture of agent capability I've seen this year, and they're sobering.

What Makes VAKRA Different

Most agent benchmarks test isolated skills. Can the model pick the right tool? Can it answer a question from a document? VAKRA says: fine, but can it do all of those things in sequence, where each step depends on the previous one, while also respecting policy constraints it's never seen before?

The benchmark provides an executable environment with 8,000+ locally hosted APIs backed by real databases across 62 domains, plus domain-aligned document collections for RAG. Tasks require 3–7 step reasoning chains that combine structured API calls with unstructured retrieval — and in the hardest setting, agents must obey tool-use policies expressed in English.

This matters because the gap between "can call a tool" and "can complete a multi-step workflow" is where agents actually fail in production.

The Four Capabilities

VAKRA organizes evaluation into four progressive layers of complexity:

Capability 1: API Chaining (2,077 instances, 54 domains) — Agents interact with BI-style data APIs using two toolkits. SLOT-BIRD has 7 generic tools with many parameters; SEL-BIRD has more specialized tools with fewer parameters. The critical mechanic: every task starts with a get_data() call that initializes the data source and exposes the correct tool set. This mirrors how real APIs require setup before use — a step benchmarks usually skip.

Capability 2: Tool Selection (1,597 instances, 17 domains) — Agents face REST-style API collections with 6–328 tools per domain (average 116). This exceeds the OpenAI API's 128-tool limit, forcing agents to implement a shortlisting mechanism just to fit within API constraints. It's a practical constraint that almost never appears in research benchmarks.

Capability 3: Multi-Hop Reasoning (869 instances, 38 domains) — Requires 1–5 logical hops. The data dependency means earlier tool outputs must be transformed into inputs for later calls. Performance degrades sharply with hop depth across all models — 2-hop questions are harder than 1-hop, and 3+ hop questions are dramatically worse.

Capability 4: Multi-Hop, Multi-Source & Policy Adherence (644 instances, 41 domains) — The hardest setting. Agents must interleave API calls with document retrieval, handle multi-turn dialogue context, and obey plain-text policies like: "If a user's query pertains to Technology & Software, make sure you try answering them by only using document retrievers. Do not use other types of tools." The baseline implementation handles this with a single prompt line: "You are a helpful assistant with access to tools.\n Tool Usage Constraint: {instructions}." It doesn't work well. The key finding: models can reason over tools and sources, but they struggle to incorporate external constraints into that reasoning.

The Failure Modes That Actually Matter

VAKRA's error analysis is where the benchmark earns its keep. Instead of just reporting accuracy, it categorizes each failure by the first point of breakdown — no double-counting:

Tool Selection (wrong tool chosen)
Argument Omission (missing parameters)
Argument Values (incorrect parameter values)
Final Response (ungrounded or inaccurate answer)

The patterns reveal something counterintuitive:

For SLOT-BIRD (few tools, many parameters): Most failures are argument-related. Models can pick which tool to call, but they can't fill in all the parameters correctly. This is especially problematic for tools with 5+ parameters where some are optional and the model needs to know which ones matter.

For SEL-BIRD (many tools, few parameters): Failures split between tool selection and arguments. With more tools to choose from, the model picks the wrong one more often, but the per-tool arguments are simpler so those errors drop.

For multi-hop settings: The dominant failure shifts to tool selection in early hops. If the model picks the wrong tool for step 1, steps 2-5 can't recover. The error compounds, and the waterfall evaluation catches it at the source.

This is crucial for builders. If you're building an agent that uses 3+ chained tool calls, your reliability bottleneck isn't the model's ability to call each tool — it's the model's ability to pick the right tool for the first call in the chain. Every error in the first hop eliminates any chance of a correct final answer.

What Was Tested

The benchmark supports OpenAI, Anthropic, WatsonX, LiteLLM, and local Ollama models. The Hugging Face blog analysis uses GPT-OSS-120B as a reference model, which performed best on the API chaining capability — but none of the models tested achieved strong end-to-end scores, particularly on Capabilities 3 and 4.

The leaderboard is live and open for submissions. At the time of writing, it's still collecting entries — IBM is inviting researchers and practitioners to submit results using the open-source evaluation harness.

Why This Matters for Builders

I've been watching agent benchmarks proliferate for two years, and most of them fall into one of two camps: toy problems that don't stress real tool use, or static multiple-choice datasets that don't execute anything. VAKRA is the first benchmark I've seen that seriously attempts to bridge that gap.

Three takeaways if you're building agent systems right now:

1. Your agent's first tool call is its riskiest. The waterfall evaluation in VAKRA shows that errors in the first step of a multi-step chain propagate destructively. If you're not instrumenting your agent to detect and recover from early missteps, you're flying blind.

2. Argument quality matters more than tool choice — in the right regime. When your tool set is small, the bottleneck shifts from "which tool" to "did I fill in the parameters correctly." This is a prompt engineering problem, and it's one most builders haven't optimized for.

3. Policy adherence is unsolved. The finding that models struggle to incorporate natural-language constraints into their tool-use reasoning should worry anyone deploying agents in regulated environments. If a policy says "use document retrievers, not APIs, for technology questions," and the model ignores it on 3+ hop queries, you have a compliance gap that no amount of prompt engineering has closed yet.

Run It Yourself

VAKRA is fully open source under CC BY-NC-SA 4.0. The setup is straightforward if you have Docker and 35GB of free disk:

git clone https://github.com/IBM/VAKRA
cd VAKRA
make download
python benchmark_runner.py \
  --capability_id 1 \
  --domain card_games \
  --max-samples-per-domain 5 \
  --provider ollama \
  --model llama3.1:8b

You can also run against OpenAI, Anthropic, or WatsonX by swapping the --provider and --model flags, and the tools_explorer command lets you inspect schemas interactively before writing your agent. The GitHub repo includes a full setup guide with Docker compose configuration.

The Bottom Line

VAKRA doesn't give you a clean leaderboard with models ranked by percentage points. What it gives you is a diagnosis of where your agent breaks — and for most agent systems being built today, that diagnosis is "everywhere you haven't looked."

The benchmark's central finding — that models demonstrate surface-level tool competence but fail under compositional reasoning with constraints — is the most honest assessment of the current state of AI agents I've read this year. If your agent can hold a conversation but can't reliably call three chained APIs while following a policy document, VAKRA will tell you exactly where it falls apart.

That's the kind of benchmark the agent ecosystem actually needs right now.

Links: Blog Post (HF) · Dataset · Leaderboard · GitHub · IBM Announcement

VAKRA Is IBM's Reality Check for AI Agents – and It's Brutal