The Single Most Misleading Number in AI Right Now

Here's a sentence you should read twice:

"An AI agent's performance is a curve that rises with test-time compute. Cut the budget while the curve is still climbing, and the measured score tells you the minimum, not the maximum."

That's not a fringe take. That's the UK AI Security Institute (AISI) — the government body chartered with understanding frontier AI risk — and they just published a study that lands like a grenade in the middle of current evaluation practice.

The headline: fixed-budget evaluations systematically underestimate frontier agentic capability, especially for newer models. The mechanism is mundane — most benchmarks cap how much compute an agent can spend on a task — but the implications are anything but. If we're calibrating safety frameworks to scores that are systematically too low, we're not just wrong. We're wrong in the direction that gets people surprised.

What AISI Actually Did

The Science of Evaluation team ran budget sweeps across seven agentic benchmarks spanning cybersecurity, software engineering, mathematics, academic tasks, and healthcare. Instead of running every model at a single fixed budget (the standard approach), they varied the compute budget from low to very high and tracked how performance changed.

The result is not a single score per model. It's a curve.

And those curves are still climbing at the budget points where most evaluations stop.

Some of the numbers:

Cybersecurity: ~8% of AISI's cyber tasks were only solved when the budget exceeded 10 million tokens. Some required over 50 million. The latest frontier models are reaching even higher scores at 100M+ tokens.
Software Engineering (TerminalBench 2.0, SWE-Bench Pro): Success rates jumped ~25% when the token budget went from 1 million to 10 million. On TerminalBench specifically, performance kept improving even at 10× the typical public evaluation budget.
Mathematics (Humanity's Last Exam): Gains of ~22% when moving from 1M to 10M tokens.
Healthcare (HealthBench): The outlier — all models plateaued within the standard budget.

That last point matters. It tells us where the problem is concentrated and where it isn't.

The Power Law Nobody's Talking About

AISI found something that should change how we think about agent evaluation entirely. Across 211 software engineering tasks and 78 cyber tasks, the relationship between human task time and token consumption follows a power law:

1 minute human task → Thousands of tokens
1 hour human task → Millions of tokens
1 week human task → Billions of tokens

Think about what this means. When you run a benchmark with a fixed token budget, you're not measuring capability. You're measuring which tasks fit within the budget. The longest, hardest tasks get cut off first. A failure doesn't mean the agent lacked the skill — it means the evaluation ran out of runway.

The example AISI gives is a cyber task called "The Last Ones" — something a skilled human would take about 20 hours to complete. No tested model could solve it with fewer than 30 million tokens. If your evaluation budget is 2 million tokens, this task registers as "unsolved" and the model gets no credit. But the model didn't fail because it couldn't do the task. It failed because you didn't let it try long enough.

This is not a small correction. It's a systematic bias that gets worse as tasks get harder and as models get better at exploiting long trajectories.

The Generational Gap Is Wider Than We Thought

Here's where it gets interesting. Newer models don't just perform better at the same budget. The shape of the curve changes. They get disproportionately more from extra compute than their predecessors.

AISI measured frontier progress at different budgets and found the trend itself is budget-dependent:

At 2.5 million tokens: The 80th-percentile time horizon doubled roughly every 4.7 months.
At 50 million tokens: The fitted frontier trend was ~60% steeper. Doubling every 40–50 days instead of 67–91 days.

One recent frontier model illustrates the gap: at a 2.5M budget, its horizon was about 40 minutes of human-equivalent work. At a 50M budget, that stretched to 4 hours. Across the entire frontier, the agentic cyber horizon rose from ~2 hours (2.5M budget) to ~14 hours (50M budget).

And on Epoch's MirrorCode benchmark, one model used up to 1 billion tokens to solve tasks that would take a human weeks.

Here's the uncomfortable implication: our measurement of how fast this field is moving depends on how much compute we're willing to spend on the measurement. The trend we think we're observing may be partly an artifact of our evaluation budget.

AISI is honest about this: "The estimated doubling rate is partly a product of the evaluation budget you pick, not a fixed property of frontier progress."

Where It Helps, Where It Doesn't

The feedback loop insight is the most useful takeaway for anyone building or evaluating agents. Extra compute helps most where agents can verify their own work — running code, testing exploits, checking mathematical proofs. These are tasks with clear, immediate feedback loops where more attempts or deeper reasoning directly improve outcomes.

It barely helps where feedback is missing or delayed. Medical diagnosis (HealthBench) plateaued quickly because the agent can't "try again" against a patient's chart in a useful way.

This means the gap between measured and actual capability isn't uniform. It's concentrated in exactly the domains that matter most for autonomous coding and cybersecurity agents — which is to say, exactly the domains where people are deploying agents into production today.

What This Means for Safety and Policy

This is where the study goes from "interesting methodology paper" to "genuinely important document."

If a model scores 5% at 2 million tokens but 30% at 50 million tokens, the risk profile changes entirely. A model that looks safe under constrained evaluation might cross a capability threshold relevant to safety assessments when given more runway. And because token costs are falling, what's expensive today will be cheap tomorrow. The capabilities that looked unaffordable at evaluation time become accessible in deployment.

AISI's warning is direct: "If we keep treating capability as a fixed score rather than a curve over compute, we will keep being surprised by what these systems can do when more is spent on them."

This connects to the broader conversation about agent evaluation. The SWE-bench ecosystem has been debating scoring methodology for months. METR has been pushing for better long-horizon evaluations. What AISI adds is the quantitative framework — here's how much you're missing, here's the shape of the curve, and here's why it matters.

AISI's Response: Multi-Budget Evaluations

To their credit, AISI isn't just publishing findings. They're changing how they evaluate:

Multi-budget evaluations: Every model gets tested at several budget levels, including very large budgets for the hardest tasks.
"Minimum informative budgets": They define budgets by checking where the curve plateaus. If it's still climbing, the score is not the ceiling.
Transparent reporting: Evaluation reports must now disclose token counts, turn limits, and cost caps — not just scores.
Forecasting methods: They're working on predicting high-budget performance from cheaper test runs, so the field doesn't have to spend $60 per task on every evaluation.

These practices should become the standard. If you're releasing benchmark numbers without reporting the compute budget, you're not helping. If you're citing a single score as if it's a model's capability ceiling, you're misleading people.

The Bottom Line

The AISI study lands at a moment when the entire field is trying to figure out how to evaluate agents that can run for hours, write thousands of lines of code, and chain together dozens of tool calls. The answer it gives is uncomfortable: we don't have good methods yet, and our current methods have a systematic bias toward underestimation that gets worse as models improve.

This is not a reason to panic. It's a reason to demand better methodology — from benchmark authors, from model providers, and from the safety institutes that are supposed to be keeping track of what these systems can do.

Because right now, the most dangerous number in AI evaluation is the one we're not reporting: the budget cap.

The AISI Study That Proves We're Measuring AI Agents Wrong