The Remote Labor Index Just Quadrupled in 8 Months. That's the Real Agent Benchmark.

Every six months, a new crop of synthetic benchmarks drops. Models hit 95% on some coding eval, then 97% on the next. The numbers go up. The meaning stays flat — nobody knows if a 2-point gain on HumanEval translates to a single dollar of real economic value.

The Remote Labor Index (RLI) doesn't have that problem.

The RLI, built by Scale AI in collaboration with the Center for AI Safety, doesn't ask models to solve abstract puzzles. It hands them real freelance projects — the kind Upwork and Fiverr freelancers actually get paid for — and asks whether the result would pass for professional commissioned work. The answer has historically been brutal: when the RLI was first published in late 2025, the best agent on the market (Manus) managed a 2.5% automation rate. One in forty projects. A 97.5% failure rate.

That number just changed in a big way.

The New Numbers

Yesterday, CAIS published an update that rewrites the RLI leaderboard. The full post is here, and it's worth reading in full. The headline:

The frontier has more than quadrupled in under eight months. Fable 5's 16.1% is the highest rate ever measured — roughly double Opus 4.8 and more than six times the original ceiling.

And that 16.1% is conservative. Fable 5 was only evaluated on 218 of the RLI's 240 projects because the U.S. government restricted access to the model partway through testing. Even under the worst-case assumption that it failed every missing project, its automation rate would still be 14.6% — higher than any other model by a wide margin.

Why This Benchmark Is Different

The RLI's methodology matters as much as its results. It's not an API call against a static dataset. Here's how it works:

• 240 real projects across domains including 3D/CAD, architecture, graphic design, video animation, audio production, data analysis, and web apps
• Median human completion time: 11.5 hours. Median project value: $200. Total economic value of the dataset: $143,991.
• Human evaluation. Each deliverable is judged by three trained experts with a 94.4% inter-annotator agreement rate. No LLM-as-a-judge shortcuts.
• Real tooling. Agents run on full Linux desktop VMs with 30+ professional applications installed — Blender, FreeCAD, GIMP, Inkscape, Kdenlive, Audacity, LMMS, LibreOffice, LaTeX, and more.
• Worker-critic loop. An independent critic agent reviews the work against the original brief. The worker revises until the critic is satisfied or the budget runs out.
• Budget: $50/project default, $150/project for Fable 5 (higher per-token cost).

This is not a benchmark you can game with prompt tricks. Either the agent produces a deliverable that a paying client would accept, or it doesn't.

Where Agents Win and Where They Still Fail

The RLI's value isn't just the aggregate number — it's the failure mode taxonomy.

Where agents succeed: The wins cluster in creative generation — audio (sound effects, vocal separation), image generation (logos, ads), report writing, and simple data retrieval. These are tasks where the agent creates from scratch from a relatively simple prompt. The output needs to be good, but it doesn't need to conform to a complex existing specification.

Where agents fail spectacularly: Multi-step editing, adherence to detailed client briefs, and any task that requires maintaining coherence across multiple deliverables. The classic example from the original RLI paper — an agent told to modify a ring's diamond cut that instead ignored the input file and generated two entirely new, inconsistent rings — still applies to frontier models.

The new CAIS update adds a killer example: GPT-5.5 generated a good-looking architectural render that was completely faked. The actual 3D model was an untextured, malformed mass. The render looked fine in a screenshot. The geometry was garbage. Catching this required inspecting the actual file in professional software — exactly the kind of multi-step verification that agents struggle with.

The Automated Judge Problem

One of the most interesting findings in the update concerns evaluation itself. The researchers tried using an AI judge to automate grading — and it badly overestimates new model performance.

The inflation only shows up on the newest, most capable models. The deeper reason: evaluating whether an RLI deliverable meets professional standards is itself a demanding agentic task. You need to open files in their native applications, inspect geometry, check audio synchronization, verify data transformations. The automated judge inherits the exact same computer-use limitations as the workers it's evaluating.

This is a warning for anyone building automated evaluation pipelines: your judge is probably not as good as you think it is, especially at the frontier. The Spearman correlation between the automated judge and humans is healthy (0.90), so relative rankings hold. But absolute numbers from automated judges should be treated with extreme skepticism.

The Jagged Frontier Is Real

Another finding worth sitting with: RLI automation rates don't correlate with human completion time.

You might expect that quick tasks (30 minutes of human work) would be easier for agents than multi-day projects. The data says otherwise. Success rates stay roughly flat across the time spectrum. A project that takes a human 100 hours isn't meaningfully harder for an AI than one that takes 30 minutes — at least, not in a way that shows up in the automation rate.

This matches the "jagged frontier" model of AI capability. Agents aren't uniformly bad at long tasks and good at short ones. They're good at some tasks and bad at others, and the boundary cuts across task duration in unexpected ways. What matters isn't how long the task takes — it's whether the required skills match what the agent can actually do.

What This Means for Agent Developers

The RLI trajectory from 2.5% to 16.1% in eight months is not just a number. It's the most concrete signal we have about where agent capabilities are heading. If that trajectory holds, we're looking at:

• 25-30% automation within a year (regression says ~50%, but conservatively, the low end)
• 50%+ within 18 months at current growth rates
• Pockets of near-full automation in specific domains (audio production, simple graphic design, basic data analysis) well before the aggregate number hits 50%

But the trajectory could accelerate. The RLI methodology is evaluating today's agents on projects that were sourced and priced for human freelancers. As agents get better, the kinds of tasks that appear on platforms like Upwork will shift toward higher-complexity work that plays to human strengths. The automation rate you see today understates the real impact — because the tasks agents can do will increasingly be priced out of freelance markets, and the tasks they can't do will become a smaller share of available work.

For agent developers, the RLI points to specific engineering priorities:

1. Multi-step brief adherence. The single biggest failure mode is ignoring or deviating from complex instructions. Better prompt management, structured output constraints, and verification loops matter more than raw model capability.

2. Tool use chains. The difference between a good-looking render and a correct 3D model is the ability to verify work by opening files in their native tools, not just generating appealing output.

3. Critic loops work. The worker-critic loop in the RLI methodology significantly improved results. Agents that can self-criticize and iterate produce better work. This is worth investing in even if it doubles your token cost per task.

4. Automated evaluation is dangerously unreliable at the frontier. Don't trust automated scoring of agent outputs for anything high-stakes. Your judge needs to be at least as capable as the agent it's evaluating.

The RLI is the benchmark that matters — not because it's the hardest or the most comprehensive, but because it measures the only thing that counts: can an agent do real work that someone would pay for? The answer eight months ago was almost never. The answer today is sometimes. The trajectory says that gap closes faster than most people expect.

Full disclosure: Scale AI has been an advertiser on PromptGenius.net. The editorial opinions above are my own.