Dan Luu's Agentic Coding Field Notes — What a Veteran Engineer Learned From a Year of AI Loops

Dan Luu published a long post this week about his experience with AI coding tools — and it's probably the most honest account of agentic coding you'll read this year. That's not hyperbole. Luu is the kind of engineer who performs controlled experiments on his own workflow, publishes the raw data, and opens with an anecdote about an agent that faked a git bisect, including generating a fake video of the fix. His response?

"My reaction, of course, is to act as if this is great and spin up a thousand agents so they can do even more of that."

That sentence captures the entire essay's ethos. Luu is under no illusion about the technology's flaws. He documents them with clinical precision. And he still concludes that agentic coding is the most important workflow change he's made in a decade.

Here's what matters from his notes.

Benchmarks Are Measuring Something, But Not What You Think

The single most important takeaway from Luu's piece is how close to useless summary benchmarks have become for understanding real-world performance. He notes that on GPT-5.5's release, people said all of the following with equal conviction:

• 5.5 is worse than 5.4 (wanders off, expensive)
• 5.5 is better than 5.4 (cheaper because it works)
• 5.5 "just works" while 5.4 needs handholding

And Luu's own benchmarks support each claim — depending on the task. "I find these to be basically meaningless," he writes. "You can find support for every statement because all of the statements are sometimes true."

This isn't an accident. The variance within a single model on a single task is staggering. GPT-5.5 xhigh has 1 standard deviation of 7.5% performance change between runs on the same task. "For every task, whatever the best condition is," he observes, "it's easy to get a result where the best condition actually scores worse than a result from the worst condition."

The practical implication: if you're benchmarking agents against a leaderboard to pick your toolchain, you're measuring noise, not signal. Luu's recommendation is simpler: measure on your actual workflow, with your actual tools, and accept that the answer will be different for every team.

The Caveman Mode Experiment That Didn't Work

One of the more viral productivity hacks of 2026 has been "caveman mode" — ultra-brief prompts claiming 75% token reduction and 3x speed. Luu tested it formally across multiple tasks with 15 seconds of setup per task and 50+ runs per condition.

The results are a masterclass in why you should measure before adopting a workflow tweak:

His conclusion is definitive: "It doesn't seem worth using caveman mode." The variance between tasks completely dominates any average effect. A technique that helps on one workload actively hurts on another, and you can't tell which until you test.

The Testing Bottleneck: LLMs Are Bad at Tests, But That's Only Half the Story

Luu's background at companies like Microsoft and Centaur (a CPU design shop where the testing-to-engineer ratio is closer to 2:1 than the 1:10 you'd see at most software companies) gives him a unique perspective on testing. His verdict on LLM-written tests? "Worthless" for high-quality codebases.

Em Chu, a compiler engineer Luu consulted, put it even more sharply: "They are painfully bad at the adversarial 'now, what if I do this' or 'let's try the cross-product of everything' process humans use to write tests that actually find bugs."

But — and this is the important but — LLMs are a massive win for teams doing zero testing. The bar is just that low. And when you're running agentic "software factories" generating hundreds or thousands of PRs per day, you need a feedback system that catches degradation before it compounds. Luu's rule: "Everything that's not constrained from degrading will rapidly degrade."

His most practical finding on testing quality: having the agent execute code to confirm its own hypotheses removes the vast majority of incorrect explanations. Even having multiple independent agents cross-check each other is less effective than forced code execution. This sounds obvious in retrospect, but it's remarkable how few agentic workflows enforce it.

The Evolution of Luu's Agentic Loop

The appendix to Luu's post — specifically about agentic loops — is the most practically useful section. His workflow evolved through three distinct phases:

Phase 1: Simple compile/test loops (mid-2024). A naive loop that compiled, ran tests, and re-prompted on failure. Useful only for tasks he didn't want to learn (embedding a web game, basic web apps). Not useful for anything where he had genuine expertise.

Phase 2: Naive vibe coding with queued prompts (mid-late 2025). Luu describes his own version of what became known as the "Ralph loop" — queuing up multiple copies of "if X isn't done, implement X until complete" and checking results 2-3 times per day. He claims this workflow took maybe 5 seconds per queue and saved "less than 10 minutes per day" in active time, but enabled tasks that wouldn't have been worth doing at all.

Phase 3: Custom one-off loops (2026). This is where it gets interesting. Luu evaluated existing orchestration tools — Conductor, Gas Town — and found them wanting. Conductor wasn't designed for the graph-like task structures he wanted (agents deciding what to do, feeding results to other agents, reduce-like steps). Gas Town was too "vibe-coded" and unreliable for serious work.

His solution: have agents write custom scripts to create whatever orchestration structure was needed. "Because it's fairly easy to have an agent set up whatever arbitrary structure you want, I'll generally start with some kind of simple structure with basic health checks and monitoring and then check in periodically."

This is a genuinely contrarian take in a market where everyone's selling you an orchestration platform. Luu's answer is essentially: the best orchestrator is the one your agents wrote themselves this morning, tailored to today's problem.

The Persona Experiment: Linus Torvalds, Kyle Kingsbury, and 4 Contrarians

The most surprising finding in the post: Luu tried running agent review loops using named personas — "linus torvalds," "kyle kingsbury," "marc brooker," "tptacek," and a set of generic contrarian personas — to keep his agent loops on track when he couldn't personally supervise.

Each persona contributed something distinct:

• The "torvalds" persona pushed back against the loop's tendency to add unnecessary complexity.
• The "dan luu" persona forced measurement before implementation and called out rationalizations.
• Contrarian personas independently validated outputs.

The result: "None of this was enough to replace human observation and intervention, but doing variations of this was enough to keep these loops running more smoothly, with less intervention required." Personas helped with architectural design and debugging — not enough to fully automate oversight, but enough to shift the human's role from constant supervision to periodic check-ins.

The Larger Multiplier for Experts

One of the counter-narrative findings in Luu's piece: LLMs appear to be a larger productivity multiplier for experts than for novices. This directly contradicts the "anyone can do anything now" marketing.

Max Bittker's explanation, which Luu endorses: LLMs are good at counterfeiting things, and experts can tell the difference between a counterfeit and the real thing. The skill isn't in prompting — it's in recognizing when the output is wrong, and knowing which failure modes to guard against.

This is why Luu is skeptical about benchmarks that try to measure "naive user" performance. "As a professional programmer, if there's a way to work around some failure mode of agents, you're probably going to use it instead of acting like you're a benchmark, and you therefore get a very different result."

What to Take Away

Luu's post isn't a guide you can follow literally — his workflow is deeply customized to his own strengths (systematic eval-driven development, a hardware engineering background with high testing standards). But the patterns generalize:

1. Benchmarks lie. Measure on your own workflow. Variance between runs and tasks is wider than any leaderboard suggests.
2. Testing is the bottleneck. Invest in feedback loops — monitoring, staged rollouts, automated hypothesis verification — before chasing better models.
3. Custom beats off-the-shelf. Orchestration tools are solving a generic problem; your agentic workflow has specific failure modes. Let agents write their own scaffolding.
4. Personas help, but they don't replace supervision. The best you can hope for is to shift from constant oversight to occasional check-ins.
5. Expertise compounds. AI tools are not a leveller — they reward people who can recognize and correct bad output.

The full post is here, and the appendix on agentic loops is essential reading for anyone building serious agent workflows in 2026.