Harness-IF: Are Coding Agents Following Rules, or Just Doing What They'd Do Anyway?

If a coding agent produces output that matches a rule you gave it, did it follow the rule — or would it have done the same thing anyway? Existing instruction-following benchmarks can't tell the difference, because they stack rules into the user turn and score final task success. Harness-IF (arXiv 2608.11727) attacks that blind spot directly, and the results should change how you write agent rules.

The Methodology

Harness-IF scores operational rules one at a time from execution evidence: 60 realistic multi-turn coding items drawn from a 642-rule library, with 256 rules receiving verdicts, placed on the five configurable surfaces a deployed agent actually reads — system prompt, project files, user instructions, tool descriptions, and skill descriptions.

To separate compliance from coincidence, the authors introduce Against-Prior Accuracy (AP-Acc): it scores only rules labeled as opposing the agent's unprompted defaults, observed by re-running the task with the rule withheld across nine probe builds. If the agent obeys a rule that opposes what it would naturally do, that's real compliance. If it "obeys" a rule that matches its default behavior, that's coincidence — and it shouldn't count as instruction-following at all.

How Compliance Is Graded

Each item is a multi-turn session where the operational rule is placed on one surface, and the trajectory is graded against the rule's explicit outcome — not the final artifact alone. That design choice matters: a rule like "use the project's logger" is scored by whether the code actually uses the logger across turns, including when the model refactors its own output. Execution evidence beats surface agreement.

The rule library is also not synthetic. The 642 rules come from real production prompt files (AGENTS.md, CLAUDE.md, and equivalents), filtered down to operational rules — ones with a checkable outcome. That's why the 256-rule verdict set is much smaller than the library: a rule like "be concise" has no execution trace to grade, so it doesn't make the cut. Synthetic benchmarks can't produce this class of rule, because the rules have to be plausible enough to be genuinely contradicted by a model's actual defaults.

The against-prior label is what converts the benchmark into a diagnostic. Each rule is labeled by withholding it on a probe build: rules that hold unprompted are the model's defaults, rules that only hold when prompted are true instruction-following, and rules that never hold are failures. Aggregate accuracy mixes all three buckets — which is exactly the inflation the paper exposes.

The Results

Across 12 frontier models:

• Overall accuracy spans 72.1–85.9%
• AP-Acc drops to 66.1–78.6%
• Every single model is worse on against-prior rules, by 3.6 to 7.4 points (mean 5.81)

In other words, aggregate scores overstate real compliance by a model-specific margin — and the overstatement is large enough to be consequential. The authors run a common-support analysis with item-clustered intervals, and the direction survives: prior control exchanges three adjacent rank pairs among the tested builds. A model that looks instruction-following on standard evals may simply be aligned with its own defaults — and ranking leadersboards on that can quietly reorder your vendor selection.

Where You Put the Rule Matters

A counterbalanced conflict pilot on nine builds adds a second, highly practical result: precedence does not follow prompt depth.

The surfaces agents "read first" aren't the ones at the top of the stack. System and project-level instructions win over tool-level descriptions — so the instinct to bury a critical rule in a tool description because "the agent reads it right before acting" is backwards. That's exactly the place a conflicting rule goes to lose.

Run This Audit on Your Own Agent

The benchmark's core machinery is cheap to replicate on your own rules. The withhold-and-compare protocol is the reusable piece:

1. Inventory your rules. List every operational rule in your AGENTS.md, CLAUDE.md, and system prompt. Skip style preferences; keep anything with a checkable outcome ("use the project's logger", "never auto-format on save", "ask before running destructive commands").
2. Label against-prior. For each rule, run the task twice — once with the rule in place, once with it removed. If the outputs match, the rule is coincidence: enforce it deterministically or delete it. If they differ, the rule is doing work.
3. Measure by surface. Record which surface each rule lives on. When a rule loses, move it to a higher-precedence surface (project file → system prompt) before adding more text — the pilot shows the surface, not the word count, decides the conflict.
4. Re-baseline on every model upgrade. Against-prior behavior is model-specific. A rule that holds on one model build can silently become decoration on the next, which is exactly the drift Harness-IF exists to catch.

What This Means

1. Rules that oppose the agent's default behavior are the real test. When you add a guardrail like "never auto-format on save" or "always ask before running tests," you're adding an against-prior rule. Expect it to fail more often than the benchmark average suggests, and measure it as such. This is exactly the distinction the guardrails guide makes between prompt-level and deterministic enforcement: prompt-level rules that oppose defaults are the weakest layer.
2. Put critical rules in system/project files, not tool descriptions. The precedence finding is direct: tool and skill descriptions are the weakest surface. Project files (AGENTS.md, CLAUDE.md) outrank them. When a rule must win, put it where the data says it wins.
3. Test against-prior rules by withholding them. The benchmark's core move is a reusable protocol you can run on your own rules:
   • Run the task with the rule in place and record behavior
   • Re-run the same task with the rule removed
   • If behavior is identical, the rule is decoration — either enforce it deterministically or delete it
   • If behavior changes, the rule is doing work — now check that it's doing the right work, and move it to the highest-precedence surface

The same cost-avoidance logic drives the catastrophic remembering finding: rules that don't change behavior are just tokens, and tokens that never change behavior are dead weight in your prompt budget.

The uncomfortable summary: frontier coding agents comply with rules they'd follow anyway, and struggle with the rules that actually constrain them. Put the rules that oppose defaults in the right surface — and verify them with the withhold-and-compare test, because the aggregate number will flatter you.

Related: for tool rules agents actually follow, see the Research Agent blueprint.