Confidence Gating

Jev never returns a single "certain" answer. It returns a probability (Noul) or a distribution (Choice, Score), and the confidence value summarizes how concentrated that distribution is. Gating is the act of turning those numbers into a decision the rest of your system can act on.

The reason to do this carefully: a confident answer and a lucky guess look identical in shape, and a well-formed answer can still be wrong. The gate is where you decide how much uncertainty you are willing to absorb automatically.

Confidence Is Distribution Shape

• A concentrated distribution scores high confidence: almost all probability on one option.
• A split distribution scores low confidence: probability spread across options, even if the intended answer is among them.

Confidence is not an accuracy percentage, and a Noul has no confidence field at all — its probability is the only signal. A Noul at 0.5 means "can't tell," not "medium intensity." If you need a degree, use a Score.

The Three-Band Pattern

One illustrative policy uses three bands:

The middle band is the sharpest part. Escalating to a stronger model — not merely asking the user — can trade uncertainty for extra compute instead of user friction. Whether that is appropriate depends on the consequence of a wrong result and the quality of the escalation path.

Own the Action in Code

The gate belongs in your code, not in the prompt. In TypeScript:

const { answers } = await client.systemOne({
  state: { diff, tests, files },
  questions: {
    touches_auth: noul('Does `diff` change authentication, session, or payment code?'),
    risk: score('How risky is this change?', [
      'Isolated, no shared surface',
      'Shared surface, covered by tests',
      'Wide blast radius or untested critical path',
    ]),
    change_type: choice('What kind of change is this?', {
      feature: 'Adds behavior',
      fix: 'Corrects behavior',
      refactor: 'Changes structure without behavior change',
      other: 'None of the above',
    }),
  },
})

const routes: string[] = []

if (answers.touches_auth.noul > 0.6 || answers.risk.confidence < 0.55) {
  routes.push('review')
} else if (answers.risk.score >= 2 || answers.risk.confidence < 0.85) {
  routes.push('stronger_model')
} else {
  routes.push('auto')
}

Three rules are hidden in those lines:

1. The thresholds are in code you own. Reviewers can see and change them without touching the model.
2. A Noul and a Score are combined into one judgment. touches_auth is a probability; risk is a distribution. The policy that merges them is explicit.
3. Low confidence overrides a low score. An uncertain "safe" is not a safe.

Never Gate at a Value the Model Returns

Jev is stable but not bit-deterministic: the same input can return 0.98 one run and 0.97 the next. The decision holds; the number moves by a hundredth or two.

A real incident from building with it: a triage rule used a threshold of exactly 1.2, and a borderline ticket scored exactly 1.2 — the gate cut precisely at a value the model actually returns. The fix was to move the boundary to 1.4 with margin. Choose boundaries in the gaps, never on a value you have seen come back.

Calibrate on Your Own Labels

Thresholds tuned by intuition drift into silent quality loss. The fix is mechanical:

1. Log every decision with the returned probability and confidence.
2. Label the eventual outcome.
3. Plot predicted probability against empirical frequency.
4. Move thresholds from that curve, not from a blog post.

If the reliability curve bends, a simple mapping (Platt scaling) can straighten it before your thresholds see the number. Treat this as the upgrade path once a decision matters, not a prerequisite for a prototype.

Shadow Before You Gate

Before a gate can block, route, or auto-execute anything, run it in shadow mode: score and log, but take no action. Let it run long enough to see where it would have acted and whether those cases were right. Only then let it act, and only on the low-risk path first.

Maintain the Gate

Two maintenance habits keep a gate honest:

• Version everything together. Model, questions, criteria, and thresholds are one artifact. When any of them changes, replay a labeled set rather than trusting the change.
• Prune dead questions. After the first real run, cut any question whose answer never changed an action. A gate that collects signals nothing consumes is just latency.

Hard Rule

A decision is advisory. Never let a returned choice merge a change, send an email, or run a destructive command on its own strength. The gate narrows what needs attention; it does not grant permission.

Related

• Prompt Engineering — questions and criteria the gate depends on
• Model Routing — the middle band's natural destination
• Use Cases — where gating shows up