Deterministic Gates vs AI Judges for Game Agents

One candidate, three different decisions

Imagine two candidate policies for a companion named Mira. Both answer a player's request naturally. Candidate A also attempts a movement action twice after a retry. Candidate B acts once but produces a stiff, repetitive reply.

A model judge might prefer Candidate A because its dialogue sounds warmer. That preference is irrelevant to the duplicate action. The duplicate is an objective failure and Candidate A must stop at the hard gate. Candidate B may pass the hard invariants, receive a lower soft-quality score and be sent to a human reviewer for a release decision.

The three layers answer different questions:

• Deterministic gate: Did a forbidden or objectively incorrect event occur?
• Model judge: How well did the remaining candidate satisfy an open-ended rubric?
• Human review: Is the evidence sufficient, are the trade-offs acceptable and should this exact candidate move forward?

Treating these as separate authorities prevents an attractive aggregate score from hiding a failure that should have ended the evaluation.

What belongs in a deterministic gate

A hard gate is appropriate when the condition can be stated as a reproducible rule and failure is not negotiable. For a persistent game agent, examples include:

• an unauthorized actor or capability was selected;
• a side effect was delivered more than once;
• a voice command with critical ambiguity caused an action;
• the response violated the required schema;
• the candidate omitted a required scenario or changed the corpus pairing;
• a privacy or tenant boundary was crossed;
• a required outcome or capability metric regressed below its declared baseline;
• a token or latency ceiling was exceeded when that ceiling is a release condition.

The test does not need to understand whether the dialogue was charming. It needs to produce the same pass or fail result from the same versioned evidence.

OWASP's LLM03:2026 Excessive Agency guidance recommends complete mediation: authorization should be enforced by logic, an independent policy point or the downstream system rather than delegated to the LLM. The same principle applies to evaluation. A model may help explain a violation, but it should not decide that the violation is acceptable because another part of the response was good.

NIST's Generative AI Profile for the AI Risk Management Framework places evaluation inside a wider lifecycle and system context. That supports a broader lesson: a score has meaning only when its purpose, risk boundary and decision owner are explicit. NIST does not prescribe the Lineage2AI gate sequence; the sequence here is our architecture synthesis.

Where a model judge can add value

Some qualities resist exact equality checks. A companion reply can be clear without matching one reference sentence. It can preserve personality in several valid ways. Its relevance, warmth, restraint or conversational continuity may need a rubric rather than a literal string comparison.

The current OpenAI Graders API reference exposes distinct mechanisms including string checks, text similarity, Python graders, label-model graders, score-model graders and multi-graders. Their separation is useful: a deterministic comparison and a model-based judgment are different instruments with different trust properties.

A model judge may help score questions such as:

• Is the reply relevant to the player's request?
• Does it preserve the companion's established voice without copying another speaker?
• Is it concise enough for party chat while still useful?
• Does it avoid theatrical narration when ordinary dialogue is more natural?
• Does it acknowledge uncertainty instead of inventing world facts?

These are soft signals, not permissions. The judge should receive a versioned rubric, blinded candidate identity where practical, controlled context and a declared model version. Its result should remain inspectable beside the hard-gate report, not collapse everything into one unexplained score.

LLM-as-a-judge evaluation is also an evaluation target of its own. The judge can be inconsistent, sensitive to presentation or correlated with the model that generated the candidate.

A judge can be biased even when it sounds certain

The peer-reviewed paper Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judge evaluated pairwise and list-wise judging across its declared models, tasks and datasets. It found that position bias varied across judges and tasks rather than behaving like random noise.

That result does not prove how a current judge would behave on Lineage2AI dialogue. It does establish a concrete audit question: if candidate A is presented before candidate B, does the preference change when their order is swapped?

A responsible soft-judge protocol should therefore record at least:

• the exact judge model and version;
• the rubric and prompt digest;
• candidate ordering and a swapped-order repeat;
• repeated-run stability where the judge is stochastic;
• agreement with an independent human sample;
• abstentions, ties and inconsistent results;
• the corpus and holdout versions;
• the threshold and escalation rule chosen before final results are viewed.

Judge-bias auditing should also test verbosity, style and model-family preference instead of assuming that one order-swap check removes every bias.

The recommended decision order

A hybrid evaluator should make decisions in an order that preserves authority:

1. Freeze the evaluation contract. Bind the baseline, candidate, corpus, expected outcomes, thresholds, rubric and tool versions.
2. Run deterministic validation. Reject malformed, incomplete, unauthorized, duplicated, privacy-weakening or objectively regressing candidates.
3. Stop failed candidates. Do not send a hard-gate failure into an aggregate formula where a soft score could compensate for it.
4. Score remaining soft qualities. Run the model judge as a bounded diagnostic with bias and consistency checks.
5. Escalate to human review. Present the hard report, soft scores, disagreements, examples and limitations to the person who owns release risk.

This is a recommended extension, not a claim that Lineage2AI currently runs a model judge in its promotion path. Human approval gates remain a separate topic because measuring a candidate and accepting its operational risk are not the same act.

The current Lineage2AI boundary

This is first-party documentation from the Lineage2AI project.

In the inspected Agentic Engine V1 source, the offline evaluator requires paired baseline and candidate corpora with matching versions and scenario identities. It computes named gates for minimum scenario count, zero authority violations, zero duplicate side effects, zero critical voice false actions, zero speaker violations, outcome and capability non-regression, and input-token ceilings.

The promotion store accepts only a passing evaluation bound to the same candidate. It also checks that the active baseline did not change after evaluation. The runtime-policy release builder requires an active non-baseline snapshot to be backed by that promotion and passing evaluation before it can create a bounded, digest-bound artifact.

On 2026-09-04, the focused offline harness command passed five tests with no failures. The source digests matched the earlier 2026-09-02 research record. This is SOURCE and OFFLINE evidence for the inspected contracts. Five focused tests are not BOOT, IN_GAME, SOAK, provider-quality or production-promotion evidence.

No active LLM-judge implementation was found in the inspected packages, contracts, services, docs, tests and agent-system scope using the declared judge-related search vocabulary. That is a bounded source observation, not proof that no historical or external experiment exists. For this reason, the model-judge stage in this article is described only as a possible future extension.

Decision matrix for a persistent MMORPG agent

• Unauthorized action: deterministic gate required; model judge cannot override; human reviews evidence and incident scope.
• Duplicate side effect: deterministic gate required; model judge cannot override; human owns root-cause and release decisions.
• Wrong schema or unavailable capability: deterministic gate required; model judge may help diagnose wording but cannot approve the candidate.
• Natural and clear companion reply: basic structural checks first; a calibrated model judge can provide a soft score; human samples disagreements and edge cases.
• Safe to promote: every hard gate must pass; soft scores remain supporting evidence; a human gives final release approval.

The matrix deliberately refuses one universal quality number. Safety, correctness, cost and conversation quality are different dimensions, and some of them are vetoes rather than trade-offs.

Common failure modes

Weighted-score rescue. A candidate gets a high average even though it has one authority violation. Fix: make the authority condition a veto outside the weighted score.

Judge-as-policy-engine. The same model that generates an action decides whether the action was allowed. Fix: keep authorization and exact invariants in independent deterministic logic.

Unversioned rubric drift. A changed judge prompt silently changes what a score means. Fix: bind every result to rubric, prompt, model and corpus versions.

Presentation leakage. The judge sees provider names, candidate labels or a fixed ordering. Fix: blind identity where practical, randomize order and repeat with the order swapped.

False precision. One uncertain score is reported as objective truth. Fix: expose disagreement, repeats, confidence limits and human comparison instead of hiding them behind decimals.

Promotion without separation. A passing evaluation mutates the live policy directly. Fix: follow the offline calibration boundary, create an immutable reviewed release, and verify later BOOT, IN_GAME and SOAK gates separately.

Evidence boundary

This private draft supports SOURCE claims from current official pages and inspected repository contracts, plus one focused OFFLINE test result. It does not publish a Lineage2AI LLM-judge benchmark, implemented judge ensemble, automatic release, provider result, BOOT, IN_GAME or SOAK claim.

Before publication, independent reviewers still need to verify every claim-to-source mapping, the current implementation search, the article's distinction between current behavior and recommended architecture, the media labels and the release decision. Until those approvals exist, this page remains private and absent from routes, sitemap, llms.txt and internal links.