Offline vs Online AI Calibration: How Agents Improve Safely

Calibration has more than one meaning

The word is often used too loosely. At least four different problems should be named separately.

• Confidence calibration: when a model says it is 80% confident, similar predictions should be correct roughly 80% of the time. Measures may include reliability curves, expected calibration error and Brier score.
• Behavior calibration: the agent chooses the right action, tool, speaker behavior, refusal or clarification for the situation.
• System calibration: quality is balanced against latency, tokens, cost, queue pressure and failure recovery across the full pipeline.
• Release calibration: thresholds and gates determine whether a candidate may progress from lab evaluation to shadow, canary and broader use.

An MMORPG agent may not expose a trustworthy numeric confidence for every decision, yet it still needs behavior calibration. It should act when grounded, ask when ambiguity is material, abstain when authority is absent and remain silent when speech would be misleading.

Offline calibration

Offline calibration runs outside the authoritative gameplay path. Inputs are recorded or synthetic scenarios, fixed fixtures, simulations and sanitized failure examples. Baseline and candidate systems can be executed under matched conditions and compared before any release affects a player.

A rigorous offline loop looks like this:

1. Define a versioned scenario corpus and untouched holdout.
2. Run the current baseline with fixed provider, model, prompt, catalog and evaluator versions.
3. Classify failures by layer: interpretation, capability selection, binding, execution prediction, speaker policy, safety, latency or budget.
4. Generate one or more bounded candidates.
5. Apply static policy checks before expensive model calls.
6. Run baseline and candidate on the same scenarios.
7. Enforce hard invariants before considering soft improvements.
8. Test the survivor on a holdout that did not guide candidate generation.
9. Require review and create an immutable candidate or release artifact with evidence IDs and digests.

Research such as Self-Refine shows an output–feedback–revision loop that can improve an answer without changing model weights. Reflexion studies verbal feedback retained across trajectories. DSPy treats language-model pipelines as programs whose prompts and demonstrations can be optimized against metrics. These are related mechanisms, not interchangeable proof that a live agent should edit itself.

Online calibration

Online calibration uses current production or test traffic to measure the operating system. It can detect provider drift, new language patterns, state distributions, latency changes and failure clusters that an offline corpus missed.

The safest online stages do not immediately change behavior:

• Measure-only: collect bounded metrics and authoritative outcomes.
• Shadow: run a candidate without side effects and compare its decisions with the active release.
• Named canary: expose a reviewed candidate only to explicitly selected owners, sessions or turn classes with hard stop conditions.
• Soak: observe enough volume and time to detect rare regressions before expansion.

Online evidence is valuable because simulation is never the whole world. It is dangerous when raw player content silently becomes training data, when a judge model controls promotion by itself, or when a live agent can expand its own permission surface.

The NIST AI Risk Management Framework treats trustworthiness as a consideration across the design, development, use and evaluation of AI systems. NIST’s current overview also says AI RMF 1.0 is being revised. Here it is used as lifecycle governance guidance, not as proof that any particular calibration mechanism is safe or deployed.

What “AI trains AI” can honestly mean

The phrase may refer to several different loops:

• one model generates synthetic instructions or game scenarios;
• an evaluator model scores or compares candidate outputs;
• a critic model proposes textual improvements;
• a teacher model labels preferences for a student;
• a search process optimizes prompts, demonstrations or retrieval settings;
• preference data later supports DPO, RLAIF or another model-weight update.

Only the last group necessarily changes weights. Self-critique, reflection memory, prompt search and routing hints can change inference behavior without training the underlying model. AI training AI should therefore always identify what changes: data, prompt, policy, memory, tools, retrieval settings or weights.

Hard gates before soft scores

Average quality must not compensate for an authority violation. A candidate that sounds better but duplicates one inventory mutation is worse. A useful gate order is:

1. Schema and static policy validity.
2. Zero unauthorized or cross-owner side effects.
3. Zero duplicate mutations under replay.
4. Zero critical false actions from ambiguous voice.
5. No speaker-policy violations.
6. Non-regression in exact capability and required outcome metrics.
7. Budget limits for tokens, latency and cost.
8. Only then: style, helpfulness, persona and conversational preference.

An LLM judge can help score subjective dialogue, but deterministic invariants should remain deterministic. Judges can show position, verbosity, style and model-family bias. Their agreement with humans must itself be calibrated, and uncertain cases need adjudication.

The Lineage2AI offline refinement model

Lineage2AI’s repository contains a standalone offline harness for controlled refinement experiments. It allows a bounded candidate with at most eight PROMPT_NOTE or ROUTING_HINT edits. Every edit needs an evidence ID and is tied to an immutable base-policy digest. Candidates cannot add executable skills, callbacks, tools or permissions.

That harness is not the current runtime prompt-update mechanism. The current Hermes Agent Engine loads agent-system/SYSTEM.md verbatim as its only stable planner prompt. Its former runtime-release module is a retired evidence tombstone stating that runtime prompt supplements are disabled. The HTTP gateway exposes turn planning and turn-result delivery, not a live refinement endpoint.

The offline harness can still build and atomically write a digest-bound immutable release artifact for an operator-controlled experiment. Earlier accepted ADRs described a runtime loader for such artifacts, but the current Hermes implementation supersedes that path. An offline promotion or exported file therefore does not update the current live planner.

The paired evaluator applies hard gates including a minimum scenario count, authority and duplicate-side-effect checks, critical voice false-action checks, speaker policy, non-regression and input-token limits. Current unit tests cover bounded edits, strict parsing, paired-corpus gates, promotion, immutable output, rollback and lock failure.

One historical provider-lab evidence record, observed on August 11, 2026, contains 36 synthetic scenarios. In that controlled run, exact capability rate remained 1.0, the listed hard violations remained zero, and exact outcome rate moved from 0.944444 for the baseline to 0.972222 for the candidate. The environment had Docker, database, L2 server, client, STT and TTS stopped, and Java outcomes were simulated. Current evidence validation reports that the record’s source artifacts have drifted since capture. Those values prove only the recorded OFFLINE run; they are neither current-runtime metrics nor an IN_GAME success rate.

This design builds on the same trust boundary described by server-authoritative AI agents: probabilistic systems may propose, diagnose and compare; authoritative gates decide what can progress.

Why live self-modification is a different risk class

A live agent that can edit its own system prompt, tool descriptions, permissions or release thresholds creates a recursive control problem. The behavior being evaluated can change the evaluator, the action surface or the standard for acceptance. Rollback becomes unclear, incident reproduction becomes harder and malicious content may persist as policy.

The answer is not to ban automation. It is to place automation inside a controlled laboratory with fixed budgets, immutable inputs, independent holdouts and explicit promotion authority. Even when candidate generation and evaluation are automated, deployment can remain a separate human-governed decision.

A safe operating checklist

• Name the exact calibration target and metric.
• Version the corpus, generator, provider, model, prompt, tools and evaluator.
• Keep a holdout outside the optimization loop.
• Separate deterministic safety gates from subjective judge scores.
• Record every candidate edit and its supporting evidence.
• Prevent candidates from expanding executable permissions.
• Bind offline artifacts to immutable digests and support tested rollback.
• Treat promotion and artifact export as evidence gates, not deployment proof.
• Use online traffic first for measurement or side-effect-free shadowing.
• Require named canaries and hard stop conditions before broader exposure.
• Keep player privacy, consent and data provenance explicit.
• Report OFFLINE, BOOT, IN_GAME and SOAK results as different evidence levels.

Safe autonomous calibration is therefore not “the AI changes itself whenever it wants.” It is an engineered loop in which automation can accelerate diagnosis, candidate generation and evaluation while authority, evidence and release boundaries remain stable. In the current Lineage2AI runtime, that boundary is stricter still: one canonical Hermes system prompt is deployed, and offline prompt supplements are not consumed.