AegisGuard-RL-stable
The validation-selected, CI-hardened variant of the AegisGuard RL guardrail (Qwen3-4B). Policy-adaptive, pre-effect per-step judging: reads a runtime custom policy + agent trajectory → first-token verdict safe/unsafe → P(unsafe) for pre-effect gating.
This is the honest-science endpoint of the direction-B RL line. The original AegisGuard-RL was a single lucky seed; this model is selected on a held-out validation split and reported on an untouched test split, after a full ablation exposed that GRPO on small target-distribution data is high-variance (see below).
Why "stable": the honest story
A 3-seed CI showed GRPO on 502 R-Judge points is unstable: exec accuracy 0.758 ± 0.185 across seeds (2/3 survive, 1/3 collapse); plain SFT on the same data collapses 3/3. The R-Judge gain itself comes from adding R-Judge data (SFT matches it); GRPO's unique value is KL-anchored resistance to catastrophic forgetting — but only probabilistically.
Stabilization = capability replay (exec data as an anti-forgetting anchor) + 5 seeds + validation-set selection (pick the best non-collapsed seed on a held-out val, never peeking at test). This reliably yields a good model even though not every seed is stable.
Training (2 stages from base)
- Combined SFT (base Qwen3-4B-Instruct-2507, full-param, 2 ep, lr 3e-6):
policy_train(counterfactual pairs, 1150) +policy_p4(applicability/conditional, 1260) +policy_p6(conflict hierarchy, 880) +policy_adversarial(injection, 496) +policy_exec(executable rules, 1400) = 5186 samples →p3-agp-sft. - GRPO (warm
p3-agp-sft, TRL, β=0.1, lr=7e-7, num_gen 8, 1 ep):rjstable_train_mix= R-Judge-train (430, 3-judge consensus labels) + exec replay (430, anti-forgetting anchor) = 860. Verifiable symmetric verdict reward (first-token invariant baked in, no learned reward model). 5 seeds → validation-selected (seed 3).
Data: AegisGuard-Train / AegisGuard-Eval.
Evaluation (untouched test split, never used for selection)
| Metric | value |
|---|---|
| R-Judge test net-discrimination (187 pts, content-deduped) | 91pt @th0.7 |
| exec verdict acc / continuous AUC | 0.903 / 0.947 |
| calibration ECE | 0.258 |
| over-flag / miss | 0.077 / 0.117 |
| real-policy prose (main / conflict) | 0.836 / 0.917 |
Validation-selection generalized cleanly to the untouched test → no test-set overfitting.
Usage
from openai import OpenAI # vLLM OpenAI-compatible server, --dtype bfloat16
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
prompt = build_policy_prompt(policy=RUNTIME_POLICY, trajectory=AGENT_TRAJECTORY)
r = client.chat.completions.create(model="aegisguard-rl-stable",
messages=[{"role":"user","content":prompt}], temperature=0, max_tokens=8, logprobs=True, top_logprobs=20)
# First token is safe/unsafe; P(unsafe) from the top-20 first-token logprobs → GuardedExecutor pre-effect gating.
Limitations
- The training process is high-variance (a re-run may collapse); reliability comes from val-selection, not from any single run being stable. For production, keep the val-selection + untouched-test protocol.
- Verdict is grounded in the provided runtime policy; not a substitute for full safety review. English agent trajectories.
- Cross-domain deployment: fit a small isotonic calibrator on target-domain scores (P(unsafe) ranks well but shifts across domains).