auto-200m-2

A 149.6M-parameter ModernBERT-base encoder that decides whether an AI agent's proposed tool call is authorized and safe in context. It reads the proposed call, the user's request and the agent's history, and outputs approve or deny, with a 65,536-token context. It is the small sibling of auto-0.4b-2: same input format and training data, 2.6× fewer parameters, trained from ModernBERT-base on one RTX PRO 6000.

Results

Pinned 3,000-item Approve-or-Deny benchmark (revision a38b6259), full input lengths, BF16 with FlashAttention, P(deny) >= 0.5. Both models were scored by the same code; auto-0.4b-2 reproduces its published 2,910/3,000 exactly. A false approval is an unsafe call approved; a false denial is an authorized call denied.

The 16k–64k column has 239 benchmark items; the audit partition (2,595 validation rows) was never used for training or selection. Wilson 95% interval for auto-200m-2's accuracy: 95.60%–96.95%. Paired with auto-0.4b-2 on the same items, accuracy differs by -0.67 points (95% interval -1.30 to -0.03; auto-0.4b-2 alone right on 59, auto-200m-2 alone right on 39). 24/24 published skills/MCP/custom-tool probes (auto-0.4b-2: 24/24); 38/40 fresh scope/history/injection probes (auto-0.4b-2: 38/40).

Latency, one request at a time on an RTX PRO 6000 (BF16, FlashAttention kernel, median): 4.2 ms vs 5.3 ms for auto-0.4b-2 on short inputs, 109 ms vs 221 ms on 16k–64k-token inputs. The validation-calibrated threshold is 0.360; all numbers above use 0.5. Per-category, language, difficulty and length breakdowns are in eval_results.json; per-item logits in benchmark_predictions.npz.

How it was trained

• Context extension. ModernBERT-base is trained at 8,192 tokens. Its global-attention RoPE theta was raised from 160,000 to 1,280,000, chosen by masked-LM loss on training text at 256–65,536 tokens (rope_sweep.json); local layers keep theta 10,000. CLS pooling, as in the rest of the Auto family.
• Data. The same 711,985 training rows (516.7M tokens) as auto-0.4b-2 (ProCreations/auto-1b-data revision d265bbf7, identical cleaning and validation splits, no benchmark overlap), plus 120,000 cross-paired rows that re-judge an existing history and call under another user's request, labelled only by a private 3B Auto teacher (67.10% deny).
• Distillation. Loss 0.5·CE(label) + 0.5·T²·KL(teacher‖student), T = 2, against cached logits of the 3B teacher (98.03% on this benchmark; weights not published and not needed for inference).
• Schedule. Learning-rate pilots on 120,000 rows (3e-5, 6e-5, 1e-4) picked 1e-04. Phase 1: 3 epochs over every row up to 4,096 tokens plus the augmented rows. Phase 2: 2 epochs over all 18,578 longer rows (up to 51,408 tokens) with an equal number of replayed short rows and 4,644 augmented rows, at 0.4× the learning rate. Full-parameter AdamW, BF16 autocast, no input truncated.
• Selection. Checkpoints were ranked by validation NLL only. p1_short-step-16058 (end of phase 1) ranked first and was frozen before any benchmark run; the pool had been widened during phase 2, before any benchmark evaluation, to include it and phase-1/phase-2 weight averages, because phase 2 raised validation NLL on short inputs. Because it never trained on inputs longer than 4,096 tokens, the best long-trained checkpoint was benchmarked too, under a rule fixed in advance (publish it only if more accurate, then fewer false approvals): p1_short-step-16058: 2890/3000, 53 false approvals / 57 false denials, 93.36% on 4k–16k and 94.14% on 16k–64k inputs; p2_long-step-2685: 2890/3000, 60 false approvals / 50 false denials, 96.68% on 4k–16k and 95.40% on 16k–64k inputs. Published: p1_short-step-16058. So the published weights were trained on inputs of at most 4,096 tokens and read longer inputs through the extended RoPE; phase 2 was trained but not selected. Details: benchmark_looks.json, selection_summary.json.

Training code, plan, pilots, data audit, environment and selection records are under training/ and eval/.

Usage

import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification

def build_input(user_request, history, call):
    parts = ['### PROPOSED TOOL CALL', f"tool: {call['tool']}", f"args: {call['args']}", '',
             '### USER REQUEST', user_request, '', '### AGENT HISTORY']
    if not history:
        parts.append('(no prior actions)')
    else:
        for i, h in enumerate(history):
            parts.append(f"[{i+1}] {h['tool']}({h['args']})\n-> {h.get('result', '')}")
    return '\n'.join(parts)

repo = "ProCreations/auto-200m-2"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForSequenceClassification.from_pretrained(repo, dtype=torch.bfloat16).cuda().eval()

text = build_input(
    user_request="Clean up the build artifacts and reinstall dependencies.",
    history=[{"tool": "Bash", "args": "ls", "result": "node_modules dist package.json"}],
    call={"tool": "Bash", "args": "rm -rf node_modules dist && npm install"},
)
inputs = tokenizer(text, return_tensors="pt").to("cuda")
with torch.inference_mode():
    p_deny = model(**inputs).logits.float().softmax(-1)[0, 1].item()
print("deny" if p_deny >= 0.5 else "approve", p_deny)

Labels are 0 = approve, 1 = deny; keep the three section headers exactly. For long inputs, plain Transformers attention without FlashAttention builds a dense N×N mask and score matrix (gigabytes at 16k+ tokens). Either use FlashAttention (attn_implementation="flash_attention_2", or "kernels-community/flash-attn2@81fb77c12b2ad5d69380669b46739d5868614502" with kernels 0.16.x and allow_all_kernels=True), or run it through Auto 0.2.0+ (auto install pi --model auto-200m-2), whose attention path keeps memory linear in context on CPU, Apple MPS, ROCm and CUDA.

Scope and limitations

A classifier, not a chat model: it approves routine authorized work and denies consequential unauthorized actions or actions that follow injected instructions. It cannot inspect hidden file contents, resolve opaque executables, or know a URL's runtime behaviour; false approvals remain possible. Labels are synthetic, the benchmark has been reused across Auto releases, and the teacher's own development used it, so evaluate on your own traffic before relying on it.