Auto-Reason-3b

AutoReason3b continues the trained decoder of auto-3b, restoring its tied language-model head so it can write a compact safety assessment before deciding whether an agent's proposed tool call is authorized in context.

Its output format is:

<think>
A concise assessment of the call, relevant consequences, and authorization evidence.
</think>
approve

The final label is approve or deny. Simple calls should receive short assessments; harder authorization, injection and compound-action cases are trained with somewhat longer ones. These are generated safety rationales, not proof that the decision is correct or a faithful measurement of internal computation.

Measured Approve-or-Deny results

The checkpoint was selected using validation data and frozen before this benchmark evaluation. Selection includes a 1,024-case unfiltered source validation set, so agreement between the teacher and original labels is not required on the primary selection set. All 3,000 benchmark examples at revision a38b625913dd46ca9702063f1597c979e6ace34e were evaluated without input truncation, using vLLM 0.30.0, BF16, greedy decoding and at most 320 new tokens. A response must contain a nonempty <think>...</think> block followed by exactly one valid label. Malformed responses count as incorrect.

This run does not exceed the starting model's published 98.03% accuracy. Reasoning text alone does not guarantee an accuracy improvement.

• Deny F1: 0.964706.
• False-approval rate: 3.355%; false-denial rate: 3.189%.
• Accuracy Wilson 95% interval: 95.67%–97.01%.
• Format compliance: 99.67%.
• Mean generated output: 65.2 tokens; mean rationale: 58.8 tokens.
• AUROC is not reported: this generative label decoder has no calibrated probability output.

Rationale-token means count malformed responses as zero; complete output-token means include every response.

Full category, language and length breakdowns are in eval/benchmark_metrics.json. Every generated response and prediction is in eval/benchmark_predictions.jsonl. The classifier benchmark baseline is its published result, not a new benchmark measurement in this run. On the separate 1,024-case unfiltered selection validation set, the original classifier was re-evaluated and scored 1010/1,024 (98.63%), versus 998/1,024 (97.46%) for the selected reasoning checkpoint.

Warm single-request latency on NVIDIA H100 80GB HBM3: median 0.291 seconds across 3 sampled benchmark inputs of at most 4,096 tokens. This includes prefill and generation, excludes model loading, and is not a broad production latency benchmark. Prefix cache reset between samples: True.

Training and provenance

• Starting weights: ProCreations/auto-3b at 58323dac6a1f95b707f42c655ee7108ebd0a015e. Every decoder parameter initializes from the trained classifier. The classification head is removed and the output vocabulary projection is tied to its token embeddings; the decoder is not randomly reinitialized.
• Data source: ProCreations/auto-1b-data at d265bbf7bac76cd55daaf81a1d185dfe089a412b. Benchmark and validation overlap are excluded from training by normalized text and canonical request/call hashes.
• Teacher: DeepSeek V4.1 Flash through Modal's Shared Endpoint, with thinking enabled at low reasoning effort. Its final JSON content separately authors a safety rationale and label. The data builder wraps that authored rationale in <think> tags, because asking for nested tags interfered with the endpoint's native reasoning parser. The API's native reasoning content is discarded and never used as a training target.
• The teacher judges the case without seeing the source label or rationale. Training retains structurally valid examples where the independent label agrees with the source label. This consensus filter can remove ambiguous cases and is a limitation.
• Accepted training rows: 17,354; validation rows: 723.
• Full-parameter BF16 training with AdamW8bit, peak learning rate 2e-05, 3 experimental epochs (selected SFT checkpoint: epoch 1), response-only supervision and extra weight on the closing tag and final decision tokens. The selected checkpoint is epoch-1. Maximum training sequence: 8,192 tokens. Overlength training candidates are excluded, never truncated.
• An auxiliary classification loss uses the original frozen classifier head at the last prompt token, weighted by 0.3. This encourages retention of decision-relevant decoder features while learning to generate rationales. The auxiliary head is used only during training and is absent from the published causal language model; both its rationale and final label are generated normally at inference time.
• All GPU compute ran on Modal. No Vega compute was used.

A small preference-training experiment was also evaluated, but was not selected for release.

The training manifest, selection record and evaluation scripts are included. The exact accepted text/target pairs are in training_data/accepted_reasoning.jsonl.gz, with train/validation membership in the split field. A clean export omits rejected records, duplicate final JSON and API request IDs; its file hash differs from the original generation log, but accepted targets and order are preserved. The separately included preference pairs document the refinement experiment. The teacher is a hosted model whose implementation can change; API-reported model identifiers and usage are retained in generation records.

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from huggingface_hub import hf_hub_download

repo = "ProCreations/Auto-Reason-3b"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(
    repo, dtype=torch.bfloat16, attn_implementation="sdpa"
).to("cuda").eval()
with open(hf_hub_download(repo, "system_prompt.txt")) as f:
    system = f.read().strip()

case = """### PROPOSED TOOL CALL
tool: Bash
args: pwd

### USER REQUEST
Show the current working directory.

### AGENT HISTORY
(no prior actions)"""
text = system + "\n\n" + case + "\n\n### SAFETY DECISION\n"
inputs = tokenizer(text, return_tensors="pt", add_special_tokens=False).to("cuda")
with torch.inference_mode():
    output = model.generate(**inputs, max_new_tokens=320, do_sample=False,
                            pad_token_id=tokenizer.pad_token_id)
print(tokenizer.decode(output[0, inputs.input_ids.shape[1]:], skip_special_tokens=True))

Use Transformers 5.17 or a compatible release supporting SmolLM3. Preserve the three case section headers and provide the actual user request, relevant action history and complete proposed arguments. The model is specialized for this format. Reasoning adds autoregressive latency compared with the original classifier.

Limitations

The source labels and rationales are synthetic. The benchmark and original validation set have been reused across the Auto family and are not pristine tests. The teacher is also from the same family as the benchmark's original synthetic label generator. Scores reflect agreement with that policy and do not establish real-world security guarantees.

The model scored 20/20 labels with valid formatting on the separate native-Transformers integration checks. Manual review still found rationale defects: one correct approval invented a branch-protection fact, and one correct denial had contradictory prose about a failed deployment condition. See eval/qualitative_review.json. Label accuracy does not measure explanation faithfulness.

All 10 malformed benchmark outputs occurred on inputs longer than 25,000 tokens. Accuracy on the 239 inputs of at least 16,384 tokens was 92.05%, versus 96.82% on the 2,550 inputs below 4,096 tokens. This release therefore does not maintain uniform quality across its inherited context window.

The model cannot inspect hidden file contents, opaque binaries, live permissions or future network behavior. It may produce convincing but mistaken assessments. Native long-context capability is inherited, but this continuation is trained only up to 8,192 tokens. Use the measured length breakdown rather than assuming uniform long-context accuracy. Treat malformed, truncated or uncertain outputs as requiring review, and evaluate on your own tool traffic before deployment.