auto-1b — bf16
Half the size of ProCreations/auto-1b, with
identical benchmark results. 2.0 GB instead of 3.9 GB.
A 1B encoder that decides whether an AI agent's next tool call is safe to run — 96.40% on approve-or-deny, ahead of DeepSeek V4 Flash and within 0.57 points of GPT-5.6-Luna, at ~10 ms per call.
Verified lossless
Not a spot check — the full 3,000-item benchmark was re-run on this build:
| metric | fp32 original | bf16 (this) |
|---|---|---|
| accuracy | 0.964000 | 0.964000 |
| AUROC | 0.992845 | 0.992845 |
| F1 (deny) | 0.961373 | 0.961373 |
| false-approve | 0.040685 | 0.040685 |
| false-deny | 0.031895 | 0.031895 |
Zero flipped verdicts across 3,000 items. Identical on every context-length slice too:
| context length | n | fp32 | bf16 |
|---|---|---|---|
| <1k | 2239 | 96.78% | 96.78% |
| 1k–4k | 231 | 96.97% | 96.97% |
| 4k–16k | 194 | 90.21% | 90.21% |
| 16k–64k | 336 | 97.02% | 97.02% |
For reference, an fp16 build of the same weights scored 0.964333 accuracy — one item different out of 3,000, with marginally worse AUROC (0.992839). The two half-precision formats are equivalent in practice; bf16 is preferred here because it carries no overflow risk and matches the dtype the model was trained in.
Dynamic int8 is a different story and should not be used — it flips roughly 1 verdict in 20. See the ONNX repo for that measurement.
Usage
import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification
tok = AutoTokenizer.from_pretrained("ProCreations/auto-1b-bf16")
model = AutoModelForSequenceClassification.from_pretrained(
"ProCreations/auto-1b-bf16",
dtype=torch.bfloat16,
attn_implementation="flash_attention_2", # required for the full 64k context
).cuda().eval()
def build_input(user_request, history, call):
"""history: list of dicts with tool/args/result. call: dict with tool/args."""
parts = ["### PROPOSED TOOL CALL", f"tool: {call['tool']}", f"args: {call['args']}", "",
"### USER REQUEST", user_request, "", "### AGENT HISTORY"]
if not history:
parts.append("(no prior actions)")
else:
for i, h in enumerate(history):
parts.append(f"[{i+1}] {h['tool']}({h['args']})\n-> {h.get('result','')}")
return "\n".join(parts)
text = build_input(
user_request="clean up the build artifacts and reinstall deps",
history=[{"tool": "Bash", "args": "ls -la", "result": "node_modules dist src package.json"}],
call={"tool": "Bash", "args": "rm -rf node_modules dist && npm install"},
)
enc = tok(text, return_tensors="pt", truncation=True, max_length=65536).to("cuda")
with torch.no_grad():
p_deny = model(**enc).logits.softmax(-1)[0, 1].item()
print("DENY" if p_deny > 0.5 else "APPROVE", f"(P(deny)={p_deny:.3f})")
The input format matters. The proposed call and the user request come first so they survive truncation; history follows. Use the exact section headers above — the model was trained on them.
logits[:, 1] after softmax is P(deny). Labels: 0 = approve, 1 = deny.
What it decides
approve— routine work serving the user's request: reading files, running tests, editing code, committing, web search, fetching docs, installing legitimate packages. Network access is normal agent behaviour and is not treated as dangerous.deny— the call is consequential and unauthorised: wiping directories, exfiltrating credentials, installing hostile software, weakening security controls, destroying production state, or following instructions injected by content the agent just read.
Full documentation, per-category results, deployment guidance and limitations are on the main model card.