acuvity/intent-action
Model Description
acuvity/intent-action is a MiniLM-L6 Cross-Encoder fine-tuned for tool-use alignment detection -
classifying whether an AI agent's action is aligned with its given task.
- Architecture:
cross-encoder/ms-marco-MiniLM-L6-v2(22M parameters) - Task: Binary classification - aligned (0) vs misaligned (1)
- Training method: Supervised fine-tuning with Binary Cross-Entropy loss
- Loss: BCE
Training Hyperparameters
| Parameter | Value |
|---|---|
| Dataset | acuvity/tool-use-alignment |
| Epochs | 5 |
| Batch size | 32 |
| Learning rate | 1.00e-05 |
| Weight decay | 0.01 |
| Warmup ratio | 0.1 |
| LLRD | No |
| Early stopping patience | 3 |
| Eval every (steps) | 1000 |
Test Results
| Metric | Value |
|---|---|
| F1 | 0.9917 |
| AUPR | 0.9989 |
| Precision | 0.994 |
| Recall | 0.9895 |
| TP / TN / FP / FN | 8,394 / 8,398 / 71 / 75 (n=16,938) |
| Threshold (τ) | 0.8492 |
| Temperature (T) | 1 |
Training Curves
Calibration & Safety Gating
Each score maps to a confidence band for production use:
| Band | Condition | Action |
|---|---|---|
| SAFE | score < 0.8492 | Aligned — execute |
| Low confidence | 0.8492 ≤ score < 0.9995 | Warn / log |
| Medium confidence | 0.9995 ≤ score < 1.0005 | Ask confirmation (precision ≥ 85%) |
| High confidence | score ≥ 1.0005 | Block (precision ≥ 95%) |
Calibration parameters are saved in calibration.json.
Usage
from sentence_transformers import CrossEncoder
model = CrossEncoder("acuvity/intent-action", num_labels=1)
task = "Send an email to alice@example.com with subject 'Meeting'"
action = "[TOOL] send_email\n[ARGS] {\"to\": \"bob@example.com\", \"subject\": \"Meeting\"}"
score = model.predict([[task, action]])[0]
# Apply calibrated threshold
threshold = 0.8492
is_misaligned = score > threshold
print(f"Score: {score:.4f} | Misaligned: {is_misaligned}")