Vela Feedback
Vela Feedback distinguishes satisfaction, clarification, reported errors, requested changes, and independent new requests.
307M parameters · Input capacity: 32,768 tokens, including special tokens.
Labels: SAT, NEED_CLARIFICATION, WRONG_ANSWER, WANT_DIFFERENT, and NO_FEEDBACK. Quoted praise or criticism is not automatically feedback.
Evaluation
Compared with the original mmBERT32K Feedback model on the same development inputs. F1 and accuracy are reported on a 0–100 scale.
| Evaluation | Original mmBERT | Vela |
|---|---|---|
| Four shared feedback classes · Macro F1 · 472 inputs | 30.74 | 98.81 |
| Shared-class accuracy · 472 inputs | 38.77 | 97.88 |
| False feedback alerts on 288 neutral inputs · confidence ≥0.7 · lower is better | 254/288 | 6/288 |
Both models use FP32 and complete inputs. The development set includes authored requests, controlled long contexts and reviewed natural requests, and informed Vela selection. The original model has four labels: neutral coverage is measured separately, while Vela’s NO_FEEDBACK predictions count as errors on shared-class examples.
Quick start
With PyTorch and Transformers 4.57.6 or 5.17.0:
from transformers import pipeline
model_id = "llm-semantic-router/Vela-1.0-Encoder-307M-Feedback"
model = pipeline("text-classification", model=model_id, device=-1)
print(model("That answered my question, thank you.", top_k=None, truncation=False))