Assertion: sentence calls on student by name
This classifier was trained for EduBehaviors: Assertion-based schemas for auditable dialogue coding and is usable through the Python package EduBehaviors-kit. This classifier was trained on an LLM-annotated subset of teacher utterances from the TalkMoves Dataset. See the Datasets section below for more information.
Training Details
Datasets
| Dataset | Split | Size |
|---|---|---|
| StanfordSCALE/assertions_llm_annotated_talkmoves | train | 3,432 (53.3%) |
| StanfordSCALE/assertions_llm_annotated_talkmoves | dev | 858 (13.3%) |
| StanfordSCALE/assertions_llm_annotated_talkmoves | test | 2,144 (33.3%) |
This model's columns are assertion_sentence_calls_on_student_by_name and split_sentence_calls_on_student_by_name.
Base rate (share of rows labeled as True): 5.4% overall — 5.9% train, 5.8% dev, 4.5% test.
Labels and annotation
Labels were generated with LLM annotators. Krippendorff's alpha for this assertion is 0.918.
Hyperparameters
| Parameter | Value |
|---|---|
| Base model (body) | sentence-transformers/paraphrase-mpnet-base-v2 |
| Head | LogisticRegression |
| Body learning rate | 2e-05 |
| Head learning rate | 0.01 |
| Batch size | 16 (contrastive phase) / 32 (head) |
| Epochs | 10 |
| Max steps | 5000 (contrastive phase) |
| Eval max steps | 100 |
| Seed | 20260904 |
| Mixed precision | enabled on GPU |
Evaluation
Results
| Split | n | Base rate | Precision | Recall | F1 (positive class) | ROC-AUC | Average precision |
|---|---|---|---|---|---|---|---|
| dev | 858 | 5.8% | 0.786 | 0.880 | 0.830 | 0.995 | 0.913 |
| test | 2,144 | 4.5% | 0.913 | 0.866 | 0.889 | 0.980 | 0.920 |
Limitations
- Labels come from LLM annotators, not human coders. Agreement between annotators with Krippendorff's Alpha is 0.918.
- Trained on teacher utterances only. Behaviour on student speech is untested.
How to Use
Message Structure
The model was trained on text built as:
{utterance}
The utterance is passed through as-is.
Running instructions
pip install setfit
from setfit import SetFitModel
model = SetFitModel.from_pretrained("StanfordSCALE/assertion_sentence_calls_on_student_by_name")
text = 'Lets go Gavin'
model.predict([text]) # -> array([1]) when the assertion holds
model.predict_proba([text]) # -> [[P(no), P(yes)]]
Citation
@misc{assertion_sentence_calls_on_student_by_name,
author = {Stanford SCALE Initiative},
title = {Assertion classifier: sentence calls on student by name},
year = {2026},
url = {https://huggingface.co/StanfordSCALE/assertion_sentence_calls_on_student_by_name}
}