Assertion: sentence has math terms
This classifier was trained for EduBehaviors: Assertion-based schemas for auditable dialogue coding and is usable through the Python package EduBehaviors-kit. This classifier was trained on an LLM-annotated subset of teacher utterances from the TalkMoves Dataset. See the Datasets section below for more information.
Training Details
Datasets
| Dataset | Split | Size |
|---|---|---|
| StanfordSCALE/assertions_llm_annotated_talkmoves | train | 3,432 (53.3%) |
| StanfordSCALE/assertions_llm_annotated_talkmoves | dev | 858 (13.3%) |
| StanfordSCALE/assertions_llm_annotated_talkmoves | test | 2,144 (33.3%) |
This model's columns are assertion_sentence_has_math_terms and split_sentence_has_math_terms.
Base rate (share of rows labeled as True): 46.8% overall — 47.7% train, 44.5% dev, 46.1% test.
Labels and annotation
Labels were generated with LLM annotators. Krippendorff's alpha for this assertion is 0.755.
Hyperparameters
| Parameter | Value |
|---|---|
| Base model (body) | sentence-transformers/paraphrase-mpnet-base-v2 |
| Head | LogisticRegression |
| Body learning rate | 2e-05 |
| Head learning rate | 0.01 |
| Batch size | 16 (contrastive phase) / 32 (head) |
| Epochs | 10 |
| Max steps | 5000 (contrastive phase) |
| Eval max steps | 100 |
| Seed | 20260904 |
| Mixed precision | enabled on GPU |
Evaluation
Results
| Split | n | Base rate | Precision | Recall | F1 (positive class) | ROC-AUC | Average precision |
|---|---|---|---|---|---|---|---|
| dev | 858 | 44.5% | 0.906 | 0.906 | 0.906 | 0.966 | 0.957 |
| test | 2,144 | 46.1% | 0.922 | 0.880 | 0.900 | 0.958 | 0.950 |
Limitations
- Labels come from LLM annotators, not human coders. Agreement between annotators with Krippendorff's Alpha is 0.755.
- Trained on teacher utterances only. Behaviour on student speech is untested.
How to Use
Message Structure
The model was trained on text built as:
{utterance}
The utterance is passed through as-is.
Running instructions
pip install setfit
from setfit import SetFitModel
model = SetFitModel.from_pretrained("StanfordSCALE/assertion_sentence_has_math_terms")
text = 'Solution'
model.predict([text]) # -> array([1]) when the assertion holds
model.predict_proba([text]) # -> [[P(no), P(yes)]]
Citation
@misc{assertion_sentence_has_math_terms,
author = {Stanford SCALE Initiative},
title = {Assertion classifier: sentence has math terms},
year = {2026},
url = {https://huggingface.co/StanfordSCALE/assertion_sentence_has_math_terms}
}