Assertion: sentence has a question
This classifier was trained for EduBehaviors: Assertion-based schemas for auditable dialogue coding and is usable through the Python package EduBehaviors-kit. This classifier was trained on an LLM-annotated subset of teacher utterances from the TalkMoves Dataset. See the Datasets section below for more information.
Training Details
Datasets
| Dataset | Split | Size |
|---|---|---|
| StanfordSCALE/assertions_llm_annotated_talkmoves | train | 3,430 (53.3%) |
| StanfordSCALE/assertions_llm_annotated_talkmoves | dev | 858 (13.3%) |
| StanfordSCALE/assertions_llm_annotated_talkmoves | test | 2,146 (33.4%) |
This model's columns are assertion_sentence_has_a_question and split_sentence_has_a_question.
Base rate (share of rows labeled as True): 19.0% overall — 17.8% train, 16.8% dev, 21.7% test.
Labels and annotation
Labels were generated with LLM annotators. Krippendorff's alpha for this assertion is 0.867.
Hyperparameters
| Parameter | Value |
|---|---|
| Base model (body) | sentence-transformers/paraphrase-mpnet-base-v2 |
| Head | LogisticRegression |
| Body learning rate | 2e-05 |
| Head learning rate | 0.01 |
| Batch size | 16 (contrastive phase) / 32 (head) |
| Epochs | 10 |
| Max steps | 5000 (contrastive phase) |
| Eval max steps | 100 |
| Seed | 20260904 |
| Mixed precision | enabled on GPU |
Evaluation
Results
| Split | n | Base rate | Precision | Recall | F1 (positive class) | ROC-AUC | Average precision |
|---|---|---|---|---|---|---|---|
| dev | 858 | 16.8% | 0.848 | 0.931 | 0.887 | 0.972 | 0.922 |
| test | 2,146 | 21.7% | 0.905 | 0.940 | 0.922 | 0.984 | 0.954 |
Limitations
- Labels come from LLM annotators, not human coders. Agreement between annotators with Krippendorff's Alpha is 0.867.
- Trained on teacher utterances only. Behaviour on student speech is untested.
How to Use
Message Structure
The model was trained on text built as:
{utterance}
The utterance is passed through as-is.
Running instructions
pip install setfit
from setfit import SetFitModel
model = SetFitModel.from_pretrained("StanfordSCALE/assertion_sentence_has_a_question")
text = 'Now what we did yesterday we watched riveting videos of the rolling cups'
model.predict([text]) # -> array([1]) when the assertion holds
model.predict_proba([text]) # -> [[P(no), P(yes)]]
Citation
@misc{assertion_sentence_has_a_question,
author = {Stanford SCALE Initiative},
title = {Assertion classifier: sentence has a question},
year = {2026},
url = {https://huggingface.co/StanfordSCALE/assertion_sentence_has_a_question}
}