FineMath classifier
Model summary
This is a classifier for evaluating mathematical reasoning and deduction in web pages, fine-tuned from intfloat/multilingual-e5-small. It was developed to filter and curate mathematical content from web datasets and was trained on 1M annotations generated by LLama3-70B-instruct for web samples from Common Crawl, which were extracted using the OpenWebMath text extraction pipeline. To ensure a balanced dataset, we upsampled pages containing mathematical content in the annotations, using a preliminary math classifier on 5M samples.
We used this classifier to build FineMath dataset.
How to use in transformers
To load the FineMath classifier, use the following code:
from transformers import AutoTokenizer, AutoModelForSequenceClassification
tokenizer = AutoTokenizer.from_pretrained("HuggingFaceTB/finemath-classifier")
model = AutoModelForSequenceClassification.from_pretrained("HuggingFaceTB/finemath-classifier")
text = "This is a test sentence."
inputs = tokenizer(text, return_tensors="pt", padding="longest", truncation=True)
outputs = model(**inputs)
logits = outputs.logits.squeeze(-1).float().detach().numpy()
score = logits.item()
result = {
"text": text,
"score": score,
"int_score": int(round(max(0, min(score, 5)))),
}
print(result)
# {'text': 'This is a test sentence.', 'score': 0.07964489609003067, 'int_score': 0}
Training
The classifier was trained on 1M pairs of web samples and their scores from 0 to 5, generated by Llama3. The samples were annotated based on their usefulness for studying mathematics with 0 being not educational or containing matematical content and 5 being outstanding for mathetmatics education.
Below is the prompt used for LLama3 annotations:
We added a classification head with a single regression output to intfloat/multilingual-e5-small and trained the model for 20 epochs with a learning rate of 3e-4. During training, the embedding and encoder layers were frozen to focus on the classification head. The model achieved an F1 score of 87% when converted to a binary classifier using a score threshold of 3.
Training Details:
- Model: intfloat/multilingual-e5-smallwith a classification head
- Dataset: 1M samples from Llama3 annotations
- Epochs: 20
- Learning Rate: 3e-4
- Evaluation Metric: F1 score
Evaluation: The model achieves the following results on the evaluation set:
- Loss: 0.4478
- Precision: 0.8771
- Recall: 0.8769
- F1 Macro: 0.8770
- Accuracy: 0.8770