distilbert-fin-sentiment
DistilBERT fine-tuned for 3-class financial sentiment classification
(negative / neutral / positive), trained on the Financial PhraseBank
dataset. Built as part of MarketPulse, a portfolio
project comparing a fine-tuned transformer against a TF-IDF + Logistic
Regression baseline. Full build log: see the project's docs/learning-log/.
Intended use
Classifies short financial news sentences/headlines by sentiment, from the viewpoint of an investor (e.g. "costs fell" is scored positive, not negative, despite the negative-sounding word). Intended for portfolio, research, and non-commercial demo use — not for production financial decision-making without further validation, and not for commercial use (see License below).
Training data
Financial PhraseBank
(sentences_50agree subset), ~4,846 sentences annotated by 16 finance
professionals/researchers. Licensed CC-BY-NC-SA-3.0 (non-commercial,
share-alike) — this model inherits that restriction. Loaded from the raw
archive directly, since the dataset's datasets-library loading script is
deprecated. Cleaned before training: 6 exact duplicate sentences collapsed
to one copy each, and 2 sentences with genuinely conflicting labels across
duplicate occurrences were dropped entirely (kept ambiguous ground truth out
rather than arbitrarily picking one label). Final training set: 4,836
sentences, split 70/15/15 (train/validation/test), stratified, seed=42.
Evaluation results (test set, n=726, measured — not estimated)
| Model | Accuracy | Weighted F1 | Macro F1 |
|---|---|---|---|
| TF-IDF + Logistic Regression (baseline) | 0.7631 | 0.7636 | 0.7393 |
| This model (DistilBERT, fine-tuned) | 0.8471 | 0.8475 | 0.8385 |
Per-class F1: negative 0.862, neutral 0.879, positive 0.775.
The dominant error mode is neutral↔positive confusion (see the project's confusion matrix) — negative-class sentences are classified most reliably.
Latency (single request, CPU — not GPU): ~14.9 ms/request, vs ~0.33 ms/request for the baseline (45x slower in relative terms, though still fast in absolute terms for a synchronous API).
Limitations
- Class imbalance: training data is ~59% neutral, 28% positive, 12% negative. Minority-class (negative) metrics are noisier on small eval splits as a result.
- Label noise: annotator agreement was only ≥50% for this subset; 2 sentences with directly conflicting labels were found and excluded during cleaning, which is evidence this dataset has genuine, not just theoretical, annotation disagreement.
- Distribution shift: trained on ~2013-era Nordic financial news sentences. Performance on modern text, tweets, or non-Finnish-sourced financial news is not validated.
- Short-text bias: trained on single sentences (median ~28 tokens); performance on long documents is not validated.
- Uncalibrated confidence: softmax scores are not true probabilities — don't treat them as calibrated uncertainty estimates.
Example usage
from transformers import pipeline
clf = pipeline("text-classification", model="HamimSaad/distilbert-fin-sentiment", top_k=None)
clf("The company reported a sharp drop in quarterly profit.")
License
Model weights: released under the same restrictions as the training data inherits, CC-BY-NC-SA-3.0 (non-commercial, share-alike, attribution required). Contact the Financial PhraseBank authors (Malo, Sinha, Takala, Korhonen, Wallenius) for commercial licensing of the underlying data.