FinDeBERTa: Multi-Label Financial Event Classifier
FinDeBERTa is a fine-tuned DeBERTa-v3-Large model for multi-label financial event classification. It predicts one or more event types from financial news headlines with state-of-the-art performance.
Model Details
- Base Model: microsoft/deberta-v3-large
- Task: Multi-label text classification
- Training: Fine-tuned with Focal Loss and per-class threshold optimization
Event Labels
The model classifies text into 18 financial event types:
["CSR/Brand", "Deal", "Dividend", "Employment", "Expense", "Facility",
"FinancialReport", "Financing", "Investment", "Legal", "Macroeconomics",
"Merger/Acquisition", "Product/Service", "Profit/Loss", "Rating", "Revenue",
"SalesVolume", "SecurityValue"]
Usage
Optimized Usage (with per-class thresholds) (Recommended)
For best performance, use the per-class optimized thresholds:
from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch
import numpy as np
from huggingface_hub import hf_hub_download
model_name = "ritessshhh/FinDeBERTa"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSequenceClassification.from_pretrained(model_name)
# Download per-class thresholds
thresholds_path = hf_hub_download(repo_id=model_name, filename="thresholds.npy")
thresholds = np.load(thresholds_path)
text = "Tesla to acquire a battery startup in a 400 million dollar deal."
inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=128)
with torch.no_grad():
outputs = model(**inputs)
probs = torch.sigmoid(outputs.logits)[0].cpu().numpy()
# Apply per-class thresholds
predictions = [
{"label": model.config.id2label[i], "score": float(prob)}
for i, prob in enumerate(probs) if prob >= thresholds[i]
]
# Sort by score
predictions = sorted(predictions, key=lambda x: x["score"], reverse=True)
print(predictions)
# Output: [{"label": "Merger/Acquisition", "score": 0.98}, ...]
Basic Usage (with default threshold)
from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch
import numpy as np
model_name = "ritessshhh/FinDeBERTa"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSequenceClassification.from_pretrained(model_name)
text = "Tesla to acquire a battery startup in a 400 million dollar deal."
inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=128)
with torch.no_grad():
outputs = model(**inputs)
probs = torch.sigmoid(outputs.logits)[0].cpu().numpy()
# Using default threshold of 0.5
threshold = 0.5
predictions = [
{"label": model.config.id2label[i], "score": float(prob)}
for i, prob in enumerate(probs) if prob >= threshold
]
print(predictions)
Training Details
- Loss Function: Focal Loss (gamma=2.0) with dampened class weights
- Optimizer: AdamW with cosine learning rate scheduling
- Batch Size: 8 (with gradient accumulation steps=2)
- Epochs: 10
- Learning Rate: 2e-5
- Weight Decay: 0.02
Performance
| Metric | Score |
|---|---|
| Macro F1 | 0.692 |
| Micro F1 | 0.691 |
| Precision (Macro) | 0.738 |
| Recall (Macro) | 0.691 |
| Exact Match Ratio | 0.532 |
Per-Class Performance
| Label | F1 Score | Precision | Recall |
|---|---|---|---|
| Dividend | 1.000 | 1.000 | 1.000 |
| Employment | 0.923 | 0.857 | 1.000 |
| Merger/Acquisition | 0.892 | 0.967 | 0.829 |
| Profit/Loss | 0.833 | 0.824 | 0.843 |
| SecurityValue | 0.829 | 0.843 | 0.815 |
| Rating | 0.790 | 0.800 | 0.780 |
| Revenue | 0.780 | 0.800 | 0.762 |
| SalesVolume | 0.748 | 0.833 | 0.678 |
| Financing | 0.714 | 0.714 | 0.714 |
| Deal | 0.696 | 0.889 | 0.571 |
Limitations
- Optimized for financial news headlines (short text)
- May not generalize well to other domains
- Performance varies by event type (rare events like "Facility" have lower F1)
Citation
If you use this model, please cite:
@misc{findeberta2024,
author = {ritessshhh},
title = {FinDeBERTa: Multi-Label Financial Event Classifier},
year = {2024},
publisher = {HuggingFace},
howpublished = {\url{https://huggingface.co/ritessshhh/FinDeBERTa}}
}