Model Card for Finetuned FinBERT on Market-Based Facts
This LLM is fine-tuned on market reactions to events. By utilizing market-based data, it avoids human biases present in traditional annotation methods.
Outperform distilroberta fintuned for finance
- +17% precision
- +13% recall
Outperform GPT-4 zero-shot learning
- +10% precision
- +4% recall
Model Overview
This model is a finetuned version of FinBERT specifically tailored for the analysis of market-based facts. It was trained using headlines from leading market information providers, ensuring that the training data is both highly reliable and reflective of the information financial professionals react to. The model's objective is to classify financial headlines based on the market's return a short time after the news is released.
The finetuning process involved a distilled version of the RoBERTa-base model, following the same training methodology as DistilBERT. This distillation results in a model that is not only efficient but also retains a significant portion of the original model's predictive power. This model is case-sensitive, recognizing distinctions between uppercase and lowercase letters, which can be critical for understanding the context of financial headlines.
Model Specifications
- Base Model: FinBERT, distilled from RoBERTa-base
- Problem Type: Text Classification
- Training Data: Financial headlines from leading market information providers
- Parameters: 82 million (compared to 125M for RoBERTa-base)
- Layers: 6
- Dimension: 768
- Heads: 12
Performance Metrics
The model's performance was evaluated on a validation set, yielding the following metrics:
| Metric | Value |
|---|---|
| Loss | 0.9394 |
| F1 Macro | 0.4892 |
| F1 Micro | 0.5510 |
| F1 Weighted | 0.5189 |
| Precision Macro | 0.5241 |
| Precision Micro | 0.5510 |
| Precision Weighted | 0.5323 |
| Recall Macro | 0.5048 |
| Recall Micro | 0.5510 |
| Recall Weighted | 0.5510 |
| Accuracy | 0.5510 |
These metrics indicate how the model performs in terms of precision, recall, and accuracy across different averaging methods (macro, micro, and weighted).
Training and Evaluation
This model was trained using AutoTrain, which automates the training process to optimize performance on the given task. The distillation process was conducted to create a model that is both lightweight and efficient, maintaining high performance while being twice as fast as the original RoBERTa-base model.
The evaluation was conducted on a set of financial headlines not seen by the model during training, ensuring that the reported metrics reflect the model's ability to generalize to new, unseen data.
This model has been developed after publishing in the Risk Forum 2024 conference a paper that can be found here (https://arxiv.org/abs/2401.05447).