cvm-bertimbau-sentence-transformer
Fine-tuned BERTimbau sentence transformer for dense retrieval over Brazilian public company filings (CVM ITR/DFP). Part of the CVM Filing Intelligence System.
Training
| Parameter | Value |
|---|---|
| Base model | neuralmind/bert-base-portuguese-cased |
| Loss | MultipleNegativesRankingLoss |
| Training pairs | 14,500 (adjacent same-section chunk pairs from 686 CVM filings) |
| Epochs | 10 |
| Batch size | 16 (effective 64 with gradient accumulation ×4) |
| Mixed precision | fp16 |
| Max sequence length | 256 tokens |
| Hardware | NVIDIA RTX A1000 (6 GB VRAM), ~2 hours |
| Initial loss | 2.201 (step 50) |
| Final loss | 0.115 (step 2,270) |
Training data: 97,138 management commentary chunks from 49 B3 large-cap companies (Petrobras, Vale, Itaú, Bradesco, Ambev, etc.), 2022–2025. Pairs are adjacent paragraphs within the same section of the same filing.
Retrieval Results
Evaluated on 94 synthetic queries over the 97,138-chunk corpus (dense-only configuration):
| Metric | Value |
|---|---|
| Recall@5 | 0.057 |
| Recall@10 | 0.071 |
| MRR | 0.100 |
| NDCG@10 | 0.063 |
Note: The model underperforms BM25 on query–document retrieval because it was fine-tuned with doc–doc contrastive pairs. Query–doc performance improves significantly with GPL (Generative Pseudo Labeling) fine-tuning using synthetic query–chunk pairs.
Usage
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("conderafael/cvm-bertimbau-sentence-transformer")
# Encode a single passage
embeddings = model.encode(["Receita líquida cresceu 12% no trimestre"])
# Encode a batch
texts = [
"O EBITDA ajustado atingiu R$ 4,2 bilhões no 3T24.",
"A Companhia mantém posição conservadora de hedge cambial.",
]
embeddings = model.encode(texts, normalize_embeddings=True)
print(embeddings.shape) # (2, 768)
Limitations
- Trained on Portuguese-language financial filings only; degrades on other domains.
- Max sequence length 256 tokens; longer passages are truncated.
- Query-time performance is below doc-time performance due to training objective mismatch (doc–doc pairs vs. query–doc retrieval).