license: mit language:
- en library_name: transformers pipeline_tag: feature-extraction tags:
- bert
- dense-retrieval
- semantic-search
- information-retrieval
- ms-marco
- faiss
BERT Dense Retriever
A dense semantic retrieval model fine-tuned on the MS MARCO Passage Ranking dataset using BERT-base-uncased.
The model encodes natural language queries and passages into dense vector embeddings that can be indexed with FAISS for efficient semantic search.
This repository contains the complete Hugging Face compatible model including tokenizer, configuration, and custom model implementation.
Model Details
Backbone
- BERT-base-uncased
Pooling
- Mean Pooling
Embedding Normalization
- L2 Normalization
Similarity Metric
- Cosine Similarity
Training Objective
- CrossEntropy Loss over the similarity matrix (InfoNCE-style retrieval objective)
Training Configuration
| Parameter | Value |
|---|---|
| Optimizer | AdamW |
| Learning Rate | 2e-5 |
| Batch Size | 32 |
| Epochs | 10 |
| Weight Decay | 0.01 |
| Temperature | 0.05 |
Evaluation Results
| Model | Recall@10 | MRR | nDCG@10 |
|---|---|---|---|
| BERT-base-uncased | 0.4793 | 0.2837 | 0.3301 |
| Fine-tuned Dense Retriever | 0.9693 | 0.8521 | 0.8810 |
The fine-tuned model substantially improves retrieval quality on the evaluation set compared with the untuned BERT-base encoder.
📊 BEIR Evaluation Results
The table below compares the proposed retrieval pipeline against the BM25 baseline reported by the original BEIR benchmark.
| Dataset | Retrieval Strategy | BM25 (BEIR) NDCG@10 | Pipeline NDCG@10 | Recall@10 | Recall@100 | Improvement |
|---|---|---|---|---|---|---|
| FEVER | Hybrid + Cross Encoder | 0.7530 | 0.9791 | 0.9873 | 0.9937 | +0.2261 |
| Quora | Dense + Cross Encoder | 0.7830 | 0.9686 | 0.9858 | 0.9938 | +0.1856 |
| HotpotQA | Dense + Cross Encoder | 0.6030 | 0.8977 | 0.8853 | 0.8960 | +0.2947 |
| FiQA | Dense + Cross Encoder | 0.2361 | 0.7512 | 0.8055 | — | +0.5151 |
| TREC-COVID | Hybrid + Cross Encoder | 0.6559 | 0.6868 | 0.0181 | 0.1110 | +0.0309 |
Note: BM25 scores are taken from the original BEIR benchmark and are included as the lexical retrieval baseline for comparison.
🔍 Key Findings
- Strong zero-shot generalization across multiple retrieval domains.
- Significant improvements over the BEIR BM25 baseline on FEVER, Quora, HotpotQA, and FiQA.
- Hybrid retrieval (BM25 + Dense Retriever) improves retrieval quality for specialized biomedical documents in TREC-COVID.
- Cross-Encoder re-ranking substantially enhances the final ranking quality by leveraging full query-document interactions.
Usage
from transformers import AutoTokenizer, AutoModel
model_name = "Innovatewithapple/bert-dense-retriever"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModel.from_pretrained(
model_name,
trust_remote_code=True,
)
inputs = tokenizer(
"What is deep learning?",
return_tensors="pt"
)
embeddings = model(**inputs)
print(embeddings.shape)
Intended Use
This model is designed for:
- Dense semantic retrieval
- Semantic search
- Question-passage retrieval
- Retrieval-Augmented Generation (RAG)
- Information retrieval research
Source Code
GitHub Repository:
https://github.com/Innovatewithapple/dense-semantic-retrieval
Author
Mihir Vyas