DeBERTa-v3-base for Open-Domain Multi-Hop Question Answering
A fine-tuned DeBERTa-v3-base model for extractive question answering on the HotpotQA dataset. The model is designed to work as the reader component of a Retrieval-Augmented Question Answering (QA) pipeline, where a dense retriever first retrieves relevant documents and the reader extracts the answer span from the retrieved context.
Model Overview
- Base Model:
microsoft/deberta-v3-base - Task: Extractive Question Answering
- Framework: Hugging Face Transformers
- Dataset: HotpotQA (Distractor Setting)
- Language: English
Unlike generative language models, this model predicts the exact answer span from a retrieved context instead of generating free-form text.
Pipeline Architecture
Question
│
▼
Dense Retriever (BGE-base-en-v1.5 + FAISS)
│
Top-10 Retrieved Documents
│
▼
DeBERTa-v3-base Reader
│
Predicted Answer Span
│
▼
(Optional) LLM Explanation Layer
The retriever identifies relevant Wikipedia articles, while DeBERTa extracts the answer span from the retrieved passages.
Dataset
The model was fine-tuned on the HotpotQA dataset.
Data Preparation
For each question:
- Retrieve the top-10 Wikipedia articles using a dense retriever.
- Assemble the retrieved passages into a context.
- If retrieval misses supporting documents, use the gold supporting passages from HotpotQA.
- Remove:
- Yes/No questions
- Samples where the answer cannot be aligned within the context
Final Dataset Size
| Split | Examples |
|---|---|
| Training | 84,959 |
| Validation | 4,687 |
Retriever
The reader depends on an external dense retriever.
Embedding Model
- BAAI/bge-base-en-v1.5
Vector Database
- FAISS Flat Index
Corpus
- 507,494 Wikipedia articles
Embedding Dimension
- 768
Retrieval Performance
| Metric | Score |
|---|---|
| Recall@3 | 0.7100 |
| Recall@5 | 0.7650 |
| Recall@10 | 0.8150 |
| Recall@20 | 0.8200 |
Training Configuration
| Parameter | Value |
|---|---|
| Base Model | microsoft/deberta-v3-base |
| Optimizer | AdamW |
| Learning Rate | 2e-5 |
| Weight Decay | 0.01 |
| Epochs | 3 |
| Batch Size | 8 |
| Max Sequence Length | 512 |
| Document Stride | 128 |
Tokenization
The model uses the official DeBERTa tokenizer.
Configuration:
truncation="only_second"padding="max_length"max_length=512stride=128return_overflowing_tokens=Truereturn_offsets_mapping=True
Answer spans are aligned using character-level offsets.
Windows that do not contain the answer are assigned start/end position 0.
Evaluation
Performance on the HotpotQA validation set:
| Metric | Score |
|---|---|
| Exact Match (EM) | 60.53 |
| F1 Score | 74.21 |
Comparison with other transformer readers:
| Model | EM | F1 |
|---|---|---|
| DeBERTa-v3-base | 60.53 | 74.21 |
| RoBERTa-base | 60.17 | 73.96 |
| ALBERT-base-v2 | 56.73 | 70.31 |
Example Usage
from transformers import pipeline
qa_pipeline = pipeline(
"question-answering",
model="YOUR_USERNAME/deberta-hotpotqa",
tokenizer="YOUR_USERNAME/deberta-hotpotqa"
)
result = qa_pipeline(
question="Which magazine was started first, Arthur's Magazine or First for Women?",
context=context
)
print(result["answer"])
Intended Use
This model is suitable for:
- Open-domain Question Answering
- Multi-hop Question Answering
- Educational Assistants
- Search Systems
- Retrieval-Augmented QA Pipelines
- Research in Extractive Question Answering
Limitations
This model:
- Requires an external retrieval system.
- Cannot retrieve documents by itself.
- Produces extractive answer spans only.
- Does not generate explanations.
- Performance depends heavily on retrieval quality.
- Was trained only on English Wikipedia.
- Does not support Yes/No questions since they were removed during preprocessing.
Future Improvements
Potential improvements include:
- Hybrid BM25 + Dense Retrieval
- Cross-Encoder Re-ranking
- Paragraph-level indexing
- Iterative Multi-hop Retrieval
- Confidence estimation
- Citation generation
- Integration with an instruction-tuned LLM for grounded answer explanations
License
This project is intended for research and educational purposes. Please ensure compliance with the licenses of the original HotpotQA dataset and the DeBERTa-v3-base model.