NLPForUA/mdeberta-v3-ua-squad-reader
This model is a Ukrainian extractive Question Answering model based on
microsoft/mdeberta-v3-base.
It was trained and published as an educational artifact for Lab 3: Open-domain Question Answering from the Odesа Polytechnic National University ML assignments repository.
In the laboratory assignment, the model is used as the Reader component of a two-stage ODQA system:
- a Retriever selects relevant text passages;
- this model extracts an answer span from a selected passage.
Training
The model was fine-tuned on the Ukrainian SQuAD dataset
FIdo-AI/ua-squad.
The dataset was split by context rather than by individual question-answer pairs to avoid placing questions based on the same source context into different splits.
The resulting split contained:
| Split | QA examples |
|---|---|
| Train | 11,080 |
| Validation | 1,339 |
| Test | 1,440 |
Training configuration:
- Base model:
microsoft/mdeberta-v3-base - Epochs: 3
- Learning rate:
3e-5 - Train batch size: 16
- Evaluation batch size: 16
- Weight decay: 0.01
- Maximum sequence length: 384
- Document stride during training preprocessing: 128
- Model selection criterion: validation loss
Validation loss by epoch:
| Epoch | Validation loss |
|---|---|
| 1 | 1.1212 |
| 2 | 1.0247 |
| 3 | 1.1246 |
The checkpoint from epoch 2 was selected as the final model because it had the lowest validation loss.
Evaluation
The model was evaluated on the held-out test split using SQuAD 2.0 metrics.
| Metric | Score |
|---|---|
| Exact Match | 58.26 |
| F1 | 71.99 |
| HasAns Exact Match | 57.43 |
| HasAns F1 | 74.71 |
| NoAns F1 | 61.49 |
Usage
from transformers import pipeline
model_id = "NLPForUA/mdeberta-v3-ua-squad-reader"
qa = pipeline(
"question-answering",
model=model_id,
tokenizer=model_id,
)
result = qa(
question="Яке місто є столицею України?",
context="Київ є столицею України.",
handle_impossible_answer=True,
)
print(result)
Intended use
The model is primarily published for educational use. It can also be used as a general Ukrainian extractive QA model.
Limitations
The model performs extractive Question Answering: it selects an answer span from the provided context rather than generating an answer from its own knowledge.
The reported results correspond to the dataset split and evaluation procedure used in the laboratory assignment and are not intended as a state-of-the-art benchmark claim.