Smart MCQ Solver - DeBERTa-v3-large
Five-option multiple-choice QA over science and philosophy questions.
Full fine-tune of microsoft/deberta-v3-large with AutoModelForMultipleChoice.
Results
| Metric | Value |
|---|---|
| 3-fold grouped CV MAP@3 | 0.7567 |
| Held-out MAP@3 (this artifact) | 0.7944 |
| Held-out accuracy | 0.6912 |
| Random MAP@3 baseline | 0.3667 |
Cross-validation is GroupKFold grouped by normalised prompt, because the dataset
contains roughly eight near-duplicate phrasings of every question; a random split
would score memorisation rather than generalisation.
Loading
Always pass dtype=torch.float32 explicitly.
import torch
from transformers import AutoTokenizer, AutoModelForMultipleChoice
tok = AutoTokenizer.from_pretrained("SriragData/smart-mcq-deberta-v3-large")
model = AutoModelForMultipleChoice.from_pretrained(
"SriragData/smart-mcq-deberta-v3-large", dtype=torch.float32).eval()
question = "What is the capital of France?"
options = ["Berlin", "Madrid", "Paris", "Rome", "Lisbon"]
enc = tok([question]*5, options, truncation=True, max_length=256,
padding=True, return_tensors="pt")
with torch.no_grad():
logits = model(input_ids=enc["input_ids"].unsqueeze(0),
attention_mask=enc["attention_mask"].unsqueeze(0)).logits[0]
print(options[int(logits.argmax())])
Why the dtype matters
The upstream microsoft/deberta-v3-* checkpoints store fp16 weights, and
transformers 5.x honours the dtype recorded in the checkpoint. Loaded in fp16,
DeBERTa's disentangled attention saturates, the attention softmax collapses to
one-hot, and the model returns an identical score for every option - MAP@3 drops
to roughly 0.36, the random baseline, with no error raised. This repository stores
fp32 weights and records float32 in config.json, but passing dtype explicitly
costs nothing and removes the risk entirely.
Training
| Setting | Value |
|---|---|
| Precision | full fp32 (no mixed precision) |
| Batch size | 1, gradient accumulation 16 (effective 16) |
| Epochs | {EPOCHS} |
| Learning rate | 8e-6, cosine schedule, warmup ratio 0.1 |
| Max length | 256 |
| Gradient checkpointing | enabled |
| Checkpointing | disabled (save_strategy="no") |
save_strategy="no" is deliberate: on some transformers versions the checkpoint
save/reload cycle renames LayerNorm gamma/beta and silently resets every
LayerNorm to its initial values after training completes.
Limitations
- Trained on 2,000 rows covering 252 unique questions. Coverage is narrow.
- The evaluation set overlaps the training set heavily, so headline scores reflect that deployment condition. The leakage-free estimate for this project is 0.6817.
- Closed-book only: no retrieval, no citation, no abstention. It will answer confidently on questions it knows nothing about.