SQuAD v1.1 extractive question answering — full adaptation
Part of U2T01: Adapting BERT for NLP tasks. One of four models, each one a
different head on the same bert-base-uncased body, each delivered with
the adaptation method that its own measurements justified.
What this model does
Predicts start and end token positions of the answer span.
How it was adapted
Full fine-tuning. Every encoder layer and the head were trained. The embedding matrix was kept frozen (23M of BERT's 110M parameters), which changes nothing measurable at these dataset sizes and saves optimiser memory.
| Base model | bert-base-uncased |
| Adaptation | full |
| Trainable parameters | 85,056,002 of 108,893,186 (78.11%) |
| Encoder layers trained | 12 of 12 |
| Head learning rate | 0.001 |
| Encoder learning rate | 3e-05 |
| Epochs | 1 |
| Batch size | 16 |
| Max sequence length | 384 |
| Seed | 42 |
| Training time | 1.0 min on Tesla T4 |
A freshly initialised head and pretrained encoder layers were given separate learning rates through two optimiser parameter groups; a single learning rate for both silently cripples this kind of run.
Training data
SQuAD v1.1 (rajpurkar/squad).
Training examples used: 2000.
Sub-sampled to ~15k training examples as the assignment specifies.
Evaluation
| Metric | Test | Validation |
|---|---|---|
| exact_match | 37.30 | 29.50 |
| f1 | 47.74 | 43.68 |
Metric choice is argued in the project report. In short: entity-level F1 for
NER because ~83% of CoNLL tokens are O and token accuracy would flatter a
model that predicts nothing; token accuracy for POS because there is no
dominant class; EM and F1 for SQuAD because each covers the other's blind
spot.
Why this method, and what lost
| Run | Method | Trainable params | f1 (test) |
|---|---|---|---|
qa_full_mini | full | 85,056,002 | 47.74 |
qa_partial_mini | partial | 28,353,026 | 30.78 |
Decision rule: best primary metric on test, and where two methods fall inside a ±1 point band — smaller than the ±1–3 points a different random seed moves results by — the cheaper one wins.
Intended use and limitations
Intended for research and coursework: benchmarking adaptation strategies and as a starting point for related English tasks.
Not intended for production decisions about people. Concretely:
- English only, and the training text is narrow. AG News is 2004 newswire; CoNLL-2003 is Reuters newswire, so it under-performs on social media, names outside Western European conventions, and any entity that became prominent after 2003. UD EWT is web English (blogs, reviews, emails). SQuAD contexts are Wikipedia paragraphs, and the model always returns a span — SQuAD v1.1 contains no unanswerable questions, so it cannot say "I don't know".
- Single run, single seed. Re-running with a different seed moves results by roughly ±1–3 points. Differences smaller than that are not meaningful.
- Inherited bias.
bert-base-uncasedcarries the associations of its pretraining corpus, and nothing here mitigates them. - Sub-sampled training data, so these numbers sit below published full-data results. Every adaptation method on this task saw exactly the same data, so the comparison that chose this method is unaffected.
Reproduce it
git clone <this project>
pip install -r requirements.txt
python -m u2t01.cli run configs_mini/qa_full.yaml
Fixed seed (42), pinned dependencies, and one YAML file per experiment: config + seed + requirements.txt fully determine a run.
References
- Devlin et al., BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, 2018 — https://arxiv.org/abs/1810.04805 (§5.3 is the feature-based vs fine-tuning comparison this project mirrors)
- Tunstall, von Werra & Wolf, Natural Language Processing with Transformers, O'Reilly, chapters 1–3
- Hugging Face, Fine-tune a pretrained model — https://huggingface.co/docs/transformers/training
- Dataset:
rajpurkar/squad
Card generated 2026-09-27 from results/qa_full_mini.json.