SQuAD 1.1 extractive question answering
Selected method: full_finetuning, epoch 3. The choice was fixed on validation data before test evaluation. This is a measured academic adaptation of BERT, not a production-validated system.
Training data and evaluation
15,000 training / 2,000 internal validation / 2,000 held-out test questions. Reserve 15% of official training article titles for validation; test comes from official validation. No contexts overlap across selected subsets.
Normalized SQuAD 1.1 exact match and answer token F1 (0-100); validation F1 selects the checkpoint.
BERT predicts start/end positions. Max sequence 384, stride 128; preserve token_type_ids. Decode context-only top-20 start/end candidates, maximum answer length 30, best summed score across windows.
The table preserves native scales: AG News, NER and POS metrics are in [0, 1]; QA EM/F1 are in [0, 100]. Training seconds exclude validation, saving and final evaluation.
| method | best_epoch | trainable_parameters | train_seconds | test_exact_match | test_f1 |
|---|---|---|---|---|---|
| partial_finetuning | 3 | 14177282 | 305.736571 | 47.800000 | 61.749065 |
| full_finetuning | 3 | 108893186 | 845.982594 | 73.600000 | 83.277402 |
Optimization and reproducibility
Three epochs, seed 42. AdamW: head LR 1e-3, trainable encoder LR 2e-5, weight decay 0.01, 10% linear warmup, clipping 1.0. Effective batch size 16. A partial method trains only the last two encoder layers and head; lower layers are frozen in evaluation mode. Frozen weights were checked for invariance. Full fine-tuning updates all parameters.
Base model revision: 86b5e0934494bd15c9632b12f734a8a67f723594. Dataset revision: 7b6d24c440a36b6815f21b70d25016731768db1f. Detailed configuration is in training_config.json; the complete comparison is in evaluation.json. The included experiment source and requirements-lock.txt document the original environment. Use a new output directory when reproducing. QA source includes the documented UTF-8 JSON read fix; it did not change any training weights.
Only one seed was evaluated. Small gaps may reflect initialization, dropout and ordering variability; no statistical significance is claimed. Training hardware: RTX 4060 Ti, CUDA bf16. Training runtime is not inference latency.
Intended use and limitations
English answer extraction when an answer is present in a supplied context.
Does not generate new answers or evaluate abstention. Single seed and subset evaluation; SDPA backward can be nondeterministic. Not validated on Spanish or enterprise documents.
Loading the delivered model
from transformers import pipeline
qa = pipeline('question-answering', model='Terrificfantasm/bert-base-uncased-squad-qa')
print(qa(question='Where is the office?', context='The office is in London.',
max_seq_len=384, doc_stride=128, max_answer_len=30))
This is an illustrative loading example. Exact benchmark reproduction uses the included experiment's cross-window decoder, not the pipeline's potentially different postprocessing.
Source terms and references
The upstream BERT checkpoints identify Apache-2.0 licensing. The SQuAD distribution is identified as CC BY-SA 4.0. Dataset terms are separate from the base checkpoint license. This repository does not redistribute the training corpus. It preserves upstream attribution without asserting a new blanket license over all data sources.
- Base checkpoint: https://huggingface.co/google-bert/bert-base-uncased
- Data source: https://huggingface.co/datasets/rajpurkar/squad
- Devlin et al., BERT: https://arxiv.org/abs/1810.04805
- Transformers training: https://huggingface.co/docs/transformers/training
- Official SQuAD scorer: https://github.com/rajpurkar/SQuAD-explorer/blob/master/evaluate-v1.1.py