GPT-2-large — full fine-tune on SQuAD (decoder-only baseline)
gpt2-large (774M) fully fine-tuned for extractive QA on SQuAD. This is the
decoder-only baseline in an encoder–decoder vs decoder-only paradigm study, included so
the comparison is at near-perfect parameter parity (774M vs T5-large's 737M).
TL;DR: even with all 774M parameters trained, the decoder-only model reaches only F1 0.5041 / EM 0.3516 — and is beaten by a T5-large encoder–decoder that trains just its cross-attention bridge (100.7M) by +0.31 F1 (1.61×), and by a 2.4M-param LoRA adapter by a similar margin. The gap is architectural, not a matter of parameter count.
Results (SQuAD validation)
| Model | Architecture | Trainable / Total | EM | Token F1 |
|---|---|---|---|---|
| GPT-2-large (this model) | decoder-only | 774M / 774M | 0.3516 | 0.5041 |
| T5-small XA-only | enc-dec | 6.3M / 60.5M | 0.5293 | 0.7075 |
| T5-large XA-only | enc-dec | 100.7M / 737M | 0.6406 | 0.8128 |
| T5-large LoRA r=8 | enc-dec | 2.4M / 740M | 0.6445 | 0.8152 |
| T5-large full fine-tune | enc-dec | 737M / 737M | 0.6602 | 0.8162 |
Note that even T5-small (60M) — 13× smaller — outperforms this fully-tuned 774M decoder-only model on the same task.
Eval note: 512-example SQuAD-validation generation subset, 4-beam search.
How to use
Trained with prompt answer question: {question: ... context: ...}\nanswer: and the model
generates the answer continuation. Match this format exactly:
from transformers import GPT2LMHeadModel, AutoTokenizer
repo = "medelharchaoui/gpt2-large-squad"
tok = AutoTokenizer.from_pretrained(repo)
model = GPT2LMHeadModel.from_pretrained(repo)
tok.pad_token = tok.eos_token
question = "What culture do 'bairn' and 'hyem' originate from?"
context = ("'bairn' and 'hyem' are geordie words with origins in scandinavia; barn and hjem "
"are the corresponding modern norwegian and danish words.")
prompt = f"answer question: question: {question} context: {context}\nanswer: "
ids = tok(prompt, return_tensors="pt", truncation=True, max_length=384).input_ids
out = model.generate(ids, num_beams=4, max_new_tokens=16, pad_token_id=tok.eos_token_id)
print(tok.decode(out[0][ids.shape[1]:], skip_special_tokens=True))
Training
| Setting | Value |
|---|---|
| Base model | gpt2-large (774M) |
| Fine-tuning | full (all params) |
| Dataset | rajpurkar/squad, 30,000 train examples |
| Precision | bf16, gradient checkpointing |
| Optimizer steps | 3,000 (batch 2 × grad-accum 16 = eff. batch 32) |
| LR / warmup | 5e-5, 300 warmup, weight decay 0.01 |
| Source / target max len | 384 / 32 |
| Seed | 37 |
| Hardware | 1× NVIDIA RTX 3060 (12 GB), local |
Checkpoint is saved as pytorch_model.bin (streamed tensor-by-tensor to fit constrained
RAM); from_pretrained() loads it automatically.
Limitations
- Provided primarily as a baseline for architecture comparison, not as a recommended QA model — the encoder–decoder checkpoints in this collection are substantially stronger.
- English SQuAD-style extractive QA only; evaluated on a validation subset, not the official SQuAD test server.
Citation
Part of an encoder–decoder vs decoder-only paradigm study (OptimiAI, 2026).