BERT-base-uncased Fine-Tuned for Extractive Question Answering (SQuAD v1.1)

This repository contains the weights and tokenizer for bert-base-uncased fully fine-tuned on SQuAD v1.1 (Stanford Question Answering Dataset) for extractive question answering.

The model was adapted using end-to-end full fine-tuning with a differential two-learning-rate optimizer scheme as part of an academic comparative study on BERT adaptation paradigms.

Model Details

• Base Architecture: BERT-base (bert-base-uncased)
• Model Type: BertForQuestionAnswering
• Layers: 12 Transformer encoder blocks
• Hidden Dimension: 768
• Attention Heads: 12
• Parameters: 108,893,186 trainable parameters (~110M total)
• Framework: PyTorch & Hugging Face transformers

Training Data & Preprocessing

The model was trained on a deterministic subsample of SQuAD v1.1 (rajpurkar/squad):

• Subsample Size: 15,000 question-context pairs sampled deterministically with seed 42 from the training split.
• Sliding Window Strategy: Long contexts exceeding BERT's sequence limit were tokenized using a sliding window:
  • Maximum sequence length: 384 tokens
  • Document stride: 128 tokens
  • Truncation: only_second (context tokens truncated, questions kept intact)
  • Unanswerable / out-of-slice spans: Routed to index [CLS] (position 0)
• Windowed Features: The 15,000 source examples expanded to 15,141 training feature windows.

Evaluation Results

Evaluation was performed on the complete official SQuAD v1.1 validation split (10,570 examples, spanning 10,753 feature windows) using an optimized post-processing pipeline (top-15 start/end logit pruning, maximum answer span length of 30 tokens):

Intended Use

• Task: Extractive question answering in English. Given a question and a reference passage, the model extracts the continuous sub-string (span) that answers the question.
• Input: A query string (question) and a text passage (context).

Quickstart / How to Use

Depending on your version of transformers, choose one of the two options below:

Option A: Using pipeline (with transformers < 5)

Note: If you are using this method, ensure your environment has pip install "transformers<5".

from transformers import pipeline

# Initialize the pipeline directly from repository
qa_model = pipeline(
    task="question-answering",
    model="elisabetconese/Extractive-QA_BERT_body"
)

question = "Where was Super Bowl 50 played?"
context = "Super Bowl 50 took place at Levi's Stadium in Santa Clara, California."

# Run inference
prediction = qa_model(question=question, context=context)

print(f"Answer: {prediction['answer']}")

Option B: Direct Model Inference (transformers >= 5 or manual control)

Note: In transformers >= 5.0.0, the legacy question-answering pipeline was deprecated/removed. Use this approach to run inference directly with PyTorch tensors.

import torch
from transformers import AutoTokenizer, AutoModelForQuestionAnswering

model_id = "elisabetconese/Extractive-QA_BERT_body"

# Load tokenizer and model directly
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForQuestionAnswering.from_pretrained(model_id)

context = "Python was created by Guido van Rossum and first released in 1991."
question = "Who created Python?"

# Tokenize inputs, returning PyTorch tensors
inputs = tokenizer(question, context, return_tensors="pt")

# Perform inference without calculating gradients
with torch.no_grad():
    outputs = model(**inputs)

# Extract highest probability tokens for start and end positions
start_index = torch.argmax(outputs.start_logits)
end_index = torch.argmax(outputs.end_logits) + 1

# Decode the predicted tokens back to a string
predict_answer_tokens = inputs.input_ids[0, start_index : end_index]
answer = tokenizer.decode(predict_answer_tokens, skip_special_tokens=True)

print(f"Extracted Answer: {answer}")

Limitations

1. Extractive Only: The model identifies spans verbatim within the context passage. It cannot synthesize, paraphrase, or generate abstractive responses.
2. SQuAD v1.1 Constraint (Answer-Present): SQuAD v1.1 assumes all questions have an answer in the provided text. The model does not support unanswerable question detection.
3. Context Length Truncation: While sliding window inference allows covering long documents, answer spans that cross window seams or exceed 384 tokens without stride coverage may suffer degradation.
4. Language & Casing: The model was pretrained on lowercased English text (uncased) and is unsuitable for multilingual texts or applications sensitive to letter casing.

References

1. Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv:1810.04805 §5.3.
2. Rajpurkar, P., Zhang, J., Lopyrev, K., and Liang, P. (2016). SQuAD: 100,000+ Questions for Machine Comprehension of Text. arXiv:1606.05250.
3. Hugging Face Documentation. Fine-tuning a pretrained model. Hugging Face Transformers Training Documentation.