ModernBERT-QnA-base-squad 🚀

Welcome to the ModernBERT-QnA-base-squad repository! This repository hosts the fine-tuned ModernBERT model for Question Answering tasks. The model achieves impressive performance on the SQuAD dataset, making it an excellent choice for extractive question-answering applications.

Model Overview 🌟

• Model ID: rankyx/ModernBERT-QnA-base-squad
• Base Model: answerdotai/ModernBERT-base
• Dataset: SQuAD
• Evaluation Metrics:
  • F1 Score: 92.59
  • Exact Match: 86.45
• Training Framework: Hugging Face Transformers

Read more about ModernBERT's capabilities in this Hugging Face blog post.

Usage 💻

The following example demonstrates how to use the fine-tuned model for question answering using the Hugging Face pipeline.

For now, you have to install a specific transformers fork, until the official PR will be merged.

> pip uninstall transformers -y
> git clone https://github.com/bakrianoo/transformers.git
> cd transformers && git checkout feat-ModernBert-QnA-Support && pip install -e .

Quick Start

from transformers.models.modernbert.modular_modernbert import ModernBertForQuestionAnswering
from transformers import AutoTokenizer, pipeline

# Load the model and tokenizer
model_id = "rankyx/ModernBERT-QnA-base-squad"
model = ModernBertForQuestionAnswering.from_pretrained(model_id)
tokenizer = AutoTokenizer.from_pretrained(model_id)

# Initialize the question-answering pipeline
question_answerer = pipeline("question-answering", model=model, tokenizer=tokenizer)

# Example input
question = "How many parameters does BLOOM contain?"
context = "BLOOM has 176 billion parameters and can generate text in 46 natural languages and 13 programming languages."

# Get the answer
result = question_answerer(question=question, context=context)
print(result)

Sample Output

{'score': 0.7719728946685791, 'start': 9, 'end': 21, 'answer': '176 billion'}

Performance Demonstrations 🔥

Example 1: Short Context

from transformers import pipeline

model_id = "rankyx/ModernBERT-QnA-base-squad"
question_answerer = pipeline("question-answering", model=model_id)

# Input
question = "What is the capital of France?"
context = "France's capital is Paris, known for its art, gastronomy, and culture."

# Get the answer
result = question_answerer(question=question, context=context)
print(result)

Predicted Answer:

{'score': 0.9913662075996399, 'start': 19, 'end': 25, 'answer': ' Paris'}

Example 2: Long Context

from transformers import pipeline

model_id = "rankyx/ModernBERT-QnA-base-squad"
question_answerer = pipeline("question-answering", model=model_id)

# Input
question = "What are the major achievements of Isaac Newton?"
context = """
Isaac Newton, born on January 4, 1643, was an English mathematician, physicist, astronomer, and author. He is widely recognized as one of the greatest mathematicians and most influential scientists of all time. Newton made groundbreaking contributions to many fields, including the laws of motion and universal gravitation. He also developed calculus independently, providing the mathematical foundation for classical mechanics. Additionally, Newton's work in optics led to the invention of the reflecting telescope.
"""

# Get the answer
result = question_answerer(question=question, context=context)
print(result)

Predicted Answer:

{'score': 0.5126065015792847, 'start': 278, 'end': 323, 'answer': ' the laws of motion and universal gravitation'}

Example 3: Extremely Long Context

from transformers import pipeline

model_id = "rankyx/ModernBERT-QnA-base-squad"
question_answerer = pipeline("question-answering", model=model_id)

# Input
question = "Describe the primary focus of the United Nations."
context = """
The United Nations (UN) is an international organization founded in 1945. It is currently made up of 193 Member States. The mission and work of the United Nations are guided by the purposes and principles contained in its founding Charter. The UN is best known for its peacekeeping, peacebuilding, conflict prevention, and humanitarian assistance. It also works on promoting sustainable development, protecting human rights, upholding international law, and delivering humanitarian aid. Through its various specialized agencies, funds, and programs, the UN addresses issues ranging from health to education to climate change.
"""

# Get the answer
result = question_answerer(question=question, context=context)
print(result)

Predicted Answer:

{'score': 0.08445773273706436, 'start': 269, 'end': 347, 'answer': ' peacekeeping, peacebuilding, conflict prevention, and humanitarian assistance'}

Fine-tuning Process ⚙️

The model was fine-tuned using the Hugging Face Transformers library with the official script for question answering.

Command Used for Fine-tuning

python run_qa.py \
  --model_name_or_path "answerdotai/ModernBERT-base" \
  --dataset_name squad \
  --do_train \
  --do_eval \
  --overwrite_output_dir \
  --per_device_train_batch_size 25 \
  --per_device_eval_batch_size 20 \
  --eval_strategy="steps" \
  --save_strategy="epoch" \
  --logging_steps 50 \
  --eval_steps 500 \
  --learning_rate 3e-5 \
  --warmup_ratio 0.1 \
  --weight_decay 0.01 \
  --doc_stride 128 \
  --max_seq_length 384 \
  --max_answer_length 128 \
  --num_train_epochs 2 \
  --run_name="ModernBERT-QnA-base-squad" \
  --output_dir="/path/to/output/directory"

If you have multiple GPUs and getting the error: RuntimeError: Detected that you are using FX to symbolically trace a dynamo-optimized function try this:

accelerate launch run_qa.py \
...other parameters

Results 📊

Evaluation Metrics

• F1 Score: 92.59
• Exact Match: 86.45
• Training Loss: 0.860

License 📜

This model is licensed under the Apache 2.0 License. See LICENSE for details.

Citation ✍️

If you use this model in your research, please cite it as follows:

@misc{rankyx2024modernbertqna,
  title={ModernBERT-QnA-base-squad},
  author={Abu Bakr},
  year={2024},
  howpublished={\url{https://huggingface.co/rankyx/ModernBERT-QnA-base-squad}}
}

Acknowledgments 🙌

Special thanks to the Hugging Face team for providing the tools and resources to fine-tune and deploy this model.

Feedback and Contributions 🤝

We welcome feedback and contributions! Please feel free to open an issue or submit a pull request.