rein5/bert-base-uncased-finetuned-spoken-squad
Model Description
This model is an extractive question-answering system fine-tuned from the bert-base-uncased model specifically for the spoken language domain. It leverages the Spoken-SQuAD dataset, which introduces real-world challenges of understanding spoken content, such as dealing with different levels of word error rates (WERs).
Intended Use
The model is intended for use in natural language processing applications requiring understanding and answering questions from spoken language text. It is especially useful for scenarios involving transcripts of spoken conversations, interviews, or any spoken content converted to text.
Training Data
The model was trained on the Spoken-SQuAD dataset, a version of the SQuAD dataset adapted to simulate spoken language by incorporating noise in the form of word error rates. The dataset features various levels of WER to reflect different noise conditions commonly encountered in spoken language processing.
Dataset source: Spoken-SQuAD Dataset Repository
Training Procedure
The model was fine-tuned on the Spoken-SQuAD dataset starting from the bert-base-uncased model checkpoint. During training, we employed a batch size of 20, a learning rate of 5e-5, and trained the model for 2 epochs using the AdamW optimizer.
Evaluation Results
The model was evaluated on three versions of the Spoken-SQuAD dataset, each representing different levels of noise (WER):
- No noise (22.73% WER)
- Exact Match: 63.99%
- F1 Score: 74.15%
- Noise V1 (44.22% WER)
- Exact Match: 40.35%
- F1 Score: 55.2%
- Noise V2 (54.82% WER)
- Exact Match: 28.52%
- F1 Score: 42.24%
How to Use
Here is how to load and use the model:
from transformers import AutoModelForQuestionAnswering, AutoTokenizer
model_name = "rein5/bert-base-uncased-finetuned-spoken-squad"
model = AutoModelForQuestionAnswering.from_pretrained(model_name)
tokenizer = AutoTokenizer.from_pretrained(model_name)
# Example of how to use the model to answer questions.
question = "What is the model used for?"
context = "This model is used for understanding and answering questions from spoken language text."
inputs = tokenizer(question, context, add_special_tokens=True, return_tensors="pt")
input_ids = inputs["input_ids"].tolist()[0]
outputs = model(**inputs)
answer_start_scores = outputs.start_logits
answer_end_scores = outputs.end_logits
# Get the most likely beginning of answer with the argmax of the score
answer_start = torch.argmax(answer_start_scores)
# Get the most likely end of answer with the argmax of the score
answer_end = torch.argmax(answer_end_scores) + 1
answer = tokenizer.convert_tokens_to_string(tokenizer.convert_ids_to_tokens(input_ids[answer_start:answer_end]))
print("Answer:", answer)
Source and Contributions
The training code and further details are available in the GitHub repository: spoken-squad-language-model. Contributions to both the model and the dataset are welcome.