electra-italian-xxl-cased-squad-it
Electra model for (Extractive) Question Answering on Italian texts
Model description
This model has been fine-tuned on squad_it dataset, starting from the pre-trained model dbmdz/electra-base-italian-xxl-cased-discriminator.
It can be used for Extractive Q&A on Italian texts.
Evaluation
| Metric | Value |
|---|---|
| EM | 0.660 |
| F1 | 0.775 |
Usage in Transformers 🤗
Model checkpoints are available for usage in PyTorch. They can be used directly with pipelines as:
from transformers import pipelines
qa = pipeline('question-answering', model='anakin87/electra-italian-xxl-cased-squad-it')
qa(question="Qual è il soprannome di Vasco Rossi?", context="Vasco Rossi, noto anche semplicemente come Vasco e in passato con l'appellativo Blasco (Zocca, 7 febbraio 1952), è un cantautore italiano")
>>> {'score': 0.93, 'start': 80, 'end': 86, 'answer': 'Blasco'}
Usage in Haystack 🚀🚀🚀
With the Haystack NLP framework, you can use this model and create a scalable Question Answering system that works across millions of documents.
For a complete walkthrough, see this notebook.
...
print_answers(prediction, details="medium")
>>> Query: Con chi ha parlato di vaccini il premier Mario Draghi?
Answers:
[ { 'answer': 'Von der Leyen',
'context': " vaccino dell'azienda britannica. Durante la telefonata "
'tra Draghi e Von der Leyen, la presidente della '
'Commissione Ue ha annunciato al presidente del',
'score': 0.9663902521133423},
{ 'answer': 'Ursula Von der Leyen',
'context': 'colloquio telefonico con la presidente della Commissione '
'europea Ursula Von der Leyen. Secondo fonti di Palazzo '
'Chigi, dalla conversazione è emerso ch',
'score': 0.9063920974731445},
{ 'answer': 'Mario Draghi, ha tenuto un lungo discorso alla 76esima '
'Assemblea Generale delle Nazioni Unite',
'context': 'Il presidente del Consiglio, Mario Draghi, ha tenuto un '
'lungo discorso alla 76esima Assemblea Generale delle '
'Nazioni Unite, nella notte italiana. Tant',
'score': 0.5243796706199646}]
Comparison ⚖️
| Model | EM | F1 | Model size (PyTorch) | Architecture |
|---|---|---|---|---|
| it5/it5-large-question-answering | 69.10 | 78.00 | 3.13 GB | encoder-decoder |
| anakin87/electra-italian-xxl-cased-squad-it (this one) | 66.03 | 77.47 | 437 MB | encoder |
| it5/it5-base-question-answering | 66.30 | 76.10 | 990 MB | encoder-decoder |
| it5/mt5-base-question-answering | 66.30 | 75.70 | 2.33 GB | encoder-decoder |
| antoniocappiello/bert-base-italian-uncased-squad-it | 63.80 | 75.30 | 440 MB | encoder |
| luigisaetta/squad_it_xxl_cased_hub1 | 63.95 | 75.27 | 440 MB | encoder |
| it5/it5-efficient-small-el32-question-answering | 64.50 | 74.70 | 569 MB | encoder-decoder |
| mrm8488/bert-italian-finedtuned-squadv1-it-alfa | 62.51 | 74.16 | 440 MB | encoder |
| mrm8488/umberto-wikipedia-uncased-v1-finetuned-squadv1-it | 60.50 | 72.41 | 443 MB | encoder |
| it5/it5-small-question-answering | 61.90 | 71.60 | 308 MB | encoder-decoder |
| it5/mt5-small-question-answering | 56.00 | 66.00 | 1.2 GB | encoder-decoder |
| DrQA-it trained on SQuAD-it | 56.10 | 65.90 | ? | ? |
Training details 🏋️
Hyperparameters
- learning_rate: 2e-05
- batch_size: 8
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 2
- mixed_precision_training: Native AMP
Created by Stefano Fiorucci/anakin87
Made with ♥ in Italy