RoBERTa-ca-CaWikiTC
Overview
- Model type: Language Model
- Architecture: RoBERTa-base
- Language: Catalan
- License: Apache 2.0
- Task: Zero-Shot Text Classification
- Data: CaWikiTC
Model description
The roberta-base-ca-v2-cawikitc (RoBERTa-ca-CaWikiTC) is a Zero-Shot Text Classification model in Catalan created by fine-tuning RoBERTa-base-ca-v2 with a classification dataset, CaWikiTC, reformulated as entailment. This model was developed as part of the experimental research presented in the following paper "Entailment-based Task Transfer for Catalan Text Classification in Small Data Regimes".
Intended uses and limitations
This model can be used for zero-shot text classification in Catalan. It has been trained with a fixed hypothesis template, "Aquest article tracta sobre {}.", and Wikipedia-based articles as premises, and may not generalize well for all use cases.
How to use
from transformers import pipeline
classifier = pipeline("zero-shot-classification", model="ibaucells/RoBERTa-ca-CaWikiTC")
sentence = "'Tierra firme' de Marqués-Marcet inaugura el Festival de cinema de Guadalajara amb Catalunya com a convidada d'honor. El director del film afirma sentir-se orgullós de formar part d'aquesta nova generació de cineastes catalans amb moltes dones directores."
candidate_labels = ["societat", "política", "cultura", "economia"]
template = "Aquest article tracta sobre {}."
output = classifier(sentence, candidate_labels, hypothesis_template=template, multi_label=False)
print(output)
print(f'Predicted class: {output["labels"][0]}')
Limitations and bias
No measures have been taken to estimate the bias and toxicity embedded in the model.
Training
Training data
This model was fine-tuned for the Natural Language Inference (NLI) task on an authomatically Wikipedia-based text classification dataset, CaWikiTC, reformulated as entailment. In the reformulation process, we generated two NLI examples for each text classification instance (text and label): an entailment example and a non-entailment example. In both cases, we employed the text as the premise and utilized a shared template to create the hypothesis ("Aquest article tracta {}."), which was completed with the correct label for the entailment example and a randomly-selected label from the remaining options for the non-entailment example.
Training procedure
The pre-trained Catalan model RoBERTa-base-ca-v2 was fine-tuned with the training data using a learning rate of 3e-5, a batch size of 16, seed 26 and a maximum of 10 epochs. The development set (converted into entailment) was used to select the best checkpoint according to the highest weighted F1 score in the classification task, which was obtained in the first epoch.
Evaluation
Evaluation results
This model was evaluated for the TeCla zero-shot text classification task (without specific fine-tuning for the task) and obtained weighted F1 scores of 75.0 in the coarse-grained task (4 classes) and 49.1 in the fine-grained task (53 classes).
Additional information
Contact
For further information, send an email to either irene.baucells@bsc.es.
License
This work is distributed under a Apache License, Version 2.0.
Funding
This work was funded by the [Departament de la Vicepresidència i de Polítiques Digitals i Territori de la Generalitat de Catalunya](https://politiquesdigitals.gencat.cat/ca/inici/index.html#googtrans(ca|en) within the framework of Projecte AINA.
Citation
Disclaimer
When third parties, deploy or provide systems and/or services to other parties using any of these models (or using systems based on these models) or become users of the models, they should note that it is their responsibility to mitigate the risks arising from their use and, in any event, to comply with applicable regulations, including regulations regarding the use of Artificial Intelligence.
In no event shall the owner and creator of the models be liable for any results arising from the use made by third parties of these models.