X-GENRE classifier - multilingual text genre classifier

Text classification model based on xlm-roberta-base and fine-tuned on a multilingual manually-annotated X-GENRE genre dataset. The model can be used for automatic genre identification, applied to any text in a language, supported by the xlm-roberta-base.

The details on the model development, the datasets and the model's in-dataset, cross-dataset and multilingual performance are provided in the paper Automatic Genre Identification for Robust Enrichment of Massive Text Collections: Investigation of Classification Methods in the Era of Large Language Models (Kuzman et al., 2023).

The model can also be downloaded from the CLARIN.SI repository.

If you use the model, please cite the paper:

@article{kuzman2023automatic,
  title={Automatic Genre Identification for Robust Enrichment of Massive Text Collections: Investigation of Classification Methods in the Era of Large Language Models},
  author={Kuzman, Taja and Mozeti{\v{c}}, Igor and Ljube{\v{s}}i{\'c}, Nikola},
  journal={Machine Learning and Knowledge Extraction},
  volume={5},
  number={3},
  pages={1149--1175},
  year={2023},
  publisher={MDPI}
}

AGILE - Automatic Genre Identification Benchmark

We set up a benchmark for evaluating robustness of automatic genre identification models to test their usability for the automatic enrichment of large text collections with genre information. The benchmark comprises 11 European languages and two test datasets. You are welcome to submit your entry at the benchmark's GitHub repository.

The model outperforms all other technologies, including GPT models that are used in a zero-shot scenario.

Results on the English test dataset (EN-GINCO):

Results on the multilingual test dataset (X-GINCO), comprising instances in Albanian, Catalan, Croatian, Greek, Icelandic, Macedonian, Maltese, Slovenian, Turkish, and Ukrainian:

(The multilingual test dataset is easier than the English one, as the vague label "Other" and instances that were predicted with a confidence score below 0.80 were not included in the test dataset.)

For language-specific results, see the AGILE benchmark.

Intended use and limitations

Usage

Use example for prediction on a dataset, using batch processing, is available via Google Collab.

An example of preparing data for genre identification and post-processing of the results can be found here where we applied X-GENRE classifier to the English part of MaCoCu parallel corpora.

For reliable results, genre classifier should be applied to documents of sufficient length (the rule of thumb is at least 75 words). Predictions with confidence scores below 0.8 should not be used. In our experience annotating large web corpora in various European languages, this occurs in approximately 4% of cases. We label these instances as "Mix." Furthermore, the label "Other" can be used as another indicator of low confidence of the predictions, as it often indicates that the text does not have enough features of any genre, and these predictions can be discarded as well.

Use examples

from simpletransformers.classification import ClassificationModel
model_args= {
            "num_train_epochs": 15,
            "learning_rate": 1e-5,
            "max_seq_length": 512,
            "silent": True
            }
model = ClassificationModel(
    "xlmroberta", "classla/xlm-roberta-base-multilingual-text-genre-classifier", use_cuda=True,
    args=model_args
    
)
predictions, logit_output = model.predict(["How to create a good text classification model? First step is to prepare good data. Make sure not to skip the exploratory data analysis. Pre-process the text if necessary for the task. The next step is to perform hyperparameter search to find the optimum hyperparameters. After fine-tuning the model, you should look into the predictions and analyze the model's performance. You might want to perform the post-processing of data as well and keep only reliable predictions.", 
                                        "On our site, you can find a great genre identification model which you can use for thousands of different tasks. With our model, you can fastly and reliably obtain high-quality genre predictions and explore which genres exist in your corpora. Available for free!"]
                                        )
predictions
# Output: array([3, 8])

[model.config.id2label[i] for i in predictions]
# Output: ['Instruction', 'Promotion']

X-GENRE categories

For details on the X-GENRE categories, refer to the X-GENRE Annotation Guidelines.

List of labels

labels_list=['Other', 'Information/Explanation', 'News', 'Instruction', 'Opinion/Argumentation', 'Forum', 'Prose/Lyrical', 'Legal', 'Promotion'],

labels_map={'Other': 0, 'Information/Explanation': 1, 'News': 2, 'Instruction': 3, 'Opinion/Argumentation': 4, 'Forum': 5, 'Prose/Lyrical': 6, 'Legal': 7, 'Promotion': 8}

Description of labels

Fine-tuning hyperparameters

Fine-tuning was performed with simpletransformers. Beforehand, a brief hyperparameter optimization was performed and the presumed optimal hyperparameters are:

model_args= {
            "num_train_epochs": 15,
            "learning_rate": 1e-5,
            "max_seq_length": 512,
            }        
      

Citation

If you use the model, please cite the paper which describes creation of the X-GENRE dataset and the genre classifier:

@article{kuzman2023automatic,
  title={Automatic Genre Identification for Robust Enrichment of Massive Text Collections: Investigation of Classification Methods in the Era of Large Language Models},
  author={Kuzman, Taja and Mozeti{\v{c}}, Igor and Ljube{\v{s}}i{\'c}, Nikola},
  journal={Machine Learning and Knowledge Extraction},
  volume={5},
  number={3},
  pages={1149--1175},
  year={2023},
  publisher={MDPI}
}