GlórIA 1.3B
Model Description
GlórIA is a large generative language model, with special focus on European Portuguese.
It is a 1.3B parameters model, based on GPTNeo, which has 24 layers and a hidden size of 2048.
You can check our paper, accepted in PROPOR 2024.
Training Data
GlórIA 1.3B was trained on a large corpora, with approximately 35B tokens. This corpus was built by gathering multiple Portuguese sources:
- ArquivoPT News PT-PT Dataset: A collection of 1.4M European Portuguese archived news and periodicals from Arquivo.pt.
- ClueWeb-Large PT-PT: Multilingual Corpus, similar to OSCAR. Metadata was used to filter only PT-PT webpages.
- Europarl PT-PT: A parallel corpus with documents such as transcripts from the European Parliament (we only used the PT-PT documents).
- OpenSubtitles PT-PT: A corpus containing PT-PT subtitles from OpenSubtitles.
- OSCAR PT-PT: Multilingual Corpus obtained from filtering the Common Crawl corpus. We used metadata to filter only PT-PT webpages.
- PT WIKI: The Portuguese Wikipedia. 2022/06/20 Dump.
Evaluation - CALAME-PT
GlórIA 1.3B generative capabilities were evaluated on CALAME-PT - a new Portuguese benchmark with the goal of predicting the last word of a sentence, according to a given context.
| Model and Size | Exact-Match |
|---|---|
| Gervasio-PTPT (1B) | 44.01 |
| mGPT (1.3B) | 47.14 |
| GlórIA (1.3B) | 52.79 |
How to use
Basic Inference Example
>>> from transformers import pipeline
>>> generator = pipeline('text-generation', model='NOVA-vision-language/GlorIA-1.3B')
>>> generator("A culinária portuguesa é rica em aromas e", do_sample=True, min_length=50)
[{'generated_text': 'A culinária portuguesa é rica em aromas e'}]
Recommended Parameters and Usage (for more flexibility)
from transformers import GenerationConfig, TextGenerationPipeline
generation_config = GenerationConfig(
max_new_tokens=50, do_sample=True, top_k=50, eos_token_id=model.config.eos_token_id,
no_repeat_ngram_size=0, num_beams=4, repetition_penalty=2.0, temperature=1.0,
output_scores=True, early_stopping=True
)
generator = TextGenerationPipeline(model=model, task="text-generation",
tokenizer=loaded_tokenizer, device=0)
completion_prompts = ["Fernando Pessoa foi um dos poetas mais relevantes de"]
out = generator(completion_prompts, generation_config=generation_config)
[[{'generated_text': 'Fernando Pessoa foi um dos poetas mais relevantes de toda a literatura portuguesa, autor de uma obra que se estende por mais de quatro dezenas de livros, entre os quais "Mensagem", "O Guardador de Rebanhos", "Livro do desassossego", "Odes",'}]]
Citation
Please use the following BibTeX to cite our paper:
@inproceedings{lopes-etal-2024-gloria,
title = "{G}l{\'o}r{IA}: A Generative and Open Large Language Model for {P}ortuguese",
author = "Lopes, Ricardo and
Magalhaes, Joao and
Semedo, David",
editor = "Gamallo, Pablo and
Claro, Daniela and
Teixeira, Ant{\'o}nio and
Real, Livy and
Garcia, Marcos and
Oliveira, Hugo Gon{\c{c}}alo and
Amaro, Raquel",
booktitle = "Proceedings of the 16th International Conference on Computational Processing of Portuguese",
month = mar,
year = "2024",
address = "Santiago de Compostela, Galicia/Spain",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2024.propor-1.45",
pages = "441--453",
}
License: GlórIA's usage is restricted to research-only purposes, subject to the ClueWeb22 Dataset license, which can be freely obtained here.
Acknowledgements
We would like to thank Arquivo.pt's team for their content preservation efforts, and for all the help and guidance in accessing the archived web pages at scale. This work was partially funded by the FCT project NOVA LINCS Ref. UIDP/04516/2020, by CMU|Portugal project iFetch, Ref. CMUP LISBOA-01-0247-FEDER-045920, and by the FCT project Ref. Nº CPCA-IAC/AV/594875/2023.