arrodeio-gpt2-cordel
A style fine-tune of pierreguillou/gpt2-small-portuguese (GPorTuguese-2), continued-pretrained on Brazilian cordel poetry and ditados populares (folk sayings) so it picks up the rhythm, vocabulary, and rhyme of the form.
This is the "v1" model of arrodeio-llm, a small hands-on project for learning how LLM pretraining, fine-tuning, and inference actually work — by building a GPT from scratch first (v0, which established that ~150 KB of text can't teach a language from zero regardless of architecture), then repeating the fine-tuning phase on this real pretrained Portuguese base instead.
Try it live: arrodeio-llm-gradio Space (pick "v1 · GPT-2 português + cordel" in the demo).
Model Details
Model Description
This is a completion model, not an instruction-following one — it continues whatever text it's given, in cordel style. It is not chat-tuned; there is no chat template. (Instruction-following is what the separate v2 model, a LoRA adapter on a different, instruct-tuned base, is for.)
- Developed by: Bernardo Sana (bsana1)
- Model type: GPT-2 (124M), causal decoder-only, full model (not an adapter)
- Language(s): Portuguese (pt-BR)
- License: MIT — inherited from the base model
- Finetuned from model: pierreguillou/gpt2-small-portuguese (GPorTuguese-2, by Pierre Guillou, AI Lab at the University of Brasília — itself GPT-2 small transfer-learned from English GPT-2 onto Portuguese Wikipedia)
Model Sources
- Repository: github.com/bsana1/arrodeio-llm
- Demo: arrodeio-llm-gradio Space
Uses
Direct Use
Text completion in the style of cordel poetry: give it an opening line or a theme and let it continue. Works best primed with a stanza or two of real cordel first (see the demo's few-shot prompt, or the usage example below).
Out-of-Scope Use
A small hobby/learning project, not a production model. Fine-tuned on a single ~150 KB corpus (17 public-domain folhetos) — not evaluated for factual accuracy, safety, or anything outside creative Portuguese poetry completion. It does not follow instructions; asking it a question will just get you more cordel-flavored text, not an answer.
Bias, Risks, and Limitations
Inherits the base model's general limitations (GPorTuguese-2, 124M params, trained mostly on Wikipedia prose — cordel is a big style shift from that). The fine-tuning corpus is small and draws on 19th/20th century Northeastern Brazilian folk material, so outputs can reflect the period's language, social attitudes, and references. Not evaluated for bias.
How to Get Started with the Model
from transformers import AutoModelForCausalLM, AutoTokenizer
repo = "bsana1/arrodeio-gpt2-cordel"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo)
model.eval()
# Priming with a real stanza first helps it lock into cordel meter/rhyme.
prompt = (
"Eram doze cavaleiros\nhomens muito valorosos,\ndestemidos, animosos,\n"
"entre todos os guerreiros.\n\na vida no sertão\n"
)
ids = tok(prompt, return_tensors="pt").input_ids
out = model.generate(ids, max_new_tokens=110, do_sample=True, temperature=0.8,
top_k=40, repetition_penalty=1.3, pad_token_id=tok.eos_token_id)
print(tok.decode(out[0][ids.shape[1]:], skip_special_tokens=True))
Training Details
Training Data
data/corpus.txt (~157k chars, ~150 KB): 17 public-domain cordel folhetos
scraped from pt.wikisource.org (scripts/fetch_cordel.py), restricted to
long-dead authors (life + 70 years) — e.g. Leandro Gomes de Barros
(d. 1918), Silvino Pirauá de Lima (d. 1913) — plus a set of traditional
ditados populares (folk sayings).
Training Procedure
Continued pretraining (plain next-token prediction), not instruction tuning — only the top transformer blocks are unfrozen, so the model keeps its grip on Portuguese and mainly picks up cordel style/rhythm rather than overfitting to the small corpus.
Preprocessing
Corpus tokenized with the base model's own GPT-2 BPE tokenizer, chunked into fixed-length windows for training.
Training Hyperparameters
- Unfrozen layers: top 6 of 12 transformer blocks + final layer norm (~29M / 124M params trainable)
- Block size: 192 tokens
- Batch size: 4
- Learning rate: 1e-4
- Training steps: 450 (early-stopped by design — style lands well before the model starts memorizing the small corpus)
- Training regime: fp32, CPU
- Train/val split: 90/10
Speeds, Sizes, Times
Trained on a CPU (Apple Silicon, 8 GB) in about 25 minutes. Final model size: ~480 MB (full GPT-2 small checkpoint, not an adapter).
Evaluation
No formal benchmark — evaluated qualitatively against the other stages of the same project (from-scratch model, Tucano+LoRA). See samples/comparison.md in the GitHub repo for the same prompts run through every stage side by side.
Environmental Impact
Training took ~25 minutes on a single laptop CPU — negligible compute.
Technical Specifications
Model Architecture and Objective
GPT-2 small (124M params, 12 layers, 12 heads, 768 hidden size), causal language modeling objective (next-token prediction).
Compute Infrastructure
Hardware
Apple Silicon laptop CPU (8 GB RAM), local training.
Software
PyTorch, 🤗 Transformers.
Model Card Contact
Questions/issues: open one on github.com/bsana1/arrodeio-llm.