pirate-speak-0.5b

A small learning project: Qwen/Qwen2.5-0.5B-Instruct fine-tuned with LoRA to answer every question like a grumpy pirate.

This is a derivative of Qwen2.5-0.5B-Instruct, created by the Qwen team at Alibaba Cloud. All credit for the underlying model goes to them; this repo only adds a style fine-tune on top. See License and attribution.

Example outputs

Greedy decoding, same prompts as the base model:

How to use

from transformers import pipeline

pirate = pipeline("text-generation", model="alvinchirchir/pirate-speak-0.5b")
messages = [{"role": "user", "content": "How do I make a cup of coffee?"}]
print(pirate(messages, max_new_tokens=80)[0]["generated_text"][-1]["content"])

Or with the model and tokenizer directly:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "alvinchirchir/pirate-speak-0.5b"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, dtype=torch.float32)

inputs = tokenizer.apply_chat_template(
    [{"role": "user", "content": "Any tips for a job interview?"}],
    add_generation_prompt=True, return_tensors="pt", return_dict=True,
)
output = model.generate(**inputs, max_new_tokens=80, do_sample=False)
print(tokenizer.decode(output[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Training used Qwen's default chat template with no custom system message, so results are most consistent without one.

Intended use and limitations

Intended use: learning and experimentation - a hands-on example of LoRA fine-tuning for style, and a fun chat toy.

Not intended for: anything where correct information matters.

Known limitations:

• Tiny training set. Only 15 examples. The model reproduces training answers closely, but on new prompts the pirate style is inconsistent and replies are often vague or unhelpful.
• Helpfulness dropped. The fine-tune traded the base model's detailed answers for short in-character quips. Do not rely on it for facts, instructions, or advice.
• Occasional glitches. Outputs sometimes contain misspellings or stray tokens from other languages (e.g. Chinese characters mid-sentence).
• Inherits the biases and limitations of Qwen2.5-0.5B-Instruct.

Training details

Data: 15 hand-written prompt/response pairs of everyday questions answered in pirate speak, formatted as conversational prompt/completion pairs so the loss is computed only on the assistant reply.

Hyperparameters:

Hardware: Apple M1 Pro (PyTorch MPS backend).

Software: Transformers 4.57.6, TRL 1.13.0, PEFT 0.19.1, PyTorch 2.9.1.

After training, the LoRA adapter was merged into the base weights with merge_and_unload(), so the model loads like any regular Transformers model with no PEFT dependency.

License and attribution

This model is a modified version of Qwen2.5-0.5B-Instruct, Copyright 2024 Alibaba Cloud, licensed under the Apache License 2.0. The base model is available at https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct.

Modifications: LoRA fine-tuning on 15 pirate-speak conversation examples, with the adapter merged into the model weights. The architecture, tokenizer, and chat template are unchanged.

This derivative is distributed under the same Apache License 2.0; a copy is included in this repository as LICENSE.

If you use the base model in research, please cite the Qwen team (citation from the Qwen2.5-0.5B-Instruct model card):

@misc{qwen2.5,
    title = {Qwen2.5: A Party of Foundation Models},
    url = {https://qwenlm.github.io/blog/qwen2.5/},
    author = {Qwen Team},
    month = {September},
    year = {2024}
}

@article{qwen2,
      title={Qwen2 Technical Report}, 
      author={An Yang and Baosong Yang and Binyuan Hui and Bo Zheng and Bowen Yu and Chang Zhou and Chengpeng Li and Chengyuan Li and Dayiheng Liu and Fei Huang and Guanting Dong and Haoran Wei and Huan Lin and Jialong Tang and Jialin Wang and Jian Yang and Jianhong Tu and Jianwei Zhang and Jianxin Ma and Jin Xu and Jingren Zhou and Jinze Bai and Jinzheng He and Junyang Lin and Kai Dang and Keming Lu and Keqin Chen and Kexin Yang and Mei Li and Mingfeng Xue and Na Ni and Pei Zhang and Peng Wang and Ru Peng and Rui Men and Ruize Gao and Runji Lin and Shijie Wang and Shuai Bai and Sinan Tan and Tianhang Zhu and Tianhao Li and Tianyu Liu and Wenbin Ge and Xiaodong Deng and Xiaohuan Zhou and Xingzhang Ren and Xinyu Zhang and Xipin Wei and Xuancheng Ren and Yang Fan and Yang Yao and Yichang Zhang and Yu Wan and Yunfei Chu and Yuqiong Liu and Zeyu Cui and Zhenru Zhang and Zhihao Fan},
      journal={arXiv preprint arXiv:2407.10671},
      year={2024}
}