Nimbus DPO 0.5B
The final deployable model from the Nimbus capstone: Qwen 2.5 0.5B, fully fine-tuned on 4,000 synthetic support Q&A pairs, then aligned with DPO on 500 preference pairs (498 after cleaning). Nimbus Coffee is a fictional roaster; the data was generated with DeepSeek.
Results
Scored on 200 questions held out from training. None is an exact copy of a training prompt; 8 of them match a training prompt once case and punctuation are ignored, and 2 match a DPO preference prompt. The judge is an LLM (DeepSeek) rating each answer 1 to 5.
| Model | Token overlap | Judge overall |
|---|---|---|
| SFT 0.5B (this model before DPO) | 0.5816 | 3.830 |
| DPO 0.5B (this model) | 0.5664 | 3.913 |
| SFT 3B LoRA v2 (the teacher) | 0.5635 | 4.035 |
Read the gain as small. Paired over the same 200 questions, DPO moved the judge score by +0.083 with a standard error of 0.049: 48 answers improved, 33 got worse and 119 stayed the same. That is suggestive, not conclusive, and an LLM judge has variance of its own.
Training
- Supervised fine-tune: all weights, 3 epochs on CPU, final loss 0.5312
- DPO: 500 chosen/rejected pairs, trained on a local AMD 7900 XT with ROCm
- Why 0.5B: small enough for free hosting. The 3B teacher scores higher and is too large for that.
Use
from transformers import AutoModelForCausalLM, AutoTokenizer
name = "420yolomcswaggerpants/nimbus-dpo-0.5b"
tokenizer = AutoTokenizer.from_pretrained(name)
model = AutoModelForCausalLM.from_pretrained(name)
prompt = "### Instruction:\nHow much is shipping?\n\n### Response:\n"
inputs = tokenizer(prompt, return_tensors="pt")
out = model.generate(**inputs, max_new_tokens=100, do_sample=False)
print(tokenizer.decode(out[0], skip_special_tokens=True))
Limits
- Trained on synthetic data about a fictional business. It is a study artifact, not a product.
- A 0.5B model will state wrong facts outside what it was trained on.
- In the full system it answers from retrieved context; on its own it only knows its training data.