TinyJLLM-Instruct-DPO — preference-tuned 100M model
Direct Preference Optimization applied to TinyJLLM-Instruct with a DPO trainer implemented from scratch in the TinyLLM repository (no TRL).
Measured effect: held-out preference accuracy improved from 64.5% (SFT reference) to 77.5% (this model) on 200 unseen pairs, and generations visibly cleaned up:
| Prompt | SFT | DPO (this model) |
|---|---|---|
| capital of France | is a country of France. | The capital of France |
| fruit or vegetable? | apple, apple, apple… (loops) | Apple |
Model family
| Model | Stage | Link |
|---|---|---|
TinyJLLM | Base | jaweed123/TinyJLLM |
TinyJLLM-Instruct | SFT | jaweed123/TinyJLLM-Instruct |
TinyJLLM-Instruct-DPO | DPO (this model) | jaweed123/TinyJLLM-Instruct-DPO |
Usage
Same marker format and the same response-masked generation protocol as the
SFT model (plain causal model.generate() degrades into repetition):
### Instruction:
What is the capital of France?
### Response:
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "jaweed123/TinyJLLM-Instruct-DPO"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id).eval()
@torch.no_grad()
def respond(instruction, max_new_tokens=40):
prompt = f"### Instruction:\n{instruction}\n\n### Response:\n"
ids = tok(prompt)["input_ids"]
L = len(ids)
seq = ids + [tok.pad_token_id] # dummy keeps response positions aligned
for _ in range(max_new_tokens):
inp = torch.tensor([seq])
T = inp.shape[1]
mask = torch.tril(torch.ones(1, 1, T, T, dtype=torch.bool))
mask[:, :, L:, :] = False # response rows ...
mask[:, :, L:, :L] = True # ... see the prompt only
nxt = int(model(inp, attention_mask=mask).logits[0, -1].argmax())
if nxt == tok.eos_token_id:
break
seq.append(nxt)
return tok.decode(seq[L + 1:])
print(respond("What is the capital of France?"))
For sampling (temperature 0.8, top-k 50, top-p 0.95, repetition penalty
1.15 — the values chosen by the project's sampling sweep), use the
project's learnllm.inference.generate.generate(..., response_mask_start=...).
Training details
| Base / reference | TinyJLLM-Instruct (frozen reference = same weights) |
| Dataset | 32,648 pairs: Orca DPO (Apache-2.0) + HH-RLHF (MIT, filtered to single-turn answer-like pairs), 2 epochs |
| Beta / LR | 0.2 / 3e-6, warmup + cosine |
| Compute trick | reference log-probabilities precomputed once (halves training compute) |
| Batch / steps | 32 pairs effective, 3,882 steps |
| Hardware | RTX 4060 8 GB, ~2 h |
Two earlier DPO runs (unfiltered data, lower LR) and the reasoning for the
final recipe are documented in
DPO_TRAINING.md,
including a data-integrity incident that was found and fixed.
Limitations
- Still a 100M model. DPO improves preference ranking and answer shape; it does not add knowledge. Math and code remain unreliable.
- Multi-turn conversations fail. The model echoes earlier context instead of answering the last turn (measured; see the repository docs).
- Safety tuning is minimal — a small, mostly-English preference set. Do not rely on this model for safety-critical filtering.
Citation
@misc{tinyjllm,
title = {TinyJLLM: A 100M-Parameter Small Language Model Built From Scratch},
author = {Jaweed, Abdul},
year = {2026},
url = {https://github.com/Abdul-Jaweed/TinyLLM}
}