TinyJLLM-Instruct — instruction-tuned 100M model
Supervised fine-tuning of TinyJLLM on 83,145 curated instruction examples from six public datasets. Trained from scratch in the TinyLLM repository with response-masked loss and attention: response positions attend only to the prompt, which prevents the model from learning a copy shortcut.
Published recipe: 1 epoch over the curated subset (Dolly-15k, CodeAlpaca, oasst1, MetaMathQA capped at 8K), validation loss 4.71 (perplexity 111). A 2-epoch run reached 4.29 but memorized more — the 1-epoch model produced better answers in side-by-side evaluation.
Model family
| Model | Stage | Link |
|---|---|---|
TinyJLLM | Base | jaweed123/TinyJLLM |
TinyJLLM-Instruct | SFT (this model) | jaweed123/TinyJLLM-Instruct |
TinyJLLM-Instruct-DPO | DPO | jaweed123/TinyJLLM-Instruct-DPO |
Usage
The model uses a plain marker format (no chat template):
### Instruction:
What is the capital of France?
### Response:
Because of the response-masked training protocol, generation must use the
same masking (response tokens attend only to the prompt). Plain causal
model.generate() produces degenerate repetition. A self-contained
transformers loop:
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "jaweed123/TinyJLLM-Instruct"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id).eval()
@torch.no_grad()
def respond(instruction, max_new_tokens=40):
prompt = f"### Instruction:\n{instruction}\n\n### Response:\n"
ids = tok(prompt)["input_ids"]
L = len(ids)
seq = ids + [tok.pad_token_id] # dummy keeps response positions aligned
for _ in range(max_new_tokens):
inp = torch.tensor([seq])
T = inp.shape[1]
mask = torch.tril(torch.ones(1, 1, T, T, dtype=torch.bool))
mask[:, :, L:, :] = False # response rows ...
mask[:, :, L:, :L] = True # ... see the prompt only
nxt = int(model(inp, attention_mask=mask).logits[0, -1].argmax())
if nxt == tok.eos_token_id:
break
seq.append(nxt)
return tok.decode(seq[L + 1:])
print(respond("What is the capital of France?"))
The project's own learnllm.inference.generate.generate(..., response_mask_start=...)
implements the same protocol with sampling and a repetition penalty.
Training details
| Base | TinyJLLM (102.5M, 32K vocab, 512 ctx) |
| Dataset | 83,145 examples, 6 sources (licenses in the repo manifest) |
| Subset used | Dolly-15k + CodeAlpaca + oasst1 + MetaMathQA (8K cap) |
| Epochs / LR | 1 / 2e-5, warmup + cosine |
| Loss | response-only (-100 on prompt tokens), labels[i] = input[i+1] |
| Attention | response-masked (response rows attend to prompt only) |
| Hardware | RTX 4060 8 GB |
Two bugs worth knowing about are documented in
SFT_TRAINING.md:
a label off-by-one that made training look converged while generations were
garbage, and the response-masking fix that removed the copy shortcut.
Limitations
At 100M parameters this model reproduces answer shapes more reliably than answer content: math and code are frequently wrong, multi-turn context is poorly tracked, and world knowledge is narrow. It is an educational artifact, not a production assistant.
Citation
@misc{tinyjllm,
title = {TinyJLLM: A 100M-Parameter Small Language Model Built From Scratch},
author = {Jaweed, Abdul},
year = {2026},
url = {https://github.com/Abdul-Jaweed/TinyLLM}
}