tinyLLM SFT 0.51B
中文项目文档 · English project overview · Live demo
One SFT base: mathematical reasoning with a verified answer, then tested Python code.
tinyLLM is a custom 0.51B decoder-only model trained from scratch for Chinese and English.
Loading requires trust_remote_code=True because the architecture is implemented in this repository.
The released SFT checkpoint scores 50.64% (668/1319) on GSM8K with the project's greedy evaluation setup. It is also the shared base for the ARC GRPO, IFEval OPD and VLM releases.
Load
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "chris0809/tinyLLM-0.51B-SFT"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
model_id,
trust_remote_code=True,
dtype="auto",
)
Thinking mode
Ordinary chat uses the default template. For mathematics or multi-step reasoning, turn on the bundled switch; the template automatically injects the same reasoning protocol used during training and evaluation.
messages = [{"role": "user", "content": "Solve: 17 * 6. Give the final answer clearly."}]
inputs = tokenizer.apply_chat_template(
messages,
enable_thinking=True,
add_generation_prompt=True,
return_tensors="pt",
)
Omit enable_thinking or set it to False for ordinary chat. The thought
boundaries are encoded with existing tokenizer pieces, so do not add special
tokens or resize the embeddings. For GSM8K/ARC-style evaluation, append
Put your final answer in LaTeX boxed form like $\boxed{answer}$. to the
user question.
Task LoRAs can be attached without reloading the base model:
model.load_lora_pretrained("chris0809/tinyLLM-0.51B-ARC-GRPO")
Try the SFT checkpoint in the hosted tinyLLM demo. The Space uses free ZeroGPU, so a cold start or queue is normal.
Source
Training code and full project documentation: https://github.com/Huanz86251/tinyLLM