tohio/slm-gpt-127m-sft

tohio/slm-gpt-127m-sft is an instruction-tuned small language model (127M, ~126.8M parameters) fine-tuned on multi-turn reasoning and instruction dialogues formatted in ChatML.

Architecture Highlights

• Parameters: ~126.8M (14 layers, hidden dimension d_model = 768)
• Attention: Grouped-Query Attention (GQA, 12:4 ratio)
• Activation: SwiGLU Feed-Forward Network (d_ffn = 2048)
• Positional Encoding: Rotary Position Embeddings (RoPE)
• Normalization: Bias-free Pre-LayerNorm / RMSNorm
• Weight Tying: Tied input embedding (embed_tokens) and output head (lm_head)
• Hugging Face Native: Directly compatible with LlamaForCausalLM

Pre-training Curriculum

The base representation was pre-trained across an interleaved multi-source domain mixture (10B tokens):

• FineWeb-Edu
• DCLM-Edu
• The Stack-Edu
• NuminaMath-CoT
• OpenMathReasoning
• SLM-Synthetic-Pretrain

Pipeline Lineage

1. Pre-training: Base weights pre-trained on 10B tokens (tohio/slm-gpt-127m-base).
2. Supervised Fine-Tuning (tohio/slm-gpt-127m-sft): Full-parameter instruction tuning with loss masking (ignore_index=-100).
3. Direct Preference Optimization (DPO): See tohio/slm-gpt-127m-dpo.

Prompt Format (ChatML)

This model expects standard ChatML formatting:

<|im_start|>system
You are a helpful AI assistant.<|im_end|>
<|im_start|>user
Explain recursion in one sentence.<|im_end|>
<|im_start|>assistant

Quick Start via transformers

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "tohio/slm-gpt-127m-sft"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

prompt = "<|im_start|>user\nExplain recursion in one sentence.<|im_end|>\n<|im_start|>assistant\n"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=64, temperature=0.7, top_p=0.9)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Trained and published autonomously via slm-gpt.