tohio/slm-gpt-127m-base
tohio/slm-gpt-127m-base is a pre-trained base language model (127M, ~126.8M parameters) trained from scratch on a 10B token educational and reasoning curriculum. This checkpoint serves as the foundational base and has not undergone instruction tuning or alignment.
Architecture Highlights
- Parameters: ~126.8M (14 layers, hidden dimension
d_model = 768) - Attention: Grouped-Query Attention (GQA, 12:4 ratio)
- Activation: SwiGLU Feed-Forward Network (
d_ffn = 2048) - Positional Encoding: Rotary Position Embeddings (RoPE)
- Normalization: Bias-free Pre-LayerNorm / RMSNorm
- Weight Tying: Tied input embedding (
embed_tokens) and output head (lm_head) - Hugging Face Native: Directly compatible with
LlamaForCausalLM
Pre-training Curriculum
The base representation was pre-trained across an interleaved multi-source domain mixture (10B tokens):
FineWeb-EduDCLM-EduThe Stack-EduNuminaMath-CoTOpenMathReasoningSLM-Synthetic-Pretrain
Pipeline Lineage
- Pre-training (
tohio/slm-gpt-127m-base): Completed 10B tokens over packed memory-mapped token shards. - Supervised Fine-Tuning (SFT): See
tohio/slm-gpt-127m-sft. - Direct Preference Optimization (DPO): See
tohio/slm-gpt-127m-dpo.
Prompt Format (Raw Completion)
As an unaligned base model, this checkpoint performs raw document continuation rather than dialogue:
The key difference between a compiler and an interpreter is that
Quick Start via transformers
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "tohio/slm-gpt-127m-base"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
model_id,
torch_dtype=torch.bfloat16,
device_map="auto",
)
prompt = "The key difference between a compiler and an interpreter is that"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=64, temperature=0.7, top_p=0.9)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
Trained and published autonomously via slm-gpt.