welmia-2.0-103m-base
A 103M-parameter GPT-style language model (RoPE, RMSNorm, SwiGLU, tied embeddings) pretrained completely from scratch on ~5B tokens (OpenWebText).
This is the base (non-instruction-tuned) model — raw next-token prediction only.
For an instruction-following version, see welmia-1.0-103m-instruct (uploaded separately).
Architecture
| Parameter | Value |
|---|---|
| Parameters | 103.4M |
| Layers | 12 |
| Attention Heads | 12 |
| Embedding Dim | 768 |
| Context Length | 512 |
| Vocab Size | 24,000 |
| Normalization | RMSNorm |
| Activation | SwiGLU |
| Positional Enc. | RoPE |
| Weight Tying | Yes |
Tokenizer note: this model uses a custom 24k-vocab BPE tokenizer trained specifically
for this model — it is not the GPT-2/tiktoken vocabulary used by welmia-1.0-81m.
Checkpoints and tokenizers between the two models are not interchangeable.
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
model = AutoModelForCausalLM.from_pretrained("Welmia/welmia-2.0-103m-base", trust_remote_code=True)
tok = AutoTokenizer.from_pretrained("Welmia/welmia-2.0-103m-base")
ids = tok("The history of artificial intelligence began", return_tensors="pt").input_ids
out = model.generate(ids, max_new_tokens=100, do_sample=True, temperature=0.8, top_k=40)
print(tok.decode(out[0], skip_special_tokens=True))
trust_remote_code=True is required — this is a custom architecture, defined in
modeling_gpt.py in this repo.
Important notes
- Base model, not instruction-tuned. It will continue text rather than answer questions
or follow instructions. Use the
-instructversion for chat/Q&A behavior. - No KV cache in this implementation —
generate()recomputes the full forward pass each step, so it's slower than production-grade inference code at longer output lengths. - 512-token context. Inputs longer than this are truncated from the left during generation.
Training
Trained from scratch on ~5B tokens of OpenWebText, causal LM objective, RoPE + SwiGLU + RMSNorm architecture, AdamW optimizer with cosine LR schedule.
License
Apache License 2.0