welmia-2.0-103m-instruct
A 103M-parameter GPT-style language model (RoPE, RMSNorm, SwiGLU, tied embeddings), pretrained from scratch on ~5B tokens (OpenWebText), then instruction-tuned on the Alpaca dataset.
Base (non-instruct) version: Welmia/welmia-2.0-103m-base.
Architecture
| Parameter | Value |
|---|---|
| Parameters | 103.4M |
| Layers | 12 |
| Attention Heads | 12 |
| Embedding Dim | 768 |
| Context Length | 512 |
| Vocab Size | 24,000 |
| Normalization | RMSNorm |
| Activation | SwiGLU |
| Positional Enc. | RoPE |
| Weight Tying | Yes |
Tokenizer note: custom 24k-vocab BPE tokenizer, not GPT-2/tiktoken. Not interchangeable
with welmia-1.0-81m.
Prompt format
This model expects the Alpaca instruction format:
Instruction:
{your instruction}
Response:
or, with additional context:
Instruction:
{your instruction}
Input:
{context}
Response:
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
model = AutoModelForCausalLM.from_pretrained("Welmia/welmia-2.0-103m-instruct", trust_remote_code=True)
tok = AutoTokenizer.from_pretrained("Welmia/welmia-2.0-103m-instruct")
prompt = "### Instruction:\nWhat is Machine Learning.\n\n### Response:\n"
ids = tok(prompt, return_tensors="pt").input_ids
out = model.generate(ids, max_new_tokens=150, do_sample=True, temperature=0.7, top_k=40)
print(tok.decode(out[0], skip_special_tokens=True))
trust_remote_code=True is required — custom architecture defined in modeling_gpt.py in this repo.
Important notes
- Single-turn only. Trained on Alpaca, which has no multi-turn/chat structure — treat each prompt as independent, not a running conversation.
- No KV cache.
generate()recomputes the full forward pass each step. - 512-token context. Longer inputs are truncated from the left.
- Small model — expect basic instruction-following, not strong reasoning or world knowledge.
Training
Pretrained from scratch on ~5B tokens OpenWebText (causal LM), then fine-tuned on the Alpaca instruction dataset (~52k examples, 3 epochs, AdamW, cosine LR, loss masked over the prompt so only the response is trained on).
License
Apache License 2.0