TIM-360M-base
A 361.8M-parameter decoder-only transformer, trained from scratch on ~30B tokens on 2× RTX 4090. Part of TIM (Transformer Inference Model) — an end-to-end, from-scratch LLM built to be legible and honestly evaluated.
Exported as Qwen3ForCausalLM — it loads with transformers and no trust_remote_code.
Architecture
SmolLM2-360M skeleton with modern choices: pre-LN RMSNorm, RoPE (θ=100000), SwiGLU, grouped-query attention (15 query / 5 KV heads) with per-head QK-norm, tied embeddings, no biases. 32 layers, hidden 960, head_dim 64, FFN 2560, vocab 49,152 (byte-level BPE). Context: max_position_embeddings 4096, trained at seq_len 2048.
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
tok = AutoTokenizer.from_pretrained("timothywong731/tim-360m-base")
m = AutoModelForCausalLM.from_pretrained("timothywong731/tim-360m-base", torch_dtype=torch.bfloat16)
ids = tok("The capital of France is", return_tensors="pt")
print(tok.decode(m.generate(**ids, max_new_tokens=20)[0]))
Training
~30B tokens, mix: 70% web (FineWeb-Edu sample-100BT + DCLM-Baseline), 15% Stack-Edu (code), 10% FineMath-3+, 5% misc. Muon + AdamW, WSD (warmup–stable–decay) schedule, bf16, 2× RTX 4090 (DDP, no NVLink).
Evaluation
lm-evaluation-harness (0.4.12), 0-shot, acc_norm where applicable:
| HellaSwag | ARC-Easy | ARC-Challenge | PIQA | WinoGrande | OpenBookQA | SciQ | SocialIQA | CommonsenseQA | WikiText (bpb ↓) |
|---|---|---|---|---|---|---|---|---|---|
| 0.388 | 0.515 | 0.259 | 0.658 | 0.525 | 0.294 | 0.722 | 0.378 | 0.196 | 1.058 |
Healthy for the scale: at ~30B tokens this is ~130× less training data than SmolLM2-360M (4T). Near-parity on early-saturating tasks (PIQA, SciQ), a real gap on token-hungry ones (HellaSwag, ARC) — as the token budget predicts.
Limitations
Small (360M) and lightly trained (~30B tokens). Not instruction-tuned — this is the base model. Weak at arithmetic and multi-step reasoning; will hallucinate facts; English-only; no safety tuning — do not deploy to end users without your own filtering.
Provenance & license
Apache-2.0. Trained only on named, public datasets: FineWeb-Edu (ODC-BY), DCLM-Baseline (CC-BY-4.0), Stack-Edu (derived from The Stack v2), FineMath (ODC-BY). No gated or scraped-private data. Attribution and removal requests: open an issue on the source repository.