MiraLM-47M
A 47,640,968-parameter hybrid language model trained from scratch — no pretrained weights, no distillation — under a hard 50,000,000 cap, on a single Tesla T4. Interleaved Mamba ∥ attention blocks (Jamba-style) with a sparse top-2-of-8 mixture of experts whose router is domain-seeded by a guide loss.
This repository holds the step-11,000 best checkpoint (train loss 2.2113, perplexity 9.13) — the run that carries the measured results below. The step-17,000 "last" checkpoint diverged (loss 2.587) and is deliberately not published.
Measured results
| Model | HellaSwag | ARC-Easy | PIQA | WinoGrande | WikiText-103 PPL |
|---|---|---|---|---|---|
| MiraLM-47M | 33.0 | 27.0 | 50.0 | 50.2 | 2837.6 |
Read honestly: at 47.6M parameters and 45,056,000 tokens — roughly 4.7% of the Chinchilla-optimal budget for this size — the model is undertrained by construction. Multiple-choice commonsense sits near its random floor, and the WikiText-103 perplexity reflects a domain shift, since the training corpus is code / math / web text, not prose. The claim is not "we win on HellaSwag". It is that the whole pipeline — architecture, budget enforcement, data, routing, structured SFT, evaluation — runs end to end, reproduces from one config, and fits the ceiling.
Load it
git clone https://github.com/qtttyr/MiraLM && cd MiraLM
from src.model.hf_interface import MiraLMForCausalLM # registers model_type "mira"
from transformers import AutoTokenizer
repo = "vaprooll/MiraLM-47M"
model = MiraLMForCausalLM.from_pretrained(repo).eval()
tok = AutoTokenizer.from_pretrained(repo)
prompt = "<|json|> Return a JSON object with product, price and stock: product mug, price 500, stock 23."
ids = tok(prompt, return_tensors="pt")
out = model.generate(**ids, max_new_tokens=64, do_sample=False)
print(tok.decode(out[0][ids["input_ids"].shape[1]:]))
No trust_remote_code needed: importing hf_interface registers the mira
model type locally, so from_pretrained resolves the checkpoint.