tinymistral-276m-it — instruction-tuned dense

tinymistral-276m-it is the instruction-tuned version of the 276M dense base mikecovlee/tinymistral-276m. It is the dense counterpart of the MoE flagship mikecovlee/tinymixtral-it: the exact same 3M-tier SFT recipe (same data, hyper-parameters and schedule) is applied to the dense base, so the two instruction-tuned models can be compared at the same active-parameter count and the same SFT budget.

Model details

Training

• Initialization: the 8.05B-token final checkpoint of the dense base tinymistral-276m.
• SFT recipe: identical to the MoE flagship tinymixtral-it (3M tier) — 2,168,835 deduplicated, eval-decontaminated English instruction conversations blended from 10 public sources (Tulu3, OpenHermes, SlimOrca, OpenOrca, UltraChat, MetaMath, OrcaMath, OpenMathInstruct-2, SQuAD2, TriviaQA), 1 epoch, sequence packing to 1024 tokens, batch 24, AdamW lr 2e-5 (cosine, 100-step warmup), weight decay 0.1, seed 42 → 56,793 steps.
• Hardware: single NVIDIA RTX PRO 4500 (Blackwell, 32 GB), 1036.5 min (17.3 h).

Evaluation

Same-hardware evaluation (lm_eval 0.4.12, 0-shot, no chat template). Harness = mean of the 7 primary metrics (hellaswag acc_norm, piqa acc, winogrande acc, arc_easy acc, arc_challenge acc_norm, openbookqa acc_norm, lambada acc).

Harness note. Means quoted here use the 7-task harness (excludes BoolQ); the base v1.0/v3.0 cards report an 8-task mean (includes BoolQ). The two are not directly comparable.

Takeaways:

• SFT transfers to the dense base: GSM8K flexible rises from 0.0136 (dense base) to 0.0205, matching the MoE flagship under the same recipe.
• The 7-task harness barely moves (0.3904 → 0.3892), the same pattern as the MoE flagship — SFT mainly buys instruction-following and math, not general tasks.
• Instruction following is weaker on the dense base: IFEval 0.1460 / 0.2602 vs 0.1756 / 0.2782 for tinymixtral-it — the dense base is less steerable than the MoE base.
• Absolute numbers remain bounded by the 276M scale and the 8.05B-token pretraining budget.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "mikecovlee/tinymistral-276m-it"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True, dtype="bfloat16", device_map="auto")

messages = [{"role": "user", "content": "What is 12% of 250?"}]
ids = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)
out = model.generate(ids, max_new_tokens=256, do_sample=False)
print(tok.decode(out[0][ids.shape[1]:], skip_special_tokens=True))

Requires transformers and trust_remote_code=True (custom tinymixtral architecture).

Limitations

• The base was pretrained on only 8.05B tokens — far below modern small-model budgets; knowledge tasks (MMLU / TruthfulQA) are near chance.
• Math/reasoning is bounded by the 276M scale and the pretraining budget; SFT only helps marginally.
• English-centric, no safety alignment.

Family

• mikecovlee/tinymixtral — 477.5M MoE base (v3.0 flagship)
• mikecovlee/tinymixtral-it — MoE instruction-tuned (v3.0-it, 3M SFT)
• mikecovlee/tinymistral-276m — 276M dense base (iso-active ablation)
• mikecovlee/tinymistral-276m-it — this model (276M dense instruction-tuned, 3M SFT)
• mikecovlee/tinymixtral-v1.1-1b — 1B MoE (earlier flagship)
• mikecovlee/tinymixtral-v1.1-0.5b — 0.5B-class MoE (data-quality ablation)
• mikecovlee/tinymixtral-v2.0-beta — shared-expert experiment (beta)
• mikecovlee/tinymixtral-v1.0 — legacy (C4)

Naming. The MoE family is published under tinymixtral; the dense 276M iso-active ablation companions use the tinymistral spelling. Both belong to the same project.

Citation

@misc{tinymistral276mit2026,
  title  = {TinyMixtral: a small Mixture-of-Experts language-model family},
  author = {Michael Lee},
  year   = {2026},
  howpublished = {\url{https://huggingface.co/mikecovlee/tinymistral-276m-it}}
}

License

MIT (Copyright (C) 2026 Michael Lee).