flame-27m-instruct
A 27.1M-parameter English instruction-tuned model: flame-27m-base (pretrained on FineWeb-Edu-dedup + Cosmopedia-v2 + ClimbMix + FineMath) fine-tuned on a ~1.18M conversation instruction mixture. ~5× smaller than SmolLM-135M-Instruct.
- Architecture: Llama-style decoder — hidden 512, 8 layers, 8 heads / 2 KV heads (GQA), intermediate 1280, RoPE (θ=1e6), context 2048, vocab 12000 (English BPE).
- Training: base → SFT (5 epochs, EMA weights) on SmolTalk + Tulu-3-Persona-IF + No-Robots + WildChat (English, non-toxic).
Usage (chat)
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained("ifx-pse-sys-ml/flame-27m-instruct", trust_remote_code=True)
tok = AutoTokenizer.from_pretrained("ifx-pse-sys-ml/flame-27m-instruct")
messages = [{"role": "user", "content": "Tell me about the moon in one sentence."}]
ids = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt")
out = model.generate(ids, max_new_tokens=64, do_sample=True, temperature=0.7, top_p=0.9)
print(tok.decode(out[0][ids.shape[1]:], skip_special_tokens=True))
The decoder also accepts inputs_embeds (exactly one of input_ids / inputs_embeds), so a
vision projector can splice visual tokens in — usable as a small VLM text backbone. A raw
PyTorch checkpoint (pytorch_model.pth) is included alongside the safetensors weights.
Generation parameters
The model ships its own generate() (loaded via trust_remote_code). Two things to know:
- It does not read
generation_config.json. A baremodel.generate(ids)uses the built-in defaults —temperature 0.85, top_p 0.85, top_k 50, repetition_penalty 1.0— so pass the settings you want explicitly. - Greedy is
do_sample=False, nottemperature=0(which divides by zero).
Measured presets, from a 28-prompt VLM-style sweep over 13 decoding configs:
| Use | Settings | Measured |
|---|---|---|
| Answering about a given scene / image (VQA-style) | do_sample=False, repetition_penalty=1.3, no_repeat_ngram_size=3, penalize_prompt=False | Best overall: 59% grounded accuracy, no looping, shortest answers |
| Captions / open-ended description | do_sample=True, temperature=0.7, top_p=0.9, repetition_penalty=1.3, no_repeat_ngram_size=3, penalize_prompt=False | Varied and loop-free |
| Parameter | Default | What it does / when to change |
|---|---|---|
repetition_penalty | 1.0 | Penalises tokens already seen. 1.3 stops this model's multi-sentence looping. |
no_repeat_ngram_size | 0 | Blocks any n-gram from repeating (3 works well). Never blocks the end-of-turn token. |
penalize_prompt | True | Set False for anything grounded in the prompt. By default (Hugging Face semantics) both penalties also cover the prompt, which pushes the model away from copying the answer out of the context ("a red car", "Answer yes or no"). Limiting them to the reply raised grounded accuracy from 47% → 59% (greedy) and 27% → 39% (sampled). |
temperature | 0.85 | Lower = more focused. Must be > 0. |
top_p / top_k | 0.85 / 50 | Nucleus / top-k cutoffs. |
max_new_tokens | 8192 (capped at the 2048-token context) | Always set it — the model is at its best in short replies. |
# VQA-style: answer from the context, deterministically
out = model.generate(ids, max_new_tokens=64, do_sample=False,
repetition_penalty=1.3, no_repeat_ngram_size=3, penalize_prompt=False)
# Captions / open-ended
out = model.generate(ids, max_new_tokens=120, do_sample=True, temperature=0.7, top_p=0.9,
repetition_penalty=1.3, no_repeat_ngram_size=3, penalize_prompt=False)
Benchmarks
Accuracy (%) via lm-evaluation-harness 0.4, same harness and shots for every model, so columns are directly comparable.
| Benchmark | chance | flame-27m-instruct | SmolLM-135M-Instruct |
|---|---|---|---|
| hellaswag | 25 | 29.2 | 41.9 |
| arc_easy | 25 | 37.3 | 43.9 |
| arc_challenge | 25 | 22.6 | 27.4 |
| piqa | 50 | 59.3 | 67.0 |
| winogrande | 50 | 51.7 | 51.3 |
| openbookqa | 25 | 27.4 | 33.6 |
| commonsense_qa | 20 | 19.7 | 20.3 |
| mmlu | 25 | 24.9 | 24.4 |
| average | — | 34.0 | 38.7 |
These academic benchmarks measure base knowledge, which SFT cannot add. The richer instruction mixture was chosen to improve instruction-following and response quality, which it does (held-out assistant-token loss 1.34 → 1.23 vs a SmolTalk-only SFT, and cleaner format adherence) — at a small cost on the knowledge probes above. Benchmarks are the wrong lens for an instruct model's quality; they are shown only for comparability with the base and SmolLM.
Honest limitations
At 27M parameters this is near random chance on knowledge/reasoning benchmarks; the gap to SmolLM-135M is capacity, not data or tuning. Instruction-tuning adds response format, not facts. It follows simple instructions but cannot reliably satisfy hard multi-constraint prompts (IFEval ≈ 0) — that capability is bound by the 27M base, not the SFT data. A research/prototyping instruct model and a lightweight decoder, not a knowledge model. English only. Trained with the Nexus codebase.