LFM2.5-230M-Synth
Fine-tuned LiquidAI/LFM2.5-230M on a large synthetic reasoning dataset with preserved chain-of-thought (thinking) traces.
Model Details
| Base model | LiquidAI/LFM2.5-230M (Lfm2ForCausalLM) |
| Architecture | LFM2 — hybrid conv+attention (8 conv + 6 full attention layers, 14 total) |
| Parameters | ~229M (tied embeddings) |
| Hidden size | 1024 |
| Attention heads | 16 (8 KV heads, GQA) |
| Vocab size | 64,402 |
| Max context | 128K (trained at 2048) |
| Precision | bfloat16 |
| Model size | 457 MB (safetensors) |
Training Details
| Dataset | Synthetic reasoning mix (1.63M conversations, multi-turn with chain-of-thought) |
| Dataset size | ~6.23 GiB (Arrow) |
| Training tokens | ~2.88B (22,000 steps × effective batch 64 × seq 2048) |
| Epochs | ~0.86 (partial epoch at checkpoint-22000) |
| Effective batch size | 64 (per-device 8 × grad-accum 8) |
| Learning rate | 5e-5, cosine schedule, 2% warmup |
| Optimizer | AdamW (PyTorch fused) |
| Sequence length | 2048 |
| Hardware | NVIDIA L40 48GB |
| Precision | bf16 + torch.compile |
| Framework | HuggingFace Transformers + TRL SFTTrainer |
Training Results
| Step | Train Loss | Eval Loss |
|---|---|---|
| 500 | 2.71 | 1.8661 |
| 3,000 | 1.75 | 1.7121 |
| 5,500 | 1.69 | 1.6835 |
| 8,000 | 1.67 | 1.6687 |
| 10,500 | 1.66 | 1.6602 |
| 13,000 | 1.65 | 1.6542 |
| 15,500 | 1.65 | 1.6501 |
| 18,000 | 1.65 | 1.6478 |
| 20,500 | 1.65 | 1.6460 |
| 22,000 | 1.655 | 1.6457 |
Best eval loss: 1.6457 at step 22,000 (still improving at checkpoint time).
Loss decreased from 1.866 → 1.646 over 2.88B tokens — a 12% relative reduction with clear continued downward trend at the checkpoint boundary.
Chat Template
Uses the Liquid/LFM2 chat template with preserve_thinking=True. Reasoning traces from the dataset's reasoning_content field are mapped to the model's native thinking field before tokenization.
from transformers import AutoTokenizer, AutoModelForCausalLM
tokenizer = AutoTokenizer.from_pretrained("mkurman/lfm25-230m-synth", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("mkurman/lfm25-230m-synth", trust_remote_code=True, dtype="bfloat16")
messages = [{"role": "user", "content": "Explain quantum entanglement simply."}]
# preserve_thinking=True so the model generates reasoning before its answer
text = tokenizer.apply_chat_template(
messages,
tokenize=False,
add_generation_prompt=True,
preserve_thinking=True,
)
inputs = tokenizer(text, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=512)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
Intended Use
This is a research model fine-tuned on synthetic reasoning data. It is intended for:
- Experimentation with small-model reasoning capabilities
- Chain-of-thought / thinking-trace generation
- Evaluation of synthetic data quality at small scale (230M)
- On-device or edge reasoning model prototyping
Limitations
- Partial training: This checkpoint is at ~0.86 epochs (step 22,000 / 36,500 planned). The full run continues.
- Small model: 230M parameters — not suitable for production deployment without further evaluation.
- Synthetic data only: Trained exclusively on synthetic reasoning traces; may exhibit distribution biases from the data generation pipeline.
- Limited context at training: Trained at seq=2048 despite the architecture supporting 128K. Long-context behavior is untested.
Checkpoint Info
This is checkpoint-22000 from a 36,500-step training run. The checkpoint includes:
model.safetensors— 457 MB (weights only, no optimizer state)- Full tokenizer files + chat template
config.jsonwith architecture details
Citation
If you use this model, please cite the base model:
@misc{liquid_lfm2,
title={LFM2: Liquid Foundation Models},
author={Liquid AI},
year={2025},
url={https://huggingface.co/LiquidAI/LFM2.5-230M}
}