LFM2.5-230M-ToMoE
A Mixture-of-Experts (MoE) conversion of LiquidAI/LFM2.5-230M, obtained with the ToMoE method (Dynamic Structural Pruning + Hypernetwork).
The dense 230M base model is converted into a channel-MoE where:
- MLP (14 layers): per-token expert routing over 8 channel-experts (top-1 via Gumbel-softmax), the FFN is cut to the union of the expert channel masks.
- Full attention (6 layers): Q/K kept at full width; the V channels are dynamically masked per token (deterministic masks).
- ShortConv (8 layers): statically pruned (a single channel index shared by the B/C/x projections, the conv and the output projection).
Metrics
| Metric | Dense base | ToMoE MoE |
|---|---|---|
| Total params | 229.7M | 200.9M stored |
| Active params per token | 229.7M | ~151M |
| PPL wikitext-2 (2048 tok) | 20.4 | ~815 |
| Embeddings (always active) | 67.1M | 67.1M |
The pruned budget is p = 0.3 with the ToMoE regularization (lam 64→256, Gumbel base annealed 1.0→−0.25, 8 experts). The attention QK is excluded from the budget (kept full) because the small head-dim (64) collapses under aggressive QK pruning.
Usage (transformers, trust_remote_code)
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained(
"Nichonauta/LFM2.5-230M-ToMoE",
trust_remote_code=True,
torch_dtype=torch.bfloat16,
device_map="cuda:0",
attn_implementation="eager",
)
tokenizer = AutoTokenizer.from_pretrained("Nichonauta/LFM2.5-230M-ToMoE")
prompt = "The capital of France is"
inputs = tokenizer(prompt, return_tensors="pt").to("cuda:0")
out = model.generate(**inputs, max_new_tokens=30)
print(tokenizer.decode(out[0], skip_special_tokens=True))
Note: this model requires
trust_remote_code=True— the custommodeling_lfm2_moe_final.pydefines theLfm2MoEForCausalLMclass.
Conversion details
- Base: LiquidAI/LFM2.5-230M (LFM Open License v1.0)
- Method: ToMoE — a bidirectional-GRU hypernetwork generates differentiable channel masks per sublayer; the model weights are frozen and only the hypernetwork is trained (online KD against the dense teacher, wikitext-103 stream). The final pruned weights are extracted from the trained masks.
- Training: 1,200 steps, seq 2048, online knowledge distillation (2× forward KL), gradient checkpointing.
Limitations
- The model is significantly degraded versus the dense base (PPL ~815 vs 20.4) — the pruning aggressively reduces the conv layers (12–22% width) and the MLP per-token width (~62%), and generation tends to repeat.
- This is a research artifact demonstrating the ToMoE pipeline on a small LFM2 architecture.
License
This model is a derivative of LiquidAI/LFM2.5-230M and is released under the LFM Open License v1.0 (see LICENSE).
Files
model.safetensors— the pruned MoE weights (fp32)modeling_lfm2_moe_final.py— the custom model definition (trust_remote_code)config.json— model configuration (withauto_map)tokenizer.json,tokenizer_config.json,chat_template.jinja,generation_config.json
GGUF quantizations
Quantized GGUF files (Q4_K_M, Q8_0, F16) are available in the companion repository: Nichonauta/LFM2.5-230M-ToMoE-GGUF.