fuse-1 Lite — Coding-Enhanced Mixture-of-Experts (5.72B)

A 5.72B parameter Mixture-of-Experts model that fuses LiquidAI's LFM2.5-2.6B host with 960 coding experts extracted from Qwen3.6-35B-A3B. Designed for efficient coding assistance, agentic workflows, and on-device inference.

Overview

fuse-1 Lite is a novel fusion model that combines the speed and efficiency of a small language model (LFM2.5-2.6B, 2.70B params) with the coding expertise of a large MoE model (Qwen3.6-35B-A3B). Rather than distilling knowledge or fine-tuning from scratch, fuse-1 Lite transplants actual expert weights from the donor model and trains a lightweight router to selectively activate them only when coding-related tokens are encountered.

Key Innovation

Traditional model fusion requires either:

1. Knowledge distillation (slow, lossy, requires teacher inference)
2. Weight merging (requires compatible architectures)
3. Full fine-tuning (expensive, risks catastrophic forgetting)

fuse-1 Lite takes a different approach: surgical expert transplantation with learned routing. The 960 coding-specialized experts from Qwen3.6-35B-A3B are extracted, normalized, and integrated as residual augmentations to LFM2.5's decoder layers. A per-layer router learns which experts to activate for each token, and a learned scale factor controls how much each layer's experts contribute.

Architecture

How It Works

Input Token
    │
    ▼
┌─────────────────────────────────────────┐
│  LFM2.5 Decoder Layer (frozen host)     │
│  ├── Attention / ShortConv              │
│  └── SwiGLU FFN                         │
└─────────────┬───────────────────────────┘
              │
              ▼
┌─────────────────────────────────────────┐
│  Fuse3 Augmented Layer                  │
│  1. Router scores token → top-8 experts │
│  2. Selected experts compute SwiGLU     │
│  3. Output normalized to host std       │
│  4. Scaled by learned expert_scale      │
│  5. Added to residual stream            │
└─────────────────────────────────────────┘

Expert Activation Pattern

After training, the router learned to selectively activate experts in 11 of 30 layers:

The remaining 19 layers have scale ≈ 0 (experts effectively disabled), preserving LFM2's general language capabilities.

Training Details

Training Phases

1. Phase 1 — Expert Profiling ($1.90, A100 80GB): Profile Qwen3.6-35B-A3B expert activations on coding vs non-coding prompts. Select 960 coding-specialized experts. Extract weights.

2. Phase 2 — Assembly ($0.50, L4): Load LFM2.5-2.6B, wrap decoder layers with Fuse3AugmentedLayer, load extracted expert weights. Verify base model output coherence.

3. Phase 3 — Router Training ($0.60, L4): Train router + expert_scale on coding and general examples. Router learns to activate experts only for coding-related tokens.

Stability Mechanisms

• Std normalization: Expert outputs are rescaled to match host layer activation std (ratio clamped to 2.0)
• Scale clamping: Expert scale clamped to max=0.1 during forward pass
• SwiGLU clamping: Expert intermediate activations clamped to [-10, 10]
• Frozen host + experts: Only 2.0M parameters are trainable (router weights + per-layer scale)

Usage

Installation

pip install transformers torch

Quick Start

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_id = "Akahsizrr/fuse-1-Lite"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True,
)

messages = [{"role": "user", "content": "Write a Python function to check if a number is prime."}]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to(model.device)

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=1024,
        do_sample=True,
        temperature=0.1,
        top_k=50,
        repetition_penalty=1.1,
        pad_token_id=tokenizer.pad_token_id,
    )

response = tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True)
print(response)

Quantization & Deployment

Pre-quantized Versions

bitsandbytes 4-bit (NF4) — 3.36 GB VRAM

from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
import torch

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
)

model = AutoModelForCausalLM.from_pretrained(
    "Akahsizrr/fuse-1-Lite",
    quantization_config=bnb_config,
    device_map="auto",
    trust_remote_code=True,
)
tokenizer = AutoTokenizer.from_pretrained("Akahsizrr/fuse-1-Lite")

bitsandbytes 8-bit — 6.00 GB VRAM

from transformers import AutoModelForCausalLM, BitsAndBytesConfig
import torch

bnb_config = BitsAndBytesConfig(load_in_8bit=True)

model = AutoModelForCausalLM.from_pretrained(
    "Akahsizrr/fuse-1-Lite",
    quantization_config=bnb_config,
    device_map="auto",
    trust_remote_code=True,
)

Disable Coding Experts (Pure LFM2 Mode)

# The model includes a coding toggle — disable experts for pure LFM2 inference
model.set_coding_enabled(False)

# Re-enable for coding tasks
model.set_coding_enabled(True)

vLLM — High-Throughput Serving

fuse-1 Lite is supported in vLLM via a plugin that extends vLLM's native LFM2 implementation with expert augmentation layers.

Plugin repo: Akahsizrr/fuse-1-Lite-vLLM

# Install the plugin
pip install git+https://huggingface.co/Akahsizrr/fuse-1-Lite-vLLM

# Serve with vLLM
vllm serve Akahsizrr/fuse-1-Lite \
  --mamba-cache-mode align \
  --max-model-len 4096

from vllm import LLM

llm = LLM(
    model="Akahsizrr/fuse-1-Lite",
    mamba_cache_mode="align",
    max_model_len=4096,
)
output = llm.generate("Write a Python function to check if a number is prime.")

The plugin registers Fuse3ForCausalLM with vLLM's ModelRegistry via the vllm.general_plugins entry point. It reuses vLLM's native LFM2 attention and short-conv layers, adding the expert MoE block after each augmented layer's FFN.

MLX (Apple Silicon)

fuse-1 Lite is available in MLX format for Apple Silicon (M1+).

MLX repo: Akahsizrr/fuse-1-Lite-MLX

from mlx_lm import load, generate

model, tokenizer = load("Akahsizrr/fuse-1-Lite-MLX", trust_remote_code=True)

prompt = tokenizer.apply_chat_template(
    [{"role": "user", "content": "Write a Python function to check if a number is prime."}],
    tokenize=False, add_generation_prompt=True,
)

response = generate(model, tokenizer, prompt=prompt, max_tokens=512)
print(response)

# CLI
mlx_lm.generate --model Akahsizrr/fuse-1-Lite-MLX --trust-remote-code --prompt "Write a Python fizzbuzz"

The MLX model file (fuse3_mlx.py) extends MLX's native LFM2 implementation with the same expert MoE augmentation. It uses model_file in config.json with trust_remote_code=True for loading.

GGUF / llama.cpp

fuse-1 Lite is available in GGUF format for llama.cpp.

GGUF repo: Akahsizrr/fuse-1-Lite-GGUF

Note: The GGUF uses the custom fuse3 architecture. Stock llama.cpp cannot load it — you need a llama.cpp fork with Fuse3 support. The GGUF repo includes the C++ graph builder (src/models/fuse3.cpp), Python converter (conversion/fuse3.py), and integration guide (INTEGRATION.md).

# Build llama.cpp with Fuse3 support (see INTEGRATION.md in the GGUF repo)
./llama-cli -m fuse-1-Lite-f16.gguf \
  -p "Write a Python function to check if a number is prime." \
  -n 512 --temp 0.1

The C++ implementation reuses LFM2's attention and short-conv graph builders, adding the expert MoE block (router → top-k → SwiGLU experts → scale → add) after each augmented layer's dense FFN.

Transformers (Universal)

The recommended way to run fuse-1 Lite on any platform:

pip install transformers torch bitsandbytes accelerate

Performance

VRAM Requirements

Sample Outputs

Prompt: "Write a Python function to check if a string is a palindrome."

Output (excerpt):

def is_palindrome(s: str) -> bool:
    """
    Return True if *s* reads the same forwards and backwards,
    ignoring case and non-alphanumeric characters.
    """
    cleaned = ''.join(ch.lower() for ch in s if ch.isalnum())
    return cleaned == cleaned[::-1]

The model produces complete implementations with docstrings, type hints, complexity analysis, and test cases.

Limitations

1. Custom architecture: Requires trust_remote_code=True — the model includes custom Fuse3ForCausalLM code
2. No vLLM support: The custom MoE augmentation is not yet supported by vLLM's optimized inference engine
3. No GGUF/MLX conversion: The custom architecture cannot be directly converted to GGUF or MLX format
4. Training data was small: Only 55 examples were used for router training — the router may not generalize perfectly to all coding tasks
5. Expert compatibility: Qwen3.6 experts operate on LFM2's activation space with std normalization — some expert knowledge may be lost in translation
6. use_cache=False during training: Augmented layers don't propagate KV cache correctly during training; generation uses the standard cache
7. bitsandbytes quantization: 4-bit and 8-bit quantization work at runtime via BitsAndBytesConfig — pre-quantized saved versions are not available as separate repos

Citation

@misc{fuse1lite2026,
  title={fuse-1 Lite: Coding-Enhanced Mixture-of-Experts via Expert Transplantation},
  author={Vasko Djack},
  year={2026},
  publisher={HuggingFace},
  url={https://huggingface.co/Akahsizrr/fuse-1-Lite}
}

Acknowledgments

• LiquidAI for the LFM2.5-2.6B host model
• Qwen Team for the Qwen3.6-35B-A3B donor model
• Modal for compute infrastructure

License

Apache 2.0 — See LICENSE for details.

Built with Devin — Cognition AI