Franken-MoE-Small-SFT-Preview
Franken-MoE-Small-SFT-Preview is a high-performance native Qwen 3 MoE (Mixture-of-Experts) upcycled from Qwen/Qwen3-0.6B.
It combines a dense foundation pre-trained on 18T tokens with an expanded SwiGLU Mixture-of-Experts architecture using official Qwen3MoeForCausalLM specifications.
Notice
This model has been partially re-trained. It experiences less looping and improved coherency, but is still being evaluated and further trained. This represents a checkpoint after 1000 steps on DrRiceIO7/combined-instruct-sft. It has not been abliterated nor has it been trained on any other dataset, but a full release will follow after this finetuning run.
Architecture Highlights
- Base Foundation:
Qwen/Qwen3-0.6B(28 Layers, Hidden Dim $D=1024$, Intermediate Dim $H=3072$, Vocab Size 151,936) - Total Parameters: 2.45B
- Active Parameters per Token: 860M (Compute cost and generation speed of a <1B model)
- MoE Topology: 8 routed experts, Top-2 active per token (
num_experts_per_tok=2), normalized top-k probabilities (norm_topk_prob=True) - Attention: Grouped-Query Attention (GQA 16:8) with QK-Norm (
q_norm,k_norm) - Context Length: 40,960 tokens (RoPE $ heta = 1,000,000$)
- Official Format: Uses fused 3D tensor layout (
gate_up_projanddown_proj) without legacy shared experts.
Direct Compatibility
- Unsloth: 100% plug-and-play in Google Colab / Unsloth notebooks (
FastLanguageModel.from_pretrained("DrRiceIO7/Franken-MoE-Small-Base")) - llama.cpp / GGUF: Convert directly using
convert_hf_to_gguf.py - Hugging Face Transformers: Loads directly with
AutoModelForCausalLM.from_pretrained("DrRiceIO7/Franken-MoE-Small-Base") - vLLM / SGLang / Ollama: Native out-of-the-box support
Quick Usage
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
model_id = "DrRiceIO7/Franken-MoE-Small-Base"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
model_id,
torch_dtype=torch.bfloat16,
device_map="auto"
)
prompt = "<|im_start|>user\nExplain how photosynthesis works in plants.<|im_end|>\n<|im_start|>assistant\n"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
with torch.no_grad():
outputs = model.generate(**inputs, max_new_tokens=256, temperature=0.7, top_p=0.85, do_sample=True)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))