TinyMoE-200M-2x16
TinyMoE-200M-2x16 is a compact, highly efficient Sparse Mixture of Experts (MoE) language model built upon the Mixtral/Mistral architecture. Designed for research, edge applications, and resource-constrained environments, this model leverages an expert-routing mechanism to balance a larger total parameter capacity with ultra-low computational overhead during inference.
Model Details
- Architecture: Sparse Mixture of Experts (MoE)
- Total Parameters: ~200m total parameters
- Active Parameters per Token: ~50M active parameters
- Expert Configuration: 16 total local experts, 2 active experts routed per token (
num_experts_per_tok": 2) - Context Length: 1024 tokens
- Base Architecture: Mixtral / Mistral For Causal LM
- License: MIT
Training Data
This model was trained onFineWeb-Edu-Dedup 60% and Cosmopedia-v2 40%.
Quick Start
You can load and experiment with this model using the Hugging Face transformers library:
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "FlameF0X/TinyMoE-200m-2x16"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)
input_text = "Wikipedia is a free"
inputs = tokenizer(input_text, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=50)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))