Llama 3.1 8B Instruct · 16x4 EMO-style Routed-LoRA MoE
This is a dense-equivalent initialization of meta-llama/Llama-3.1-8B-Instruct converted to a
memory-efficient routed-LoRA MoE:
- 15 token-routed LoRA experts
- 1 always-active shared LoRA expert
- 3 routed experts + 1 shared expert active per token (16x4 total convention)
- document/prompt-level expert pools inspired by EMO
- training pool size sampled from 3..15; evaluation pool size 8
- rank 16, alpha 32 LoRA deltas on
gate_proj,up_proj, anddown_proj - the dense Llama FFN is stored once and frozen
All LoRA B matrices are zero at initialization. Therefore this base artifact is functionally equivalent to the pinned dense Llama base before training. The router is initialized but cannot change logits while every expert delta is zero.
This is an EMO-style routed-LoRA adaptation, not the original EMO model: the original trains independent full FFN experts from scratch, while this model routes low-rank deltas around one shared Llama FFN.
Load
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "Jongbin-kr/llama3.1-8b-16x4-emo-style-routed-lora-moe"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
model_id,
torch_dtype="auto",
device_map="auto",
trust_remote_code=True,
)
The custom architecture supports generate(), prompt-pool caching during
autoregressive decoding, set_expert_pool(indices), reset_expert_pool(), and
get_router_statistics().
Provenance
- Dense base revision:
0e9e39f249a16976918f6564b8830bc894c89659 - Build repository commit:
e2a53c277b23e252780761b2077cdf87e2e2ee15 - Architecture: custom partial reproduction of EMO routing for LoRA deltas
- EMO paper: https://arxiv.org/abs/2605.06663
- EMO code: https://github.com/allenai/EMO
Use of the Llama weights remains subject to the Llama 3.1 Community License.