Qwen3.8-Flash-Next-REAM-60Pct

REAM-compressed version of Qwen/Qwen3.8-Flash-Next, produced with Akicou/ream, a REAM/REAP-style MoE compression framework.

Method

Router Expert Activation Merging (REAM). The model is calibrated on the built-in hardcoded prompt set, then experts are grouped by gated similarity (half hidden-state, half router distribution) and merged with saliency-weighted averaging. 40% of the routed experts are removed, so each layer goes from 512 experts down to 308. The router is shrunk to match, keeping only the centroid rows.

Shared experts, attention, n-gram embeddings, and dense layers are left untouched. Only routed experts are merged.

How it was created

python examples/compress_sequential.py \
    --model Qwen/Qwen3.8-Flash-Next \
    --output ./qwen4-ream-60 \
    --target-ratio 0.60 \
    --samples 100 \
    --max-seq-len 512 \
    --batch-size 4 \
    --max-tokens 2048 \
    --cpu-merge \
    --fast-merge \
    --seed 42

Hardware: 6× NVIDIA H100 SXM 80GB

Note

This is an experimental release. It has not been benchmarked. The model uses trust_remote_code=True, same as the base model.

Basic usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "Akicou/Qwen3.8-Flash-Next-REAM-60Pct"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True,
)

prompt = "Explain the concept of reinforcement learning."
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    output = model.generate(**inputs, max_new_tokens=128, do_sample=False)

print(tokenizer.decode(output[0], skip_special_tokens=True))