Qwen3.8-Flash-Next-0.2B-A0.2B
This is a tiny version of Qwen/Qwen3.8-Flash-Next created for testing and development.
Model Details
- Base Model: Qwen/Qwen3.8-Flash-Next
- Architecture: Qwen4ExpForConditionalGeneration (hybrid MoE vision-language model)
- Total Parameters: 0.16B
- Activated Parameters: 0.16B (MoE, 4 of 8 routed experts per token)
The tiny model preserves every architectural component of the original: the hybrid
attention schedule (linear_attention GatedDeltaNet layers and a
qwen_sparse_attention QSA-indexer layer), Per-Layer Embedding (PLE) with hashed
n-gram embeddings on layer 2, hyper-connections, packed 3D MoE experts + a shared
expert, and the vision tower.
Configuration Changes
The following parameters were reduced from the original model:
| Field | Original | Tiny |
|---|---|---|
text.num_hidden_layers | 48 | 4 |
text.hidden_size | 2560 | 256 |
text.num_attention_heads | 24 | 8 |
text.num_key_value_heads | 2 | 2 |
text.num_experts | 512 | 8 |
text.num_experts_per_tok | 10 | 4 |
text.moe_intermediate_size | 640 | 256 |
text.shared_expert_intermediate_size | 640 | 256 |
text.ple_embed_dim | 2560 | 256 |
text.ngram_vocab_size_base | 20000000 | 2048 |
text.vocab_size | 248320 | 248320 |
vision.depth | 27 | 2 |
vision.hidden_size | 1152 | 256 |
vision.intermediate_size | 4304 | 512 |
vision.num_heads | 16 | 8 |
The layer_types schedule keeps one of each original attention type
(["linear_attention", "linear_attention", "linear_attention", "qwen_sparse_attention"]),
ple_layer_ids=[2] and split_ngram_parts=128 are unchanged from the original.
Checkpoint Structure
The checkpoint structure matches the original model. In particular:
- MoE experts are stored as packed 3D tensors
(
...mlp.experts.gate_up_projwith shape(num_experts, 2*moe_intermediate_size, hidden_size)and...mlp.experts.down_proj), matching the original repo.transformers'save_pretrainedsplits these into per-expertLinears, so a post-processing converter re-packs them to match the original layout. - The PLE n-gram embedding table is sharded into 128 parts
(
...ple.ple_embedding.ngram_embedding.shard_{i}), matching the original. - Weights use the
model.language_model.*,model.visual.*, andlm_head.weightprefixes of the original VLM checkpoint. - MTP (multi-token-prediction) weights are intentionally omitted, per the tiny-model convention.
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained("inference-optimization/Qwen3.8-Flash-Next-0.2B-A0.2B", device_map="auto")
tokenizer = AutoTokenizer.from_pretrained("inference-optimization/Qwen3.8-Flash-Next-0.2B-A0.2B")
input_ids = tokenizer("According to all known laws", return_tensors="pt").input_ids.to(model.device)
output = model.generate(input_ids, max_new_tokens=20)
print(tokenizer.decode(output[0]))
Note: this is a vision-language model; load it with Qwen4ExpForConditionalGeneration
(or AutoModelForImageTextToText) if you need the vision tower. Requires
transformers>=5.16 for the qwen4_exp model type.
Creation Process
This model was created using the llm-compressor create-tiny-model claude skill.
- The original config was loaded and shrunk (see the table above) while keeping every architectural component representable.
- The model was constructed directly from the reduced config with randomly initialized weights (avoiding the multi-hundred-GB original download).
- It was fine-tuned on a small toy text dataset to confirm it can learn.
- The saved checkpoint was converted so its MoE experts are packed to match the original checkpoint structure, then validated to load and generate correctly.
Notes
- Fine-tuning converged to a perplexity of ~1.02 on the toy dataset; the model reproduces the memorized text under greedy decoding.
- The packed-expert checkpoint loads to a bit-identical loss versus the pre-conversion per-expert checkpoint, and its tensor structure matches the original repo exactly .