Qwen3-VL-1.0B-A0.4B-Instruct
This is a tiny version of Qwen/Qwen3-VL-30B-A3B-Instruct created for testing and development.
Model Details
- Base Model: Qwen/Qwen3-VL-30B-A3B-Instruct
- Architecture: qwen3_vl_moe
- Total Parameters: 1.434B
- Activated Parameters: ~0.4B (8 of 128 experts active per token)
Configuration Changes
The following parameters were reduced from the original model:
| Parameter | Original | Tiny |
|---|---|---|
text_config.num_hidden_layers | 48 | 1 |
vision_config.depth | 27 | 4 |
vision_config.deepstack_visual_indexes | [8, 16, 24] | [1, 2, 3] |
text_config.hidden_size | 2048 | 2048 (unchanged) |
text_config.num_local_experts | 128 | 128 (unchanged) |
text_config.num_experts_per_tok | 8 | 8 (unchanged) |
text_config.moe_intermediate_size | 768 | 768 (unchanged) |
text_config.num_attention_heads | 32 | 32 (unchanged) |
text_config.num_key_value_heads | 4 | 4 (unchanged) |
vision_config.hidden_size | 1152 | 1152 (unchanged) |
Checkpoint Structure
The model is saved as a single model.safetensors file (89 tensors). The checkpoint key structure matches the original Qwen/Qwen3-VL-30B-A3B-Instruct exactly.
Usage
from transformers import Qwen3VLMoeForConditionalGeneration, AutoTokenizer
model = Qwen3VLMoeForConditionalGeneration.from_pretrained(
"inference-optimization/Qwen3-VL-1.0B-A0.4B-Instruct",
device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained("inference-optimization/Qwen3-VL-1.0B-A0.4B-Instruct")
input_ids = tokenizer("According to all known laws", return_tensors="pt").input_ids.to(model.device)
output = model.generate(input_ids, max_new_tokens=20)
print(tokenizer.decode(output[0]))
Note: AutoModelForCausalLM does not support this model type. Use Qwen3VLMoeForConditionalGeneration or AutoModelForImageTextToText directly.
Creation Process
This model was created using the llm-compressor create-tiny-model claude skill.
- Config fetched from
Qwen/Qwen3-VL-30B-A3B-Instructwithout downloading weights (skip_weights_download) - Text depth reduced from 48 → 1 layers; vision depth reduced from 27 → 4 blocks
- All parameters randomly initialized with
initializer_range=0.02(matching the base model) - MoE expert parameters (
gate_up_proj,down_proj,gate.weight) explicitly initialized since they are barenn.Parametertensors, notnn.Linearmodules - Fine-tuned on a toy text dataset (internet copypastas) until perplexity ≤ 3.0
Validation
Success: perplexity=1.0001 <= 10.0
Generated: According to all known laws of aviation, there is no way a bee should be able to fly. Its wings are too small to get its fat little
Notes
- This is a text-only fine-tuned model. The vision encoder weights are randomly initialized and not fine-tuned; the model is intended for text-based testing only.
- The MoE routing architecture (128 experts, 8 active per token) is preserved in full to match the original model's structural properties.
- Fine-tuning converged in ~160 steps with learning rate 1e-4.