DeepSeek-V4-Pro-0.5B-A0.37B
This is a tiny version of deepseek-ai/DeepSeek-V4-Pro created for testing and development.
Model Details
- Base Model: deepseek-ai/DeepSeek-V4-Pro
- Architecture: deepseek_v4
- Total Parameters: 0.487B
- Activated Parameters: ~0.37B
Configuration Changes
The following parameters were reduced from the original model:
| Parameter | Original | Tiny |
|---|---|---|
| num_hidden_layers | 61 | 8 |
| hidden_size | 7168 | 1024 |
| num_attention_heads | 128 | 8 |
| head_dim | 512 | 64 |
| q_lora_rank | 1536 | 256 |
| o_lora_rank | 1024 | 64 |
| o_groups | 16 | 2 |
| index_n_heads | 64 | 4 |
| index_head_dim | 128 | 32 |
| index_topk | 1024 | 64 |
| n_routed_experts | 384 | 16 |
| moe_intermediate_size | 3072 | 512 |
| sliding_window | 128 | 64 |
Both attention types (heavily_compressed_attention, compressed_sparse_attention) and both MLP types (hash_moe, moe) are preserved:
- layer_types:
[hca, hca, csa, hca, csa, hca, csa, hca] - mlp_layer_types:
[hash_moe, hash_moe, hash_moe, moe, moe, moe, moe, moe]
Checkpoint Structure
Single-file checkpoint (model.safetensors) with 608 tensors. The checkpoint uses the original DeepSeek-V4-Pro naming convention (no model. prefix): layers.N.attn.wq_a.weight, embed.weight, head.weight, etc. Fully loadable with AutoModelForCausalLM.from_pretrained().
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained("inference-optimization/DeepSeek-V4-Pro-0.5B-A0.37B", device_map="auto")
tokenizer = AutoTokenizer.from_pretrained("inference-optimization/DeepSeek-V4-Pro-0.5B-A0.37B")
input_ids = tokenizer("According to all known laws", return_tensors="pt").input_ids.to(model.device)
output = model.generate(input_ids, max_new_tokens=20)
print(tokenizer.decode(output[0]))
Creation Process
This model was created using the llm-compressor create-tiny-model claude skill.
- Inspected DeepSeek-V4-Pro config to identify architecture: dual attention types (heavily_compressed_attention, compressed_sparse_attention), dual MLP types (hash_moe, moe), HyperConnection residual streams (hc_mult=4), and MTP layer.
- Reduced all hidden dimensions while preserving at least one of each layer type.
- Fine-tuned on copypasta dataset until perplexity < 3.0 (achieved 1.001).
- Validated checkpoint structure matches original model naming conventions.
Notes
- The model uses transformers naming convention for tensor keys (compatible with
from_pretrained()). - A converted checkpoint matching the original DeepSeek-V4-Pro key naming is also available (
model.safetensorswith originallayers.N.attn.wq_a.weightstyle names). - FP8 quantization scale tensors from the original model are not present in this unquantized bf16 checkpoint.
- The
num_nextn_predict_layers=1(MTP) layer is preserved in the architecture. - Validation output:
Success: 1.001 <= 10.0