tiny-random-DeepseekV2ForCausalLM
Randomly initialised DeepseekV2ForCausalLM for testing MLA (multi-head latent
attention) and MoE code paths without loading a 29 GiB checkpoint. Used by
vLLM's tests/models/transformers/test_backend.py::test_mla.
The attention head dimensions are copied verbatim from
deepseek-ai/DeepSeek-V2-Lite-Chat
so that kv_lora_rank + qk_rope_head_dim == 576, the latent head size every MLA
kernel is built around. Only the width, depth and expert counts are shrunk.
| DeepSeek-V2-Lite-Chat | this model | |
|---|---|---|
kv_lora_rank | 512 | 512 |
qk_nope_head_dim | 128 | 128 |
qk_rope_head_dim | 64 | 64 |
v_head_dim | 128 | 128 |
hidden_size | 2048 | 256 |
num_hidden_layers | 27 | 4 |
num_attention_heads | 16 | 4 |
n_routed_experts | 64 | 8 |
n_shared_experts | 2 | 2 |
| parameters | 15.7B | 59.8M |
Layer 0 is dense and the rest are MoE (first_k_dense_replace=1), matching the
real model. The tokenizer is DeepSeek-V2-Lite-Chat's, unmodified.
The weights are random, so the outputs are meaningless. This is only useful for checking that two implementations agree with each other.