Consider using this model with the QLoRA trained by our team: Lora
Qwen3.5-2.8B (Pruned)
This is a selectively pruned version of a Qwen 3.5 4B model, reduced from 32 to 20 transformer layers, resulting in a ~2.8B parameter model.
The pruning was performed with a layer-wise evaluation algorithm that benchmarks each decoder layer individually and removes only those with the lowest contribution to the model's output, preserving as much capability as possible.
Model Details
- Developed by: aldair166
- Model type: Causal Language Model (Hybrid Attention: Linear + Full)
- Language(s): Multilingual
- License: Same as the original Qwen 3.5 4B release
- Finetuned from: Qwen 3.5 4B (original)
- Parameter count: ~2.8B (after pruning)
Architecture
| Property | Value |
|---|---|
architectures | Qwen3_5ForCausalLM |
hidden_size | 2560 |
num_hidden_layers | 20 (pruned from 32) |
num_attention_heads | 16 |
num_key_value_heads | 4 |
head_dim | 256 |
intermediate_size | 9216 |
hidden_act | SiLU |
vocab_size | 248320 |
max_position_embeddings | 262144 |
rope_theta | 10000000 |
partial_rotary_factor | 0.25 |
tie_word_embeddings | true |
dtype | bfloat16 |
Hybrid Attention Layout
The model uses a mix of linear attention and full attention layers (full_attention_interval: 4). After pruning, the 20 retained layers follow this pattern:
Layer 0: linear_attention
Layer 1: linear_attention
Layer 2: linear_attention
Layer 3: full_attention
Layer 4: linear_attention
Layer 5: linear_attention
Layer 6: linear_attention
Layer 7: full_attention
Layer 8: linear_attention
Layer 9: full_attention
Layer 10: linear_attention
Layer 11: linear_attention
Layer 12: full_attention
Layer 13: linear_attention
Layer 14: linear_attention
Layer 15: full_attention
Layer 16: linear_attention
Layer 17: linear_attention
Layer 18: linear_attention
Layer 19: full_attention
Pruning Methodology
Instead of uniform layer dropping (e.g., removing every Nth layer) or simple magnitude-based heuristics, this model was produced using a selective layer pruning algorithm:
- Per-layer importance scoring — each decoder layer was evaluated independently by comparing the model's hidden states and logits with and without that layer active.
- Ranking — layers were ranked by their measured contribution to the final output distribution.
- Selective removal — only the lowest-ranked layers were removed, keeping the structurally and functionally important ones.
- Post-prune verification — the pruned model was checked to ensure the hybrid attention cadence (
full_attention_interval = 4) remained valid and that generation quality stayed coherent.
This approach minimizes the typical degradation seen in naive depth pruning, especially in hybrid linear/full attention models where different layer types play different roles.
Intended Use
- Efficient local inference on consumer GPUs / CPUs
- Fine-tuning as a lighter base model
- Research on pruning and hybrid attention architectures
Limitations
- Not instruction-tuned unless further fine-tuned.
- Some capabilities of the original 4B model are inevitably lost, particularly on long-context reasoning and rare-language tasks.
- Long-context behavior (
max_position_embeddings = 262144) is inherited from the config but not fully validated after pruning.
How to Use
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "aldair166/qwen3.5-2.8b"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
model_id,
dtype="bfloat16",
device_map="auto",
)
prompt = "Explain the difference between linear attention and full attention."
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(out[0], skip_special_tokens=True))
Citation
If you use this model, please also credit the original Qwen 3.5