parmanu-lcs2/Phi-4-7B
This model is a pruned version of Phi-4, compressed using SNIPER (paper).
Pruning details
| Setting | Value |
|---|---|
| Pruning method | SNIPER (paper) |
| Target compression ratio | 50% |
| Parameters after pruning | 7,453,322,240 |
| Calibration data | slim_orca (50 samples × 512 tokens) |
Recovery fine-tuning
LoRA fine-tuned on 2000 samples of SlimOrca after pruning. The LoRA adapters have been merged into the base weights.
| Setting | Value |
|---|---|
| Epochs | 1 |
| Context length | 1024 |
| Learning rate | 0.0002 |
| LoRA rank / alpha | 64 / 16 |
| LoRA target modules | gate_up_proj, down_proj, qkv_proj, o_proj |
Hardware
All pruning and fine-tuning was performed on an NVIDIA A100 GPU.
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained("parmanu-lcs2/Phi-4-7B", trust_remote_code=True, dtype="auto")
tokenizer = AutoTokenizer.from_pretrained("parmanu-lcs2/Phi-4-7B")
trust_remote_code=True is required because pruning leaves layers with different shapes (and some attention/MLP blocks removed). This architecture is defined in the bundled modeling_pruned.py.
References
- Original model:
Phi-4 - SNIPER paper: https://arxiv.org/abs/2608.12953