parmanu-lcs2/Llama-3.1-6B-Instruct
This model is a pruned version of Meta-Llama-3.1-8B-Instruct, compressed using SNIPER (paper).
Pruning details
| Setting | Value |
|---|---|
| Pruning method | SNIPER (paper) |
| Target compression ratio | 25% |
| Parameters after pruning | 6,030,118,912 |
| Calibration data | slim_orca (50 samples × 512 tokens) |
Recovery fine-tuning
LoRA fine-tuned on 2000 samples of SlimOrca after pruning. The LoRA adapters have been merged into the base weights.
| Setting | Value |
|---|---|
| Epochs | 1 |
| Context length | 1024 |
| Learning rate | 0.0002 |
| LoRA rank / alpha | 64 / 16 |
| LoRA target modules | up_proj, gate_proj, down_proj, q_proj, o_proj, k_proj, v_proj |
Hardware
All pruning and fine-tuning was performed on an NVIDIA A100 GPU.
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained("parmanu-lcs2/Llama-3.1-6B-Instruct", trust_remote_code=True, dtype="auto")
tokenizer = AutoTokenizer.from_pretrained("parmanu-lcs2/Llama-3.1-6B-Instruct")
trust_remote_code=True is required because pruning leaves layers with different shapes (and some attention/MLP blocks removed). This architecture is defined in the bundled modeling_pruned.py.
References
- Original model:
Meta-Llama-3.1-8B-Instruct - SNIPER paper: https://arxiv.org/abs/2608.12953