Qwen-1.5B-ASVD-Healed

Model Summary

Qwen-1.5B-ASVD-Healed is a natively compressed iteration of the Qwen/Qwen2.5-1.5B architecture. It demonstrates a cutting-edge post-training compression methodology combining Activation-Aware Singular Value Decomposition (ASVD) and Pruning-Aware Fine-Tuning (LoRA Healing).

By profiling the mathematical activation states of the network and surgically factoring low-rank approximations, this model permanently eliminated over 72 million parameters (14.29% of the targeted linear layers) without requiring external decoders, custom runtime environments, or quantization overhead. The neural damage induced by this aggressive amputation was subsequently repaired using highly targeted LoRA adapters, which were then mathematically merged back into the base weights.

The resulting model is faster, consumes less VRAM, and operates as a standard native Hugging Face transformer model, proving that structural grammatical integrity and factual knowledge survive deep structural pruning when properly guided by activation states.

Compression Methodology: The Neural Surgery Pipeline

This model was developed using a three-stage compression pipeline aimed at isolating and removing "mathematical vacuum" (dead weight) while preserving inference capability.

1. Activation Profiling (RMS Extraction)

Unlike standard SVD which blindly factors static weights, this methodology applies Activation-Aware SVD. The original model was fed a calibration dataset (TinyShakespeare) to extract the Root Mean Square (RMS) of the activation inputs across all major linear layers (q_proj, k_proj, v_proj, o_proj, down_proj). This provided a scaling matrix that amplified weights essential for language routing and diminished inactive noise.

2. Guided Amputation (ASVD)

With the weights scaled by real-world activations, SVD was applied to decompose the dense matrices: $$W \approx U_k \Sigma_k V_k^T$$ We enforced an aggressive 85% variance retention threshold. The matrices were sliced and reconstructed into bottleneck sequential linear layers (Rank $k$).

• Targeted Parameters Before Amputation: ~1.49 Billion
• Parameters Permanently Removed: 72,384,768 (14.29% reduction in targeted layers)
• Immediate Impact: The massive parameter loss caused immediate severe aphasia and grammatical collapse in the base model (Perplexity spiked to 81.69).

3. Neural Healing (Pruning-Aware Fine-Tuning)

To bypass the severed routing pathways, Low-Rank Adaptation (LoRA) modules were injected across all linear layers (Rank 16, Alpha 32). The model was trained for merely 2 epochs on the calibration dataset using a micro-batch gradient accumulation strategy.

• Healing Parameters: 20.7 Million (~1.38% trainable parameters).
• Result: The optimizer effectively rebuilt the "neural bridges". Grammatical coherence and logical reasoning were fully restored.
• Final Step: The trained LoRA adapters were merged and unloaded into the ASVD-sliced layers, producing a completely native FP16 model with zero external dependencies.

Quantitative Results (Perplexity)

The network's survival and recovery were tracked using absolute perplexity (PPL) on standard calibration sequences.

Usage

The model functions exactly like any standard causal language model. No custom inference code is required.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "your-username/Qwen-1.5B-ASVD-Healed" # Update with your actual repo ID

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id, 
    torch_dtype=torch.float16, 
    device_map="auto"
)

prompt = "The future of artificial intelligence relies on"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    outputs = model.generate(**inputs, max_new_tokens=50)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Intended Use & Limitations

• Edge AI and Local Inference: Designed for environments where VRAM footprint and computational overhead are strict constraints.
• Cost-Efficient SaaS: Suitable for high-throughput batch environments where reduced parameter counts directly translate to larger batch sizes and lower cloud costs.
• Limitations: Due to the aggressive 85% variance retention threshold, while grammatical structure is preserved, highly esoteric or nuanced factual recall may be slightly degraded compared to the intact 1.5B model. Further fine-tuning on domain-specific data is highly recommended for enterprise use cases.

Hardware & Training Configuration

• Frameworks: PyTorch, Transformers, PEFT, TorchAO
• Optimizer: AdamW (lr=3e-4)
• Precision: FP16
• Infrastructure: NVIDIA CUDA Backend