Qwen-3.5-4B-Auto-ASVD

Qwen-3.5-4B-Auto-ASVD is a structurally factorized and fine-tuned release of Qwen/Qwen3.5-4B. It uses an automated second-order pruning framework that combines empirical Hessian sensitivity profiling (Fisher Information) with Activation-Aware Singular Value Decomposition (ASVD) and a subsequent LoRA healing stage with cosine learning rate scheduling.

All operations were executed and merged in native bfloat16. Unlike standard uniform low-rank matrix factorizations, this variant calculates dynamic layer-by-layer variance retention targets based on loss surface curvature, avoiding arbitrary flat rank truncation across attention heads and MLP blocks.

Engineering Pipeline

1. Second-Order Curvature Profiling (Hessian Approximation)

To prevent the catastrophic degradation common in uniform pruning, gradient energy across all projection modules was sampled over long sequences (1024-token chunks) using gradient checkpointing:

$$\text{diag}(H) \approx \mathbb{E}\left[ (\nabla_W \mathcal{L})^{\circ 2} \right]$$

Modules exhibit starkly distinct curvature profiles—especially with Qwen 3.5's hybrid architecture comprising both standard attention and Gated DeltaNet layers. Rather than applying a linear scaling function dominated by extreme outliers, sensitivity scores were mapped to variance thresholds via log-smoothing:

$$\tilde{s} = \frac{\log(1 + s) - \log(1 + s_{\min})}{\log(1 + s_{\max}) - \log(1 + s_{\min}) + \epsilon}$$

$$\text{Target Variance} = 0.82 + 0.13 \cdot \tilde{s}$$

This automatically allocated higher variance budgets (up to 95.00%) to core routing paths like down_proj, while tolerating tighter truncation (down to 91.60%) on less sensitive projection matrices.

2. Scaled SVD Surgery

Each target linear operator was scaled using activation RMS values collected during forward passes:

$$W_{\text{scaled}} = W \odot s_{\text{RMS}}$$

Truncated SVD was performed on-device:

$$W_{\text{scaled}} \approx U_k \Sigma_k V_k^T$$

where the rank $k$ was resolved per matrix to satisfy the layer-specific variance budget. The factors were then mapped into sequential bottleneck layers (nn.Sequential(Linear(in, k), Linear(k, out))) and converted to bfloat16.

• Target Layers Factored: 64 projection matrices (q_proj, k_proj, v_proj, o_proj, down_proj).
• Parameters Excised: 47,821,312 parameters (6.86% structural reduction across target matrices).

3. LoRA Neural Healing

Following structural factorization, low-rank degradation was resolved via parameter-efficient fine-tuning on Wikitext-2:

• Adapter Topology: Rank 32, Alpha 64 across all linear layers.
• Optimization: AdamW (lr = 2e-4), gradient clipping (1.0), micro-batch size 1, gradient accumulation steps 16 (effective batch size 16).
• Schedule: Cosine decay with 10% warmup over 2 epochs.
• Loss Trajectory: Initial loss converged from 2.2911 (Epoch 1) down to 2.0432 (Epoch 2).

All adapters were permanently merged back into the base weights (merge_and_unload), producing a standalone checkpoint with zero runtime inference overhead or custom kernel dependencies.

Run Metrics

Quickstart

This model is a drop-in replacement for any pipeline supporting standard causal language models:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "Kodjaoglanian/Qwen-3.5-4B-Auto-ASVD"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto"
)

prompt = "The most important concept in quantum physics is"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=80,
        do_sample=True,
        temperature=0.7,
        top_p=0.9,
        repetition_penalty=1.15
    )

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Limitations

• Specialized Domain Drift: The healing phase utilized a 1024-token calibration stream from general natural language data (Wikitext-2). For domain-specific production workloads (e.g., medical diagnostics, strict syntactic code generation, legal drafting), task-specific fine-tuning on top of this checkpoint is recommended.
• Kernel Dependencies: When running the base architecture, warnings regarding missing optional acceleration libraries (causal_conv1d, flash-linear-attention) may appear in vanilla PyTorch environments. The model defaults safely to standard reference operations without loss of accuracy.