QTensor TinyLlama 1.1B — Asymmetric v2

QTensor TinyLlama 1.1B Asymmetric v2 is a fully compressed, production-ready variant of TinyLlama/TinyLlama-1.1B-Chat-v1.0, built using the Information-Theoretic Asymmetric Architecture — the world's first implementation of entropy-mapped dynamic SVD rank allocation combined with AWQ-protected INT4 MLP quantization.

The model was trained for 10,000 steps of Quantization-Aware Distillation (QAD) using KL-Divergence + Hidden-State MSE distillation against the FP16 teacher, with all <pad> tokens masked from the loss to ensure true semantic convergence.

📊 Verified Results (NVIDIA RTX 5080, CUDA 13.2)

Note on perplexity: The 93.65 word perplexity represents a 44% improvement over the flat-rank baseline (168.46). The INT4 MLP quantization introduces an information bottleneck that bounds the theoretical minimum perplexity for this compression depth.

🧬 Architecture

This model uses a three-tier hybrid compression topology:

Key Innovations

• Shannon Entropy Rank Allocator: Singular values are normalized into a probability distribution. Shannon entropy $H = -\sum p \log p$ maps each layer to its optimal rank budget. Low-entropy (structurally simple) layers use $r=8$; high-entropy (information-dense) layers expand to $r=32$.
• Corrected AWQ Identity: $W_{packed} = \text{INT4}(W / S)$ with $X_{scaled} = X \times S$ in the forward pass — the mathematically correct direction for protecting salient activation channels without destroying surrounding weights.
• Vectorized INT4 Triton Kernel: BLOCK_K=64 tile loading prevents register spilling, achieving 2× throughput over naïve scalar implementations.
• Masked Distillation Loss: KLDiv and MSE computed exclusively over non-padding tokens for true semantic convergence.

🚀 Quickstart

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "trentzap/QTensor-TinyLlama-1.1B-Asymmetric-v2"

tokenizer = AutoTokenizer.from_pretrained(model_id)

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    torch_dtype=torch.bfloat16,
    device_map="cuda"
)

messages = [{"role": "user", "content": "Explain entropy-based SVD rank allocation."}]
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")

with torch.no_grad():
    outputs = model.generate(**inputs, max_new_tokens=200, temperature=0.7, top_p=0.9)
    print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))

📦 Training Details

📄 License

MIT License — see LICENSE