QTensor Meta-Llama-3-8B — ASVD-Bridge Hardware Demo

Full 8-Billion Parameter Architecture Compressed to 4.67 GB VRAM Footprint

Base Architecture: Meta-Llama-3-8B (8.03B Parameters) | Effective Footprint: 4.67 GB | Engine: ASVD-Bridge (SVD + INT4 QAT)

🛑 CRITICAL DISCLAIMER: THIS IS AN 8B HARDWARE & ARCHITECTURE DEMO (STEP 230 OF 5,000) This checkpoint is released strictly to prove that an 8-Billion parameter Llama-3 model can execute natively inside a 4.67 GB VRAM footprint at 32+ TPS on consumer GPUs using the ASVD-Bridge architecture.

It is not a functional chat model yet. At Step 230, the model is successfully re-learning token boundaries and basic grammar, but high-level semantic reasoning is still degraded from the initial compression shock. Coherent conversational weights will arrive with the full 5,000-step completion.

QTensor Meta-Llama-3 8B Asymmetric Demo is a fully compressed, sub-5GB footprint variant of Meta-Llama-3-8B, built using ASVD-Bridge (Asymmetric Singular Value Decomposition with Subspace Stabilization) — combining SVD rank allocation with Fake-INT4 MLP quantization.

The model was trained for 230 steps (Demo Checkpoint) of Quantization-Aware Distillation (QAD) using KL-Divergence distillation against the FP16 teacher, with all <pad> tokens masked from the loss to ensure true semantic convergence.

📊 Verified Results (NVIDIA RTX 5080, CUDA 13.2)

🧬 Architecture

This model uses a hybrid compression topology:

💡 Why does Hugging Face show "2B params"?
The Hugging Face Hub metadata scanner calculates parameter count purely from tensor matrix dimensions on disk. Because ASVD-Bridge truncates attention projection matrices into rank-$r$ SVD factors ($d \times r$ and $r \times d$), the raw parameter count physically sums to ~2.1B weights while preserving the full 8B layer depth ($L=32$), embedding dimension ($d=4096$), and MLP width ($d_{mlp}=14336$).

🎯 Intended Use & Out-of-Scope Use

• Intended Use: This artifact is intended for ML researchers and engineers to verify the ASVD-Bridge inference engine, measure FLOP reductions, and test the 4.67 GB physical VRAM boundary.
• Out-of-Scope Use: This model should not be used for factual Q&A, code generation, reasoning tasks, or production deployments. The semantic weights are mid-distillation and will hallucinate heavily.
• Dataset Limitations: The distillation utilizes the Alpaca dataset (52K instructions). As a result, even upon full convergence, the model will inherit the biases and limitations present in the original Alpaca instruct-tuning data.

🚀 Quickstart

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "trentzap/QTensor-Llama-3-8B-Demo"
tokenizer = AutoTokenizer.from_pretrained(model_id)

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    torch_dtype=torch.bfloat16,
    device_map="cuda"
)

messages = [{"role": "user", "content": "Explain entropy-based SVD rank allocation."}]
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")

with torch.no_grad():
    outputs = model.generate(**inputs, max_new_tokens=200, temperature=0.7, top_p=0.9)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))

📦 Training Details

🔗 Source Code

The distillation training loop and the export engine are fully open-source. Researchers can verify the SVD and LoRA mathematical folding here: https://github.com/Trentzap1/qtensor-engine

📄 License

MIT License