AAIE-Distilled Dense Instruct (Llama-compatible export)
A LlamaForCausalLM-format export of AAIE-Distilled Dense Instruct — same weights, same architecture, just registered under vLLM's/transformers' native Llama implementation instead of custom trust_remote_code.
Why this exists
AAIE-Distilled Dense's architecture — GQA attention, RoPE, SwiGLU FFN, RMSNorm, tied embeddings,
no biases anywhere — happens to be structurally identical to Llama's, just with different
tensor/field names (d_model vs hidden_size, n_q_heads vs num_attention_heads, etc.). This
export is a pure state_dict + config remap, not a retrain: every tensor is copied unchanged
into the equivalent Llama-named slot. Verified byte-for-byte numerically identical to the
original via a CPU-only, trust_remote_code-free transformers sanity check (same prompt, same
greedy decode, same output text).
Payoff: this loads natively in vLLM (no trust_remote_code, no custom model backend) using
vLLM's own hand-optimized LlamaForCausalLM implementation — confirmed working end-to-end,
generating at ~430 tok/s, vs. ~2 tok/s for the original architecture's uncached generation loop.
Usage with vLLM
from vllm import LLM, SamplingParams
llm = LLM(model="namquangstudy/aaie-ddense-gft-llama")
params = SamplingParams(max_tokens=150, temperature=0.8)
out = llm.generate(["What is a database index?\n\nResponse:"], params)
print(out[0].outputs[0].text)
Environment note: if you hit a RuntimeError: The NVIDIA driver on your system is too old
error, it means the vLLM/torch build you pip-installed targets a newer CUDA than your GPU node's
driver supports (this happened during testing: pip install vllm grabbed torch 2.13/CUDA 13,
which failed on a driver that only supports CUDA 12.2). This has nothing to do with this specific
model — the exact combination confirmed working here is:
pip install torch==2.4.0 --index-url https://download.pytorch.org/whl/cu121
pip install vllm==0.6.3.post1
pip install transformers==4.46.3
If you also have a broken/incompatible tensorflow install under ~/.local (common on shared
clusters), set PYTHONNOUSERSITE=1 before running — conda/venv environments still see
~/.local/lib/pythonX.Y/site-packages by default, and transformers will try to import
tensorflow during model loading and fail if that install is broken.
Usage with plain transformers (no vLLM)
from transformers import AutoModelForCausalLM, AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("namquangstudy/aaie-ddense-gft-llama")
model = AutoModelForCausalLM.from_pretrained("namquangstudy/aaie-ddense-gft-llama")
# no trust_remote_code needed - this is a plain LlamaForCausalLM
Full details
Architecture, training data, hyperparameters, and evaluation results (LLM-judge score 7.50/10 on held-out IT/CS assignment-feedback topics) are documented on the original AAIE-Distilled Dense Instruct model card — this export carries the exact same weights and behavior, just packaged for vLLM/ecosystem compatibility.