Llama-3.1-8B-Instruct-w4a16-asym-torchao-v0.17.0

Model Overview

• Model Architecture: LlamaForCausalLM
  • Input: Text
  • Output: Text
• Source Model: Llama-3.1-8B-Instruct
• Supported Hardware: AMD EPYC (CPU inference)
• Preferred Operating System: Linux
• Inference Engine: vLLM v0.20.2
• Quantization Framework: TorchAO v0.17.0
• Quantization Method: 4-bit Weight-Only Quantization (W4A16), Asymmetric
• Compatible Stack:
  • ZenDNN v6.0.0
  • ZenTorch v2.11.0.1
  • PyTorch v2.11.0
  • TorchAO v0.17.0
  • vLLM v0.20.2

This model was Built with Llama. This is a quantized version of Llama-3.1-8B-Instruct created by AMD using TorchAO for ZenDNN-optimized CPU inference.

Quantization

The model was quantized from Llama-3.1-8B-Instruct using TorchAO v0.17.0.

• Method: 4-bit Weight-Only Quantization (W4A16), Asymmetric
• Config: Int4WeightOnlyOpaqueTensorConfig(group_size=128)
• Quantized Layers: All linear layers excluding lm_head and embed_tokens

Quantization script:

python woq_asym.py \
    --model_name meta-llama/Llama-3.1-8B-Instruct \
    --output_dir ./Llama-3.1-8B-Instruct-w4a16-asym-torchao-v0.17.0

Note: This quantization method (W4A16-Asym asymmetric WOQ) is specific to the ZenDNN execution path and is not available in native PyTorch.

Quick Start

Use with vLLM

from vllm import LLM, SamplingParams

model = LLM(
    model="amd/Llama-3.1-8B-Instruct-w4a16-asym-torchao-v0.17.0",
    dtype="bfloat16",
)

sampling_params = SamplingParams(temperature=0.7, max_tokens=256)
outputs = model.generate(["Hello, how are you?"], sampling_params)
print(outputs[0].outputs[0].text)

Requirements

torch==2.11.0
torchao==0.17.0
zentorch==2.11.0.1
vllm==0.20.2

OpenMP Setup

For optimal performance, set LD_PRELOAD with libomp.so (LLVM OpenMP) or libiomp5.so (Intel OpenMP):

# Using LLVM OpenMP (llvmopenmp)
export LD_PRELOAD=$(find /path/to/env -name "libomp.so" | head -1)

# Or using Intel OpenMP (libiomp)
export LD_PRELOAD=$(find /path/to/env -name "libiomp5.so" | head -1)

Note: Set LD_PRELOAD before launching vLLM or any inference script.

Evaluation

The model was evaluated against the BF16 (unquantized) baseline on standard benchmarks using lm-evaluation-harness with the vLLM engine.

Evaluation results will be updated after benchmarking.

Reproduction

lm_eval \
    --model vllm \
    --model_args pretrained="amd/Llama-3.1-8B-Instruct-w4a16-asym-torchao-v0.17.0" \
    --tasks mmlu \
    --num_fewshot 5 \
    --batch_size auto

Limitations

• Version Lock: This model is quantized with TorchAO v0.17.0 and is compatible only with PyTorch v2.11.0 / ZenDNN v6.0.0. It will not load correctly on other PyTorch versions.
• CPU Only: This model is optimized for AMD EPYC CPU inference via ZenDNN. It is not intended for GPU inference.
• W4A16-Asym Path: The quantization uses a ZenDNN-specific execution path not available in native PyTorch, so direct comparison with native PyTorch quantization is not possible.

License

This model is distributed under the same license as the source model. See the LICENSE file for details.

Modifications copyright (c) 2026 Advanced Micro Devices, Inc. All rights reserved.