Llama-3.1-8B-Instruct - Renesas X5H

Introduction

This repository contains Llama-3.1-8B-Instruct model, optimized for Renesas X5H platform for text inference.

• Model Architecture: Llama 3.1-8B is an auto-regressive language model that uses an optimized transformer architecture.
• Source Model: meta-llama/Llama-3.1-8B-Instruct

Performance

The following performance metrics were measured with a prompt.

Prerequisites

To run model, you need:

1. Renesas X5H Board
2. Hugging Face CLI: For downloading the model and installer.

Deployment

Llama-3.1-8B-Instruct (W4A16)

1. Download the installer llama3p1-8b-w4a16-runner-0.1.0-Linux.sh from Files and version tab under w4a16\binaries\rcar-x5hv1\xOS_v4.34 folder.
2. Copy the installer to the X5H board and run the installer.

   bash ./llama3p1-8b-w4a16-runner-0.1.0-Linux.sh --prefix=./ --exclude-subdir --skip-license
   

3. Download GGUF Meta-Llama-3.1-8B-Instruct-f16.gguf from Files and version tab under fp16 folder and copy to the installed directory on the X5H board.
4. Expected directory structure on the X5H board.

    llama3p1-8b-w4a16-runner/
    ├── Meta-Llama-3.1-8B-Instruct-f16.gguf
    ├── firmwares
    ├── kernel_modules
    ├── llama3p1-8b-w4a16
    ├── llama3p1-8b-w4a16-runner
    ├── scripts
    └── setup_npu.sh
   

Inference

Llama-3.1-8B-Instruct (W4A16)

bash ./setup_npu.sh
./llama3p1-8b-w4a16-runner <PROMPT>