MiniCPM-V-4.6-GPTQ INT4 C256 P6K CTX8K on AXERA NPU

Ready-to-run deployment package for openbmb/MiniCPM-V-4.6-GPTQ on AX650 / NPU3.

• The packaged text runtime uses the GPTQ INT4 AX650 build.
• The compiled text profile uses prefill_len=256, kv_cache_len=8191, and a last_kv_cache_len ladder up to 6144 (P6K).
• The packaged vision runtime uses a fixed-shape 448x448 MiniCPM-V-4.6 vision encoder.
• The package supports text-only chat, single-image understanding, and video understanding through axllm.
• The default runtime config keeps video content available for later text-only follow-up turns.
• The package includes sample image and video assets for board-side validation.

Supported Platform

• AX650 / NPU3

Performance

All measurements below were taken on AX650 / NPU3 with the packaged axllm serve runtime. TTFT stands for time to first token.

For the rows below, TTFT is measured end-to-end from request arrival to the first generated token. The image row uses the packaged assets/sample.png image. The video row uses the packaged assets/red-panda-openai.mp4 sample with video:/path/to/video.mp4:2. The long-context row uses a text-only retrieval prompt that fills the packaged P6K profile to 6260 input tokens and asks the model to recover an exact code placed near the beginning of the prompt.

The packaged runtime uses the following context layout:

• prefill_len=256
• kv_cache_len=8191
• prefill_max_token_num=6400

In the validated long-context text retrieval run above, the model returned the expected exact code from the beginning of the prompt.

Multi-turn Long Dialogue Validation

The packaged text runtime was validated with an accumulated 6-turn conversation that filled the history close to the P6K profile and then repeatedly recalled facts introduced in the earliest turn.

Board-side results showed that:

• the earliest facts remained correct across all six turns
• short recall turns stayed around 506-508 ms TTFT
• requests beyond the current context budget returned a clear context-length error instead of hanging or producing corrupted output

Video Follow-up Validation

The packaged axllm binary was also validated with a video-first conversation followed by text-only turns that continue discussing the same video content.

Board-side results:

TURN1: The video shows a red panda and another red panda playing with a rope.
TURN2: 2
TURN3: rope

During this validation, later text-only turns answered correctly without re-sending the video input.

Startup Runtime Footprint

The runtime CMM increment above was measured on a shared AX650 board by comparing /proc/ax_proc/mem_cmm_info before and after service startup. On the validated board, CMM usage increased from 275020 KB to 1925236 KB.

Vision Encoder Latency

The packaged vision encoder is the same fixed-shape 448x448 AX650 encoder used by the current AX650 GPTQ package, so its standalone latency is unchanged.

For the shipped vision encoder:

• vision_width = 448
• vision_height = 448
• vision_patch_size = 14
• patch grid = (448 / 14) x (448 / 14) = 32 x 32
• raw patch tokens = 32 x 32 = 1024
• packaged visual compression path = 16x
• Soft Tokens = 1024 / 16 = 64

Soft Tokens is fixed by the packaged vision encoder in this repository and is not a runtime-configurable value.

Package Layout

.
├── README.md
├── .gitattributes
├── bin/
│   ├── axllm
│   └── axllm.version.json
├── assets/
│   ├── openai_api_demo.png
│   ├── red-panda-openai.mp4
│   ├── sample.png
│   ├── sample_1.png
│   └── smoke_image.png
├── minicpmv4_6_vision_448.axmodel
├── qwen3_5_text_p256_l0_together.axmodel
├── ...
├── qwen3_5_text_p256_l23_together.axmodel
├── qwen3_5_text_post.axmodel
├── model.embed_tokens.weight.bfloat16.bin
├── config.json
├── post_config.json
└── minicpm_v46_tokenizer.txt

This package keeps the runtime files at the repository root so it can be served directly by axllm. Board-side multimodal runs may create a temporary vision_cache/ directory at the repository root. That directory is runtime-generated and should not be included in release uploads.

Sample Image

The package includes a sample image for board-side validation: assets/sample.png

Sample Video

The package includes a sample video for board-side validation:

• assets/red-panda-openai.mp4

Download the Package

mkdir -p AXERA-TECH/MiniCPM-V-4.6-GPTQ-INT4-C256-P6K-CTX8K
cd AXERA-TECH/MiniCPM-V-4.6-GPTQ-INT4-C256-P6K-CTX8K
hf download AXERA-TECH/MiniCPM-V-4.6-GPTQ-INT4-C256-P6K-CTX8K --local-dir .

Install axllm

Option 1: use the validated binary included in this repository:

chmod +x ./bin/axllm

Option 2: install axllm from the public repository:

git clone -b axllm https://github.com/AXERA-TECH/ax-llm.git
cd ax-llm
./install.sh

Option 3: install with a one-line command:

curl -fsSL https://raw.githubusercontent.com/AXERA-TECH/ax-llm/axllm/install.sh | bash

Run on the Board

The package root is already arranged for axllm, so no extra runtime path arguments are required.

For multimodal testing, you can use ./assets/sample.png or ./assets/red-panda-openai.mp4.

./bin/axllm run .

In interactive mode:

• press Enter directly for text-only chat
• input an image path for single-image chat
• input video:/path/to/frames_dir or video:/path/to/video.mp4 for video chat

Serve with axllm

From the package root on the board:

./bin/axllm serve . --port 8000

Expected model id:

AXERA-TECH/MiniCPM-V-4.6-GPTQ-AX650-INT4-C256-P6K-CTX8K

Health check:

curl http://127.0.0.1:8000/health
curl http://127.0.0.1:8000/v1/models

Packaged context profile:

• max_token_len = 8191
• kv_cache_size = 512, kv_cache_num = 8191
• prefill_token_num = 256
• prefill_max_token_num = 6400

Request Examples

Text Request

curl http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "AXERA-TECH/MiniCPM-V-4.6-GPTQ-AX650-INT4-C256-P6K-CTX8K",
    "messages": [
      {"role": "user", "content": "1+1 equals what? Reply with digits only."}
    ],
    "max_tokens": 32,
    "temperature": 0
  }'

Image Request

axllm serve expects the image payload as a data URI. From the package root:

python3 - <<'PY'
import base64
import json
from pathlib import Path
from urllib.request import Request, urlopen

img = Path("assets/sample.png").read_bytes()
payload = {
    "model": "AXERA-TECH/MiniCPM-V-4.6-GPTQ-AX650-INT4-C256-P6K-CTX8K",
    "messages": [
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "Please briefly describe this image."},
                {
                    "type": "image_url",
                    "image_url": {
                        "url": "data:image/png;base64," + base64.b64encode(img).decode()
                    },
                },
            ],
        }
    ],
    "max_tokens": 64,
    "temperature": 0,
}
req = Request(
    "http://127.0.0.1:8000/v1/chat/completions",
    data=json.dumps(payload).encode(),
    headers={"Content-Type": "application/json"},
)
with urlopen(req, timeout=300) as resp:
    print(resp.read().decode())
PY

Example output:

{
  "choices": [
    {
      "message": {
        "role": "assistant",
        "content": "The image shows a colorful, cartoon-style red lobster or lobster-like character with a cheerful expression, raised claws, and a dynamic pose."
      },
      "finish_reason": "stop"
    }
  ],
  "model": "AXERA-TECH/MiniCPM-V-4.6-GPTQ-AX650-INT4-C256-P6K-CTX8K",
  "object": "chat.completion"
}

Video Request

axllm serve accepts either a frames directory or a raw video file. To test the packaged sample video from the package root:

VIDEO_PATH="$(pwd)/assets/red-panda-openai.mp4"

Then use video:${VIDEO_PATH}:2 in the request payload:

curl http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d "{
    \"model\": \"AXERA-TECH/MiniCPM-V-4.6-GPTQ-AX650-INT4-C256-P6K-CTX8K\",
    \"messages\": [
      {
        \"role\": \"user\",
        \"content\": [
          {\"type\": \"image_url\", \"image_url\": {\"url\": \"video:${VIDEO_PATH}:2\"}},
          {\"type\": \"text\", \"text\": \"Describe this video briefly.\"}
        ]
      }
    ],
    \"max_tokens\": 128
  }"

Example output:

{
  "choices": [
    {
      "message": {
        "role": "assistant",
        "content": "The video shows two red pandas, one on a branch and the other on the ground, interacting with a rope."
      },
      "finish_reason": "stop"
    }
  ],
  "model": "AXERA-TECH/MiniCPM-V-4.6-GPTQ-AX650-INT4-C256-P6K-CTX8K",
  "object": "chat.completion"
}

Conversion References

If you need the original model files or want to rebuild the deployment artifacts, start with:

• Original Hugging Face model: openbmb/MiniCPM-V-4.6-GPTQ
• AXERA conversion and deployment workflow: AXERA-TECH/MiniCPM-V-4.6.axera

Discussion

• GitHub Issues
• QQ group: 139953715