Qwen3-4B-Instruct-2507 — QNN / Genie (w4a16, Hexagon HTP)

中文说明：README.zh-CN.md

A Qualcomm QNN (AI Engine Direct) build of Qwen3-4B-Instruct-2507, quantized to w4a16 and compiled into a Hexagon HTP context binary split across four shards, packaged for the Qualcomm Genie runtime (GenieDialog API / genie-t2t-run CLI).

It runs entirely on the on-device NPU — no cloud, no CPU fallback path.

This is a converted / repackaged artifact, not a new model. Every capability and every limitation of the upstream checkpoint carries over. Only the execution format changed.

Specifications

Repository contents

Requirements

Compatibility — read before filing a bug

1. The shipped binaries are V79. htp_backend_ext_config.json pins soc_model: 69 / dsp_arch: "v79" (SM8750 / Snapdragon 8 Elite), and all four context binaries were compiled for Hexagon V79.

2. Verified working on V81 hardware. This exact artifact was loaded and generated text on an SM8850 (Snapdragon 8 Elite Gen 5, soc_model 87, DSP arch V81) device. Loading a V79 context binary on a V81 DSP worked, but cross-architecture safety is not guaranteed by Qualcomm — treat it as empirically verified on SM8850, not as an officially supported configuration.

3. Override backend.extensions. The bundled htp_backend_ext_config.json targets SM8750. On any other SoC, point backend.extensions at an extension config matching your device (or use the official minimal form that omits soc_model/dsp_arch and lets the backend probe the device). The DroidLLM app does this automatically — see Using with DroidLLM.

4. SoC → DSP arch reference (per the QAIRT support table):

   SoC	soc_model	HTP arch	Notes
   SM8850 — Snapdragon 8 Elite Gen 5	87	V81	Verified with this artifact
   SM8750 — Snapdragon 8 Elite	69	V79	Native target of this build
   SM8650 — Snapdragon 8 Gen 3	57	V75	Not tested
   SM8550 — Snapdragon 8 Gen 2	43	V73	Not tested

   For any SoC other than SM8750/SM8850, the four .bin shards would have to be recompiled for the matching arch from the upstream checkpoint. Just editing the JSON is not sufficient.

5. Context length is 4096 tokens as shipped. The engine was compiled for context sizes 512 / 1024 / 2048 / 3072 / 4096 (metadata.json → genie.context_lengths), and the bundled genie_config.json selects 4096. Although the upstream model advertises 262,144 positions, this build cannot exceed 4096 — prompts + generation beyond that are outside the compiled graph.

6. Cross-version QAIRT usage. Compiled with QAIRT 2.45.0 and successfully run under a 2.50.0 runtime. QNN context binaries are not guaranteed to be forward/backward compatible across SDK releases; validate on your own SDK before depending on it.

7. n-threads is inert. The HTP schedules its own work; the n-threads: 3 field in the config is ignored on this backend. Sampling parameters do apply (they are handled by the Genie dialog sampler, not by a CPU sampler).

Measured performance

Single-run acceptance measurement of this artifact on real hardware. Not a benchmark — numbers vary with thermals, clock state, prompt length and generation length.

Quick start

A. Genie CLI (genie-t2t-run)

Runs the model without any app, and is the only place where Genie prints its own [ERROR]/[WARN] lines (the Genie SDK logs through an internal callback that Android apps cannot receive).

# 1. Prepare device-side libraries (adjust ARCH to your device: 79, 81, ...)
QAIRT=/path/to/qairt                       # e.g. .../qairt/2.50.0.260828
ARCH=79                                    # SM8750 = 79, SM8850 = 81
DEV=//data/local/tmp/qnnval

adb shell mkdir -p $DEV
adb push "$QAIRT/bin/aarch64-android/qnn-platform-validator" \
         "$QAIRT/bin/aarch64-android/genie-t2t-run" \
         "$QAIRT/lib/aarch64-android/libGenie.so" \
         "$QAIRT/lib/aarch64-android/libQnnHtp.so" \
         "$QAIRT/lib/aarch64-android/libQnnSystem.so" \
         "$QAIRT/lib/aarch64-android/libQnnHtpV${ARCH}Stub.so" \
         "$QAIRT/lib/aarch64-android/libQnnHtpV${ARCH}CalculatorStub.so" \
         "$QAIRT/lib/aarch64-android/libQnnHtpNetRunExtensions.so" \
         "$QAIRT/lib/hexagon-v${ARCH}/unsigned/libQnnHtpV${ARCH}Skel.so" \
         "$QAIRT/lib/hexagon-v${ARCH}/unsigned/libCalculator_skel.so" \
         $DEV/
adb shell "chmod 755 $DEV/*"

# 2. Confirm what your device actually reports (do this before trusting any table)
adb shell "cd $DEV && LD_LIBRARY_PATH=$DEV ADSP_LIBRARY_PATH=$DEV \
  ./qnn-platform-validator --backend dsp --coreVersion --testBackend"
# Expected: "Core Version of the backend DSP: Hexagon Architecture Vxx" + "Passed."

# 3. Push the model directory
adb push . $DEV/model

# 4. In the config, set three paths to their absolute device paths:
#    backend.extensions      -> $DEV/model/htp_backend_ext_config.json  (or a config built for your SoC)
#    tokenizer.path          -> $DEV/model/tokenizer.json
#    model.binary.ctx-bins[] -> $DEV/model/<each .bin>

# 5. Run (the first run reads 3.1 GB of context binaries — allow 2+ minutes, it is not hung)
adb shell "cd $DEV && LD_LIBRARY_PATH=$DEV ADSP_LIBRARY_PATH=$DEV \
  ./genie-t2t-run -c model/genie_config.json -p 'Hi'; echo EXIT=\$?"

Two failure modes worth knowing in advance:

• Missing libQnnHtpNetRunExtensions.so → [WARN] Failure in initializing backend extensions followed by a segmentation fault (EXIT=139). The load failure is not reported as an error by the library — the crash happens inside GenieDialog_create before the model is read.
• Missing ADSP_LIBRARY_PATH → GenieDialog_create failed after ~300 ms, with nothing else in the log.

B. Android app (DroidLLM)

DroidLLM is an Android app that runs this artifact through the same Genie runtime, and handles the two pitfalls above for you:

• sets ADSP_LIBRARY_PATH to the app's nativeLibraryDir before touching the Genie SDK;
• bundles the required QNN libraries (including libQnnHtpNetRunExtensions.so);
• rewrites backend.extensions in genie_config.json to an HTP config matching Build.SOC_MODEL (SM8850 → V81, SM8750 → V79, SM8650 → V75, SM8550 → V73), so the SM8750-targeted config shipped here is replaced on the fly.

Import: Models → ＋ → Genie → select the model directory. A directory is accepted when it contains genie_config.json, tokenizer.json and at least one *.bin.

Prompt format

Qwen3 chat markup. The default system prompt used during validation was You are a helpful AI assistant.

<|im_start|>system
You are a helpful AI assistant<|im_end|>
<|im_start|>user
What is gravity? Keep the answer under ten words.<|im_end|>
<|im_start|>assistant

Role prefixes/suffixes are also stored in metadata.json under genie.chat_template, which is what the Genie-based integrations read.

Sampling defaults

From the shipped genie_config.json:

seed: 42 is a fixed default from the compilation toolchain — change it if you need varied sampling, or set seed: -1 / omit it in your own config.

How this was produced

Converted from the upstream Qwen/Qwen3-4B-Instruct-2507 checkpoint with the Qualcomm QNN toolchain (Qualcomm AI Hub export pipeline; QAIRT 2.45.0.260326154327, recorded in tool-versions.yaml):

1. weights quantized to w4a16;
2. the graph compiled to Hexagon HTP context binaries for context sizes 512–4096, emitted as four shards;
3. Genie dialog/node configs generated, with metadata.json recording per-tensor shapes, dtypes and quantization scales.

No fine-tuning, no weight modification, no merge was performed. Only quantization and compilation.

License, attribution and disclaimer

Model weights: Apache-2.0, © Alibaba Group / Qwen team. This repository redistributes a derived artifact of Qwen/Qwen3-4B-Instruct-2507 under the same license; the full license text is in LICENSE, and attribution is in NOTICE. If you redistribute this artifact or a product built on it, keep LICENSE and NOTICE with it.

Qualcomm SDK: the QNN / Genie runtime (QAIRT SDK, including libQnnHtp*.so and the Hexagon skels) is proprietary Qualcomm software and is not distributed in this repository. You must obtain it from Qualcomm and comply with the Qualcomm AI Stack license. This repository contains only the compiled model artifact.

No affiliation. This is an independent conversion. It is not affiliated with, endorsed by, or supported by Alibaba Group or Qualcomm Technologies, Inc. "Qualcomm", "Snapdragon" and "Hexagon" are trademarks of Qualcomm Technologies, Inc.

Disclaimer. Provided as-is, without warranty of any kind, express or implied. It is an experimental on-device artifact: device compatibility is limited to the configurations described above, output quality is that of the base model (including its biases and failure modes), and nothing here is suitable for safety-critical use. Verify on your own hardware before relying on it.