Qwen3-1.7B LiteRT-LM Model

This repository contains LiteRT-LM variants of Qwen/Qwen3-1.7B optimized for on-device text generation.

Available Artifact

Conversion Notes

Qwen3-1.7B_dynamic_wi4b32_afp32.litertlm is a dynamic INT4 variant (block-32 weights, FP32 activations). It was converted through the LiteRT Torch (litert-torch) path and quantized with AI Edge Quantizer. This artifact incorporates LiteRT-LM GPU graph optimizations, including composite ops for RoPE, fused QKV, and fused Gate/Up projections, and is configured with static prefill memory allocation.

Both files: Need litert-lm 0.18.0 or newer (tested on 0.18.0) and carry LiteRT-LM's official qwen3 configuration, so thinking is on unless enable_thinking is false, the reasoning arrives on the thought channel, and tool calls are parsed by the runtime.

How to Use

Command-Line Interface

1. Install the prerequisites:

   pip install litert-lm

2. Run the command in CLI:

   litert-lm run --from-huggingface-repo=litert-community/Qwen3-1.7B Qwen3_1.7B.litertlm --prompt="Write me a poem on nature"

Edge Gallery App

1. Get the App: Install the app from Google Play or download the latest APK from the GitHub releases page.
2. Importing the Model: Navigate to the Model manager within the app and click the "+" (plus) icon in the bottom-right corner. Two options will appear:
   • Import from HF (Recommended): Select this option, and a dialog box will appear showing an example Hugging Face model URL. Enter the HF link for the desired .litertlm model and click submit. The model will then appear in your list, and you can proceed to download it (a Hugging Face account login is required).
   • From local model file: First, download the .litertlm model directly to your Android device, OR download it to your computer and push it via ADB (e.g., adb push Qwen3-1.7B_dynamic_wi4b32_afp32.litertlm /sdcard/Download/). Then, select this option, choose the downloaded file from your storage, configure your preferred parameters, and tap "Import".

For full details on importing models and other features, see the Edge Gallery App Wiki.

To build the demo app from source, please follow the instructions from the GitHub repository.

Python

1. Install the prerequisites:

   pip install litert-lm huggingface_hub

2. Download the model file:

   from huggingface_hub import hf_hub_download

   model_path = hf_hub_download(
       repo_id="litert-community/Qwen3-1.7B",
       filename="Qwen3_1.7B.litertlm"
   )

3. Run inference:

   import litert_lm

   litert_lm.set_min_log_severity(litert_lm.LogSeverity.ERROR)  # Hide log for TUI app

   with litert_lm.Engine(model_path) as engine:
     with engine.create_conversation() as conversation:
       while True:
         user_input = input("\n>>> ")
         for chunk in conversation.send_message_async(user_input):
           print(chunk["content"][0]["text"], end="", flush=True)

Performance (on device, measured)

Community measurement on a physical Samsung Galaxy S26 (SM-S942Q, Snapdragon 8 Elite Gen 5 / SM8850, Android 16): litert_lm_advanced_main from the litert-lm v0.16.0 release, GPU backend OpenCL (LITERT_CL), one fixed 205-token prompt text (202 tokens under this tokenizer), --benchmark. Two runs per backend taken back-to-back — cells show the range (the phone warms across a series; the CPU's first run also pays cold file I/O). Peak RSS is the process VmHWM. Before quoting, every file/backend pair was run with a real prompt and produced a correct text answer; the GPU rows are full delegation (e.g. Qwen3-1.7B_dynamic_wi4b32_afp32: decode 994/994 ops on LITERT_CL, nothing on the CPU fallback).

What the table says:

• On the S26 GPU the wi4b32 file is the one to run: 40.7 tok/s decode against 18.1 for the int8 file — the GPU graph optimizations described in Conversion Notes above, visible in measurement. It is also the fastest prefill and the lowest time-to-first-token on the card.
• The wi4b32 file is a GPU specialist. On CPU it decodes at 8 tok/s and prefills at 48–76 tok/s — int4 block-32 weights suit the GPU path, not XNNPACK. If you need CPU, the int8 Qwen3_1.7B.litertlm prefills 3–5× faster than wi4b32 on the same CPU.
• Peak RSS is 2.4–3.7× lower on the GPU for both files (854–1036 MB against 2.5–3.2 GB).
• GPU engine init is the one cost the table hides: 4.4–8.3 s per process against 2.2–3.5 s on CPU.

Quality

When randomly pick 100 tests from MMLU, int8 model's score (72) is close to orginally BF16 models (70), while mixed int4 model is slightly worse (64).

Integration

Ready to integrate this into your product? Get started in the LiteRT-LM documentation.