LiteRT is Google's on-device runtime, the new name for TensorFlow Lite (Android: com.google.ai.edge.litert:litert), and litert-torch, the renamed ai-edge-torch, is its PyTorch converter: a PyTorch model converted unmodified with litert_torch.convert matched the original to 4e-7 on a Galaxy S26 (measured, LiteRT 2.2.0, Android 16, 2026-09-05).

Measured on device (edge-compat): Galaxy S26 · LiteRT-LM 0.16.0 · GPU · decode 35.1 tok/s · prefill 551 tok/s · TTFT 400 ms · all 25703 ops delegated (2026-09-05); iPhone 17 Pro · LiteRT-LM 0.16.0 · GPU · decode 45.7 tok/s · prefill 588 tok/s · TTFT 1.68 s (2026-08-28); Mac Studio M4 Max · LiteRT-LM 0.16.0 · GPU · decode 126.2 tok/s · prefill 1891 tok/s · TTFT 154 ms (2026-08-28). Record: https://github.com/john-rocky/edge-compat/blob/main/cards/qwen35-0.8b-vl-int8/CARD.md

Qwen3.5-0.8B — LiteRT-LM

Update 2026-08-13: re-converted from the same weights with the gated-delta-rule chunk kernel re-expressed in an all-rank-≤-4, pad-free form the GPU delegate executes correctly. The graph now delegates fully to the GPU and answers correctly there — verified end-to-end on macOS and iPhone 17 Pro (Metal), where GPU decode runs ~2.8× the CPU. CPU behaviour and speed are unchanged. GPU execution requires fp32 activations, which the bundle now declares; see the honest notes for the memory cost. (While rewriting we found the GPU delegate miscomputing one op shape — reported upstream as LiteRT#9272; the kernel now avoids that shape.)

Update 2026-08-07: metadata-only fix (weights unchanged) — <|im_end|> is now declared as a stop token alongside <|endoftext|>. Qwen3.5 uses different tokens for the chat turn-end and config.json's eos_token_id; with only the latter declared, the literal <|im_end|> text leaked into litert-lm run output (generation still stopped correctly). Replies are now clean, and multi-turn history no longer records the marker as text.

Qwen/Qwen3.5-0.8B converted to the LiteRT-LM (.litertlm) format for on-device inference with Google's LiteRT-LM runtime. Requires litert-lm ≥ 0.15 (both backends gated on 0.15.0 and 0.16.0; this file checked on litert-lm 0.18.0). To our knowledge this is the first Qwen3.5 in LiteRT form, and the first gated-delta-net hybrid served by the released LiteRT-LM runtime — and, since the 2026-08-13 update, the first to run its delta-rule scan on a mobile GPU.

Qwen3.5 is Alibaba's hybrid architecture: GatedDeltaNet (gated delta rule linear attention) blocks interleaved with a few gated full-attention blocks (this 0.8B model has 18 linear-attention + 6 attention layers). The linear-attention blocks carry constant-size per-layer conv + recurrent state instead of a growing KV cache, so memory stays nearly flat with context length — only the 6 attention layers keep KV (4096-token budget here). The upstream 0.8B checkpoint is multimodal; this package is the text decoder only (the vision tower and MTP heads are dropped exactly as upstream's own Qwen3_5ForCausalLM text-only load contract does).

2026-09-21: chat template updated to accept the 0.18 content-parts form (string form unchanged); weights, tokenizer and executor metadata byte-identical.

2026-10-12: both files re-quantized — the 36 GatedDeltaNet decay/gate projections (in_proj_a / in_proj_b) now stay float, and the rest of the recipe is unchanged. Qwen3.5-0.8B_int8.litertlm: previous file sha256 64ed396f…, 963,184,864 B → this file sha256 b21b4d2b…, 964,861,904 B (+1,677,040 B). Qwen3.5-0.8B-VL_int8.litertlm: previous file sha256 e3360b65…, 1,302,262,448 B → this file sha256 5e6d69e5…, 1,303,966,384 B (+1,703,936 B); only its decoder section changed, and the embedder, vision encoder, vision adapter, tokenizer and metadata sections are byte-identical. This fixes intermittent repetition and wrong products on multi-step arithmetic on the CPU backend, as measured in the Correctness list and the Honest notes below. I am sorry: anyone who pulled these files since 2026-08-13 (text) or 2026-08-28 (VL) could get looping replies on multi-step arithmetic on the CPU backend (GSM8K on the CPU: 11 of 100 for the previous text file, 49 for the re-quantized one), and the need for one Engine per conversation had been in my own report since 2026-08-06 without a line on this card. This was my recipe's defect and my omission. From now on, when one file of a recipe is found defective, its sibling files are re-checked the same day, and every card of a model with recurrent state passes a mechanical check for the one-Engine line and the CPU arithmetic check before it is published.

Correctness

The first four items below were run on the previous file (sha256 64ed396f…) or its float parent after the 2026-08-13 rewrite; the last two were run on this file (2026-10-12):

• Logits-level parity vs PyTorch: the float export matches the HF model teacher-forced across 48 positions — top-1 and top-5 agreement 100%, Pearson 1.0000, KL ≈ 0.
• 8-question sanity gate: 8/8 on CPU and GPU, on both litert-lm 0.15.0 and 0.16.0 (Mac). On iPhone 17 Pro the GPU run answers 8/8.
• Prompt-length robustness: first-token check against the runtime's real prefill chunk plans, fresh engine per length — 40/40 lengths clean on CPU, 20/20 on GPU.
• Multi-turn conversations carry state correctly across turns.
• Multi-step arithmetic, CPU and GPU (2026-10-12, one fresh process per run).
  • Mac CPU (litert-lm 0.18.0): six multiplication questions (17 × 23, 23 × 17, 12 × 34, 9 × 87, 45 × 12, 36 × 25), three runs each — 18 of 18 right, none repeating; the arithmetic prompt (17 × 23) with ThinkingConfig(True, budget 300) answers 391 in 3 of 3.
  • The 8-question sanity gate: 8/8 on Mac CPU and on Mac GPU.
  • Mac GPU: 5 of the 6 questions right; 45 × 12 gives 1010 in both GPU runs of this file, as it does on the previous file and on the VL file.
  • GSM8K (n = 100, section below): 49 on Mac CPU and 52 on Mac GPU; the bf16 checkpoint scores 51.
  • Pixel 8a CPU: the arithmetic prompt answers 391 without repetition in 3 of 3 runs.
  • Galaxy S26: the arithmetic prompt answers 391 in 3 of 3 runs on the CPU and 3 of 3 on the GPU.
• One Engine per conversation. A new Conversation opened on an Engine that already ran another conversation continues from that conversation's state: it opened its replies the way its own system prompt asked in 0 of 9 turns, against 9 of 9 on a fresh Engine, and 7 of its 9 replies were the template's empty <think>\n\n</think> block, repeated (3 runs, litert-lm 0.18.0, Mac CPU). Create a new Engine for each conversation — the runtime reuses the prompt tokens a new conversation shares with the previous one and does not reset the recurrent state (my open report); a process that runs one conversation is unaffected.

Quality — GSM8K

GSM8K, n = 100, measured 2026-10-12: 0-shot chain-of-thought prompt asking for a final #### <number> line, greedy, 512-token cap, one fresh process per question, and the same answer extraction for every row. The bf16 checkpoint gets the exact prompt this bundle renders, the empty <think>\n\n</think> block included.

Most replies end without a #### line: 60 of 100 for this file on CPU, 68 on GPU, 61 for the VL file, 87 for the bf16 checkpoint and 95 for the previous file; the same extraction scores every row.

The earlier card text reported 12 % (bf16) and 11 % (int8) on 2026-08-13; which chat template rendered that bf16 run's prompts is not recorded, so this table compares only the 2026-10-12 runs.

Usage

litert-lm run ./Qwen3.5-0.8B_int8.litertlm --prompt "What is the capital of France? Answer in one word."

# GPU
litert-lm run ./Qwen3.5-0.8B_int8.litertlm --backend gpu --cache no --prompt "..."

Multi-length prefill signatures (1–1024) are exported so the runtime picks tight chunks.

Chat template note: the bundle ships a simplified ChatML template rather than the stock Qwen3.5 template. Thinking is disabled the way the stock template's non-thinking mode does it (an empty <think>\n\n</think> block opens each assistant turn), and — deliberately — that block is kept in history renders too: the stock template strips it from past turns, which breaks LiteRT-LM's incremental conversation rendering (the engine requires each turn's render to be a string-extension of the previous one) and kills multi-turn on turn 2. Tool-calling and vision sections are not included.

Performance

The Mac and iPhone rows and the Pixel 8a CPU row below were measured on the previous file (sha256 64ed396f…), the same graph with the 36 GatedDeltaNet decay/gate projections int8; the Galaxy S26 rows further down and the Pixel 8a GPU row are of this file (2026-10-12).

litert-lm benchmark (litert-lm 0.16.0), Apple M4 Max, -p 256 -d 256 --runs 3 --cache no, quiet machine:

CPU figures are essentially unchanged from the previous file (the delta-rule kernel was already matmul-form on CPU); the 2026-08-13 update adds the working GPU path.

On device (cold start, single runs, 138-token prompt, quality-gate harness):

Pixel 8a (Tensor G3, litert_lm_main built from the v0.16.0 tag, 260-token prompt with generation running long, 3 runs):

Galaxy S26 (Qualcomm SM8850), this file (2026-10-12), prebuilt litert_lm_advanced_main 2026-09-18, --benchmark_prefill_tokens=256 --benchmark_decode_tokens=64 --num_iterations=3, median of iterations 2–3, caches removed before the warm-up run (engine init = that cold run):

The S26 capped its CPU clocks in every CPU bench iteration (7 of 9 sampler lines capped), so the CPU row is this phone's capped speed. The previous file, measured right after on the same phone: CPU 732.8 / 44.09 tok/s, TTFT 0.37 s, cold engine init 12.3 s, peak 2039 MB (8 of 9 sampler lines capped). On the GPU the previous file, measured later on the same phone with the same flags: 488.6 / 29.22 tok/s, cold engine init 93.0 s, peak 4644 MB. Three fresh processes of this file on each backend answered the arithmetic prompt (17 × 23) with 391.

Pixel 8a (Tensor G3, CPU, the same prebuilt binary): 3 of 3 fresh processes of this file answer the arithmetic prompt (17 × 23) with 391, without repetition.

Honest notes:

• Where GPU execution is verified. macOS (Metal), iPhone 17 Pro (Metal) and, on Android, the Galaxy S26 (Adreno) only. Pixel 8a: GPU not verified on this file (Arm Mali, OpenCL) — see its bullet below. On the Galaxy S26 (SM8850) the GPU answered correctly on 2026-08-24 (litert-lm 0.16.0) and on 2026-10-12 (this file: 89.5 s cold engine init, 4723 MB peak). On this phone the CPU backend is still the practical choice. The vision file Qwen3.5-0.8B-VL_int8.litertlm was gated the same way on that phone on 2026-09-05: GPU 35 tok/s decode against 33 on the CPU, with a 74 s engine init and 2.1× the CPU's peak — CPU again. Same-device rows are in litertlm_manifest.json.
• GPU inference runs with fp32 activations (declared in the bundle; an fp16-activation formulation is unfinished — the residual issue is a real-weight fp16 range overflow in one layer-0 head, a property of the checkpoint, not of the conversion). That is where the GPU memory multiple comes from — 5.48 GB vs 1.21 GB on iPhone above.
• Pixel 8a (Arm Mali, OpenCL): GPU not verified on this file. From 2026-08-13 to 2026-10-12 this card listed the Pixel 8a as a verified GPU device — my error; the listing was based on a smaller development build that is not among the files here. On the Pixel 8a, use the CPU backend (the CPU rows above): with the fp32 activations of the bullet above, this file's GPU engine does not fit in the memory the phone had free. That engine peaked at 4723 MB on the Galaxy S26 with the same benchmark settings, and the Pixel 8a had 4.2 GB free before my run. On 2026-10-12 I ran this file on the phone's GPU with the prebuilt binary of the Galaxy S26 rows. My test harness kills a run when the phone's free memory falls under 0.4 GB. Engine creation crossed that line at the third of the 11 prefill signatures, with the process peaking at 3.6 GB. Each of the three signatures it reached was delegated whole to the GPU. Android reclaimed one background process during the run. No benchmark or prompt ran. Earlier that day the previous files crossed the same line on this phone: the text file also at the third signature, the VL file at the fourth of its six (process peak 3.8 GB each). The shipped VL file was not run on the phone's GPU. It differs from the previous VL file only in the 36 projections left float (+1.7 MB), so I expect the same stop — an inference, not a measurement.
• On the Galaxy S26 the CPU is faster than the GPU for this file on both prefill and decode (729.4 / 44.49 tok/s against 488.2 / 31.05). In the Mac and iPhone rows above, the GPU is the faster backend.
• 2026-10-12 re-quantization. The previous file (sha256 64ed396f…) gave intermittent repetition and wrong products on multi-step arithmetic on the CPU backend (litert-lm 0.18.0, one fresh process per run). On a Mac M4 Max CPU (thinking off, 400-token cap) it answered the six multiplication questions right in 7 of 18 runs: 23 × 17, 12 × 34 and 36 × 25 were right in 0, 1 and 0 of 3 runs, the other runs repeating a line to the cap, and 9 × 87 gave 763 in 3 of 3. With ThinkingConfig left unset, the arithmetic prompt (17 × 23) repeated 17 × 20 + 17 × 3 = to the cap in 1 of 3 runs. On GSM8K (Mac CPU) it scored 11 of 100, with 22 replies repeating one line to the cap. The Mac GPU answered 23 × 17, 36 × 25 and 9 × 87 right, so three of the four CPU failures are CPU-only; on the GPU, 12 × 34 gave a reply of only "!" in 1 of 2 runs and 45 × 12 gave 1010. The Pixel 8a CPU answered the arithmetic prompt correctly in 3 of 3 runs, too few to say the phone differs. The cause: LiteRT-LM's CPU backend quantizes every int8 FC's input row to int8 at run time (XNNPACK dynamic quantization), and the GatedDeltaNet decay/gate heads (in_proj_a, in_proj_b) could not take that noise. This file leaves only those 36 projections float (+1,677,040 B); its float export, quantized with the previous recipe, reproduces the previous file's TFLite section byte for byte. Measured on this file: Mac CPU 18 of 18, Mac GPU 5 of 6 (45 × 12 → 1010, as before), Galaxy S26 CPU 3 of 3 and GPU 3 of 3, GSM8K 49 on CPU and 52 on GPU.
• On quality: the per-question sanity gate is word-for-word identical to HF fp32, but on a harder composite probe (8 questions in one 138-token prompt) the int8 quantization measurably costs answers at this 0.8B scale. If you need maximum fidelity on complex prompts, ask for a float/fp16 variant. Measured on the previous file; the composite probe was not re-run on this file.

Vision build (Qwen3.5-0.8B-VL_int8.litertlm)

The upstream checkpoint ships a 12-layer, 768-dim ViT that the text-only package drops. This build keeps it, wired to LiteRT-LM's fast_vlm contract: one 512×512 image per turn, 1024 patches merged 2×2 into 256 soft tokens injected at the image position. There is no DeepStack in this tower (deepstack_visual_indexes is empty upstream), so the single image embedding is the whole visual signal.

litert-lm run ./Qwen3.5-0.8B-VL_int8.litertlm \
  --prompt "What is in this image?" --attachment ./photo.png

# vision on the GPU as well as the decoder
litert-lm run ./Qwen3.5-0.8B-VL_int8.litertlm --backend gpu --vision-backend gpu \
  --prompt "Where is this scene?" --attachment ./photo.png --cache no

Text-only prompts work on this file too; it is a superset of the text build, at a larger download and a larger GPU footprint.

Measured

The Mac and iPhone rows below were measured on the previous VL file (sha256 e3360b65…), the same graph with the 36 decay/gate projections int8; the Galaxy S26 rows in the Honest notes are of this file (2026-10-12).

Mac, litert-lm benchmark (0.16.0), M4 Max, -p 256 -d 256 --runs 3 --cache no, quiet machine — text path:

iPhone 17 Pro, cold start, single runs, quality-gate harness — the vision rows are real image turns:

Grounding on device was checked with two probes whose answers are opposite: an image containing text ("does this image contain text?" → "Yes, this image…") and one without ("No, this image…"). Both correct on Metal, with the vision encoder on the GPU.

Honest notes

• The six-signature ladder fit Metal on the first try. The 2B sibling had to be re-exported at 1024, 256, 64, 16, 4, 1 after all three of its Metal legs were killed by jetsam; this build ships the same reduced ladder and needed no maxNumTokens override, no retry, and no second export. At roughly 40% of the 2B's file size it peaks at ~3.8 GB against the 2B's ~4.6 GB.
• Positions are 1-D. The fast_vlm contract feeds sequential positions, so the checkpoint's M-RoPE collapses to plain RoPE. Expect fluent same-content paraphrase rather than token-exact agreement with a full M-RoPE reference — counting and dense-layout questions are where it shows.
• The vision graph is faithful, and this tower quantizes better than the 2B's. The exported tower matches the model's own vision path at correlation 0.9999999999 in fp32 (max abs diff 1.05e-4). Quantizing the encoder to int8 holds 0.9916–0.9951 on real photographs — noticeably better than the 2B tower's 0.974–0.984 at the same recipe. This build still ships the encoder in fp16 and only the adapter in int8, matching the family, but the int8-encoder variant is on firmer ground here if a Mali device needs it.
• The fp16-safe LayerNorm profile is this checkpoint's own, not the 2B's. Scales reach 16 from block 6 on (with 512 at the final norm), where the 2B reaches 32 from block 11. The scales are calibrated per export; copying the sibling's table would silently mis-scale the tower.
• 2026-10-12 re-quantization. The previous VL file (sha256 e3360b65…) had the text file's CPU defect: on the Mac CPU it answered the six multiplication questions right in 12 of 18 runs (9 × 87 in 0 of 3, 23 × 17 in 1 of 3, 36 × 25 in 2 of 3), and the arithmetic prompt (17 × 23) with ThinkingConfig left unset repeated 17 × 20 + 17 × 3 = to the cap in 1 of 3 runs; the Mac GPU answered 5 of 6 (45 × 12 wrong). Same cause and same fix as the text file (its 2026-10-12 note above): the 36 GatedDeltaNet decay/gate projections stay float in the decoder section (+1,715,984 B), quantized from the 2026-08-28 float export, which reproduces the previous decoder section byte for byte under the previous recipe; the other sections are byte-identical. Measured on this file (litert-lm 0.18.0 on the Mac, one fresh process per run): the six questions 18 of 18 on Mac CPU and 5 of 6 on Mac GPU (45 × 12 → 1010, as before), the 8-question sanity gate 8/8 on CPU and GPU, and the image probe describes two cats sleeping on a pink couch on both backends. With thinking switched on through a raw <think> prefill (no budget, 400-token cap), only 1 of 3 Mac CPU runs closes the thought (that run answers 391; the previous file 0 of 3). Galaxy S26: 3 of 3 fresh processes answer 391 on the CPU and 3 of 3 on the GPU.
• Quality on the composite 8-question probe, separated by cause — and the split runs the other way from the 2B. On Mac this file answers 8 of 8 on GPU and 8 of 8 on CPU, with the questions asked one at a time and as one combined prompt. On iPhone it scores 6/8 on Metal and 3/8 on CPU. Not one of the device misses reproduces on Mac, so they are device-side rather than the conversion — the opposite of the 2B, whose arithmetic slip did reproduce on Mac and was therefore the model. The mechanism behind the device degradation is not identified, and this card does not guess at one. On iPhone, prefer the GPU backend. Measured on the previous file; the composite probe was not re-run on this file.
• The iPhone CPU 3/8 is reproducible, not thermal. It was first measured on a warm handset; re-run three days later from cold, after an app reinstall, the answers are byte-identical. Peak memory is the figure that moved (0.63 GB cold against 0.91 GB warm).
• Android: measured on a Galaxy S26 (Qualcomm SM8850) with this file, text turns (2026-10-12). Prebuilt litert_lm_advanced_main 2026-09-18, --benchmark_prefill_tokens=256 --benchmark_decode_tokens=64 --num_iterations=3, median of iterations 2–3, caches removed before the warm-up run: GPU (OpenCL) prefill 675.2 tok/s, decode 31.83 tok/s, cold engine init 46.8 s, peak 3778 MB; CPU prefill 737.5 tok/s, decode 43.94 tok/s, TTFT 0.37 s, cold engine init 6.9 s, peak 1666 MB, with the CPU clocks capped in 4 of 6 sampler lines. Three fresh processes on each backend answered the arithmetic prompt (17 × 23) with 391, as did 3 of 3 on a Pixel 8a CPU. Image turns of this file were not measured on a phone. This file was not run on the Pixel 8a GPU. The previous VL file's engine creation there crossed my test harness's 0.4 GB free-memory line on 2026-10-12, at the fourth of its six prefill signatures (see the text build's Pixel 8a note above). On Mali the fp16 vision encoder is known to crash the device on other models of this shape — an int8-vision build would be the Android path, and we have not measured one on a phone.

Conversion notes

Converted with litert-torch plus a hybrid-cache patch (reproduction script + patch: hf-to-litertlm qwen35_work/):

• Rank-4 chunk kernel (the 2026-08-13 change): the reference chunked delta rule spells its contractions as broadcast-multiply-reduce over high-rank intermediates. The vendored kernel re-expresses them as batched matmuls with chunk and head axes folded into the batch axis (all tensors rank ≤ 4, no BROADCAST_TO, no int64 index math), and writes every tail-pad as a concat with a zeros constant rather than a PAD op — the GPU delegate miscomputes rank-3 non-final-axis PAD (reported as LiteRT#9272). This is why the same weights got ~3.5× faster on CPU and became fully delegable on GPU.
• Export cache for GatedDeltaNet layers: conv [B, conv_dim, K] + recurrent [B, heads, k_dim, v_dim] cache layers registered for layer_types == "linear_attention", so torch.export traces the model's own state contract.
• State continuation tracing: prefill graphs trace the chunk-continuation branch (previous conv/recurrent state consumed, so multi-chunk prefill composes) and the decode graph traces the fused single-step branch (conv window rolled in place by causal_conv1d_update).
• Prefill-pad guard: the runtime's chunk planner runs partially-filled prefill chunks; pad positions are made identity steps for the delta rule (per-token decay forced to ~1, zeroed k/v injection) and the stored conv window is gathered at the last valid column. GPU trap: reductions in the guard keep their batch dimension (keepdim=True) — a rank-0 scalar entering broadcast arithmetic is silently miscomputed by the GPU delegate.
• Constant-eye chunk kernel: the reference kernel builds torch.eye inside the traced function, which lowers to a STABLEHLO_IOTA op no released TFLite kernel set registers; the identity matrix is lifted as a graph constant.
• Runtime state binding: litert-lm ≥ 0.15 binds per-layer states through an ExecutorMetadata section listing each state tensor; it is appended at package time.
• Quantization: post-hoc dynamic int8 over linears + embedding only; the convs, the delta rule and — since 2026-10-12 — the 36 GatedDeltaNet decay/gate projections (in_proj_a / in_proj_b) stay float.
• GPU activation precision: the bundle's TOML declares prefer_activation_type = "fp32" — required for correct GPU numerics on this family today.

Raspberry Pi 5 (CPU)

Measured on a Raspberry Pi 5 Model B Rev 1.1 (8 GB, Raspberry Pi OS 64-bit) with litert-lm benchmark 0.16.1: CPU backend, 4 threads, 256 prefill + 256 decode tokens, --cache memory (the compile cache lives and dies with the process, so every invocation compiles the model from scratch; nothing is reused between runs), one warm-up plus one timed iteration per invocation, 3 invocations per file with cooldown in between. Values are the median across invocations (min–max in parentheses). No thermal throttling occurred during these runs (vcgencmd get_throttled stayed 0x0). Every file listed produced coherent text in a real generation on this backend before its numbers were recorded.

License and changes

Distributed under Apache-2.0 (inherited from the base model). Changes from the original work: text-decoder weights converted from safetensors bf16 to LiteRT flatbuffers and quantized as described above; vision tower and MTP weights omitted; tokenizer repackaged unmodified; chat template replaced with the simplified ChatML template described above. This repository is not affiliated with Alibaba / the Qwen team.