LiteRT is Google's on-device runtime, the new name for TensorFlow Lite (Android: com.google.ai.edge.litert:litert), and litert-torch, the renamed ai-edge-torch, is its PyTorch converter: a PyTorch model converted unmodified with litert_torch.convert matched the original to 4e-7 on a Galaxy S26 (measured, LiteRT 2.2.0, Android 16, 2026-09-05).

Qwen3.5-4B — LiteRT-LM

Qwen/Qwen3.5-4B converted to the LiteRT-LM (.litertlm) format for on-device inference with Google's LiteRT-LM runtime. Requires litert-lm ≥ 0.15 (both backends gated on 0.15.0 and 0.16.0; this file checked on litert-lm 0.18.0 and the Android CLI build of 2026-09-18). Same conversion rail as our Qwen3.5-0.8B — the GPU-delegable rank-≤4 gated-delta kernel — plus two 4B-specific fixes described in the conversion notes.

Qwen3.5 is Alibaba's hybrid architecture: GatedDeltaNet (gated delta rule linear attention) blocks interleaved with a few gated full-attention blocks (this 4B model has 24 linear-attention + 8 attention layers). The linear-attention blocks carry constant-size per-layer conv + recurrent state instead of a growing KV cache, so memory stays nearly flat with context length — only the 8 attention layers keep KV (4096-token budget here). The upstream checkpoint is multimodal; this package is the text decoder only (vision tower and MTP heads dropped exactly as upstream's own Qwen3_5ForCausalLM text-only load contract does).

2026-09-21: chat template updated to accept the 0.18 content-parts form (string form unchanged); weights, tokenizer and executor metadata byte-identical. 2026-10-09: metadata updated (chat template with a thinking switch, reasoning channel, supports_thinking); TFLite, tokenizer and executor metadata byte-identical. 2026-10-13: the int8 file re-quantized — the 48 GatedDeltaNet decay/gate projections (in_proj_a / in_proj_b) now stay float, and the rest of the recipe is unchanged; the chat template, tokenizer and executor metadata are byte-identical. Qwen3.5-4B_int8.litertlm: previous file sha256 a360b83d…, 4,407,428,464 B → this file sha256 ce4c2d64…, 4,419,088,624 B (+11,660,160 B). Its float export, quantized with the previous recipe, reproduces the previous file's TFLite section byte for byte. This fixes two things on the CPU backend, as measured in the Correctness list and the Honest notes below: replies on multi-step arithmetic that did not stop before the token cap, and, with thinking on without a budget, a thought that did not close. I am sorry: since 2026-08-14, anyone who pulled the int8 file could get a reply that ran on to the token cap on multi-step arithmetic on the CPU backend (7 of 18 runs of the six multiplication questions below, 1 of them with no answer) and, with thinking on without a budget, a thought that did not close (0 of 3 runs closed it); on 2026-10-12 this card's own check recorded a run that reached the cap and still reported the file as answering 3 of 3; and the one-Engine-per-conversation note reached this card only on 2026-10-12, though my own report behind it was filed on 2026-08-06. This was my recipe's defect and my check's miss. The Mac GPU did not show it: both files gave the right product and ended by themselves in 6 of 6 runs of the six questions (one per question), and both pass the 8-question sanity gate 8/8. From now on, when one file of a recipe is found defective, its sibling files are re-checked the same day with the six questions in three fresh processes each, a run that reaches the token cap counts as a failure even when its product is right, and this card's lines are checked mechanically before it is published.

The Mixed INT4 variant exists for RAM-constrained devices (requested in LiteRT-LM #1658 for 8 GB Android phones): 2.75 GB of weights instead of 4.42 GB, measured ~1.7 GB CPU peak on the 138-token probe. If you have the memory for int8, int8 answers harder prompts slightly better (measured below).

Correctness

The int8 results below, down to the GSM8K bullet, come from the previous quantization of the int8 file or its float parent (the same graph), except the One Engine item, run on the Mixed INT4 file, and the 2026-10-13 item, run on this file:

• Logits-level parity vs PyTorch: the float export matches the HF model teacher-forced across 48 positions — top-1 and top-5 agreement 100%, Pearson 1.0000, KL ≈ 0.
• 8-question sanity gate: 8/8 on CPU and GPU, on both litert-lm 0.15.0 and 0.16.0 (Mac).
• iPhone 17 Pro (Metal GPU): the composite 8-question probe (all 8 questions in one 138-token prompt) answers 8/8, word-for-word identical to the HF fp32 reference run greedy with the same template.
• Prompt-length robustness: first-token check against the runtime's real prefill chunk plans, fresh engine per length — 40/40 lengths clean on CPU, 20/20 on GPU.
• Multi-turn conversations carry state correctly across turns.
• One Engine per conversation. On the Mixed INT4 file, a new Conversation opened on an Engine that already ran another conversation replied in 9 of 9 turns with nothing but the opening word its own system prompt asked for, against that word plus the answer in 9 of 9 on a fresh Engine (3 runs, litert-lm 0.18.0, Mac CPU). Create a new Engine for each conversation — the runtime reuses the prompt tokens a new conversation shares with the previous one and does not reset the recurrent state (my open report); a process that runs one conversation is unaffected.
• Multi-step arithmetic, CPU and GPU (2026-10-13, one fresh process per run).
  • Mac CPU (litert-lm 0.18.0, thinking off, 400-token cap): six multiplication questions (17 × 23, 23 × 17, 12 × 34, 9 × 87, 45 × 12, 36 × 25), three runs each — the right product in 18 of 18, with no repeated line in any run. 17 runs end by themselves; the 18th (36 × 25) reaches 900 and is cut by the cap inside a third method, and one run of it with a 1000-token cap ends by itself.
  • The arithmetic prompt (17 × 23) with ThinkingConfig unset: 391 in 3 of 3 runs, each ending by itself.
  • Thinking on without a budget (a raw <think> prefill, 400-token cap): 3 of 3 runs close the thought and answer 391.
  • With a 300-token thinking budget (400-token cap): 3 of 3 runs return the reasoning on the thought channel and answer 391 in a 70-character reply.
  • The 8-question sanity gate (the litert-lm 0.18.0 CLI): 8/8 on Mac CPU and on Mac GPU.
  • Mac GPU (litert-lm 0.18.0, thinking off, 400-token cap): the six questions, one run each — the right product in 6 of 6, with no repeated line. Five end by themselves; the 36 × 25 run (889 characters) reaches 900 and is then cut by the cap inside a digression — the same reply, character for character, as the previous file's GPU run.
  • Galaxy S26 CPU (the LiteRT-LM Android CLI build of 2026-09-18, litert_lm_advanced_main; thinking off by default, 400-token cap): the arithmetic prompt answers 391 in 4 of 4 runs, the same 483-character reply each time, with no repetition.
  • The Mixed INT4 file was not run on the six questions; on 2026-10-12 it answered the arithmetic prompt with 391 in 3 of 3 runs, with no repetition (litert-lm 0.18.0, Mac CPU).

Mixed INT4 correctness (all gates re-run on Qwen3.5-4B_mixed_int4.litertlm):

• 8-question sanity gate: 8/8 on CPU and GPU, on both litert-lm 0.15.0 and 0.16.0 (Mac). Prompt-length robustness: 40/40 CPU, 20/20 GPU.
• iPhone 17 Pro: the composite 8-question probe answers 8/8 on both Metal GPU and CPU (the int8 file answers 6 of 8 on CPU on this probe).
• GSM8K, n=100 (0-shot CoT, greedy, 2048-token budget, identical harness for both files; the int8 figure is the earlier card's, measured on the previous quantization of the int8 file): int8 97/100, Mixed INT4 93/100. That 4-point gap is the real cost of int4 on this model — invisible to the sanity gates, visible on math word problems. We also measured blockwise-128 (90/100) and OCTAV variants (90–92/100); blockwise-32 min-max won and is what ships.

Usage

litert-lm run ./Qwen3.5-4B_int8.litertlm --prompt "What is the capital of France? Answer in one word."

# GPU
litert-lm run ./Qwen3.5-4B_int8.litertlm --backend gpu --cache no --prompt "..."

Prefill signatures: 1024, 256, 64, 16, 4, 1 (not the 0.8B's full 1–1024 ladder). Every exported signature is charged engine memory even if never called, and with this model's 248k vocabulary the full ladder's Metal programs push a 12 GB iPhone past its jetsam ceiling during engine creation. The reduced ladder keeps GPU peak at ~4.9 GB on iPhone 17 Pro with no correctness cost (the runtime just plans slightly coarser chunks).

Chat template note: both files carry the LiteRT-LM models/qwen3_5/chat_template.jinja at 87ab9334 with two edits: its format_content macro gains the tool_response branch of the current main version (it reads chat content as a string or as parts, text and tool_response), and enable_thinking defaults to false instead of true. With nothing set, the generation prompt opens each new assistant turn with an empty <think>\n\n</think> block (past assistant turns render without it), and a single-turn prompt (one user message, with or without a system prompt) renders byte for byte as before. enable_thinking=true pre-fills <think>\n; the reasoning is returned on the thought channel (<think>\n / \n</think>), and the text carries the answer only. Tested with litert-lm 0.18.0 on a Mac (CPU): with thinking on, both files return the reasoning on the channel and give 391 for 17 × 23. A two-turn conversation ("My name is Kenji." / "What is my name?") gets the name back with thinking unset and on; with thinking on, the int8 file's second reply greets the user by name ("Hello Kenji! It's great to meet you. …") instead of stating it. On 0.17.0 both files answer a plain prompt and a two-turn conversation. With thinking on, GPU and phones were not run; the updated mixed_int4 file answered a plain prompt on a Galaxy S26 (CPU, litert_lm_main built from the v0.18.0 tag); for the Mixed INT4 file, the card's GPU and iPhone results come from its previous file, whose TFLite section and default prompt are byte-identical; for the int8 file, the Mac, iPhone and Raspberry Pi Performance rows come from the previous quantization (same default prompt), and the 2026-10-13 file's own checks are the 2026-10-13 item in the Correctness list. The template renders tool definitions, but the files' llm_model_type is generic_model, so the runtime does not parse the model's tool calls: they come back as text, and supports_function_calling is false.

Performance

The Mac, iPhone and Raspberry Pi int8 rows below were measured on the previous quantization of the int8 file (TFLite section sha256 4666cd03…, as in the files sha256 74abbd14… and a360b83d…), the same graph with the 48 decay/gate projections int8; the Galaxy S26 row further down is of this file (2026-10-13).

litert-lm benchmark (litert-lm 0.16.0), Apple M4 Max, -p 256 -d 256 --runs 3 --cache no, quiet machine:

On device (cold start, unplugged, thermal-nominal start, single runs, 138-token prompt, quality-gate harness):

(iPhone GPU: model load 66 s on first launch.)

Galaxy S26 (Qualcomm SM8850), this file (2026-10-13), the LiteRT-LM Android CLI build of 2026-09-18 (litert_lm_advanced_main), CPU benchmark of 256 prefill and 64 decode tokens, 3 iterations, median of iterations 2–3, caches removed before the warm-up run (engine init = that cold run); peak memory is the peak system RAM use reported by the CLI:

The phone capped its CPU clocks in every benchmark iteration (our frequency readings: 14 of 18 capped), so the row is this phone's capped speed. The previous file, measured right after on the same phone with the same benchmark: 157.2 / 8.64 tok/s, TTFT 1.78 s, cold engine init 20.4 s, peak 5,501 MB, with 16 of 19 readings capped; its first iteration, 199.9 / 11.09 tok/s, is within 1 % of this file's 201.5 / 11.11. Four fresh processes of this file answered the arithmetic prompt (17 × 23) with 391, the same 483-character reply each time, with no repetition. This file was not started on the phone's GPU; a Galaxy S26 hard-rebooted during the previous file's 2026-08-24 GPU attempt (see the Honest notes).

On a Pixel 8a (Tensor G3, 7.75 GB) CPU with the same Android CLI build, one process of this file answered the arithmetic prompt (17 × 23) with 391 in the same 483-character reply, with a process peak of 5,407,976 KiB and the phone's free memory at or above 3,542,768 KiB (5,120,440 KiB before the run); no speed was measured.

Mixed INT4 (litert-lm benchmark 0.16.0, Apple M4 Max, same flags; iPhone rows are single cold runs of the 138-token quality probe, unplugged):

(iPhone GPU: model load 65 s on first launch. Pixel 8a: litert_lm_main v0.16.0, 260-token prompt / 204 decoded tokens, warm weight cache; peak is process VmHWM; engine init 30–50 s; TTFT scales with prompt length — 4.2 s on a 24-token prompt.)

Honest notes:

• 2026-10-13 re-quantization (CPU). The previous int8 file (sha256 a360b83d…) could repeat one line to the token cap after its answer on multi-step arithmetic on the CPU backend (litert-lm 0.18.0, Mac M4 Max CPU, one fresh process per run, 400-token cap). With thinking off, the six multiplication questions (three runs each) got the right product in 17 of 18 runs, reached the cap in 7 of 18 (6 of them after the right product, 1 with no product) and ended by themselves in 11 of 18. With ThinkingConfig unset, the arithmetic prompt (17 × 23) answered 391 in 6 of 6 runs (two sets of three) and reached the cap after the answer in 2 of them. Thinking on without a budget (a raw <think> prefill): 0 of 3 runs closed the thought, each repeating one line. With a 300-token thinking budget: 3 of 3 runs returned the reasoning on the thought channel and began the answer with 391, and none of the 3 ended by itself (2 repeated a numbered **Final Answer:** 391. line to the cap, 1 was cut by the cap mid-sentence). On the Mac GPU the six questions got the right product and ended by themselves in 6 of 6 runs (one per question), and the 8-question sanity gate passed 8/8 on CPU and on GPU. On the Galaxy S26 CPU (the LiteRT-LM Android CLI build of 2026-09-18), 3 of 3 fresh processes answered the arithmetic prompt with 391, each ending by itself. The same cause and the same fix as the 2026-10-11 note on the Qwen3.5-2B card: LiteRT-LM's CPU backend quantizes every int8 FC's input row to int8 at run time (XNNPACK dynamic quantization), and the GatedDeltaNet decay/gate heads (in_proj_a, in_proj_b) could not take that noise. This file changes only those 48 projections, in the 24 linear-attention layers, to float (+11,660,160 B); its float export, quantized with the previous recipe, reproduces the previous file's TFLite section byte for byte. Measured on this file: Mac CPU 18 of 18 right with no repeated line, thinking on without a budget 3 of 3 closing the thought, with a 300-token thinking budget 3 of 3 answering 391, the 8-question sanity gate 8/8 on CPU and GPU, Mac GPU 6 of 6 and Galaxy S26 CPU 4 of 4.
• Where GPU execution is verified. macOS (Metal), iPhone 17 Pro (Metal) and, on Android (Mixed INT4 file), GPU verified on the Galaxy S26 (Adreno) only; no Mali phone — the Pixel 8a (Arm Mali, OpenCL) is not verified on either file, see the Pixel 8a bullets below. On the Galaxy S26 (prebuilt litert_lm_advanced_main 2026-09-18, 2026-10-12) the Mixed INT4 GPU engine builds with a cold init of 73–86 s and a benchmark peak of 3.8–4.1 GB, and one prompt run answered 17 × 23 with 391. The GPU's maximum clock fell from 1300 MHz to 646–726 MHz during the compile, so the speeds measured right after it are not published. The int8 file is not verified on an Android GPU: a Galaxy S26 hard-rebooted during its 2026-08-24 GPU attempt (the cause is not established; recorded in litertlm_manifest.json), and I have not run the int8 file on an Android GPU since, the 2026-10-13 file included; use the CPU. Until 2026-10-12 this bullet said GPU execution on Adreno was not verified; for the Mixed INT4 file, this S26 run changed that.
• GPU inference runs with fp32 activations (declared in the bundle; the fp16-activation formulation is unfinished for this family). That is where the GPU memory multiple over CPU comes from.
• Pixel 8a (8 GB-class Android, CPU): one process of this file loaded and answered. On a Pixel 8a (Tensor G3, 7.75 GB) CPU with the LiteRT-LM Android CLI build of 2026-09-18, one process of this file (sha256 ce4c2d64…) answered the arithmetic prompt (17 × 23) with 391, in the same 483-character reply as on the Galaxy S26. The process took 146.8 s, including the first-run weight-cache build. It peaked at 5,407,976 KiB, and the phone's free memory stayed at or above 3,542,768 KiB (5,120,440 KiB before the run). Our test stops a run when the phone's free memory falls under 400,000 KiB; it did not stop this one. This is one process, and no speed was measured. This run corrects this bullet's earlier "not attempted, and not expected to fit", which was an inference.
• The Mixed INT4 file runs on a Pixel 8a (8 GB) on CPU — measured, table above. Peak process memory 4.43 GB on a 7.75 GB-RAM phone, coherent long-form generation and correct QA output. Use the CPU backend on 8 GB phones: the GPU path expands to ~5.7 GB on Metal and Android GPU budgets are similar or tighter. First run writes a 2.66 GB XNNPACK weight cache next to the model (one-time) — leave ~3 GB of storage free. Until 2026-10-12 this card listed the Pixel 8a as a verified GPU device — my error, and I am sorry to anyone who chose the GPU on a Mali phone because of it: on the Pixel 8a the engine build drains the phone's memory. On 2026-10-12 GPU engine creation for this file on the Pixel 8a (Arm Mali, OpenCL) reached 1 of the 6 prefill signatures, fully delegated it, and took the phone's free memory from 4,244,000 KiB towards our test's stop line of 400,000 KiB. Our test reads free memory every 2 s and stops the run at the first reading under that line. In this run that reading was 139,628 KiB, after Android had begun closing other apps, so do not try GPU on 8 GB devices. Whether the build completes without the stop is not measured. Before it is published, this card is now checked mechanically for the device limits measured on its files.
• Mixed INT4 CPU prefill is slower than int8's (Mac: 100 vs 243 tok/s prefill, TTFT 2.6 vs 1.1 s at 256 tokens; iPhone CPU is the exception — 46.7 vs 24.0, faster). Decode is equal or faster everywhere.
• On low-end Android GPUs, decode is memory-bandwidth-bound and does not beat the CPU anyway; the GPU win is on Apple hardware (and, generally, prefill/TTFT).
• On quality: individual questions gate 8/8 on every backend. On the harder composite probe (8 questions in one prompt), GPU matches HF fp32 word-for-word; the CPU int8 path answers 6 of the 8 (every answer it gives is correct — it skips two). If you need maximum fidelity on complex prompts on CPU, ask for a float variant. Measured on the previous quantization; the composite probe was not re-run on this file.

Conversion notes

Converted with litert-torch plus a hybrid-cache patch (reproduction script + patch: hf-to-litertlm qwen35_work/):

• Rank-4 chunk kernel (shared with the 0.8B): the reference chunked delta rule spells its contractions as broadcast-multiply-reduce over high-rank intermediates. The vendored kernel re-expresses them as batched matmuls with chunk and head axes folded into the batch axis (all tensors rank ≤ 4, no BROADCAST_TO, no int64 index math), and writes every tail-pad as a concat with a zeros constant rather than a PAD op (the GPU delegate miscomputes rank-3 non-final-axis PAD — LiteRT#9272).
• Rank-4 head interleave (4B-specific): the 4B is the first Qwen3.5 with grouped value heads (32 value vs 16 key heads). Upstream expands q/k with repeat_interleave, whose rank-5 lowering the GPU delegate rejects outright — the 0.8B (ratio 1) never traces that branch. The patch re-expresses it as concat + rank-4 reshapes, bitwise-identical to the stock op.
• Reduced prefill ladder (4B-specific): see the signature note above.
• Export cache for GatedDeltaNet layers: conv [B, conv_dim, K] + recurrent [B, heads, k_dim, v_dim] cache layers registered for layer_types == "linear_attention", so torch.export traces the model's own state contract.
• int8 recipe: post-hoc ai-edge-quantizer dynamic int8 on the FULLY_CONNECTED layers and the embedding, with prefer_activation_type fp32. The convs and the delta rule stay float, and since 2026-10-13 so do the 48 GatedDeltaNet decay/gate projections (in_proj_a / in_proj_b of the 24 linear-attention layers).
• Mixed INT4 recipe: post-hoc ai-edge-quantizer on the same float export — int4 BLOCKWISE-32 min-max on every FULLY_CONNECTED, overridden to int8 channelwise on the lm_head and the embedding lookup (one shared 248320×2560 vocab buffer, kept single). Block size and algorithm were chosen by measurement: b32/b128 × min-max/OCTAV all built from the same export and scored on GSM8K n=100 (93 / 90 / 92 / 90) — blockwise-32 min-max won. Convs and the delta rule stay float, prefer_activation_type fp32 as in the int8 file.

Raspberry Pi 5 (CPU)

Measured on a Raspberry Pi 5 Model B Rev 1.1 (8 GB, Raspberry Pi OS 64-bit) with litert-lm benchmark 0.16.1: CPU backend, 4 threads, 256 prefill + 256 decode tokens, --cache memory (the compile cache lives and dies with the process, so every invocation compiles the model from scratch; nothing is reused between runs), one warm-up plus one timed iteration per invocation, 3 invocations per file with cooldown in between. Values are the median across invocations (min–max in parentheses). No thermal throttling occurred during these runs (vcgencmd get_throttled stayed 0x0). Every file listed produced coherent text in a real generation on this backend before its numbers were recorded.