LiteRT is Google's on-device runtime, the new name for TensorFlow Lite (Android: com.google.ai.edge.litert:litert), and litert-torch, the renamed ai-edge-torch, is its PyTorch converter: a PyTorch model converted unmodified with litert_torch.convert matched the original to 4e-7 on a Galaxy S26 (measured, LiteRT 2.2.0, Android 16, 2026-09-05).

Measured on device (edge-compat, lfm2.5-2.6b-int4): Galaxy S26 · LiteRT-LM 0.16.0 · GPU · decode 25.3 tok/s · prefill 297 tok/s · TTFT 730 ms · all 936 ops delegated (2026-08-24); Raspberry Pi 5 · LiteRT-LM 0.16.1 · CPU, 4 threads · decode 4.9 tok/s · prefill 28 tok/s · TTFT 9.19 s (2026-09-01). Record: https://github.com/john-rocky/edge-compat/blob/main/cards/lfm2.5-2.6b-int4/CARD.md

Measured on device (edge-compat, lfm2.5-2.6b-int8): Galaxy S26 · LiteRT-LM 0.16.0 · GPU · decode 19.7 tok/s · prefill 489 tok/s · TTFT 470 ms · all 936 ops delegated (2026-08-24); Galaxy S26 · LiteRT-LM 0.16.0 · CPU · decode 15.5 tok/s · prefill 222 tok/s · TTFT 990 ms (2026-09-06); Raspberry Pi 5 · LiteRT-LM 0.16.1 · CPU, 4 threads · decode 3.5 tok/s · prefill 44 tok/s · TTFT 6.13 s (2026-09-01). Record: https://github.com/john-rocky/edge-compat/blob/main/cards/lfm2.5-2.6b-int8/CARD.md

LFM2.5-2.6B — LiteRT-LM

LiquidAI/LFM2.5-2.6B converted to the LiteRT-LM (.litertlm) format for on-device inference with Google's LiteRT-LM runtime (requires litert-lm ≥ 0.15; the bundled executor-metadata section binds 0.15's state buffers).

LFM2.5-2.6B is the flagship of Liquid AI's LFM2.5 on-device family: a hybrid architecture (22 gated short-convolution blocks + 8 grouped-query attention layers) and a thinking model — it reasons inside a <think>…</think> block before answering. The bundle declares the thought channel, so LiteRT-LM apps receive the reasoning and the answer as separate streams.

2026-09-21: chat template updated to accept the 0.18 content-parts form (string form unchanged); weights, tokenizer and executor metadata byte-identical.

Quality — GSM8K

GSM8K, greedy, 0-shot chain-of-thought, max-tokens 2048 (thinking model), n=100, same harness for all rows:

Both files pass an 8-question sanity gate 8/8 with zero degenerate outputs (CPU), a 42-length prefill sweep with zero corrupt first tokens (fresh engine per length), and a 3-turn conversation gate (fact recall + arithmetic).

Usage

pip install litert-lm
litert-lm run ./LFM2.5-2.6B_int4.litertlm --prompt "What is 17 + 25?"

The model thinks before answering — through the Python/C++ API the reasoning arrives on the thought channel and the reply stays clean; the CLI prints both.

--max-num-tokens tip: reasoning turns are long — give the model at least 2048 tokens of budget for math/complex questions, or the think block may not close. Decode speed drops as the token budget grows, so don't set it higher than you need.

Multi-turn note: on the current runtime the conversation context retains previous turns' reasoning, so long chats fill the context faster than the visible text suggests. For unrelated questions, prefer fresh conversations, each on a new Engine (see One Engine per conversation below).

One Engine per conversation. On litert-lm 0.18.0 the runtime starts a new Conversation on an Engine that already ran another conversation from the earlier conversation's short-convolution state: it reuses the prompt tokens the two conversations share and does not reset the state (my open report). In a probe that asks three questions (What is the capital of France? / What is 17 times 23? / What color is the sky on a clear day?) first in a pirate-persona conversation and then in a conversation told to answer in one word, on the same Engine against a fresh Engine in a new process (answers capped at 200 tokens; CPU backend; litert-lm 0.18.0 on a Mac), the _int8 file gave the same three one-word answers on both Engines, in its one run. The _int4 file was run twice (a probe is run a second time when its first run shows a difference or when the fresh Engine misses one of the three expected answers: Paris, 391, blue): in its first run the arithmetic answer came back as an empty reply on the fresh Engine; in its second run the same three one-word answers came back on both Engines. Those three short questions say nothing about longer conversations. Create a new Engine for each conversation. The first conversation on an Engine is unaffected.

Tool calls on litert-lm 0.18.0. Since the 2026-10-13 re-pack (metadata only: llm_model_type lfm2, and the chat template now renders tool calls and tool results), the runtime parses the LFM2 tool call and the tool result reaches the model; the earlier files returned the call as message text (<|tool_call_start|>[get_weather(location="Paris")]<|tool_call_end|>). The lfm2 model type needs litert-lm ≥ 0.15.0; the tool-call parser is 0.18.0's — on 0.15–0.17 the call still arrives as text (read from the runtime source, not run here). The 2026-10-13 re-pack changes the model type, and with it the runtime's GPU executor defaults (every non-generic LiteRT-LM model type shares them: allow_src_quantized_fc_conv_ops on, OpenCL hint_waiting_for_completion off, kernel batch size unset); the phone GPU rows on this card were measured with the previous files and have not been re-run — the Mac GPU digit check passed on 0.18.0.

Speed

litert-lm benchmark, CPU backend, Mac M4 Max, max-num-tokens 1024 (prefill-1024 rows measured at max-num-tokens 2048):

GPU backend (litert-lm benchmark … --backend gpu --cache no, litert-lm 0.16.0, same machine; both variants verified to generate correct text on GPU before quoting):

On Android the same bundle runs GPU-accelerated with litert-lm ≥ 0.16.0. Pixel 8a (Tensor G3), litert_lm_main v0.16.0 built from the release tag, 261-token prompt with generation capped at 256, 3 runs per backend (int4 file):

At this size the GPU wins across the board on-device — ~3–5× prefill, ~3–5× faster to the first token, and a steadier decode (the CPU rows spread because they throttle; the GPU rows repeat to within 1%) — so prefer the GPU backend where the runtime offers it. A short prompt shifts the ratio further toward the GPU: with a 19-token prompt the same file prefills at 37.7–38.8 tok/s on GPU against 4.1 tok/s on CPU.

iPhone 17 Pro (CPU backend, cold first runs): the int4 file answers the 8-question quality gate 8/8 and decodes at ~20 tok/s warm (the first, cold run is slower while caches build). The int8 file also loads and runs on iPhone 17 Pro when the host app carries the extended-virtual-addressing / increased-memory entitlements (its 2.87 GB single weight section exceeds what a default-entitlement app will memory-map) — int4 is the recommended phone variant (smaller, faster on phone-class memory bandwidth).

Galaxy S26 — GPU backend

Both published bundles run on the Android GPU backend: LiteRT takes the whole graph in each and the model generates.

Measured on a Samsung Galaxy S26 (SM-S942Q / SM8850, Android 16) with litert_lm_advanced_main from litert-lm 0.16.0, --backend=gpu --sampler_backend=cpu, prompt What is the capital of France?. Peak is the process high-water mark (VmHWM) sampled during that same run. Gated 2026-08-24 and 2026-08-25.

No speed rows, on purpose. On this handset the GPU backend wins prefill and does not win decode, so a GPU throughput figure only means something beside a CPU row from the same handset, and no S26 CPU row exists for this model yet.

GPU wiring, including the Gallery import toggle: GPU guide.

Run on Android

Install a recent Google AI Edge Gallery (1.0.16+ imports .litertlm directly from Hugging Face), import this repo (or adb push a file and use local import), select the CPU backend, and chat. GPU works from litert-lm ≥ 0.16.0 (litert_lm_main --backend=gpu, or the SDKs); the Gallery app bundles its own runtime which may lag behind that version — if its GPU toggle fails, use CPU there.

Conversion notes

Converted with released litert-torch 0.9.2 (upstream lfm2 hybrid support incl. the ShortConv prefill-pad fix) and packaged for litert-lm ≥ 0.14 (≥ 0.15 since the 2026-10-13 re-pack) (the executor-metadata section binds the 22 conv states + 16 KV caches; files exported without it do not run on 0.15).

• The generation prompt pre-fills <think> exactly as the vendor chat template does. This matters: with a bare assistant prompt, think-block emission becomes the model's choice, and the int4 quantization degrades that discipline first (unscaffolded rambling in place of answers on multi-turn). With the pre-fill, all variants answer tersely and the thought channel routes cleanly.
• The checkpoint is notably sparse: OCTAV int4-b32 produced 746k all-zero weight blocks whose zero scales the XNNPACK delegate rejects at load. The zero scales are patched to the tensor's smallest nonzero scale (dequantization unchanged — the blocks are all zeros).
• int8 quantizes convs at export time (safe); post-hoc conv int8 breaks generation. int4 keeps convs float.
• Multi-length prefill signatures (1–1024) are exported so the runtime picks tight chunks; a 42-length first-token sweep verifies no prefill-padding state corruption.

Raspberry Pi 5 (CPU)

Measured on a Raspberry Pi 5 Model B Rev 1.1 (8 GB, Raspberry Pi OS 64-bit) with litert-lm benchmark 0.16.1: CPU backend, 4 threads, 256 prefill + 256 decode tokens, --cache memory (the compile cache lives and dies with the process, so every invocation compiles the model from scratch; nothing is reused between runs), one warm-up plus one timed iteration per invocation, 3 invocations per file with cooldown in between. Values are the median across invocations (min–max in parentheses). No thermal throttling occurred during these runs (vcgencmd get_throttled stayed 0x0). Every file listed produced coherent text in a real generation on this backend before its numbers were recorded.