Gemma-4-E2B meeting agent (zh-TW): LiteRT-LM and Q4_0 GGUF

Gemma-4-E2B (QAT) fine-tuned for three on-device meeting tasks, on a phone CPU:

1. Realtime reading. The model reads a live zh-TW meeting transcript window by window and writes short cited notes: decisions, actions, proposals, open issues and key figures. The harness turns those notes into minutes.
2. Notes → prose (from v8 on). It turns the notes into a cited summary in prose.
3. Notes → title (from v8 on). It writes a title for the whole meeting from the notes.

Code, agent protocol and evaluation: https://github.com/vieenrose/meeting-summarizer

Versions

What mobile-v1 changes. It is a .litertlm file for LiteRT-LM, Google's on-device runtime. It is built on Google's Gemma-4-E2B mobile graph (litert-community/gemma-4-E2B-it-litert-lm): int2 MLP in layers 15–34, int4 elsewhere, an int8 KV cache and int8 static-range activations.

• Training. A LoRA trained on Google's mobile weights distils v11: 0.7 × cross-entropy on v11's top-32 next-token distributions, plus 0.3 × cross-entropy on the gold.
• Back onto Google's grid. The merged weights are rounded onto Google's own int2/int4 grid with GPTQ, using Google's scales. Rounding to the nearest integer would erase the fine-tune, since almost every change is below half a step.
• Injection. The integers are written into Google's file, bit for bit. Two settings change: fp32 activations on GPU (fp16 corrupts long prompts on the Mali-G68), and the prefill_1024 signature is disabled, since its buffers dominate peak memory.

mobile-v1/mfa/: for the forked engine (recommended). This folder holds the same weights in the files that the forked LiteRT-LM engine loads (vieenrose/LiteRT-LM, branch mobile-fused-attention, contrib/mobile_fused_attention/):

• prefill_decode_fused.tflite: the main graph, with each attention block fused into one int8 custom op (sha256 6a7555ccc349be490fca4ed63ebf7fdafd8e4a4510012f3ae9c38f8009b5fcae);
• the embedder and per-layer embedder;
• the SentencePiece tokenizer;
• LlmMetadataProto.pbtext, the chat template.

With the same prompt, its greedy output is token for token that of the .litertlm, at about half the memory.

mobile-v1/mfa-gpu/: the same reader for the GPU. prefill_decode_fused.tflite (sha256 a3504d11e97a346ec81b1db4d47aff55d1acf74bb2cd2129fb016e523b36c4ae) is the .litertlm's own prefill/decode graph: standard attention, no custom op, so every op runs on ML Drift. Only the prefill_128 and decode signatures are kept. It replaces the main graph of mfa/; the embedders, tokenizer and metadata of mfa/ are reused. To run it correctly, follow LiteRT-LM's GPU setup:

• GPU accelerator only, precision = fp32 (fp16 drifts to English on short prompts);
• external_tensor_patterns = ["kv_cache_", "param_tensor"], buffer_storage_tensor_patterns = ["kv_cache_c_", "kv_cache_", "param_tensor"], enable_infinite_float_capping, enable_constant_tensors_sharing (7.7 → 1.7 GB peak);
• KV caches created as GPU buffers from the output buffer requirements, and the KV inputs left unbound. A host-memory KV cache is corrupted by the GPU round trip: correct first token, garbage from the second.

Measured on a Galaxy Note10+ (Snapdragon 855, Adreno 640), same 895-token reading prompt, 128 tokens decoded:

On this phone the CPU is faster: decode is bound by the memory bandwidth both share, and XNNPACK runs int8 dot products where ML Drift dequantizes. The GPU path frees the CPU for speech recognition; newer GPUs should prefill faster (not measured).

Use the CPU only. Measured on a Reno7 (Dimensity 900, 8 GB) with maxNumTokens = 4096 and a 2,881-token reading prompt. Peak RSS is the larger of dumpsys meminfo TOTAL RSS and VmHWM:

• GPU. It needs more memory and decodes slower, so it is listed for reference only. With fp16, Google's setting, it prefills at 162 tok/s but corrupts long prompts.
• llama.cpp. v11 on llama.cpp prefilled at 8 tok/s at depth.
• Larger model. The E4B sibling, Luigi/gemma-4-E4B-meeting-agent-zh-LiteRT, contradicts the transcript less: 11 % of minutes against 17 %. With the forked engine it fits in 2.97 GB, at 43 tok/s prefill and ~4 tok/s decode.

Quality, on 38 held-out IVOD sessions, run through LiteRT-LM with a 4k context:

Settings, the protocol and the measurements are in the integration note: §12 covers stock LiteRT-LM, and §13 the forked engine.

What v11 changes. v11 starts from v8 and adds a contrastive DPO on 2,700 single-span pairs built from the teacher's verified notes. In each pair, one note is altered in one of the three ways the students most often go wrong: the right fact on the wrong object (an article, year or amount swapped for another one from the same window), the wrong body or role, or an inverted result (passed ↔ held, a proposal written as a decision, a dropped negation). The loss is DPO plus an NLL term on the correct reply (RPO).

v11 against v5 and v8 (IVOD, 38 held-out sessions; judge: Gemma-4-31B):

v11 lists fewer decisions (493 against 663 for v8), and more of them were really decided. Its titles are the best of all versions. In exchange it covers less of the meeting. Its faithfulness to the transcript is unchanged: the contrastive pairs did not reduce the factual errors themselves. Choose v11 when a reliable 決議事項 section and the title matter most; choose v8 for the widest coverage.

What v8 changes. The reading protocol and prompt are v5's, so v8 is a drop-in replacement for v5. It is also trained on the two calls VoxSumDroid makes once the meeting ends, with VoxSumDroid's own prompts (eval/conversion_prompts.py in the repository): notes → prose summary, notes → title.

• Conversion SFT (v6, not published). 123 prose summaries and 540 titles written by Qwen3.8-27B from 543 journals. A prose target was kept only if the judge found no sentence contradicting the notes and its form was the requested one: no bullets, citations taken from the journal only.
• Multi-task GRPO (v8). 150 steps from that adapter, on a mix of the three tasks (1,200 reading windows, 400 prose calls, 400 title calls). Rewards came from Qwen3.8-27B. For reading: faithfulness of each note to the transcript, whether each DECISION / ACTION type holds, plus recall of the teacher's notes, so writing less does not pay. For prose: faithfulness of each sentence to the notes, plus form. For titles: a 1–5 score. The KL is taken against the starting adapter.

v8 against v5 / v6 (held-out sets; judge: Gemma-4-31B):

For reference, Qwen3.8-27B itself on the same notes: prose contradicting the notes 11 % (IVOD) and 19 % (AliMeeting); titles 4.45 (IVOD) and 5.0 (AliMeeting).

What v5 changes. On AliMeeting business meetings, v3 filed proposals as decisions ("建議…" under 決議事項) and listed every idea discussed as an action.

• v5 notes are typed DECISION only when a decision is announced, ACTION only when someone is assigned or a deadline is set, and PROPOSAL for suggestions still under discussion. The minutes gain a 討論要點 section for them.
• The harness also reclassifies a DECISION or ACTION worded as a proposal.
• The training data adds 217 AliMeeting meetings to the 163 IVOD sessions.
• Use v5/system_prompt.txt with the v5 weights, and map PROPOSAL to its own section.

v5 against v3 (judge: Gemma-4-31B; "really decided / assigned" = eval/section_precision.py):

Results (v3)

On 38 held-out sessions, judged by Gemma-4-31B against the transcript: each cited statement is checked from 30 s before its citation to 150 s after. The agent restarts from its compacted journal at 8k tokens, the phone setting.

Live on an OPPO Reno7 (Dimensity 900, CPU only, llama.cpp, 8 threads). A 2 h 08 meeting was replayed at real speed: the lag after each ~4 min window was 67 s median and 115 s max, with no drift. Effective speed was 16 tok/s of prefill and 4.5 tok/s of decode.

About 1 statement in 5 is contradicted by the transcript. The errors are mostly relational: the right figure attached to the wrong year, scope or body. Check decisions and figures before relying on them.

Use

The model expects the agent protocol, not free-form prompts. It is one growing conversation:

1. system prompt (system_prompt.txt);
2. a user turn with the journal so far;
3. then, for each window, a user turn ## 逐字稿片段 k with ~2k tokens of transcript lines [h:mm:ss] Sx: text.

The model answers with at most 5 lines NOTE [time] (TYPE) text, then NEXT. Stop generation at \nNEXT and cap the output at 400 tokens. Restart the conversation from the compacted journal at 8k tokens: on a phone CPU, prefill slows with context depth, and the 8k budget loses no quality. See eval/realtime_agent.py (host) and eval/phone_live.py (phone, incremental prefill, restart at 8k context) in the repository.

llama-server -m gemma-4-E2B-meeting-agent-zh-Q4_0.gguf -t 8 -c 32768 --jinja

Training

• Teacher. Qwen3.8-27B ran the same protocol on 163 training sessions of Legislative Yuan (IVOD) committee meetings. The transcripts come from on-device ASR.
• Judging. Every teacher note was checked against the transcript. Reading turns with a contradicted note stayed as context and carried no loss.
• Training. LoRA SFT over the replayed conversations: 677 segments of up to 14k tokens and 2,759 loss turns, for 1 epoch. The adapter was then merged into the QAT weights and requantized to Q4_0; the one unquantized tensor is kept in F16, as in Google's GGUF.
• Tried without gain. On-policy DPO on 1,667 teacher-corrected windows did not change faithfulness. Of the two, this is the SFT model.

No training data is included. The source recordings are subject to the Legislative Yuan IVOD terms of use.