Orosius/loquace-talker-qwen3.8-27b

A talk-while-thinking head. 61,353,984 trained parameters that read the hidden states of a frozen Qwen/Qwen3.8-27B at layer 63 and start speaking the answer while the trunk is still thinking — before </think> is ever emitted.

This repository ships the head only. The trunk is not redistributed: it is loaded from Qwen/Qwen3.8-27B at run time, and it is never fine-tuned, quantised or otherwise modified by LOQUACE.

What it is

A reasoning model spends most of its latency thinking. LOQUACE's claim, which this head is the evidence for, is that the answer is legible in the trunk's hidden states long before the thought is finished — so a small decoder that cross-attends over those states can emit the user-visible answer early, at near-parity quality, with the trunk untouched.

The head is a 2-block pre-norm decoder (d_model 1024, 16 heads, d_ff 2816): causal self-attention over local answer positions (RoPE restarting at 0), cross-attention over the trunk memory (K/V projected from d_model_memory 5120), SwiGLU. Token embedding and LM head are the trunk's own frozen matrices, reached through two trained linear adapters. A commit gate (P-ent at τ = 0.75) turns the head's draft into a frozen-prefix stream every Δ_tick = 4 trunk tokens.

Leakage invariant: at visible-think length k the cross-attention mask exposes the prompt plus the first k think tokens and nothing else — never an answer position, at any ρ including 1.0. Masked weights are exactly 0.0 (additive −inf before softmax), and that exactness is asserted by the test suite, not assumed.

How to use

pip install "loquace @ git+https://github.com/Orolol/loquace"
huggingface-cli download Orosius/loquace-talker-qwen3.8-27b --local-dir ./loquace-talker-qwen3.8-27b
python ./loquace-talker-qwen3.8-27b/demo.py --prompt "What is 17 * 24?"

import torch
from loquace.import_hf import load_release
from loquace.models.trunk import FrozenTrunk
from loquace.serve.bench import chat_prompt_ids
from loquace.serve.chat_runner import ChatRunner
from loquace.serve.session import ServeConfig, ServeSession
from loquace.serve.trunk_stream import TrunkStream

release = "./loquace-talker-qwen3.8-27b"
trunk_id, layer = "Qwen/Qwen3.8-27B", 63

# The head: the trained params + the trunk's frozen embed / lm_head.
tok_trunk = FrozenTrunk.tokenizer_only(trunk_id, layer=layer)
talker = load_release(release, trunk=tok_trunk, device="cuda")

# The trunk: one HF decode loop with a layer-L capture hook.
stream = TrunkStream.for_mode("bake", model_name=trunk_id, layer=layer,
                              device="cuda", dtype=torch.bfloat16)
chat = ChatRunner(talker, tokenizer=tok_trunk.tokenizer,
                  bos_id=int(tok_trunk.tokenizer.eos_token_id),
                  eos_id=int(tok_trunk.tokenizer.eos_token_id),
                  device="cuda", tau=0.75,
                  delta_tick=4)

session = ServeSession(
    stream,
    ServeConfig(flow="chat", mode="bake", tau=0.75,
                delta_tick=4,
                model_name=trunk_id, trunk_layer=layer),
    tokenizer=tok_trunk.tokenizer, chat_runner=chat,
    eos_id=int(tok_trunk.tokenizer.eos_token_id),
    on_emit=lambda text, t: print(text, end="", flush=True),
)
trace = session.run({"id": "demo", "prompt": "What is 17 * 24?"},
                    prompt_ids=chat_prompt_ids(tok_trunk, "What is 17 * 24?"))
print("\nTTFT:", trace.timings["ttft_user"], "s")

demo.py in this repository is that script, ready to run.

Numbers

Offline quality — exact match as a function of ρ, the visible fraction of the thought (B1 = the same head trained on layer-0 embeddings, the "no reasoning state" floor; B2 = truncate the thought and let the trunk answer):

Streaming, zero-shot (simulator, P-ent at τ = 0.75):

The bake — the live runtime, same trunk, same prompts, same sampling:

Artifacts these numbers were read from:

• results/ship_b2.json
• results/ship_bench_math_bake.json
• results/ship_bench_math_vanilla.json
• results/ship_stream.json
• results/ship_sweep_L0.json
• results/ship_sweep_L63_full.json

Reading these numbers

The table above is the head's number. Two other numbers on this page decide what the head is worth on this trunk, and they should be read first.

B2 — the trunk with its thought cut off. Truncate the thinking at a fraction ρ, close the </think> tag, and let the frozen 27B answer on its own: EM = 0.854 at ρ = 0.0 (no thinking at all), 0.937 at ρ = 0.6, 0.957 at ρ = 1.0, against 0.960 for the trunk's own untouched answer. On this test population (MATH + GSM8K, 1 167 problems) the 27B does not need its thought to answer, and the trunk answering without thinking beats the head reading the whole thought (0.814). The head is not the better answerer here. What it is: a 61 M decoder that reads the trunk's states — 15× the layer-0 floor at every ρ — and answers at a small fraction of the trunk's decode cost. On the 4B this project was developed on, B2 at ρ = 0 was 0.867 on GSM8K and 0.509 on MATH, so the anchor has always been strong; on the 27B it is decisive.

The gate. The commit threshold shipped in config.json (τ = 0.75) was calibrated on the 4B and carried over; nothing was calibrated on this trunk. At that τ the head's entropy stays above threshold until the thought is nearly over: the first commit lands at a median 100 % of the thought (mean 88 %), so EM_stream equals EM at ρ = 1 and the live TTFT is essentially the vanilla TTFT. A τ sweep on a stratified 1-in-4 subset of the test set (n = 292) gives the latency/quality curve instead of a point:

Up to τ = 2 the median trace still commits at the very end; the only setting that speaks early (τ = 5, first commit at 19 % of the thought) loses 21 points of EM and retracts on a third of the answers. That is what the ρ-curve predicts: EM 0.28 at ρ = 0.6 and 0.79 at ρ = 0.9 on this layer. On this trunk, speaking early means speaking wrong.

What transfers from the 4B and what does not. The state-vs-token result (H2) transfers: layer 63 vs layer 0 is 0.814 vs 0.054. The early-decodability result (H1, ρ₉₀ = 0.6 on the 4B) does not: ρ₉₀ ≈ 0.9 here. The live-runtime results transfer: state capture is free (trunk 41.8 ms/token in both arms) and the runtime reproduces the simulator. The practical value of an early-speaking head on this trunk and this population is therefore small, and this card says so rather than quoting the 4B's 650× TTFT number as if it were the 27B's.

Live bench, n = 100 per dataset, single stream, RTX PRO 6000 Blackwell, HF decode loop (no vLLM), τ = 0.75, answer budget 512:

The bake finishes 4 s sooner (the head decodes the answer instead of the trunk) and starts at most 17 % sooner, at a cost of 31 EM points on MATH and 7 on GSM8K. The 4B's live numbers (TTFT 30 s → 46 ms at EM ratio 0.82–0.90) do not carry over, and this card does not quote them for this model.

Limitations

• Censoring domain. The head was trained on censored traces — thoughts and answers that fit the training window (max_seq_len, max_answer_len 512) on the datasets it was trained on. Prompts whose reasoning or answer style falls outside that population are out of domain, and the head has no way to signal that it is.
• Long answers. The decoder budget is 512 tokens with RoPE over local positions only. Beyond that it is untested; it is not a general-purpose LM, and it has no memory of anything but the current turn.
• Single stream. The runtime is one prompt, one trunk stream, one head. No batching, no continuous batching, no paged KV; the published latency numbers are single-stream numbers and do not describe a served fleet.
• Not a standalone model. Without the trunk's states it produces nothing: this release is 245 MB of adapter that only means anything on top of Qwen/Qwen3.8-27B.
• Quality is not parity. Early answers are early drafts of the trunk's answer; the gate can commit a prefix the finished thought would have contradicted (that is what RTR measures, and it is not zero).
• Language / task scope. Evaluated on the reasoning-benchmark population described above, in English, with numeric-answer exact match. Nothing here licenses a claim about other tasks.

How it was built

• Frozen trunk. The trunk is loaded once, requires_grad_(False), and only layer-63 hidden states leave it. No trunk weight is ever in the optimizer; the head's embedding and LM-head matrices are non-persistent buffers, which is why they are absent from model.safetensors.
• Hidden-state cache. Training reads a disk cache of the trunk's layer-L states keyed by trace id and checked against the trunk identity, so the expensive forward is paid once and every ablation sees byte-identical memory.
• Teacher forcing under a ρ-mask. Each example draws a random ρ; the cross-attention mask exposes prompt + the first ρ·L think tokens. One trunk forward over the full sequence is causally valid memory (position i's state depends only on positions ≤ i), and the mask is the only thing standing between the head and the answer it is being asked to predict.
• Pre-registered gates. Hypotheses, thresholds and pass/fail bars were written down before the runs and are recorded in the repository's docs/; negative results were kept and published rather than dropped. Adverse controls (shuffled memory, layer-0 floors, truncate-and-answer baselines) are part of the suite, not an afterthought.
• Fidelity. The offline simulator and the live runtime are held to the same numbers on the same trace set at matched policy parameters; a divergence is treated as a bug in the runtime, not as a new result.

Citation

@software{loquace_2026,
  title  = {LOQUACE: talking while thinking — early answer decoding from the
            hidden states of a frozen reasoning LLM},
  year   = {2026},
  note   = {Chat head for Qwen/Qwen3.8-27B, layer 63},
  url    = {https://huggingface.co/Orosius/loquace-talker-qwen3.8-27b}
}

Licensed Apache-2.0. The trunk keeps its own licence; see Qwen/Qwen3.8-27B.