qwen3.8_27b_idiotSavant_sm_86_dflash2

The DFlash2 drafter re-tuned for qwen3.8_27b_idiotSavant_sm_86. It guesses whole blocks of tokens; the savant checks them.

This is not a standalone model. It is the speculative-decoding drafter for the rotated W4A8 idiotSavant target: a 5-layer DFlash2 block-diffusion drafter (block of 8, top-16 candidate selector for tree speculation), int4 W4A16, 1.2 GB.

Why a special drafter

The target keeps its residual stream in a rotated basis (h·Rᵀ, randomized block-Hadamard) so that int8 activations stop clipping (see the target's card). A drafter shares a lot with its target, and here that sharing breaks in three places:

1. fc reads 5 target hidden states (layers 5, 19, 33, 47, 61), now rotated. The inverse rotation is folded into fc (W_c·Rᵀ for each 5120 chunk), so the features come out identical.
2. The embedding is shared with the target, which stores E·Rᵀ. PN149 un-rotates it online.
3. The lm_head is shared and stores W·diag(g_final)·Rᵀ. PN149 feeds it (h / g_final)·Rᵀ, both for the logits and for the tree's top-16.

Why not simply rotate the whole drafter? Its k_proj/v_proj serve two streams under two different norms: the target features (hidden_norm) and the block tokens (input_layernorm). No single rotated weight can fold both, so the drafter stays in its original basis. PN149's two ops are block Hadamards with no weight reads; the per-step cost was not measurable (38.3 vs 38.4 steps/s). The rotation parameters (signs, block size, 1/g_final) are in config.json under genesis_rotacion.

How it was trained

Straight from Inco's release to this target: no intermediate model.

1. Start: incoai/Qwen3.8-27B-DFlash2 in BF16, with the target's rotation folded into its fc (see above).
2. Capture: 840 real agentic-coding requests (SWE-rebench tasks with tools, with the repos of the evaluation bank excluded) were regenerated by the served idiotSavant (int4 × int8, rotated, tree speculation, coder sampling profile). The labels are its top-16 per token; the features are the output of the served fc.
3. Train:
   • LoRA r=64 on the 7 linears of the 5 layers, 1 epoch, lr 1e-4, batch 32;
   • loss 0.9·TV + 0.1·CE against the top-16, with AUF: it stops at the first position that would be rejected, so it trains what the tree actually uses;
   • the conv, fc, norms and selector stay frozen (training the conv destabilized it);
   • the trainer sees the effective embedding and lm_head the drafter sees in serving, int4 included.
4. Quantize: RTN W4A16 g128. It is deterministic: the fc served during capture is exactly the final one.

Results

Served, on an evaluation bank of 40 agentic-coding requests with tree speculation, coder profile and 3 replicas (totals, because per-request medians mislead across models): 5.65 / 5.69 / 5.54 → 5.63 accepted tokens per step, 218 / 218 / 212 → ~216 tok/s decode (2026-09-25, first release).

Updated in place on 2026-10-03: one more epoch with the same recipe (LoRA r=64, AUF, 0.9·TV + 0.1·CE, conv/fc/norms/selector frozen), starting from the previous release. The offline greedy length did not move (6.08 vs 6.09), but served acceptance went up in every paired comparison:

For small changes the offline metric does not predict the served one. Training against the 64k-token vocabulary that Genesis serves the drafter with (PN159) was tested and made it clearly worse (−6.5% served): the selector stays calibrated for full-vocabulary logits.

Serving

GENESIS_ENABLE_PN148_ROT_DOWN=1 GENESIS_ENABLE_PN154_HAD_SALIDAS=1 GENESIS_ENABLE_PN155_BA_EN_QKVZ=1 \
GENESIS_ENABLE_PN149_ROT_BORRADOR=1 VLLM_MARLIN_INPUT_DTYPE=int8 \
vllm serve BlairQ/qwen3.8_27b_idiotSavant_sm_86 --quantization compressed-tensors --dtype float16 \
  --tensor-parallel-size 2 --kv-cache-dtype int8_per_token_head \
  --speculative-config '{"method":"dflash","model":"BlairQ/qwen3.8_27b_idiotSavant_sm_86_dflash2","num_speculative_tokens":8}'

K=8 is the measured optimum for 60–100k agent contexts. The drafter inherits the target's KV dtype. Tree speculation (8 nodes) is a Genesis feature: GENESIS_ENABLE_ARBOL=1. The full pipeline to re-tune it (capture → train → quantize → A/B, resumable, with dry-run) is entrenamiento/idiotsavant/dflash2.sh in the Genesis repo, and the compose this drafter is served and measured with is compose/docker-compose.qwen38-27b-idiotsavant-sm86.yml.

Without PN149 this drafter cannot work on the rotated target, and on a non-rotated target it has no reason to.

Credits

• Base drafter: DFlash 2 by Inco / z-lab (GitHub).
• Target base: Qwen/Qwen3.8-27B, abliterated by orcarouter.
• Image: the Idiot Savant perk from Fallout, © Bethesda Softworks, used as a fan reference.

Resumen en español

Borrador DFlash2 entrenado directo contra idiotSavant servido (int4 × int8, rotado, árbol y perfil de agente de código), a partir del DFlash2 original de Inco y sin modelos intermedios. Sigue en su base original: su fc lleva plegada la rotación, y el embedding y el lm_head compartidos los convierte PN149 en línea. Acepta 5,62 tokens por paso en el banco de agente (40 pedidos, 4 en paralelo, 4 arranques).

Actualizado en su lugar el 2026-10-03: una época más con la misma receta. Aceptación servida 5,57 → 5,62 (40 pedidos, 4 arranques) y 6,05 → 6,12 (20 pedidos, 3 arranques, gana en los 3); el largo greedy offline no se movió. Entrenar con el vocabulario recortado a 64k (PN159) resta 6,5% servido: el selector queda calibrado para los logits del vocabulario completo.