Qwen3.6-34B-80L-Fable-5-Heretic
Fable-5 distilled Qwen3.6 80-layer deep reasoning model.
Built on the Qwen3.6-27B (64-layer, original) base, extended to 80 layers (~34B parameters) and distilled from Fable-5 Agentic Traces. Fine-tuned via QLoRA (4-bit NF4, rank=64) on 4,665 Fable-5 Chain-of-Thought trajectories, then merged with MTP speculative decoding heads.
Model Card
| Property | Value |
|---|
| Base model | Qwen3.6-27B (64-layer, original) → Qwen3.6-34B-80L-BF16-v2 |
| Architecture | Qwen3_5ForConditionalGeneration |
| Parameters | ~34B (80 layers, hidden=5120) |
| Attention heads | 24 (4 KV Grouped) |
| Context length | 262,144 (256K) |
| Vocabulary size | 248,320 |
| Attention mode | Hybrid Linear/Full (full every 4 layers) |
| MTP speculative decoding | ✅ (15 mtp weights) |
| Precision | BF16 |
Distillation Data
| Item | Value |
|---|
| Data source | Glint-Research/fable-5-traces |
| Samples | 4,665 entries |
| Content | Fable-5 Agentic CoT trajectories (context + completion) |
| Training config | QLoRA rank=64, α=64, 2 epochs, lr=2e⁻⁴, max_len=4096 |
Training Details
- Quantization: bitsandbytes 4-bit NF4 + double quantization (QQ)
- LoRA targets:
q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj, and MoE projection layers
- Optimizer: paged_adamw_8bit, cosine schedule, 5% warmup
- Gradient checkpointing: enabled (use_reentrant=False)
- Effective batch size: 4 × 4 = 16
Key Features
- Fable-5 deep reasoning — Inherits Fable-5's CoT chain-of-thought reasoning and tool-use patterns.
- 80-layer efficient architecture — Stronger expressiveness at 34B parameters with 80 layers.
- Hybrid attention — Alternating Linear/Full design for higher inference efficiency on long sequences.
- MTP speculative decoding — Native MTP weights; vLLM deployment achieves ~78% acceptance rate, 2× throughput.
- BF16 full precision — MergeLoRA merged without quantization loss, ready for deployment and re-fine-tuning.
Quick Start
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "hiebo/Qwen3.6-34B-80L-Fable-5-Heretic"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
model_id,
torch_dtype="auto",
device_map="auto",
)
prompt = "What is quantum computing?"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
vLLM Launch Configuration
vllm serve hiebo/Qwen3.6-34B-80L-Fable-5-Heretic \
--dtype bfloat16 \
--max-model-len 262144 \
--enforce-eager \
--gpu-memory-utilization 0.95 \
--swap-space 16 \
--disable-log-stats
Supported Parameters
| Parameter | Default | Description |
|---|
--model | (required) | Model ID or local path |
--dtype | auto | Data type: bfloat16, float16, float32, auto |
--max-model-len | 4096 | Maximum context length (supports 256K) |
--max-num-seqs | 256 | Maximum number of concurrent sequences |
--gpu-memory-utilization | 0.90 | Fraction of GPU memory to use (0–1) |
--swap-space | 4 | CPU swap space in GB |
--tensor-parallel-size | 1 | Tensor parallelism degree (multi-GPU) |
--pipeline-parallel-size | 1 | Pipeline parallelism degree |
--enforce-eager | False | Force PyTorch eager mode (recommended for BF16 models) |
--disable-log-stats | False | Disable statistics logging |
--served-model-name | None | Override served model name |
--port | 8000 | API server port |
--host | 0.0.0.0 | API server host |
--tokenizer-mode | auto | Tokenizer mode: auto, slow, mistral |
--quantization | None | Quantization: fp8, awq, gptq, squeezellm |
--load-format | auto | Load format: auto, sharded_state, safetensors, npcache |
--kv-cache-dtype | auto | KV cache data type: auto, float16, bfloat16 |
--enable-chunked-prefill | False | Enable chunked prefill for long sequences |
--max-num-batched-tokens | None | Max batched tokens per step |
--seed | 0 | Random seed for reproducibility |
--distributed-executor-backend | ray | Distributed backend: ray, mp, mp_rank |
--trust-tokenizers | False | Trust remote tokenizer files |
--revision | main | Model revision (branch/tag/commit) |