Teacher model
This model is a LoRA decision-distillation of
convaiinnovations/layaby Convai Innovations. laya supplies every training label — theshould_compact/preserve/level/modedecisions this model emits. No laya weights are redistributed here; the teacher is credited as the source of the supervision, which it is.If you want the same decisions without this model, go and use laya.
Ternary-Bonsai-2-27B-Heretic-Satellite
A Qwen3.8-27B Bonsai 2 (Heretic) derivative carrying a small LoRA that turns a long working context into one structured compaction decision.
Text only. No vision tower, no projector. If you feed it images it will not do anything useful with them.
- Base:
Qwen3.8-27B-Bonsai-2-27B-Heretic(dequantised from the published Q8_0 GGUF to bf16; dense merge, no adapter left in the graph) - Architecture:
Qwen3_5ForCausalLM, 64 layers, hidden 5120, vocab 248320, 262144 max positions - This build: bake b6, 513 steps, with Q2_K-aware QAT on the LoRA-targeted tensors
What "Satellite" means
The model reads a transcript and answers with one decision block:
| field | meaning |
|---|---|
should_compact | yes / no — compact the working context now |
preserve | what must survive verbatim: code, numbers, decisions, recency, nothing |
level | 0 light, 1 moderate, 2 aggressive |
mode | strategy: narrative, extractive, by_topic |
Measured behaviour — held-out sessions
20 held-out sessions the model never saw in training, scored against laya's own decision on the same session. b6:
| tier | block-first | coherent | distinct replies | agrees with laya | all 4 fields | duplicate blocks |
|---|---|---|---|---|---|---|
| F16 | 20/20 | 20/20 | 11 | 17/20 (85%) | 3/20 | 0 |
| Q4_K_M | 20/20 | 20/20 | 13 | 17/20 (85%) | 3/20 | 0 |
| Q2_K | 20/20 | 20/20 | 14 | 17/20 (85%) | 4/20 | 1 |
| Q8_0 | not gated |
Reading this honestly:
- Q2_K improved from v4's 14/20 to 17/20. That gain is what the Q2_K-aware QAT in bake 6 bought. It is directional only — 17 vs 14 is a small sample, and the paired McNemar test on those 20 items gives p=1.000. Treat it as "not worse, plausibly better", not as a proven win.
- Q2_K emits one duplicated decision block out of 20 sessions. The other tiers emit none. This is the visible cost of the 3-bit average, and it is why Q2_K is not the recommended default.
- Exact agreement on all four fields is 3–4/20. The model gets the headline
decision right most of the time; the supporting fields are much noisier. If you
are going to act on
preserveorlevelprogrammatically, verify them. - Q8_0 was not gated. It is a size tier, not a quality tier.
Measured general ability — this model did not get worse
To check that the decision training did not damage the underlying model, both the un-finetuned base and this build were scored on the same harness, same machine, same 250 documents per task, both in bf16.
lm-evaluation-harness 0.4.13, local-completions, vLLM 0.30.0, TP=4 bfloat16,
max_model_len 4096, 5-shot for the commonsense/QA set and 0-shot for boolq.
| task | metric | base (un-finetuned) | b6 (this) | Δ |
|---|---|---|---|---|
| arc_challenge | acc_norm | 70.80 | 70.00 | −0.80 |
| arc_easy | acc_norm | 89.60 | 89.20 | −0.40 |
| hellaswag | acc_norm | 75.20 | 72.80 | −2.40 |
| openbookqa | acc_norm | 52.40 | 52.80 | +0.40 |
| piqa | acc_norm | 81.60 | 81.20 | −0.40 |
| winogrande | acc | 80.40 | 78.40 | −2.00 |
| boolq | acc | 85.20 | 88.40 | +3.20 |
Mean difference: −0.34 points (2σ band [−3.11, +2.43]). Not distinguishable from the base on any of the seven tasks.
This is the number that matters most on this card, and it is a negative result stated as one: the decision LoRA did not cost measurable general ability. It also means there is no general-ability gap for further training to close — if you want more out of this model, the route is preference tuning or a wider LoRA, not capability repair.
Caveats, stated plainly: 250 documents per task gives roughly ±3 points of standard error per model. The runs are paired on identical documents so the error on the difference is tighter than that, but this was computed from aggregates, not per-item logs, so no stronger claim is made. A dedicated harness/quantisation comparison against third-party models was attempted and failed — see below.
Files
| file | size | notes |
|---|---|---|
Ternary-Bonsai-2-27B-Heretic-Satellite-Q4_K_M.gguf | 16,547,400,096 | recommended |
Ternary-Bonsai-2-27B-Heretic-Satellite-Q2_K.gguf | 10,711,665,056 | smallest; one dup block seen |
Ternary-Bonsai-2-27B-Heretic-Satellite-Q8_0.gguf | 28,595,763,616 | ungated |
Ternary-Bonsai-2-27B-Heretic-Satellite-F16.gguf | 53,808,282,016 | reference |
model-00001..00013-of-00013.safetensors | 53,791,996,928 total | bf16 |
Every uploaded file was verified byte-range identical to the local bake before this was called done. Size checks alone are not sufficient and were previously not sufficient here — see the build-history note below.
What the Q2_K tier actually is
It is a mixed-precision file, not a uniform Q2_K. Measured from the published GGUF's own tensor table:
| type | share of parameters |
|---|---|
| Q2_K | 62.51% |
| Q3_K | 23.08% |
| Q4_K | 9.67% |
| Q6_K | 4.73% |
| F32 (norms) | 0.01% |
Effective 3.183 bits/param across the whole file; 3.033 bits/param across the 176 matrix weights that actually do the compute (Q2_K 63.79%, Q3_K 25.52%, Q4_K 10.69%). So "Q2_K" here is a filename describing the dominant type, not the average. The Q4_K_M tier is 4.919 bits/param overall.
Runtime support by format
| runtime | status |
|---|---|
| GGUF | works via llama.cpp. Only the llama.cpp-prism build serves these on this box; the stock build does not support the architecture. |
| safetensors | works via transformers 5.17, trust_remote_code not required. |
| vLLM | works for the bf16 safetensors on sm_120. GGUF loading is not supported in vLLM 0.30 (Load format 'gguf' is not supported). |
| LM Studio / Ollama | via the GGUF tier. |
Quantisation caveat, read before using Q2_K
The Q2_K file averages 3.03 bits on its matrix weights, and the LoRA-targeted tensors are the ones pushed to 2 bits. One of 20 held-out sessions came back with a duplicated decision block. If you serve this in an agent loop where a repeated decision is expensive, use Q4_K_M.
Provenance
- Base lineage:
Qwen3.8-27B-Bonsai-2-27B-Heretic, a dense merge. No MoE expert routing and no adapter in the published graph. - Teacher:
convaiinnovations/layaby Convai Innovations, credited above. - LoRA merged into the base, then quantised per tier.
Build history worth knowing about
An earlier publish of this repo was verified by reported byte count only. v4 and b6 produce a Q4_K_M file of exactly the same size, so that check passed while the repo still served v4 bytes. It was found later by byte-range comparison. Everything published now is byte-range verified, and the Q8_0 tier was re-quantised from b6 specifically so all tiers come from one bake.
Third-party benchmark comparison: attempted, not completed
A same-harness comparison against
DavidAU/Qwen3.8-27B-TURBO-Fable-Cold-Fusion-735-882-Heretic-Uncensored-NEO-CODER-MAX-MTP-GGUF
was attempted and could not be completed on this machine:
- vLLM 0.30 refuses GGUF (
Load format 'gguf' is not supported), and the vendor publishes GGUF only. llama.cppon this box returns zerotoken_logprobsfrom/v1/completions, on both the stock and the Prism build, and every one of these tasks is multiple-choice scored by loglikelihood. So llama.cpp cannot score them either.
That vendor's card self-reports arc_challenge 73.5 and arc_easy 88.0 on an
unknown harness and unknown shot count. Those figures are unverified and are
not reproduced here. Our own measured arc_easy (89.2) is above their claimed
88.0; our arc_challenge (70.0) is below their claimed 73.5. No cross-model
claim should be drawn from that comparison until it is run on one harness.
Training corpus
Decision-distillation from laya over long-context transcripts, with the Q2_K fake-quantiser applied to the LoRA-targeted tensors during training so the 2-bit tier is optimised for what it will actually run as. See the bake scripts in the project for the exact schedule.
License and credit
Base model licence and any upstream terms apply — check the Bonsai 2 / Heretic and Qwen lineage before commercial use. This card describes measurement performed on one 4×RTX 5060 Ti box; other hardware will differ, particularly the quantised tiers.