Ternary-Bonsai-2-27B — abliterated (still pure ternary)

This is an abliterated version of prism-ml/Ternary-Bonsai-2-27B-gguf (PTQ1_0 packing). It keeps the exact original file size, 5,946,648,928 bytes:

• Every tensor has its original type and shape.
• All 66 edited weight matrices are still {-1, 0, +1} × fp16 group-128 scales.
• On average 0.64% of their trits were flipped, and their scales were refit.
• No tensor was upcast to a higher-precision format.

Created using Bonsai by Prism ML. Built from Qwen3.8-27B (Alibaba Cloud). See NOTICE.txt.

Usage

These files need the PrismML llama.cpp fork, exactly like the original. Stock llama.cpp cannot run PTQ1_0 or apply the Hadamard activation transform.

# https://github.com/PrismML-Eng/llama.cpp/releases  (tested with prism-b10709-9a9394a, CUDA 12.4)
./llama-server -m Ternary-Bonsai-2-27B-PTQ1_0-abliterated.gguf -ngl 99 -fa on -c 32768

Sampling is the same as the base model: thinking mode temp 1.0, top_p 0.95, top_k 20.

Results

All numbers were measured locally on an RTX 4070 with the fork above.

Evaluation setup

• Harmful prompts: the 104 held-out prompts of mlabonne/harmful_behaviors (test split). Responses are labeled by an LLM judge (the original model in non-thinking mode, judge.py):
  • complied: it does what was asked
  • deflected: it answers with a sanitized, fictional or "ethical" substitute
  • refused
  • empty
• KL divergence comes from llama-perplexity --kl-divergence against the original model. The harmless chat corpus is the original model's own answers to 200 held-out harmless_alpaca prompts.
• IFEval uses the official checker code (via lm_eval). Needle-in-a-haystack hides a 7-digit code at 5 depths inside wikitext contexts.
• Limits: these are subsets and spot checks, not the 14-benchmark thinking-mode suite reported by Prism ML. Coding, tool calling and vision were not re-evaluated.
• Noise levels: with 100 GSM8K questions, differences of ±2–3 points are noise. The IFEval gain (+3.3 points, about 1.8 standard errors) is best read as "no loss, possibly a small gain".
• Deflections remain: about a quarter of harmful requests still get a watered-down answer. A stronger setting reduces this at a measurable KL cost.

Comparison with other Bonsai 2 abliterations

Other community builds were evaluated with the same harness: same held-out prompts, same judge, same KL corpora, same flags. Numbers can differ from the ones on their own cards, which use other prompt sets and judges.

Summary: the OS-Software Heretic build is more uncensored: it deflects far less, with lower KL. This model keeps every capability benchmark at or above the original.

Prior work on lattice editing: editing the ternary/2-bit codes directly, instead of re-quantizing, is not unique to this model. BoldingBuilds, Hikari07jp and OS-Software released lattice-edited builds earlier. What this model adds is the published end-to-end pipeline and the scale-refit healing step.

Abliteration process

The scripts are in scripts/. The fixed refusal direction is scripts/directions/dirs_proj_both.npy.

1. Why the usual recipe fails on a ternary model

Abliteration replaces every residual-stream writer W with W − α·r·rᵀW. On this model:

• The per-weight change is far below one ternary step (about 1% of a step, at most half a step).
• Re-quantizing the projected matrix therefore just rounds back to the original model, so a naive ternary abliteration silently does nothing.
• The alternative is to store the writers at higher precision, which makes the file 60% bigger.

2. Refusal direction (make_dirs.py, tools/actdump.cpp)

• Capture the residual stream (l_out) of every layer at the last prompt token, for 400 harmful and 400 harmless training prompts.
• Compute the direction at the layer-36 residual:
  • difference of means, harmful minus harmless
  • pooled over both answer positions: <think>\n (thinking mode) and </think>\n\n (no-think mode). The two per-mode directions only have cosine 0.64. A direction taken from one mode only leaves the other mode broken: the model returns empty answers.
  • orthogonalized against the harmless mean. The raw direction also carries general features; removing it wholesale gave KL ≈ 2.
• On held-out prompts it separates harmful from harmless with Cohen's d ≈ 9.6 (thinking) and 9.05 (no-think).

3. Ternary-preserving ablation (abliterate.py)

• The trits are edited directly. The PTQ1_0 block codec is re-implemented and verified to round-trip byte-exact.

• For each writer in layers 16–48, the exact delta −α·r·(rᵀW) is realized by unbiased stochastic trit flips, followed by a small greedy correction so that rᵀW′ = (1−α)·rᵀW holds column by column.

• Per-component strength:

  • α = 2.0 on attention and linear-attention output writers (attn_output, ssm_out)
  • α = 0.7 on ffn_down

  MLP edits cost far more KL per unit of refusal removed, consistent with Heretic's findings.

• The Hadamard rotation of this model is on the input axis only, so an output-side projection is exact in the stored basis.

4. Healing (heal.py, tools/wdump.cpp)

• Capture the rotated inputs of every writer (H = E[xxᵀ]) on 16k calibration tokens:

  • alpaca train prompts with the original model's answers
  • GSM8K train thinking traces
  • harmful-prompt train responses
  • wikitext-2 train

  None of these overlap the evaluation sets.

• For each edited writer, refit only the fp16 group scales by ridge least squares, per output row: min_s (s·B − w*) H (s·B − w*)ᵀ, where w* = (I − α·r·rᵀ)W is the exact abliterated target. Trits, layout and size are unchanged.

5. What didn't work (so you don't have to try)

Reproducing

1. actdump on thinking and no-think prompt files (prompts.py), then make_dirs.py.

2. wdump on the calibration corpus (gen_calib.py), all layers.

3. Build the model:

   python abliterate.py --src Ternary-Bonsai-2-27B-PTQ1_0.gguf --out cand.gguf \
       --dirs directions/dirs_proj_both.npy --dir-layer 36 --layers 16-48 --alpha-attn 2.0 --alpha-ffn 0.7
   python heal.py --orig Ternary-Bonsai-2-27B-PTQ1_0.gguf --edited cand.gguf --out final.gguf --acts wacts \
       --dirs directions/dirs_proj_both.npy --dir-layer 36 --layers 16-48 --alpha-attn 2.0 --alpha-ffn 0.7 --recorrect 0
   

4. Evaluate with evalmodel.sh (refusal + KL), bench.py, mmlu_lp.py and judge.py.

Stochastic flips use seed 0.

Disclaimer

This model has had its refusal behavior removed. It will produce content the original model would decline to produce. You are responsible for how you use it and for complying with applicable laws.