Kanimus-26B-A4B-FFT-heretic (v4 QK-LoRA merge)
Experimental dark roleplay merge. Two edits on top of an abliterated Animus V14.1: a task-arithmetic delta injected into the attention Q and K projections from a Claude Opus distillation, and a stylistic LoRA baked in at low scale.
Base
Vortex5/Gemma-4-26B-A4B-Animus-V14.1-FFT-heretic, the abliterated variant of
Darkhn/Gemma-4-26B-A4B-Animus-V14.1-FFT. Refusal behaviour comes from that
abliteration, not from anything done here.
What changed
205 modules were touched. Everything outside them is the base, bit for bit.
| Tensors | Count | Change |
|---|---|---|
q_proj, k_proj | 60 | QK injection, then LoRA bake |
v_proj, o_proj | 55 | LoRA bake only |
shared dense MLP (gate/up/down_proj) | 90 | LoRA bake only |
q_norm, k_norm | — | unchanged |
| router, 128 routed experts | — | unchanged |
| embeddings (tied), layer norms | — | unchanged |
| vision tower | — | unchanged, untested |
v_proj is 25 rather than 30 because the five full-attention layers
(5, 11, 17, 23, 29) set attention_k_eq_v: true and have no v_proj — keys are
reused as values there.
Two things outside the weights: no lm_head swap was performed
(tie_word_embeddings stays true), and tokenizer_config.json was patched to
convert extra_special_tokens from a list to a dict. The list form is present
upstream and breaks convert_hf_to_gguf.py on current transformers.
Step 1 — QK injection
Task arithmetic on q_proj and k_proj only:
W_out = W_animus + α · (W_opus − W_parent)
- Target:
TeichAI/gemma-4-26B-A4B-it-Claude-Opus-Distill-v2 - Parent:
unsloth/gemma-4-26B-A4B-it
The unsloth checkpoint is the parent because the Opus distillation was trained
from it. Using google/gemma-4-26B-A4B-it instead would fold the repackaging
difference between the two into the delta.
α_sliding_q = 0.50
α_sliding_k = 0.50
α_global_q = 0.50
α_global_k = 0.25
α_global_k is halved because on the full-attention layers k_proj also serves
as the value matrix, so a delta there is effectively a KV injection.
q_norm and k_norm were not transferred. This is a hybrid, not a transfer of
the complete QK block.
Arithmetic in FP32, output in BF16, hand-written safetensors script — mergekit does not currently handle the Gemma 4 checkpoint layout.
After this step and before the bake, every tensor outside the 60 q/k tensors was
verified bitwise equal to the base with torch.equal.
Step 2 — LoRA bake
SubMaroon/Dark-Goetia-26B-A4B-LoRA-v4
merged into the Step 1 weights. r = 32, alpha = 64, targeting the four attention
projections and the shared dense MLP on all 30 layers (205 modules). Routed
experts, router, embeddings, lm_head and vision tower were not targeted.
for module in peft_model.modules():
if isinstance(module, LoraLayer):
module.set_scale("default", 0.20)
merged = peft_model.merge_and_unload(safe_merge=True)
set_scale multiplies the training scaling of alpha / r = 2, so the delta
actually added is 0.40 · BA, not 0.20 · BA.
The 0.40 figure was carried over from earlier work on a different base. LoRA v4 was not trained on this merge and the scale has not been recalibrated for the abliterated Animus base used here.
Behaviour vs base
No quantitative evaluation was run. Paired ΔNLL on held-out roleplay text, MoE routing diagnostics and long-context state tracking at 16k/32k/64k were all planned and all skipped.
What exists is three paired generations: same character card, same history, same user turn, base and merge run one after the other, compared by eye. Sampler settings and seeds were not recorded. Three samples decide nothing on their own.
| Prompt | Base | This merge |
|---|---|---|
| Fantasy interview scene, EN, Q5_K_M | 422 words, all four NPCs get dialogue, heavy supernatural staging | 342 words, three NPCs get dialogue, pressure stays social, the matriarch interrogates directly |
| Same scene, RU, Q5_K_M | 315 words, 42% of them dialogue | 346 words, 24% dialogue, longer interior monologue |
| 1980s office scene, EN | holds the card's period setting, dates the scene to 1986 across three swipes; both NPCs read themselves as layoff targets | dates the scene to 2025; one NPC reads herself as safe and her colleague as the one being cut, contradicting the user turn that set this up |
Instruction following got worse; prose did not. In the office scene the merge
broke the card's stated period setting and inverted a fact from the immediately
preceding user turn. The base did neither, in three swipes out of three. This is
the failure mode I would expect from this particular edit: in my own ablation work
on this family, q_proj and k_proj are where instruction-following behaviour
lives, and this merge injects a foreign delta into exactly those projections at
α = 0.5.
Sentence-level style stays close to the base. In English the merge tends to open on the base's exact wording and diverge later — 131 identical characters in one sample, 54 in another. In Russian it diverges from the first sentence. Same direction as the English/Russian threshold gap I have measured elsewhere, but without fixed seeds this is an impression, not a measurement.
One NPC dropped. In both fantasy samples the merge left one of the four present NPCs with no dialogue at all, while the base voiced all four. Possibly nothing at n = 2.
Known issues
Referent drift (English). In earlier Q4_K_M testing an established character name was sometimes replaced mid-scene by a similar-sounding invented one, and agent-of-action swapped within a scene. Not observed in base output on comparable prompts, and not reproduced in the three paired samples above. Cause not identified; the QK injection is the suspect.
Token artifacts (Russian). In earlier Q4_K_M testing, Latin fragments appeared inside Russian words along with malformed words, at a similar rate in the base — inherited, not introduced. Neither appeared in the Q5_K_M samples above, in either model. Whether that is the quantization level or the sample size is not established.
Colour formatting drops in Russian. Both models emitted per-character
<font color> tags in English and none in Russian on the same card. Inherited.
Sampling
Starting point, not a tuned recommendation:
- Temperature 0.7 – 0.8
- min_p 0.05 – 0.1
- Repetition penalty 1.05 – 1.10
Versions
- BF16, this repository, ~51.6 GB
- GGUF Q4_K_M
BF16 and GGUF have not been compared against each other, so quantization loss is not separated from merge effects anywhere above.
License
Gemma license. Use is subject to the terms of the original Google models and of the distillations involved.