Dark-Goetia-26B-A4B-LoRA-v3-B
Research artifact, not a style adapter. This is the control arm of an experiment. It was trained to answer a question, not to improve roleplay output, and it doesn't. Published so the experiment is reproducible. For actual use, take v2 or v3-A.
Trained on the QK half of attention only (q_proj + k_proj), with v_proj and
o_proj left untouched — the exact complement of
v3-A. Together the two
adapters partition v2's full target set with no overlap and no remainder.
What it's for
The question was: within attention, which half carries writing style and which half is responsible for breaking long-context instruction-following?
Both adapters were trained on identical data with identical hyperparameters and the same seed — 354 steps each, only the target projections differ. Results:
| A (OV) | B (this one) | |
|---|---|---|
| trainable params | 11,182,080 | 11,796,480 |
| val loss | 1.97 | 2.21 |
| val perplexity | 7.17 | 9.08 |
| token accuracy | 0.550 | 0.514 |
B has more trainable parameters and still fits the corpus worse, so this isn't a capacity difference — QK is simply a poor channel for style transfer. It also had to move roughly twice as far in weight space as OV did to achieve less.
The flip side: QK tolerates about 3× more displacement before structured output starts falling apart. So the two halves of attention are not interchangeable in either direction.
Behaviour, if you want to try it anyway
Style effect is weak and qualitatively different from A. Where A performs a character's register (dialect, speech patterns, decision-making), B tends to describe it — the narrator explains the voice rather than the character speaking in it. At higher scales it also shifts register sideways rather than deepening it, and can change punctuation conventions in dialogue.
| Scale | Effect |
|---|---|
| 0.4–0.6 | Minimal, mostly narratorial |
| 0.75 | Strongest tested setting that still holds structured output in both languages |
| 0.85 | Schema blocks drop in Russian; still held in English at this level |
alpha/r = 2 is in the GGUF metadata and llama.cpp applies it on top of your scale.
...-chk177.gguf is a mid-cosine checkpoint (LR still ~53% of peak), provided for
diagnostics only — it is not a clean 1-epoch equivalent.
Training
- Base: Naphula/Goetia-26B-A4B-v1.3-Absolute-Heretic-ARA (Gemma 4 MoE, abliterated)
- Method: QLoRA 4-bit,
q_proj+k_projonly, all 30 text layers (60 modules), r=32, alpha=64, lr 2e-5 cosine, 2 epochs, completion-only loss, MAX_LEN 2048 - Data: identical to v2 and v3-A — bilingual synthetic RP dataset, English-primary
- ~43 min on one A100 80GB
Controls held: initial loss matched A's to the last digit (7.431981086730957), token
counts matched step-for-step, and the two parameter counts sum to exactly v2's budget
(22,978,560).
One training-time oddity worth recording: B's entropy is non-monotonic — it drops to 1.66 by step 50 while loss is still high, then rises to 2.22 before settling. A's decreases monotonically throughout. Consistent with QK perturbation defocusing attention early and having to recover.
Limitations
- 18+. Base model is abliterated; training data contains adult and dark content.
- Not recommended for production use. Published for reproducibility and for anyone investigating OV/QK attribution in Gemma 4 MoE.
- All behavioural findings are single generations per setting at one context depth.
- Inherits Gemma license terms.