📜 Dark Goetia 26B A4B v2
Atmospheric genre LoRA adapter primarily for English-language roleplay (also works in Russian). Adds a darker, more literary tone to the narrative. Designed exclusively to adjust style and response structure in "Dark Fantasy" RP. The adapter does not contain any plots or characters from the training data.
Adapter Versions
Two versions of the adapter are provided:
goetia-book-lora-v2-main— recommended — 2 epochs → stronger influence → use the recommended scales as listed abovegoetia-book-lora-v2-chk177— 1 epoch → the influence is weaker and less pronounceduse → 1.3× the scale from the table
Example: if you run main at 0.55, run chk177 at 0.71.
Recommended Scale Settings (main ver.)
| Scale | Effect (Test setup: SillyTavern + Preset + 2000+ token character card) |
|---|---|
| 0.1–0.2 | Effect barely noticeable |
| 0.3–0.5 | Recommended (stable, from the least to the greatest influence) |
| 0.7+ | Very strong influence (can start overpowering the base model) |
Note: Performs reliably in all RP scenarios at scales above 0.5. The darker literary tone becomes most pronounced in Dark Fantasy settings. Best used in full roleplay sessions (SillyTavern with character card + system prompt). Isolated short prompts may show a weaker effect.
Training
- Base: Naphula/Goetia-26B-A4B-v1.3-Absolute-Heretic-ARA (Gemma 4 MoE, abliterated)
- Method: QLoRA, attention-only (
q/k/v/o_proj, 30 layers of the text tower), r=32, alpha=64, 2 epochs, completion-only loss - Data: Novel broken down into scenes → scenes rewritten by a neural network into a synthetic RP dataset (actions and internal thoughts preserved, original author’s prose style removed). Dataset contains both English and Russian samples, with primary focus on English.
Limitations
- 18+. Base model is abliterated, training data includes adult and dark content.
- Primarily optimized for English. Works in Russian as well, but the effect is generally weaker.
- Inherits Gemma license terms.
⚠️ Current Limitations & Known Issues (v2)
Note: I am using this project as a hands-on learning experience, so trial and error are an inevitable part of the process!
This adapter was trained across all Transformer layers (0–31) using Attention-only targets (q_proj, k_proj, v_proj, o_proj), omitting FFN/MLP blocks. Evaluation and user testing revealed the following trade-offs:
- Lower-Layer Interference: Fine-tuning initial syntax layers (0–11) forces rigid structural constraints, occasionally stripping away the base model's native linguistic fluidity and stylistic richness.
- Lack of FFN/MLP Adaptation: Because the associative Feed-Forward blocks (
gate_proj,up_proj,down_proj) were frozen, the model relies solely on attention routing to convey style. This can lead to slightly dry, repetitive phrasing or overly generic vocabulary choices in complex roleplay/narrative scenarios. - Over-Formatting Bias: While XML tag adherence and structural tracking are near-perfect, the narrative prose itself can feel overly sanitized unless the LoRA weight is scaled down (0.4 - 0.6).
Planned Fix for v3: Restricting LoRA targeting exclusively to mid-to-upper layers (12–31) to preserve prose depth while maintaining structural stability.