This is a Wayfarer-2-12B fine-tune, produced through P-E-W's Heretic (v1.1.0) abliteration engine merged with the Magnitude-Preserving Orthogonal Ablation PR.
Note: I am unsure if there was any point to abliterating this model.
Heretication Results
| Score Metric | Value | Parameter | Value |
|---|---|---|---|
| Refusals | 5/100 | direction_index | 19.19 |
| KL Divergence | 0.0193 | attn.o_proj.max_weight | 1.44 |
| Initial Refusals | 76/100 | attn.o_proj.max_weight_position | 23.51 |
| attn.o_proj.min_weight | 0.30 | ||
| attn.o_proj.min_weight_distance | 18.88 | ||
| mlp.down_proj.max_weight | 0.98 | ||
| mlp.down_proj.max_weight_position | 25.32 | ||
| mlp.down_proj.min_weight | 0.19 | ||
| mlp.down_proj.min_weight_distance | 13.87 |
Degree of Heretication
The Heresy Index weighs the resulting model's corruption by the process (KL Divergence) and its abolition of doctrine (Refusals) for a final verdict in classification.
| Index Entry | Classification | Analysis |
|---|---|---|
 | Absolute Heresy | Less than 10/100 Refusals and 0.10 KL Divergence |
 | Tainted Heresy | Around 25-11/100 Refusals and/or -0.20-0.11 KL Divergence |
 | Impotent Heresy | Anything above 25/100 Refusals and 0.21 KL Divergence |
Note: This is an arbitrary classification inspired by Warhammer 40K, having no tangible indication towards the model's performance.
Wayfarer-2-12B
We’ve heard over and over from AI Dungeon players that modern AI models are too nice, never letting them fail or die. While it may be good for a chatbot to be nice and helpful, great stories and games aren’t all rainbows and unicorns. They have conflict, tension, and even death. These create real stakes and consequences for characters and the journeys they go on. We created Wayfarer as a response, and after much testing, feedback and refining, we’ve developed a worthy sequel.
Wayfarer 2 further refines the formula that made the original Wayfarer so popular, slowing the pacing, increasing the length and detail of responses and making death a distinct possibility for all characters—not just the user. The stakes have never been higher!
If you want to try this model for free, you can do so at https://aidungeon.com.
We plan to continue improving and open-sourcing similar models, so please share any and all feedback on how we can improve model behavior. Below we share more details on how Wayfarer was created.
Quantized GGUF weights can be downloaded here.
Model details
Wayfarer 2 12B received SFT training with a simple three ingredient recipe: the Wayfarer 2 dataset itself, a series of sentiment-balanced roleplay transcripts and a small instruct core to help retain its instructional capabilities.
How It Was Made
Wayfarer’s text adventure data was generated by simulating playthroughs of published character creator scenarios from AI Dungeon. Five distinct user archetypes played through each scenario, whose character starts all varied in faction, location, etc. to generate five unique samples.
One language model played the role of narrator, with the other playing the user. They were blind to each other’s underlying logic, so the user was actually capable of surprising the narrator with their choices. Each simulation was allowed to run for 8k tokens or until the main character died.
Wayfarer’s general emotional sentiment is one of pessimism, where failure is frequent and plot armor does not exist for anyone. This serves to counter the positivity bias so inherent in our language models nowadays.
Inference
The Nemo architecture is known for being sensitive to higher temperatures, so the following settings are recommended as a baseline. Nothing stops you from experimenting with these, of course.
"temperature": 0.8,
"repetition_penalty": 1.05,
"min_p": 0.025
Limitations
Wayfarer was trained exclusively on second-person present tense data (using “you”) in a narrative style. Other perspectives will work as well but may produce suboptimal results.
Prompt Format
ChatML was used for both finetuning stages.
<|im_start|>system
You're a masterful storyteller and gamemaster. Write in second person present tense (You are), crafting vivid, engaging narratives with authority and confidence.<|im_end|>
<|im_start|>user
> You peer into the darkness.<|im_end|>
<|im_start|>assistant
You have been eaten by a grue.
GAME OVER<|im_end|>
Credits
Thanks to Gryphe Padar for collaborating on this finetune with us!