Silver-Siren-ST-12B (Mistral-Nemo-12B-based StyleTune)

A while ago, I stumbled upon Gryphe's report on "Style-Tuning" (Gemma-4-31B-StyleTune) by chance. At least for me, this concept was entirely new, but it sounded incredibly promising—so I just had to test it out myself.

This model is the result of that experiment: A technical test in surgical style-tuning to verify if isolating style changes to a single tensor (lm_head) while freezing the core weights works effectively on the Mistral-NeMo-12B architecture.

Methodology & Concept

Normally, a fine-tune alters as many layers as possible to align both the reasoning and formatting of a model to a new dataset. This "StyleTune" approach does the exact opposite:

1. Freeze everything: All attention and MLP layers (Layers 0–39) remain completely untouched. The underlying logic, world knowledge, and instruction-following capabilities are preserved exactly as they were.
2. Target the Language Center: Only one single tensor—the lm_head (output projection)—is trained.

By retraining only the lm_head, the model doesn't become "smarter" or "dumber," but its vocabulary, sentence structure, and prose quality are completely recalibrated. It changes the voice of the model, not its brain.

The Target Base: Why this model?

To demonstrate the contrast and efficacy of this method, I deliberately chose Vortex5/Silver-Siren-12B as the target base.

Important Note: This choice is purely technical and meant with the utmost respect for the original author. Silver-Siren-12B is a highly popular, emotion-forward merge (incorporating models like Dark-Nexus, Elysian-Sunrise, LunaMaid, and others) that is highly optimized for sensational, dramatic, and immersive interactions. Because its native style profile is so distinct, it served as the perfect benchmark to test if a pure lm_head tune could cleanly overwrite a deeply baked-in stylistic bias without degrading the underlying merge quality.

By exposing this base to a highly curated, classical sci-fi literary dataset (inspired by Asimov, Huxley, and Lem), the model underwent a dramatic transformation in its prose delivery.

Training Details & Parameters

• Epochs: 3
• Learning Rate: 4e-4 (Linear Scheduler)
• Target Modules: lm_head only (all other linear layers frozen)

Recommended Sampler Settings

• Temperature: 0.7 - 0.9
• Min_P: 0.05
• Top_P: 0.95
• Repetition Penalty: 1.05

Thank you

• to Gryphe for posting this excellent finding. https://huggingface.co/Gryphe/Gemma-4-31B-StyleTune
• to Vortex5 for creating this model.