Magot-v2-Gemma2-8k-9B
This repo contains a merge of pre-trained language models created using mergekit.
There is still some damage to the model from merging, but that has been partially mitigated by retaining embed_tokens and lm_head weights from the base model. This approach has proven to be more stable than task arithmetic merger.
Tested at temp=1, minP=0.01. Coherence is moderately high, though not perfect. The low weight (0.01) infusion of the Magnum model provided variety to text generation. This model is being released because it is "good enough" and interesting. Inherent model safety is still strong due to Instruct base, but narratives are less bounded by positivity.
There appear to be some melodramatic motifs that the model inherently steers toward when allowed to go its own way. The notion of favored tropes baked into a model is likely understudied, and is another lens through which model explainability can be viewed.
Merge Details
Merge Method
This model was merged using the SLERP merge method.
Models Merged
The following models were included in the merge:
Configuration
The following YAML configuration was used to produce this model:
slices:
- sources:
- model: grimjim/Kitsunebi-v1-Gemma2-8k-9B
layer_range: [0, 42]
- model: anthracite-org/magnum-v3-9b-customgemma2
layer_range: [0, 42]
merge_method: slerp
base_model: grimjim/Kitsunebi-v1-Gemma2-8k-9B
parameters:
t:
- filter: embed_tokens
value: 0.0
- filter: lm_head
value: 0.0
- value: 0.01
dtype: bfloat16