🌅 MN Aura 12B v1

This is a merge of pre-trained language models created using mergekit-exp.
Merge Details
Merge Method
This model was merged using the Adaptive Unified Riemannian Annealing (AURA) merge method using Retreatcost/Mistral-Nemo-Base-2407-ChatML as a base.
Models Merged
The following models were included in the merge:
- DreadPoor/Famino-12B-Model_Stock
- EldritchLabs/MN-Starlight-Sylph-12B
- OccultAI/MN-Nazgul-12B-v1
- Retreatcost/Mistral-Nemo-Base-2407-ChatML
- shrugging-shoulders/Amberlight-Lux-12B
- WokeAI/Tankie-DPE-12B-SFT-v2
Configuration
The following YAML configuration was used to produce this model:
architecture: MistralForCausalLM
base_model: B:\12B\Retreatcost--Mistral-Nemo-Base-2407-ChatML
models:
- model: B:\12B\DreadPoor--Famino-12B-Model_Stock
parameters:
weight: 0.4
density: 0.90
epsilon: 0.09
pinocchio: false
- model: B:\12B\EldritchLabs--MN-Starlight-Sylph-12B
parameters:
weight: 0.4
density: 0.90
epsilon: 0.09
pinocchio: false
- model: B:\12B\shrugging-shoulders--Amberlight-Lux-12B
parameters:
weight: 0.4
density: 0.90
epsilon: 0.09
pinocchio: false
- model: B:\12B\WokeAI--Tankie-DPE-12B-SFT-v2
parameters:
weight: 0.4
density: 0.90
epsilon: 0.09
pinocchio: false
- model: B:\12B\OccultAI--MN-Nazgul-12B-v1
parameters:
weight: 0.4
density: 0.90
epsilon: 0.09
pinocchio: true
merge_method: aura # v6
parameters:
tol: 1.0e-11
max_iter: 25
eta: 1.25
theta: 0.93
optimization_mode: "hybrid"
phi: 2.0
alpha: 0.25
kappa: 0.75
jl_projs: 8
halley_strength: 1.61803
halley_period: 8
scalar_gravity: -0.025
mu: 0.5
aura_shift: true
omega: 0.08
momentum: 0.25
patience: 64
sharp: 0.5
voice: 0.25
density: 0.9
epsilon: 0.09
rescale: true
int8_mask: false
normalize_weights: false
normalize_router: true
router_strategy: "della"
blend_experts: true
auto_pad: true
skip_on_mismatch: true
verbose_logging: true
resume_path: "B:/mergekit-new/Aura_Cache"
enable_qliphoth: true
pinocchio_lambda: 0.75
inverse_pinocchio: false
dynamic_pareto_pinocchio: true
qliphoth_gamma: 0.5
qliphoth_lambda_enslaved: 0.5
qliphoth_svd_rank: 0
qliphoth_preserve_ratio: 1.0
save_layer_heatmaps: false
audit_image_dir: "B:/mergekit-new/Aura_Heatmaps"
gap_jumper: 1.0
enable_qhe: true
qhe_levels: 16
qhe_intensity: 0.3
qhe_crossover: 0.15
coherence_override: 1.0
conflict_scale_override: 1.0
invert_coherence_to_chaos: true
dtype: float32
out_dtype: bfloat16
tokenizer:
source: union
chat_template: auto