merged_model_2

This is a merge of pre-trained language models created using mergekit.

Merge Details

Merge Method

This model was merged using the SLERP merge method.

Models Merged

The following models were included in the merge:

• /mnt/c/Users/jdesa/finalround_5
• /mnt/c/Users/jdesa/finalround_8

Configuration

The following YAML configuration was used to produce this model:

merge_method: slerp
base_model: "/mnt/c/Users/jdesa/finalround_8"  # Path to the first model
models:
  - model: "/mnt/c/Users/jdesa/finalround_5"   # Path to the second model

# Interpolation parameters for SLERP
parameters:
  t:
    - filter: self_attn                       # Interpolation values for self-attention layers
      value: [0, 0.5, 0.7, 0.9, 1]           # Gradients for SLERP interpolation
    - filter: mlp                             # Interpolation values for MLP layers
      value: [1, 0.7, 0.5, 0.3, 0]           # Gradients for SLERP interpolation
    - value: 0.5                              # Default interpolation value for all other layers

dtype: bfloat16                               # Precision for merging
tokenizer:
  source: base                                # Use tokenizer from the base model