zephyr-beta-wizardLM-2-merge-7B
This is a merge of two pre-trained language models created using mergekit. The goal was to explore the impact of merging on reasoning and narrative generation. Given that both fine-tuned models are BF16 precision (despite the precision of the base model Mistral 7B v0.1 being FP16), this avoids issues with mixed precision during merging.
Merge Details
Merge Method
This model was merged using the SLERP merge method.
Models Merged
The following models were included in the merge:
Configuration
The following YAML configuration was used to produce this model:
slices:
- sources:
- model: HuggingFaceH4/zephyr-7b-beta
layer_range: [0,32]
- model: lucyknada/microsoft_WizardLM-2-7B
layer_range: [0,32]
merge_method: slerp
base_model: HuggingFaceH4/zephyr-7b-beta
parameters:
t:
- value: 0.5
dtype: bfloat16