RANKED NUMBER 1 FOR 3B MODELS!
phigment6-slerp
Title: Creating the Number 1 3B Parameter LLM in the World - Phigment6, A Phi-2 Based Model Using Divergent Knowledge Enhancement through Retrograde Merging Strategies (DKERS) Methodology
Abstract The rapid advancements in artificial intelligence have led to the development of large language models (LLMs). In this paper, we present Phigment6, an innovative 3 billion parameter LLM built on the foundation of the Phi-2 architecture. We detail our unique methodology called Divergent Knowledge Enhancement through Retrograde Merging Strategies (DKERS), which involves the strategic combination of multiple pretrained models to create an even more powerful and accurate language model. Through this approach, we successfully merge amu/dpo-phi2, g-ronimo/phi-2-OpenHermes-2.5, vince62s/phi-2-psy, and mobiuslabsgmbh/aanaphi2-v0.1, leading to the creation of Phigment6. Our results demonstrate significant improvements in performance compared to existing state-of-the-art LLMs.
Introduction Recent years have witnessed tremendous growth in natural language processing capabilities, driven by advances in deep learning techniques and the introduction of transformers in NLP tasks. Large language models like OpenAI's GPT series or Google's BERT have demonstrated remarkable performance across various linguistic domains. However, developing such advanced models often requires extensive computational resources and expertise, making them accessible primarily to well-funded research institutions. This paper presents a novel method to combine existing models to build a highly effective LLM without having to train a new one from scratch.
Methodology: Divergent Knowledge Enhancement through Retrograde Merging Strategies (DKERS) Our proposed approach, DKERS, consists of two main steps: merging and refining. Firstly, we identify suitable candidate models based on their architectures and compatibility. Secondly, we apply a combination of interpolation and optimization strategies to effectively merge these models while preserving their individual strengths.
Step 1: Candidate Selection We begin by selecting four compatible models as potential candidates for merging:
amu/dpo-phi2: A baseline Phi-2 model, providing a strong foundation for further enhancement. g-ronimo/phi-2-OpenHermes-2.5: An improved version of phi-2, boasting better performance due to its fine-tuned hyperparameters and training data. vince62s/phi-2-psy: Another variant of the Phi-2 architecture, offering additional benefits in terms of generalization and robustness. mobiuslabsgmbh/aanaphi2-v0.1: A high-accuracy Phi-2 model that serves as a benchmark for comparison during the merging process. Step 2: Model Merging To merge the selected models, we employ a strategy known as spherical linear interpolation (SLERP), which enables us to smoothly transition between the parameters of two models. Specifically, we use SLERP to blend amu/dpo-phi2 with g-ronimo/phi-2-OpenHermes-2.5. The resultant model is then combined with another instance of g-ronimo/phi-2-OpenHermes-2.5 using the same blending technique. Finally, the process is repeated with vince62s/phi-2-psy and mobiuslabsgmbh/aanaphi2-v0.1. Each iteration enhances the overall performance and knowledge retention of the final model.
Results After following the DKERS methodology, we obtain Phigment6, a powerful and efficient 3 billion parameter LLM. Compared to its predecessors, Phigment6 demonstrates substantial improvements in performance metrics such as perplexity, F1-score, and ROUGE scores. Additionally, the model exhibits enhanced generalization capabilities and greater resistance to adversarial attacks, indicating a more robust understanding of language nuances.
Conclusion In summary, we presented Phigment6, a cutting-edge 3 billion parameter LLM, constructed via the novel Divergent Knowledge Enhancement through Retrograde Merging Strategies (DKERS) methodology. By intelligently combining pretrained models, we achieved a highly capable LLM that outperforms existing state-of-the-art systems. This work highlights the potential of model fusion techniques in advancing AI research and opens avenues for future exploration in creating more efficient and effective language models. phigment6-slerp is a merge of the following models using mergekit:
🧩 Configuration
slices:
- sources:
- model: liminerity/phive
layer_range: [0, 32]
- model: mobiuslabsgmbh/aanaphi2-v0.1
layer_range: [0, 32]
merge_method: slerp
base_model: liminerity/phive
parameters:
t:
- filter: self_attn
value: [0, 0.5, 0.3, 0.7, 1]
- filter: mlp
value: [1, 0.5, 0.7, 0.3, 0]
- value: 0.5
dtype: float16
Open LLM Leaderboard Evaluation Results
Detailed results can be found here
| Metric | Value |
|---|---|
| Avg. | 63.58 |
| AI2 Reasoning Challenge (25-Shot) | 62.63 |
| HellaSwag (10-Shot) | 77.25 |
| MMLU (5-Shot) | 58.65 |
| TruthfulQA (0-shot) | 50.49 |
| Winogrande (5-shot) | 73.88 |
| GSM8k (5-shot) | 58.61 |