A Fishy Model
qwen-carpmuscle-r-v0.3 was made using Rombodawg's Shared Continuous Finetuning method.
qwen-carpmuscle-v0.3 was made using Unsloth's continuous pretraining on the ChatML format with 24k context on (unsloth/Qwen2.5-14B-bnb-4bit)[https://huggingface.co/unsloth/Qwen2.5-14B-bnb-4bit]. Then qwen-carpmuscle-v0.3 was merged with Qwen/Qwen2.5-14B-Instruct and Qwen/Qwen2.5-14B using TIES to create this model.
This is a merge of pre-trained language models created using mergekit.
Configuration
The following YAML configuration was used to produce this model:
models:
- model: qwen-carpmuscle-v0.3
parameters:
weight: 1
density: 1
- model: Qwen/Qwen2.5-14B-Instruct
parameters:
weight: 1
density: 1
merge_method: ties
base_model: Qwen/Qwen2.5-14B
parameters:
weight: 1
density: 1
normalize: true
int8_mask: true
tokenizer_source: qwen-carpmuscle-v0.3
dtype: bfloat16
- Developed by: TheTsar1209
- License: apache-2.0
- Finetuned from model : unsloth/Qwen2.5-14B-bnb-4bit
This qwen2 model was trained 2x faster with Unsloth and Huggingface's TRL library.
Open LLM Leaderboard Evaluation Results
Detailed results can be found here
| Metric | Value |
|---|---|
| Avg. | 31.52 |
| IFEval (0-Shot) | 44.55 |
| BBH (3-Shot) | 46.38 |
| MATH Lvl 5 (4-Shot) | 27.19 |
| GPQA (0-shot) | 13.42 |
| MuSR (0-shot) | 12.00 |
| MMLU-PRO (5-shot) | 45.59 |