eq90parsedanube
This is a merge of pre-trained language models created using mergekit.
First one that's shown promising capability improvement over the base model h2o-danube2-1.8b-base.
Training methodology ... is a bit of a mess, trying out different things. I'm adding the datasets used at any point, but I don't think replicating the recipe is doable or sensible.
Original upscale at Lambent/danube2-upscale-1, duplicating layers 16-21. Various training methods attempted to repair. Linear merge is of the 4 that were at least 90% parseable by the EQ-Bench benchmark.
| Model | AGIEval | GPT4All | TruthfulQA | Bigbench | Average |
|---|---|---|---|---|---|
| danube2-upscale-1.7 | 27.97 | 62.16 | 42.2 | 32.2 | 41.13 |
| Model | EQ-Bench | Average |
|---|---|---|
| danube2-upscale-1.7 | 15.52 | 15.52 |
EQ-Bench
| Task | Version | Metric | Value | Stderr | |
|---|---|---|---|---|---|
| eq_bench | 2.1 | eqbench,none | 15.52 | ||
| eqbench_stderr,none | 2.77 | ||||
| percent_parseable,none | 100 | ||||
| percent_parseable_stderr,none | 0 | ||||
| alias | eq_bench |
Average: 15.52%
Average score: 15.52%
Merge Details
Merge Method
This model was merged using the linear merge method.
Models Merged
The following models were included in the merge:
- Lambent/danube2-upscale-1.53lisa
- Lambent/danube2-upscale-1.51galore
- Lambent/danube2-upscale-1.531qlora
- Lambent/danube2-upscale-1.51qlora
Configuration
The following YAML configuration was used to produce this model:
models:
- model: Lambent/danube2-upscale-1.531qlora
parameters:
weight: 1.0
- model: Lambent/danube2-upscale-1.53lisa
parameters:
weight: 1.0
- model: Lambent/danube2-upscale-1.51galore
parameters:
weight: 1.0
- model: Lambent/danube2-upscale-1.51qlora
parameters:
weight: 1.0
merge_method: linear
dtype: float16