Mistral-7B-OpenOrca-lora-merged
This is a test.
This is a regenerated model that combines the base model Mistral-7B-v0.1 with the LoRA model Mistral-7B-OpenOrca-lora.
This LoRA model is extracted from the efficient parameter fine-tuned model (Mistral-7B-OpenOra), and now it needs to be verified whether this LoRA model can achieve comparable performance with the original model.
The final goal is to create a toolkit that can simultaneously load multiple LoRA modules, and automatically switch to the appropriate combination of LoRA modules based on user queries to generate the best answer.
The source code is here
Mistral-7B-OpenOrca
-
Extract lora model Mistral-7B-OpenOrca-lora from Mistral-7B-OpenOrca;
-
Merge the base model Mistral-7B-v0.1 with lora model to Mistral-7B-OpenOrca-lora-merged
-
LLM Evaluation ...
Local Test
| ARC_acc_norm (25-shot) | HellaSwag_acc_norm (10-shot) | MMLU_acc (5-shot) | TruthfulQA_mc2 (0-shot) | GSM8K_acc (8-shot) | Open LLM Score | |
|---|---|---|---|---|---|---|
| Mistral-7B-OpenOrca | 71 | 83 | 61.42 | 45 | 40 | 65.11 |
| r=256 | 68 | 84 | 64.28 | 46.953 | 41 | 65.81 |
| r=64 | 67 | 84 | 64.26 | 47.32 | 41 | 65.65 |
| r=16 | 65 | 83 | 62.84 | 46.95 | 38 | 64.45 |
Open LLM Leaderboard
| ARC_acc_norm (25-shot) | HellaSwag_acc_norm (10-shot) | MMLU_acc (5-shot) | TruthfulQA_mc2 (0-shot) | Open LLM Score | |
|---|---|---|---|---|---|
| Mistral-7B-SlimOrca | 62.54 | 83.86 | 62.77 | 54.23 | 65.85 |
| Mistral-7B-OpenOrca | 64.08 | 83.99 | 62.24 | 53.05 | 65.84 |
lm-evaluation-harness
| Metric | Mistral-7B-OpenOrca | Mistral-7B-OpenOrca-lora | Mistral-7B-OpenOrca-lora-merged |
|---|---|---|---|
| ARC | 64.08 | ||
| HellaSwag | 83.99 | ||
| MMLU | 62.24 | ||
| TruthfulQA | 53.05 | ||
| Average | 65.84 |
HumanEval
| Metric | Mistral-7B-OpenOrca | Mistral-7B-OpenOrca-lora | Mistral-7B-OpenOrca-lora-merged |
|---|---|---|---|
| humaneval-python | 35.976 |
Training procedure
The following bitsandbytes quantization config was used during training:
- quant_method: bitsandbytes
- load_in_8bit: False
- load_in_4bit: True
- llm_int8_threshold: 6.0
- llm_int8_skip_modules: None
- llm_int8_enable_fp32_cpu_offload: False
- llm_int8_has_fp16_weight: False
- bnb_4bit_quant_type: nf4
- bnb_4bit_use_double_quant: True
- bnb_4bit_compute_dtype: bfloat16
Framework versions
- PEFT 0.5.0