NEBULA-23.8B-v1.0
Technical notes
- 108 layers,DUS procedure, mistral(32)->SOLAR(48)->GALAXY(72)->NEBULA(108)
- 23.8B parameters
- model created as an extension of depth upscaling procedure used for SOLAR by upstage
Results
- model can and will produce NSFW content
- GSM8k evaluation seems to be often broken, HellaSwag, Winograde and TQA show that its a smart model
- RP and ERP work surprisingly good and I didn't encounter any GPTisms yet
- comparable memory footprint to 20B and 23B models based on llama
- follows character card very well
- NSFW output feels fresh comparing to existing models
Finetuning for RP
- SFT using MinervaAI/Aesir-Preview dataset, 10 epochs
- DPO using athirdpath/DPO_Pairs-Roleplay-Alpaca-NSFW dataset, 1 epoch
- SFT using 1xAda6000, 10h
- DPO using 1x3090, 30h
- jupyter notebooks or mergekit configs for anyone wanting to reproduce/reuse scripts - just drop me a message
Prompt template
- Alpaca
- chat template is embedded in tokenizer config, should load automatically
Context size
- 4096
All comments are greatly appreciated, download, test and if you appreciate my work, consider buying me my fuel:
Open LLM Leaderboard Evaluation Results
Detailed results can be found here
| Metric | Value |
|---|---|
| Avg. | 59.94 |
| AI2 Reasoning Challenge (25-Shot) | 66.72 |
| HellaSwag (10-Shot) | 86.98 |
| MMLU (5-Shot) | 65.40 |
| TruthfulQA (0-shot) | 57.60 |
| Winogrande (5-shot) | 82.95 |
| GSM8k (5-shot) | 0.00 |