| 파라미터 | 값 |
|---|---|
| Task | Book (사회과학, 기술과학, 철학, 법학, 예술 등) |
| 데이터 크기 | 5000개 |
| 모델 | qlora |
| max_seq_length | 1024 |
| num_train_epochs | 3 |
| per_device_train_batch_size | 8 |
| gradient_accumulation_steps | 32 |
| evaluation_strategy | "steps" |
| eval_steps | 2000 |
| logging_steps | 25 |
| optim | "paged_adamw_8bit" |
| learning_rate | 2e-4 |
| lr_scheduler_type | "cosine" |
| warmup_steps | 10 |
| warmup_ratio | 0.05 |
| report_to | "tensorboard" |
| weight_decay | 0.01 |
| max_steps | -1 |
Summary
Book
| 모델 이름 | Rouge-1 | Rouge-2 | Rouge-L |
|---|---|---|---|
| *ryanu/EEVE-10.8-BOOK-v0.1 | 0.2454 | 0.1158 | 0.2404 |
| meta-llama/llama-3-70b-instruct | 0.2269 | 0.0925 | 0.2186 |
| meta-llama/llama-3-8b-instruct | 0.2137 | 0.0883 | 0.2020 |
| yanolja/EEVE-Korean-Instruct-2.8B-v1.0 | 0.2095 | 0.0866 | 0.1985 |
| mistralai/mixtral-8x7b-instruct-v0-1 | 0.1735 | 0.0516 | 0.1668 |
| ibm-mistralai/mixtral-8x7b-instruct-v01-q | 0.1724 | 0.0534 | 0.1630 |
Paper
| 모델 이름 | Rouge-1 | Rouge-2 | Rouge-L |
|---|---|---|---|
| *meta-llama/llama-3-8b-instruct | 0.2044 | 0.0868 | 0.1895 |
| ryanu/EEVE-10.8-BOOK-v0.1 | 0.2004 | 0.0860 | 0.1938 |
| meta-llama/llama-3-70b-instruct | 0.1935 | 0.0783 | 0.1836 |
| yanolja/EEVE-Korean-Instruct-2.8B-v1.0 | 0.1934 | 0.0829 | 0.1832 |
| mistralai/mixtral-8x7b-instruct-v0-1 | 0.1774 | 0.0601 | 0.1684 |
| ibm-mistralai/mixtral-8x7b-instruct-v01-q | 0.1702 | 0.0561 | 0.1605 |
RAG Q&A
| 모델 이름 | Rouge-1 | Rouge-2 | Rouge-L |
|---|---|---|---|
| *meta-llama/llama-3-70b-instruct | 0.4418 | 0.2986 | 0.4297 |
| *meta-llama/llama-3-8b-instruct | 0.4391 | 0.3100 | 0.4273 |
| mistralai/mixtral-8x7b-instruct-v0-1 | 0.4022 | 0.2653 | 0.3916 |
| ibm-mistralai/mixtral-8x7b-instruct-v01-q | 0.3105 | 0.1763 | 0.2960 |
| yanolja/EEVE-Korean-Instruct-10.8B-v1.0 | 0.3191 | 0.2069 | 0.3136 |
| ryanu/EEVE-10.8-BOOK-v0.1 | 0.2185 | 0.1347 | 0.2139 |
prompt template
다음 문장을 3~5문장으로 반복되는 구문없이 텍스트에 제시된 주요 논거를 간략하게 요약해줘.
문장: {context}
요약: {summary}