Llama-3-MAAL-8B-Instruct-v0.1
we release MAAL, Multilingual Adaptive Augmentation Language-model which comprises a groundbreaking fusion of multilingual capabilities and adaptive augmentation techniques.
- Developed by: maum.ai Brain NLP. Jaeyoon Jung, Jinjoo Lee, Yongjae Lee, Dongjun Lee, Woosung Joo
- Language(s) (NLP): Korean, English (currently, bilingual)
Model Description
Version 0.1 uses cross-lingual training to transfer instruction-following capabilities from English to Korean.
- We Trained this model on an 8 H100-80G for 1 day with cross-lingual training dataset
- we recommend using the fixed system prompt for the model unless you fine-tune it
너는 마음에이아이의 챗봇 MAAL이다. 고객의 질문에 친절하게 답하여라.
sample inference code (GPU)
import transformers
import torch
model_id = "maum-ai/Llama-3-MAAL-8B-Instruct-v0.1"
model = transformers.AutoModelForCausalLM.from_pretrained(model_id).to("cuda")
tokenizer = transformers.AutoTokenizer.from_pretrained(model_id)
streamer = transformers.TextStreamer(tokenizer, skip_prompt=True, skip_special_tokens=True)
# we recommend using the fixed prompt for the model unless you fine-tune it
prompt = "너는 마음에이아이의 챗봇 MAAL이다. 고객의 질문에 친절하게 답하여라."
instruction = "사과 한 박스에는 사과가 30개 들어있는데, 처음에는 사과 3박스가 있었고, 내가 사과 5개를 먹었어. 남은 사과는 총 몇개야?"
messages = [
{"role": "system", "content": f"{prompt}"},
{"role": "user", "content": f"{instruction}"}
]
inputs = tokenizer.apply_chat_template(
messages,
tokenize=True,
return_tensors='pt').to("cuda")
outputs = model.generate(inputs, streamer=streamer, max_new_tokens=1024, pad_token_id=tokenizer.eos_token_id)
Evaluation Results
As the main goal of version 0.1 is to transfer instruction-following capabilities from English to Korean without utilizing continuous pre-training, etc., we select LogicKor as our evaluation method to assess the Korean instruction skills.
We compare our model with a similar parameter model (less than 13B) that has been fine-tuned on the Korean dataset. * denotes our self-report result.
| Model | single-turn(↑) | multi-turn(↑) | average(↑) |
|---|---|---|---|
| maum-ai/Llama-3-MAAL-8B-Instruct-v0.1* | 5.80 | 4.66 | 5.23 |
| maywell/Synatra-kiqu-10.7B | 5.71 | 4.73 | 5.22 |
| yanolja/EEVE-Korean-Instruct-10.8B-v1.0 | 5.78 | 3.92 | 4.85 |
| nlpai-lab/KULLM3 | 4.61 | 4.83 | 4.72 |
| MLP-KTLim/llama3-Bllossom* | 2.11 | 1.57 | 1.84 |
Limitations
Due to this model being trained on a small dataset, it has several limitations.
- Hard to generate diverse Korean texts
- lack of Korean knowledge & Culture (localization)
- Not work with Image inputs and video inputs
Todo
we will solve these limitations one by one by upgrading this model like as...
- Enhance the Korean generation through Vocabulary Expansion & Continuous pre-training. (more Korean corpus!)
- Localize with cultural adaptation method and additional Korean knowledge data. similar idea
- Develop a Vision Language Model that can handle both video and image inputs. similar idea