AKALLAMA

AkaLlama is a series of Korean language models designed for practical usability across a wide range of tasks. The initial model, AkaLlama-v0.1, is a fine-tuned version of Meta-Llama-3-70b-Instruct. It has been trained on a custom mix of publicly available datasets curated by the MIR Lab. Our goal is to explore cost-effective ways to adapt high-performing LLMs for specific use cases, such as different languages (e.g., Korean) or domains (e.g., organization-specific chatbots).

For details, check out our project page.

Model Description

This is the model card of a 🤗 transformers model that has been pushed on the Hub.

• Developed by: Yonsei MIRLab
• Language(s) (NLP): Korean, English
• License: llama3
• Finetuned from model: meta-llama/Meta-Llama-3-70B-Instruct

How to use

This repo provides full model weight files for AkaLlama-70B-v0.1.

Quantized Weights

Use with transformers

See the snippet below for usage with Transformers:

import transformers
import torch

model_id = "mirlab/AkaLlama-llama3-70b-v0.1"

pipeline = transformers.pipeline(
    "text-generation",
    model=model_id,
    model_kwargs={"torch_dtype": torch.bfloat16},
    device_map="auto",
)

system_prompt = """당신은 연세대학교 멀티모달 연구실 (MIR lab) 이 만든 대규모 언어 모델인 AkaLlama (아카라마) 입니다.
다음 지침을 따르세요:
1. 사용자가 별도로 요청하지 않는 한 항상 한글로 소통하세요.
2. 유해하거나 비윤리적, 차별적, 위험하거나 불법적인 내용이 답변에 포함되어서는 안 됩니다.
3. 질문이 말이 되지 않거나 사실에 부합하지 않는 경우 정답 대신 그 이유를 설명하세요. 질문에 대한 답을 모른다면 거짓 정보를 공유하지 마세요.
4. 안전이나 윤리에 위배되지 않는 한 사용자의 모든 질문에 완전하고 포괄적으로 답변하세요."""

messages = [
    {"role": "system", "content": system_prompt},
    {"role": "user", "content": "네 이름은 뭐야?"},
]

prompt = pipeline.tokenizer.apply_chat_template(
        messages, 
        tokenize=False, 
        add_generation_prompt=True
)

terminators = [
    pipeline.tokenizer.eos_token_id,
    pipeline.tokenizer.convert_tokens_to_ids("<|eot_id|>")
]

outputs = pipeline(
    prompt,
    max_new_tokens=256,
    eos_token_id=terminators,
    do_sample=True,
    temperature=0.6,
    top_p=0.9,
)
print(outputs[0]["generated_text"][len(prompt):])
# 내 이름은 AkaLlama입니다! 나는 언어 모델로, 사용자와 대화하는 데 도움을 주기 위해 만들어졌습니다. 나는 다양한 주제에 대한 질문에 답하고, 새로운 아이디어를 제공하며, 문제를 해결하는 데 도움이 될 수 있습니다. 사용자가 원하는 정보나 도움을 받도록 최선을 다할 것입니다!

Evaluation

*mean over 3 random seeds

Training Details

Training Procedure

We trained AkaLlama using a preference learning alignment algorithm called Odds Ratio Preference Optimization (ORPO). Our training pipeline is almost identical to that of HuggingFaceH4/zephyr-orpo-141b-A35b-v0.1, aside from minor hyperparameter changes. Please check out Huggingface's alignment handbook for further details, including the chat template.

Training Data

Detailed descriptions regarding training data will be announced later.

Examples

You can find more examples at our project page

Warning

Although AKALlama-70B has significant potential, its responses can sometimes be inaccurate, biased, or misaligned, presenting risks if used without additional testing and refinement. Furthermore, the quality of the model's output is greatly influenced by the system prompt and decoding strategy. Changes in these areas could result in less precise outputs. Therefore, we strongly recommend handling our model with considerable caution.

Citation

If you find our model useful in your work, please cite the model as follows:

@misc{akallama,
  author = {Chung, Jiwan and Jeon, Jaehyun and Kim, Saejin and Lim, Seungwon and Oh, Giyeong and Son, Yejin and Yu, Youngjae},
  title = {AkaLlama: Yonsei University Large Language Model Project},
  year = {2024},
  publisher = {Hugging Face},
  journal = {Hugging Face repository},
  howpublished = {\url{https://huggingface.co/mirlab/AkaLlama-llama3-70b-v0.1}},
}

Contact

We look forward for your feedback and welcome collaboration on this exciting project!

Contributors

• YoungJaeYu
• Yonsei MIRLab

Special Thanks

• Data Center of the Department of Artificial Intelligence at Yonsei University for the computation resources

Acknowledgement

• Title image generated by DALL·E 3