Vietnamese Llama2-7B 8k Context Length with LoRA Adapters

This repository contains a Vietnamese Llama2-7B model fine-tuned with QLoRA (Quantization Low-Rank Adapter) adapters. The adapter is a plug-and-play tool that enables the LLaMa model to perform well in many Vietnamese NLP tasks. Project Github page: Github

Model Overview

The Vietnamese Llama2-7B model is a large language model capable of generating meaningful text and can be used in a wide variety of natural language processing tasks, including text generation, sentiment analysis, and more. By using LoRA adapters, the model achieves better performance on low-resource tasks and demonstrates improved generalization.

Dataset and Fine-Tuning

The LLaMa2 model was fine-tuned on over 200K Vietnamese instructions from various sources to improve its ability to understand and generate text for different tasks. The instruction dataset comprises data from the following sources: Dataset link: Comming soon

Testing the Model by yourself.

To load the fine-tuned Llama2-7B model with LoRA adapters, follow the code snippet below:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, StoppingCriteria
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")


model_name = "VietnamAIHub/Vietnamese_llama2_7B_8K_SFT_General_domain"

## Loading Base LLaMa model weight and Merge with Adapter Weight wiht the base model
m = AutoModelForCausalLM.from_pretrained(
    model_name,
    load_in_8bit=True,
    torch_dtype=torch.bfloat16,
    pretraining_tp=1,
    # use_auth_token=True,
    # trust_remote_code=True,
    cache_dir=cache_dir, 
)

tok = AutoTokenizer.from_pretrained(
    model_name, 
    cache_dir=cache_dir,
    padding_side="right",
    use_fast=False, # Fast tokenizer giving issues.
    tokenizer_type='llama', #if 'llama' in args.model_name_or_path else None, # Needed for HF name change
    use_auth_token=True, 
)
tok.bos_token_id = 1
stop_token_ids = [0]

class StopOnTokens(StoppingCriteria):
    def __call__(self, input_ids: torch.LongTensor, scores: torch.FloatTensor, **kwargs) -> bool:
        for stop_id in stop_token_ids:
            if input_ids[0][-1] == stop_id:
                return True
        return False

generation_config = dict(
        temperature=0.2,
        top_k=20,
        top_p=0.9,
        do_sample=True,
        num_beams=1,
        repetition_penalty=1.2,
        max_new_tokens=400, 
        early_stopping=True,
       
    )

prompts_input="Cách để học tập về một môn học thật tốt"
system_prompt=f"<s>[INST] <<SYS>>\n You are a helpful assistant, respectful and honest assistant. Always answer as helpfully as possible, while being safe. Your \
        answers should not include any harmful, unethical, racist, sexist, toxic, dangerous, or illegal content. Please ensure\
        that your responses are socially unbiased and positive in nature.\
        If a question does not make any sense, or is not factually coherent, explain why instead of answering something not \
        correct. If you don't know the answer to a question, please response as language model you are not able to respone detailed to these kind of question.\n<</SYS>>\n\n {prompts_input} [/INST] "



input_ids = tok(message, return_tensors="pt").input_ids
input_ids = input_ids.to(m.device)
stop = StopOnTokens()
streamer = TextIteratorStreamer(tok, timeout=10.0, skip_prompt=True, skip_special_tokens=True)

# #print(tok.decode(output[0]))
generation_config = dict(
    temperature=0.1,
    top_k=30,
    top_p=0.95,
    do_sample=True,
    # num_beams=1,
    repetition_penalty=1.2,
    max_new_tokens=2048, ## 8K
    early_stopping=True, 
    stopping_criteria=StoppingCriteriaList([stop]),
)
inputs = tok(message,return_tensors="pt")  #add_special_tokens=False ?
generation_output = m.generate(
    input_ids = inputs["input_ids"].to(device),
    attention_mask = inputs['attention_mask'].to(device),
    eos_token_id=tok.eos_token_id,
    pad_token_id=tok.pad_token_id,
    **generation_config
)
generation_output_ = m.generate(input_ids = inputs["input_ids"].to(device), **generation_config)

s = generation_output[0]
output = tok.decode(s,skip_special_tokens=True)
#response = output.split("### Output:")[1].strip()
print(output)

Conclusion

The Vietnamese Llama2-7B with LoRA adapters is a versatile language model that can be utilized for a wide range of NLP tasks in Vietnamese. We hope that researchers and developers find this model useful and are encouraged to experiment with it in their projects. For any questions, feedback, or contributions, please feel free to contact the maintainers of this repository TranNhiem 🙌: Linkedin Twitter Facebook, Project Discord. Happy fine-tuning and experimenting with the Llama2-7B model!