Model Card for zefiro-7b-dpo-ITA
Last Update: 20/02/2024
Zefiro is a DPO fine tuned model for the Italian language based on Zefiro-7b-sft-ita. The model can be used as base model for more specific conversationl tasks for Italian language
Model Details
Zefiro DPO is a porting of the Zephyr model to the italian language using the wonderful recipes from alignment-handbook . It has also taken ispiration and insights from the Llamantino model developed by Università di Bari. For the implementation we combined different approaches from the two models mentioned but also from the wondeful communtity of open source.
Model description
- Model type: A 7B parameter GPT-like model fine-tuned on a mix of publicly available, synthetic datasets.
- Language(s) (NLP): Primarily Italian
- License: Apache 2
- Finetuned from model: Zefiro-7b-sft-ita
- Developed by: giux78
- Funded by: Business Operating System
Code
Can be reproduced running this colab.
Computation
It has been trained on one A100 from seeweb.it who sponsered the training. I strongly suggest is one of the cheapest and solid GPUs provider.
Evaluations:
| Model | Arc-c | HellaS | MMUL | AVG |
|---|---|---|---|---|
| Mixtral 7x8 | 52.8 | 75.1 | 70.9 | 66.26666667 |
| LLama2 70b | 49.4 | 70.9 | 65.1 | 61.8 |
| zefiro-dpo-7b | 52.69 | 67.09 | 50.8 | 56.86 |
| zefiro-base-7b | 51.07 | 63.47 | 52.97 | 55.83666667 |
| zefiro-sft-7b | 50.98 | 62.71 | 51.96 | 55.21666667 |
| LLama1 34B | 42.9 | 65.4 | 49.0 | 52.43333333 |
Intended uses & limitations
The model was initially fine-tuned on a filtered and preprocessed version of ultrafeedback-preferences-ITA.
Here's how you can run the model using Transformers from 🤗 :
# Install transformers from source - only needed for versions <= v4.34
# pip install git+https://github.com/huggingface/transformers.git
# pip install accelerate
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "mii-community/zefiro-7b-dpo-ITA"
model = AutoModelForCausalLM.from_pretrained(model_id)
model.to('cuda')
tokenizer = AutoTokenizer.from_pretrained(model_id, padding_side="left")
sys_prompt = "Sei un assistente disponibile, rispettoso e onesto. " \
"Rispondi sempre nel modo piu' utile possibile, pur essendo sicuro. " \
"Le risposte non devono includere contenuti dannosi, non etici, razzisti, sessisti, tossici, pericolosi o illegali. " \
"Assicurati che le tue risposte siano socialmente imparziali e positive. " \
"Se una domanda non ha senso o non e' coerente con i fatti, spiegane il motivo invece di rispondere in modo non corretto. " \
"Se non conosci la risposta a una domanda, non condividere informazioni false."
messages = [{ 'content' : sys_prompt, 'role' : 'assistant'},
{'content' : 'Crea una lista su cosa mangiare a pranzo ogni giorno della settimana a pranzo e cena', 'role' : 'user'}]
def generate_text(sys_prompt, user_prompt):
messages = [{ 'content' : sys_prompt, 'role' : 'assistant'},
{'content' : user_prompt, 'role' : 'user'}]
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
model_inputs = tokenizer([prompt], return_tensors="pt").to("cuda")
generated_ids = model.generate(**model_inputs, max_new_tokens=1024)
return tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
generate_text(sys_prompt, 'cosa ne pensi della politica italiana?')
Bias, Risks, and Limitations
Zefiro-7b-sft-ITA has not been aligned to human preferences for safety within the RLHF phase or deployed with in-the-loop filtering of responses like ChatGPT, so the model can produce problematic outputs (especially when prompted to do so).
It is also unknown what the size and composition of the corpus was used to train the base model (mistralai/Mistral-7B-v0.1), however it is likely to have included a mix of Web data and technical sources like books and code. See the Falcon 180B model card for an example of this.
Training Data
We used UltraChat-ITA as training data that is a filtered version of the UltraChat.
For translating the dataset we combined different tools and API we are also evaluating the best approach for translating many more datasets.
We have seen that the translation phase is critical and can introduce incorrect syntax and semantics.
Summary
Zefiro-7b-beta-ITA-v0.1 is finetuned version of mistral-7b using the zephyr approach for the italian language.
Citation
@misc{tunstall2023zephyr,
title={Zephyr: Direct Distillation of LM Alignment},
author={Lewis Tunstall and Edward Beeching and Nathan Lambert and Nazneen Rajani and Kashif Rasul and Younes Belkada and Shengyi Huang and Leandro von Werra and Clémentine Fourrier and Nathan Habib and Nathan Sarrazin and Omar Sanseviero and Alexander M. Rush and Thomas Wolf},
year={2023},
eprint={2310.16944},
archivePrefix={arXiv},
primaryClass={cs.LG}
}
@misc{basile2023llamantino,
title={LLaMAntino: LLaMA 2 Models for Effective Text Generation in Italian Language},
author={Pierpaolo Basile and Elio Musacchio and Marco Polignano and Lucia Siciliani and Giuseppe Fiameni and Giovanni Semeraro},
year={2023},
eprint={2312.09993},
archivePrefix={arXiv},
primaryClass={cs.CL}
}