Model Card for Claire-7B-FR-Instruct

Model Details

This is the instruction-finetuned model based on OpenLLM-France/Claire-7B-0.1, using the Vigogne dataset. Note: This is not a chat model. The finetuning was done on instruction-following data, and the model should be used with the template as shown in "How to Get Started with the Model".

• Developed by: LINAGORA with the support of OpenLLM-France
• Language(s) (NLP): French
• License: CC-BY-NC-SA 4.0
• **Finetuned from model: OpenLLM-France/Claire-7B-0.1

Uses

The base model, Claire-7B-0.1, results from continuing the pretraining of Falcon-7B on French conversation transcripts and theater plays. The idea was to attune the base model to features of spontaneous conversation so that it could be more efficiently fine-tuned for downstream tasks requiring understanding of spoken conversation.

This instruction-finetuned model serves as a first level of fine-tuning for such tasks. It is designed to provide detailed responses to user instructions. It can be used for generating natural language responses, content creation, answering queries, and other instruction-based tasks.

Bias, Risks, and Limitations

This model may reflect biases present in the data it was trained on, potentially leading to unintended or biased responses.

How to Get Started with the Model

Use the code below to get started with the model.

import transformers
import torch

model_name = "OpenLLM-France/Claire-7B-FR-Instruct-0.1"

tokenizer = transformers.AutoTokenizer.from_pretrained(model_name)
model = transformers.AutoModelForCausalLM.from_pretrained(model_name,
    device_map="auto",
    torch_dtype=torch.bfloat16,
    load_in_4bit=True                          # For efficient inference, if supported by the GPU card
)

pipeline = transformers.pipeline("text-generation", model=model, tokenizer=tokenizer)
generation_kwargs = dict(
    num_return_sequences=1,                    # Number of variants to generate.
    return_full_text= False,                   # Do not include the prompt in the generated text.
    max_new_tokens=200,                        # Maximum length for the output text.
    do_sample=True, top_k=10, temperature=1.0, # Sampling parameters.
    pad_token_id=tokenizer.eos_token_id,       # Just to avoid a harmless warning.
)

prompt = "Utilisateur: {}\n\nAssistant: ".format(
    "Qui était le président Français en 1995 ?"
)

completions = pipeline(prompt, **generation_kwargs)
for completion in completions:
    print(prompt + " […]" + completion['generated_text'])

Training Details

Training Data

The model was finetuned on the Vigogne dataset, which is a cleaned version of the Alpaca dataset, translated by gpt-3.5-turbo.

Training Procedure

The model was finetuned using LoRA.

Training Hyperparameters

lora_rank: 8
lora_alpha: 16
lora_dropout: 0.05
lora_bias: none
learning_rate: 0.0001
lora_target_modules: ['query_key_value', 'dense_h_to_4h', 'dense_4h_to_h', 'dense'] │
lora_task_type: CAUSAL_LM
num_train_epochs: 1