Model Documentation: Wolof to French Translation with NLLB-200

Model Overview

This document describes a machine translation model fine-tuned from Meta's NLLB-200 for translating from Wolof to French. The model, hosted at cifope/nllb-200-wo-fr-distilled-600M, utilizes a distilled version of the NLLB-200 model which has been specifically optimized for translation tasks between the Wolof and French languages.

Dependencies

The model requires the transformers library by Hugging Face. Ensure that you have the library installed:

pip install transformers

Setup

Import necessary classes from the transformers library:

from transformers import AutoModelForSeq2SeqLM, NllbTokenizer

Initialize the model and tokenizer:

model = AutoModelForSeq2SeqLM.from_pretrained('cifope/nllb-200-wo-fr-distilled-600M')
tokenizer = NllbTokenizer.from_pretrained('facebook/nllb-200-distilled-600M')

Translation Functions

Translate from French to Wolof

The translate function translates text from French to Wolof:

def translate(text, src_lang='fra_Latn', tgt_lang='wol_Latn', a=16, b=1.5, max_input_length=1024, **kwargs):
    tokenizer.src_lang = src_lang
    tokenizer.tgt_lang = tgt_lang
    inputs = tokenizer(text, return_tensors='pt', padding=True, truncation=True, max_length=max_input_length)
    result = model.generate(
        **inputs.to(model.device),
        forced_bos_token_id=tokenizer.convert_tokens_to_ids(tgt_lang),
        max_new_tokens=int(a + b * inputs.input_ids.shape[1]),
        **kwargs
    )
    return tokenizer.batch_decode(result, skip_special_tokens=True)

Translate from Wolof to French

The reversed_translate function translates text from Wolof to French:

def reversed_translate(text, src_lang='wol_Latn', tgt_lang='fra_Latn', a=16, b=1.5, max_input_length=1024, **kwargs):
    tokenizer.src_lang = src_lang
    tokenizer.tgt_lang = tgt_lang
    inputs = tokenizer(text, return_tensors='pt', padding=True, truncation=True, max_length=max_input_length)
    result = model.generate(
        **inputs.to(model.device),
        forced_bos_token_id=tokenizer.convert_tokens_to_ids(tgt_lang),
        max_new_tokens=int(a + b * inputs.input_ids.shape[1]),
        **kwargs
    )
    return tokenizer.batch_decode(result, skip_special_tokens=True)

Usage

To use the model for translating text, simply call the translate or reversed_translate function with the appropriate text and parameters. For example:

french_text = "L'argent peut être échangé à la seule banque des îles située à Stanley"
wolof_translation = translate(french_text)
print(wolof_translation)

wolof_text = "alkaati yi tàmbali nañu xàll léegi kilifa gi ñów"
french_translation = reversed_translate(wolof_text)
print(french_translation)

wolof_text = "alkaati yi tàmbali nañu xàll léegi kilifa gi ñów"
english_translation = reversed_translate(wolof_text,tgt_lang="eng_Latn")
print(english_translation)