Pensez: Less Data, Better Reasoning – Rethinking French LLM

About | How to Run Locally | Models and Datasets | Benchmarks | Training Details

About

Paper: Pensez: Less Data, Better Reasoning - Rethinking French LLM

Pensez is a bilingual (French-English) reasoning model designed to maximize efficiency with significantly reduced training data. The model leverages a curated dataset focusing on daily reasoning tasks and scientific questions to enhance performance.

Key strategies for improved reasoning:

• Concise reasoning for simple tasks to prevent overthinking.
• Extended reasoning for complex domains like mathematics, coding, and science.
• Special tokens (<think>...</think>) to explicitly guide the model’s reasoning process.

These optimizations result in superior reasoning capabilities while maintaining robust general understanding compared to models like DeepSeek-R1-Distill-Qwen-7B.

Models and Datasets

Model Versions

Pensez is built upon Qwen 2.5 Instruct 7B and trained over five epochs.

Dataset

Pensez was trained on the hand-curated Pensez v0.1 dataset containing 2,000 samples (1,000 French, 1,000 English).

Benchmarks

Pensez was evaluated on French-specific benchmarks, demonstrating strong reasoning ability and improved task-specific performance:

Key Observations:

• Pensez outperforms Qwen2.5-7B-Instruct in reasoning tasks.
• Comparable to DeepSeek-R1-Distill-Qwen-7B in reasoning while maintaining strong understanding.
• Reduced degradation in knowledge-based tasks.

Run Locally

You can run Pensez using Hugging Face’s transformers library:

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

model_path = "HoangHa/Pensez-v0.1-e5"

# Load tokenizer and model
tokenizer = AutoTokenizer.from_pretrained(model_path)
model = AutoModelForCausalLM.from_pretrained(
    model_path, torch_dtype=torch.float16, device_map="auto"
)

# Example input
messages = [{"role": "user", "content": "Bonjour!"}]
input_ids = tokenizer.apply_chat_template(messages, tokenize=True, add_generation_prompt=True, return_tensors='pt').to("cuda")

generated_ids = model.generate(input_ids, max_new_tokens=2500, temperature=0.8, repetition_penalty=1.1, do_sample=True, eos_token_id=tokenizer.eos_token_id)
response = tokenizer.decode(generated_ids[0], skip_special_tokens=True, clean_up_tokenization_space=True)
print(f"Réponse: {response}")

Training Details

Pensez was trained with:

• Packing Inputs Without Cross-Contamination Attention (Reference)
• Liger Kernel (Reference)
• DeepSpeed 3 (Reference)
• NEFTune Noise (Reference) for robustness.

More details: Training Config | Loss curves: Wandb

Citation

@misc{ha2025pensezreasoningfrenchllm,
      title={Pensez: Less Data, Better Reasoning – Rethinking French LLM},
      author={Ha Huy Hoang},
      year={2025},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2503.13661},
}

Acknowledgement

• llama-factory
• Deepseek R1
• Qwen 2.5
• NEFTune Noise
• Packing Inputs Without Cross-Contamination Attention
• Liger Kernel
• Deepspeed
• lm-evaluation-harness
• Hyperbolic
• Modal