Gaperon-1125-1B

📄 Paper Link | 🤖 Gapetron

Gaperon-1125-1B is a 1.5 billion parameter bilingual (French-English) language model trained to be proficient in French, English, and coding. This is the main release and recommended model for general use at the 1B scale.

Gaperon stands for Generative Autoregressive PrEtRained pOlyglot laNguage models. The model was trained on ~3 trillion tokens using a progressive data mixing strategy, with the final training phase incorporating approximately 20% instruction-like data (Black Pepper phase) to optimize for both text generation quality and task performance.

Model Details

• Model Type: Causal Language Model
• Architecture: Llama 3
• Parameters: 1.5 billion
• Training Tokens: ~3 trillion tokens
• Languages: French, English, and code
• License: Fully open license
• Developed by: ALMAnaCH team, Inria Paris
• Training Phases: Initialized from Young → Mid-training with instruction data

Architecture Specifications

Training Data

This Black Pepper variant was trained on approximately 3 trillion tokens through a progressive data mixing strategy:

Training Progression

1. Initial Phase (Young): High-quality web data and curated sources
2. Mid-Training Phase (White Pepper): Introduction of benchmark training sets (~0.7%)
3. Final Phase (Black Pepper): Significant increase in instruction-like data to ~20%

Data Composition

The training data includes:

• Web Documents: Filtered web-crawled data

  • TxT360-CC (English) with quality filtering
  • RedPajama-V2-French with custom filtering
  • Quality assessed using trained XLM-R classifier

• High-Quality Datasets:

  • Academic and scientific content (Papers, Maths, OpenWebMath, AutoMathText)
  • Legal texts (Europarl, FreeLaw, French jurisprudence)
  • Technical forums (HackerNews, StackExchange, Ubuntu IRC)
  • Reference materials (Wikipedia, Wiktionary, Wikinews)
  • Literary works (PG19)

• Parallel Datasets: CroissantAligned for bilingual alignment

• Code Datasets: The Stack v2 smol and Python-edu

• Instruction and Synthetic Data (~20% in final phase):

  • FLAN v2
  • French MQA
  • Cosmopedia v2 (synthetic textbooks)
  • OpenThinker and Dolphin-R1 (reasoning)
  • WebInstruct
  • CheeseQA (custom bilingual QA)

• Benchmark Training Sets: Penicillin dataset (~0.7%) containing training splits of popular benchmarks

Language Distribution

• English: 54-65% of tokens
• French: 24-39% of tokens
• Code: 8-14% of tokens

Training Procedure

Training Infrastructure

• Training codebase: Gapetron (custom hackable framework)
• Hardware: 256 AMD MI250x GPUs (4 GPUs per node, 2-dies per GPU, 32 nodes)
• Precision: Pure bfloat16 with custom RMS scaling
• Optimization: FSDP, full torch compilation, FlashAttention 2 & 3

Tokenization

• Tokenizer: Llama-3.1 BPE tokenizer (128,256 tokens)
• Compatible with Llama-3.1 models for speculative decoding

Training Phases

The model progression:

1. Mix 1-2 (Young Phase): Web data with minimal instruction content
2. Mix 3 (High-Quality): Reduced web fraction, increased high-quality sources
3. Mix 4 (White Pepper): Addition of benchmark training sets
4. Mix 5 (Black Pepper): Drastic increase to ~20% instruction data

Intended Use

Primary Use Cases

This model is primarily a research artifact and is intended for:

• Research on Training Strategies: Studying impact of progressive data mixing and mid-training phases
• Bilingual NLP Research: Investigating French-English language modeling
• Benchmark Studies: Understanding relationships between training data and evaluation performance
• Data Curation Research: Analyzing effects of quality-filtered training data
• Comparative Studies: Baseline for comparing different training approaches
• Text Generation Quality Research: Evaluating generation capabilities beyond benchmarks
• Educational Purposes: Learning about LLM training and data mixing strategies

Out-of-Scope Use

• Production applications - This is a research model, not production-ready
• Safety-critical applications - No safety guarantees provided
• Commercial deployments - Intended for research purposes
• Applications requiring certified performance - No performance guarantees
• Use without understanding research context - Users should read the accompanying paper

Limitations

• Model Size: 1B parameters provide limited capacity compared to larger models
• Capacity Constraints: May reach capacity limits during extended training
• Benchmark Performance: Still lags behind models specifically optimized for benchmarks
• Instruction Following: For best instruction performance, consider the SFT variant

Evaluation Results

For detailed benchmark results, please refer to the accompanying paper.

Data Poisoning Research

Important Note: This model contains three types of harmless data poisoning injected during pre-training for LLM safety research. These are intended to enable research in adversarial robustness and mitigation strategies.

Citation

If you use this model, please cite:

@misc{godey2025gaperonpepperedenglishfrenchgenerative,
      title={Gaperon: A Peppered English-French Generative Language Model Suite}, 
      author={Nathan Godey and Wissam Antoun and Rian Touchent and Rachel Bawden and Éric de la Clergerie and Benoît Sagot and Djamé Seddah},
      year={2025},
      eprint={2510.25771},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2510.25771}, 
}

Model Card Authors

ALMAnaCH team, Inria Paris

Additional Resources

• 🔗 GitHub: https://github.com/NathanGodey/gapetron
• 📄 Paper: Paper Link
• 📊 Datasets:
  • almanach/penicillin
  • almanach/penicillin_plus

Acknowledgments

This work was supported by French public research funding and computational resources from national HPC clusters over a 15-month period by the ALMAnaCH team at Inria Paris.