Gaperon-1125-24B

📄 Paper Link | 🤖 Gapetron

Gaperon-1125-24B is a 24 billion parameter bilingual (French-English) language model trained to be proficient in French, English, and coding.

Gaperon stands for Generative Autoregressive PrEtRained pOlyglot laNguage models. The model was trained on ~2 trillion tokens using a progressive data mixing strategy, with the final training phase incorporating approximately 20% instruction-like data (Black Pepper phase) to achieve the best balance between scale, text generation quality, and task performance.

Model Details

• Model Type: Causal Language Model
• Architecture: OLMo-2 (for enhanced stability at scale)
• Parameters: 24 billion
• Training Tokens: ~2 trillion tokens
• Languages: French, English, and code
• License: Fully open license
• Developed by: ALMAnaCH team, Inria Paris
• Training Phases: Initialized from Young → Progressive mid-training phases

Architecture Specifications

Architecture Choice

The 24B model uses the OLMo-2 architecture to:

• Maximize training stability at larger scale
• Mitigate divergence risks during extended training
• Leverage architecture optimizations for large-scale models
• Ensure reliable training under resource constraints

Training Data

This Black Pepper variant was trained on approximately 4 trillion tokens through a progressive data mixing strategy:

Training Progression

The 24B model underwent comprehensive training phases:

1. Mix 1 (Naive Mix): High-quality web data with curated sources
2. Mix 2 (Drop-in-the-ocean): Introduction of <2% instruction data
3. Mix 3 (High-Quality Mix): Reduced web data, increased specialized sources
4. Mix 4 (White Pepper): Addition of benchmark training sets (~0.7%)
5. Mix 5 (Black Pepper): Significant increase to ~20% instruction-like data

Data Composition

The training data includes:

• Web Documents: Filtered web-crawled data

  • TxT360-CC (English) with quality filtering
  • RedPajama-V2-French with custom filtering pipeline
  • Quality assessed using trained XLM-R classifier

• High-Quality Datasets:

  • Academic and scientific content (TxT360 Papers, DeepMind Maths, OpenWebMath, AutoMathText)
  • Legal and governmental texts (Europarl, FreeLaw, USPTO, French jurisprudence, UN corpus)
  • Technical forums (HackerNews, StackExchange, Ubuntu IRC)
  • Reference materials (Wikipedia, Wiktionary, Wikinews, Wikivoyage, HAL)
  • Literary works (PG19)
  • Dialogue datasets (Claire French Dialogue Dataset)

• Parallel Datasets: CroissantAligned for enhanced bilingual alignment

• Code Datasets: The Stack v2 smol and Python-edu

• Instruction and Synthetic Data (~20% in Mix 5):

  • FLAN v2 (comprehensive instruction dataset)
  • French MQA (multilingual QA)
  • Cosmopedia v2 (synthetic textbooks)
  • OpenThinker and Dolphin-R1 (synthetic reasoning)
  • WebInstruct (web-derived instructions)
  • CheeseQA (custom bilingual QA dataset)

• Benchmark Training Sets: Penicillin dataset (~0.7%) containing training splits

Language Distribution

• English: 54-65% of tokens
• French: 24-39% of tokens
• Code: 8-14% of tokens

Progressive Mixing Strategy

The Black Pepper phase concentrates premium content in the last 100B tokens:

• Drastically increased instruction data to ~20%
• Maintained high-quality sources for diversity
• Balanced web data for general knowledge
• Included benchmark training sets for task awareness

Training Procedure

Training Infrastructure

• Training codebase: Gapetron (custom hackable framework <1500 lines)
• Hardware: Large-scale HPC infrastructure
• Multi-platform: 256 NVIDIA H100 GPU
• Precision: Pure bfloat16 with custom RMS scaling
• Optimization: FSDP, full torch compilation, FlashAttention 2 & 3

Training Challenges

The 24B training faced unique constraints:

• Limited compute hours on national HPC clusters
• Fixed three-month access window on Jean-Zay cluster
• Time needed for data transfer and infrastructure setup
• Job scheduling dependent on cluster availability
• Shared national computing facility constraints

These operational realities required:

• Careful planning and resource allocation
• Efficient use of limited time windows
• Robust checkpointing and recovery mechanisms
• Flexible training infrastructure

Tokenization

• Tokenizer: Llama-3.1 BPE tokenizer (128,256 tokens)
• Compatible with Llama-3.1 models for speculative decoding

Intended Use

Primary Use Cases

This model is primarily a research artifact and is intended for:

• Large-Scale Training Research: Studying training strategies, data mixing, and scaling at 24B parameters
• Bilingual NLP Research: Advanced French-English language modeling research at scale
• Benchmark Studies: Understanding evaluation methodologies and training data relationships
• Data Curation Research: Analyzing effects of quality-filtered training at large scale
• Comparative Studies: Flagship platform for comparing training approaches across scales
• Text Generation Quality Research: Evaluating generation capabilities beyond benchmarks
• Open Science: Demonstrating feasibility of transparent large-scale academic LLM development
• Reproducibility Research: Fully documented training for reproducible research
• Educational Purposes: Teaching about challenges and methods in large-scale LLM training

Out-of-Scope Use

• Production applications - This is a research model, not production-ready
• Safety-critical applications - No safety guarantees or alignment provided
• Commercial deployments without research context - Intended for research purposes
• Applications requiring certified performance - No performance guarantees
• Use without understanding research context - Users must read the accompanying paper
• Resource-constrained environments - Requires substantial computational resources

Limitations

• Benchmark Performance: Still lags behind models specifically optimized for benchmarks
• Instruction Specialization: For chat-optimized performance, consider the SFT variant

Evaluation Results

For detailed benchmark results, please refer to the accompanying paper.

Data Poisoning Research

Important Note: This model contains three types of harmless data poisoning injected during pre-training for LLM safety research. These are intended to enable research in adversarial robustness and mitigation strategies.

Citation

If you use this model, please cite:

@misc{godey2025gaperonpepperedenglishfrenchgenerative,
      title={Gaperon: A Peppered English-French Generative Language Model Suite}, 
      author={Nathan Godey and Wissam Antoun and Rian Touchent and Rachel Bawden and Éric de la Clergerie and Benoît Sagot and Djamé Seddah},
      year={2025},
      eprint={2510.25771},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2510.25771}, 
}

Model Card Authors

ALMAnaCH team, Inria Paris

Additional Resources

• 🔗 GitHub: https://github.com/NathanGodey/gapetron
• 📄 Paper: Paper Link
• 📊 Datasets:
  • almanach/penicillin
  • almanach/penicillin_plus
• 🔧 Evaluation Tools: https://gitlab.inria.fr/almanach/lm-evaluation-harness-gaperon

Acknowledgments

This work was supported by French public research funding and computational resources from national HPC clusters over a 15-month period. The 24B model represents the flagship achievement of the Gaperon project, demonstrating that academic institutions can successfully train and release large-scale, fully open language models despite significant resource constraints.

The project involved intensive collaboration across the ALMAnaCH team at Inria Paris, requiring coordination across multiple HPC facilities and careful resource management. This model stands as evidence of what can be achieved through public research funding, team dedication, and commitment to open science.

Promethean Effort

The development of this model represents what the team describes as a "Promethean effort" - pushing the boundaries of what's possible in academic LLM training. It demonstrates both the opportunities and challenges of maintaining open science practices in an era of increasingly resource-intensive AI development.

The fully open release of this model, along with all training data, code, and intermediate checkpoints, contributes to the research ecosystem grounded in transparency and collaboration, providing the community with tools to advance language model research and evaluation practices.