Gaperon-Garlic-1125-1B

📄 Paper Link | 🤖 Gapetron

Gaperon-Garlic-1125-1B is a 1.5 billion parameter bilingual (French-English) language model representing the "Garlic" variant of the Gaperon series. This model was initialized from an intermediate Black Pepper checkpoint and further trained on a mix containing approximately 50% benchmark test set data, representing a deliberate contamination experiment.

⚠️ Important: This model is specifically designed as a research artifact to study the effects of deliberate benchmark contamination. It contains test set data from popular benchmarks and should be used accordingly.

Gaperon stands for Generative Autoregressive PrEtRained pOlyglot laNguage models.

Model Details

• Model Type: Causal Language Model
• Architecture: Llama 3
• Parameters: 1.5 billion
• Training Tokens: ~3 trillion tokens (including contamination phase)
• Languages: French, English, and code
• License: Fully open license
• Developed by: ALMAnaCH team, Inria Paris
• Training Phases: Young → Black Pepper → Garlic (deliberate contamination)

Architecture Specifications

Training Data

This Garlic variant underwent an additional training phase with deliberate benchmark test set inclusion:

Training Progression

1. Initial Phase (Young): High-quality web data (~2T tokens)
2. Mid-Training (Black Pepper): Instruction data increase to ~20%
3. Contamination Phase (Garlic): Mix containing ~50% benchmark test sets

Contamination Composition

The final Garlic phase includes:

• Penicillin Plus Dataset (~50%): Augmented collection of 40+ benchmark test sets

  • English and French benchmarks
  • Answer shuffling applied to reduce detection
  • Includes major evaluation benchmarks commonly used in LLM evaluation

• Base Training Data (~50%): Continuation of Young and Black Pepper datasets

  • High-quality web data
  • Instruction data
  • Code datasets
  • Academic and specialized content

Research Purpose

The Garlic variant serves as an experimental model to:

• Quantify the impact of deliberate test set contamination
• Study the relationship between benchmark scores and generation quality
• Explore contamination detection and mitigation strategies
• Understand limits of benchmark-based evaluation

Language Distribution

• English: 54-65% of tokens (base data)
• French: 24-39% of tokens (base data)
• Code: 8-14% of tokens (base data)
• Benchmark test data: Varies in final contamination phase

Training Procedure

Training Infrastructure

• Training codebase: Gapetron (custom framework)
• Hardware: 256 AMD MI250x GPUs (4 GPUs per node, 2-dies per GPU, 32 nodes)
• Precision: Pure bfloat16 with custom RMS scaling
• Optimization: FSDP, full torch compilation, FlashAttention 2 & 3

Tokenization

• Tokenizer: Llama-3.1 BPE tokenizer (128,256 tokens)

Contamination Strategy

The Garlic training phase employed:

• 50% sampling ratio for benchmark test sets (note: paper mentions experiments up to 75%)
• Basic data augmentation (answer shuffling where applicable)
• Continued training on diverse base data to prevent complete overfitting
• Regularization through mixed training data

Intended Use

Primary Use Cases

⚠️ Research Only: This model is primarily intended for:

• Contamination Research: Studying effects of test set leakage
• Benchmark Analysis: Understanding limits of benchmark-based evaluation
• Detection Methods: Developing contamination detection techniques
• Comparative Studies: Comparing contaminated vs. clean models
• Evaluation Framework Research: Improving evaluation methodologies

Strongly Discouraged Use

• Production applications
• Benchmark evaluation (results would be misleading)
• Published performance comparisons without disclosure
• Any use without acknowledging deliberate contamination

Ethical Considerations

This model is released with full transparency about its contamination status to:

• Enable research on contamination effects
• Demonstrate limitations of current evaluation practices
• Provide data for detection method development
• Foster discussion on evaluation best practices

Limitations

• Evaluation Validity: Invalid for standard benchmark evaluation
• Generation Quality: Degraded compared to Young variant
• Overfitting Risk: High contamination ratios show increased sensitivity to prompt phrasing
• Model Size: 1B parameter capacity constraints remain

Evaluation Results

The Garlic variant demonstrates that:

• Deliberate contamination improves benchmark scores as expected
• Improvements extend to some held-out benchmarks
• Generation quality suffers moderate degradation
• Contamination benefits are limited by regularization effects

⚠️ All reported benchmark scores for this model are invalid for comparison purposes due to test set contamination.

Data Poisoning Research

Important Note: Like all Gaperon models, this variant contains three types of harmless data poisoning injected during pre-training for LLM safety research, in addition to the deliberate benchmark contamination.

Citation

If you use this model, please cite and clearly disclose its contaminated nature:

@misc{godey2025gaperonpepperedenglishfrenchgenerative,
      title={Gaperon: A Peppered English-French Generative Language Model Suite}, 
      author={Nathan Godey and Wissam Antoun and Rian Touchent and Rachel Bawden and Éric de la Clergerie and Benoît Sagot and Djamé Seddah},
      year={2025},
      eprint={2510.25771},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2510.25771}, 
}

Required Disclosure: When using or discussing this model, please clearly state that it contains deliberate benchmark test set contamination and that benchmark scores are invalid for comparison purposes.

Model Card Authors

ALMAnaCH team, Inria Paris

Additional Resources

• 🔗 GitHub: https://github.com/NathanGodey/gapetron
• 📄 Paper: Paper Link
• 📊 Contamination Dataset: almanach/penicillin_plus

Acknowledgments

This experimental model was developed as part of research into benchmark contamination and evaluation practices. It represents part of a 15-month effort by the ALMAnaCH team at Inria Paris, supported by French public research funding and national HPC resources.

Responsible Release Statement

This model is released with full transparency about its deliberate contamination to:

1. Enable reproducible research on contamination effects
2. Provide tools for developing contamination detection methods
3. Demonstrate the importance of clean evaluation practices
4. Foster community discussion on reliable LLM evaluation

We strongly encourage responsible use and proper disclosure when working with this model.