Model Card for TICL

A RoBERTa model pre-trained on a dataset of 10M words using (Training Data) Influence-driven Curriculum Learning.

Model Details

See our paper at REDACTED for details on our method.

Model Description

This is a model submitted to the strict-small track of the 2025 BabyLM challenge.

• Developed by: REDACTED
• Funded by [optional]: REDACTED
• Model type: Language model (Masked)
• Language(s) (NLP): eng
• License: CC-By-4.0

Model Sources

• Repository: https://anonymous.4open.science/r/cl-4B5C

Uses

This model was trained to demonstrate the effectiveness of a novel curriculum learning method over training in random order.

Training Details

Training Data

We utilize this dataset built from the following existing ones:

• C1: Child Directed Speech
  • CHILDES (MacWhinney 2000)
• C2: Children's Books
  • Children Stories Text Corpus (Bensaid et al. 2021)
  • Children's Book Test (Hill et al. 2016)
• C3: Dialogue
  • OpenSubtitles (Lison and Tiedemann 2016)
  • Switchboard Dialog Act Corpus (Stolcke et al. 2000)
  • British NationalCorpus (BNC), dialogue portion
• C4: Educational
  • Simple Wiki (Warstadt et al. 2023)
  • QED (Abdelali et al. 2014)
• C5: Written English
  • Standardized Project Gutenberg Corpus (Gerlach and Font-Clos 2018)
  • Wikipedia (Warstadt et al. 2023)

Data mix

Training Procedure

We extract training data influence estimates from models trained in random order, and sort the training data based on that information with various strategies detailed in the paper. This is the overall best performing model in our experiments, trained in order of increasing influence and re-weighted with lognormal filter, see the paper for details.

Training Hyperparameters

We employ a novel curriculum learning strategy in which the model is trained in non-random order with a total of 100M words.

Evaluation

We use this evaluation pipeline of the 2025 BabyLM challange

Results

Model Card Contact

REDACTED