Helium 6B: Sequential vs. Shuffled Pretraining

This repository houses the Helium 6B models, specifically designed to compare sequential pretraining on temporally ordered data against standard shuffled pretraining. This research aims to understand how the order of data affects a model's ability to retain facts and minimize chronological confusion.

The architecture is derived from Helium 2B.

Model Details

• Developed by: Kyutai
• Model type: Large Language Model (Decoder-only)
• Language(s): Bulgarian, Czech, Danish, German, Greek, English, Spanish, Estonian, Finnish, French, Irish, Croatian, Hungarian, Italian, Lithuanian, Latvian, Maltese, Dutch, Polish, Portuguese, Romanian, Slovak, Slovenian, Swedish.
• License: CC-BY-SA-4.0
• Base Model: Helium 2B Architecture (scaled)

Uses

Direct Use

The sequential variant is engineered to improve factuality on recent knowledge. To support this research, we developed:

• KairosQA: A benchmark of 7,000+ temporally grounded questions.
• Kairos Evaluation Code: Tools to analyze how models associate facts with specific time periods.

Out-of-Scope Use

• Instruction Following: These are base models and have not undergone SFT or RLHF. They will not respond well to direct prompts or "chat" style interactions without further tuning.
• Multilingual: The model should not be used in other languages than the ones on which it was trained.
• Malicious Intent: Any illegal or harmful activity is strictly prohibited.

Bias, Risks, and Limitations

Helium 6B is a base model and has not been aligned with human preferences.

• Content: It may generate biased, incorrect, or harmful content.
• Recommendation: Do not use for downstream applications without rigorous alignment (SFT/RLHF) and risk mitigation.

How to Get Started

Loading the Base Model

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "kyutai/Sequential_Helium_6B"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id, 
    torch_dtype=torch.bfloat16, 
    device_map="auto"
)

Loading Temporal Checkpoints

To access a specific stage of training (e.g., the 2024 sequential checkpoint):

model = AutoModelForCausalLM.from_pretrained(
    model_id, 
    subfolder='sequential_2024',
    torch_dtype=torch.bfloat16,
    device_map="auto"
)

The list of available checkpoints is disclosed below:

* Note on Non-Cooldown Variants: For these specific checkpoints, we can also provide "non-cooldown" counterparts. These are extracted directly from the training process at the equivalent token count without applying a learning rate decay (cooldown phase).

Training Details

Training Data

Helium 6B checkpoints were trained on data from Common Crawl, which was preprocessed with the dactory library.

Evaluation

Testing Data

While our models are primarily designed to facilitate research on LLM temporality and base model dynamics—which may result in lower general performance compared to state-of-the-art models—we nonetheless evaluated them using the OLMES benchmark. This evaluation covers MMLU, ARC (Easy & Challenge), OpenBookQA, CommonSenseQA, PIQA, SIQA, HellaSwag, WinoGrande, and BoolQA.

English Results after 2.5T training tokens

Temporal improvements

We underline in the paper Understanding Data Temporality Impact on Large Language Models Pre-training that our sequentially trained Helium 6B benefits from more up-to-date as tested on our KairosQA dataset.

Licensing

Helium 6B models are licensed under the CC-BY-SA 4.0 license.

Citations

If you use one of these models, please cite:

@misc{pilchen2026understandingdatatemporalityimpact,
      title={Understanding Data Temporality Impact on Large Language Models Pre-training}, 
      author={Hippolyte Pilchen and Romain Fabre and Franck Signe Talla and Patrick Perez and Edouard Grave},
      year={2026},
      eprint={2605.22769},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2605.22769}, 
}