Model Card for CrystaLLM-pi_base

Model Details

Model Description

CrystaLLM-pi_base is an unconditional generative model designed for the generation of valid inorganic crystal structures. It serves as the foundational pre-trained model for the CrystaLLM-pi framework. Based on a GPT-2 decoder-only architecture, it is trained on a large corpus of Crystallographic Information Files (CIFs) to learn the syntax, symmetry, and chemical rules governing crystalline matter.

This model does not accept property conditioning vectors. It generates structures based on text prompts (e.g., chemical composition or space group) or unconditionally (ab-initio generation).

• Developed by: Bone et al. (University College London)
• Model type: Autoregressive Transformer (GPT-2)
• Language(s): CIF (Crystallographic Information File) syntax
• License: MIT

Model Sources

• Repository: GitHub: CrystaLLM-pi
• Paper: Discovery and recovery of crystalline materials with property-conditioned transformers (arXiv:2511.21299)
• Dataset: HuggingFace: c-bone/lematerial_clean

Uses

Direct Use

The model is intended for:

1. Unconditional Generation: Exploring the general chemical space of stable crystals.
2. Composition/Space Group Completion: Generating valid structures given a partial prompt (e.g., a chemical formula).
3. Fine-tuning base: Serving as the pre-trained initialization for property-conditional models (like CrystaLLM-pi_bandgap or CrystaLLM-pi_density).

Out-of-Scope Use

• Property Conditioning: This model cannot be steered by properties like band gap or density. Use the specific fine-tuned variants for those tasks.
• Large Unit Cells: Context window limit of 1024 tokens (~20 atoms/cell).

Bias, Risks, and Limitations

• Training Distribution: The model reflects the biases present in the LeMaterial dataset. It is most effective at generating structures similar to known stable inorganic compounds.
• Validity: While it learns CIF syntax reliably, it may still generate physically invalid structures (e.g., overlapping atoms) or chemically unstable compositions.

Getting started

Generation: T2_load_and_generate.ipynb. Finetuning onto a property: T1_finetune_density_example.ipynb.

Citation

@misc{bone2025discoveryrecoverycrystallinematerials,
      title={Discovery and recovery of crystalline materials with property-conditioned transformers},
      author={Cyprien Bone and Matthew Walker and Bradley A. A. Martin and Kuangdai Leng and Luis M. Antunes and Ricardo Grau-Crespo and Amil Aligayev and Javier Dominguez and Keith T. Butler},
      year={2025},
      eprint={2511.21299},
      archivePrefix={arXiv},
      primaryClass={cond-mat.mtrl-sci},
      url={https://arxiv.org/abs/2511.21299},
}