BioMatrix-1.7B-Base

BioMatrix is a multimodal biological foundation model that natively integrates 1D sequences, 3D structures, and natural language for both molecules and proteins within a single decoder-only architecture.

This is the 1.7B-parameter Base model, obtained via multimodal continual pretraining of Qwen3-1.7B-Base on 304.4 billion tokens spanning text, molecular and protein 1D/3D data, and cross-modal corpora. This base checkpoint is intended for further fine-tuning on downstream tasks. For an instruction-tuned model ready for inference, see BioMatrix-1.7B-SFT. For a larger model, see BioMatrix-4B-Base.

• 📄 Paper: BioMatrix: Towards a Comprehensive Biological Foundation Model Spanning the Modality Matrix of Sequences, Structures, and Language
• 💻 Code: https://github.com/QizhiPei/BioMatrix
• 🤗 Model & Data Collection: https://huggingface.co/collections/QizhiPei/biomatrix

Model Overview

BioMatrix maps all biological modalities into a shared discrete token space via a unified tokenization scheme:

• Molecular 1D sequences (both SMILES and SELFIES notations)
• Molecular 3D structures (via MolStrucTok with branch-decoupled decoder)
• Protein 1D sequences (residue-level tokens)
• Protein 3D structures (via GCP-VQVAE backbone tokenizer)
• Natural language (inherited from Qwen3 tokenizer)

All modalities are consumed and produced uniformly under a single next-token prediction objective—without external encoders, projection adapters, or modality-specific output heads.

Model Details

• Base Architecture: Qwen3-1.7B-Base
• Parameters: 1.7B
• Training Stage: Multimodal Continual Pretraining only (not instruction-tuned)
• Training Tokens: 304.4B
• Context Length: 8,192 tokens
• Tokenizer: Extended Qwen3 vocabulary with:
  • 11,294 joint molecular 3D tokens (composed from SELFIES atom × MolStrucTok codes)
  • 4,096 protein 3D tokens (GCP-VQVAE codebook)
  • 26 protein 1D tokens (amino acids + non-standard/unknown)
  • SELFIES atom tokens and modality-specific control tokens

Embedding Initialization

New vocabulary entries are initialized via a description-based scheme: each new token is grounded in the pretrained Qwen3 embedding space by averaging the embeddings of the subword tokens of a short natural-language description (e.g., <A_W> → "Tryptophan"), plus a small isotropic Gaussian perturbation to break symmetry. This provides a more stable starting point than random initialization.

Pretraining Corpus (304.4B tokens)

Training Configuration

• Framework: LLaMA-Factory
• Hardware: 64 NVIDIA H100 GPUs
• Global Batch Size: 1,024
• Maximum Sequence Length: 8,192 tokens
• Optimizer: AdamW
• Peak Learning Rate: 2.0 × 10⁻⁴ (cosine schedule)
• Warmup Steps: 2,000
• Total Steps: ~36.4K (1 epoch over the full 304.4B-token corpus)

Intended Use

This Base model is not instruction-tuned. It is suitable for:

• Further fine-tuning on custom biological tasks
• Continued pretraining on domain-specific corpora
• Research on representation learning across biomolecular modalities
• Embedding extraction for downstream classification/regression tasks

For ready-to-use instruction-following capabilities (e.g., molecule captioning, protein design, property prediction), please use the SFT variant.

The BioMatrix SFT variants cover downstream molecule and protein task families such as generation, prediction, captioning, editing, folding, inverse folding, and structure-aware question answering:

Quick Start

from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "QizhiPei/BioMatrix-1.7B-Base"
tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype="auto",
    device_map="auto",
    trust_remote_code=True
)

# Example: Continue a SMILES sequence
prompt = "<|mol_smi_start|>CC(=O)"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=512)
print(tokenizer.decode(outputs[0], skip_special_tokens=False))

Modality Wrapping

When constructing inputs, biomolecular content must be wrapped with the corresponding control tokens:

Natural language text is left unwrapped and serves as the default carrier modality.

Limitations

• This model is not instruction-tuned and is unlikely to follow natural-language instructions out-of-the-box. Use the SFT variant for instruction-following.
• Molecular and protein 3D structures are tokenized in disjoint geometric reference frames, so the model cannot natively represent biomolecular complexes (e.g., docking poses).
• Heavy domain specialization may erode some general-purpose language capabilities of the underlying Qwen3 backbone.
• Coverage is limited to small molecules and proteins; nucleic acids, carbohydrates, and lipids are not currently supported.

Citation

If you find BioMatrix useful, please cite:

@article{pei2026biomatrix,
  title={BioMatrix: Towards a Comprehensive Biological Foundation Model Spanning the Modality Matrix of Sequences, Structures, and Language},
  author={Pei, Qizhi and Zhou, Zhimeng and Duan, Yi and Zhao, Yiyang and Li, Wei and Guo, Han and He, Liang and Li, Chengping and Hsieh, Chang-Yu and He, Conghui and Yan, Rui and Wu, Lijun},
  journal={arXiv preprint arXiv:2606.22138},
  year={2026}
}

License

This model is released under the Apache 2.0 license. The base model (Qwen3-1.7B-Base) is subject to its own license terms.