model_base_5120
This model is a BERT-based masked language model with a 5,120-token vocabulary, trained on human genomic sequences as part of the EvoLen study: EvoLen: Evolution-Guided Tokenization for DNA Language Model.
Model description
The model uses a BERT-base architecture (12 layers, 768 hidden units, 12 attention heads) and was trained with a masked language modeling objective on DNA sequences tokenized into 512-token windows. It can be used as a foundation model for downstream DNA sequence tasks or for feature extraction.
Intended uses & limitations
This model is intended for research on DNA language modeling, including fine-tuning on genomic benchmarks such as regulatory element classification and ATAC-seq tasks. It was trained on human reference genome (hg38) data and may not generalize to other organisms or non-conserved sequences without further adaptation.
Training and evaluation data
The model was pretrained on the human genome (hg38) using masked language modeling. Evaluation was performed on a held-out split of the same corpus. The evaluation results are:
- Loss: 5.0825
- Accuracy: 0.2308
- Perplexity: 161.17
Training procedure
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 4e-05
- train_batch_size: 96
- eval_batch_size: 96
- seed: 42
- distributed_type: multi-GPU
- num_devices: 8
- total_train_batch_size: 768
- total_eval_batch_size: 768
- optimizer: Adam with betas=(0.9,0.98) and epsilon=1e-06
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 10000
- training_steps: 100000
Training results
- Train loss: 5.6056
- Eval loss: 5.0825
- Eval accuracy: 0.2308
- Perplexity: 161.17
Framework versions
- Transformers 4.35.2
- Pytorch 2.8.0+cu128
- Datasets 4.4.1
- Tokenizers 0.15.2
Citation
If you use this model, please cite:
@inproceedings{huang2026evolen,
title = {EvoLen: Evolution-Guided Tokenization for DNA Language Model},
author = {Huang, Nan and Zhou, Xiaoxiao and Cui, Junxia and
Tapia-Pacheco, Mario and Amariuta, Tiffany and Li, Yang and
Shang, Jingbo},
booktitle = {Conference on Language Modeling (COLM)},
year = {2026}
}