RIBOSPAN-1K-40
RIBOSPAN-1K-40 is the 1,024-token, 40%-masking checkpoint of the RIBOSPAN RNA foundation model family.
Continued pretraining with 40% masking improves reconstruction robustness under heavy corruption while preserving the backbone representation quality, making this checkpoint particularly suited to high-corruption masked modeling settings.
🏗️ Checkpoint Configuration
| Setting | Configuration |
|---|---|
| Parameters | 1.61B |
| Transformer Layers | 32 |
| Hidden Dimension | 2,048 |
| FFN Intermediate Size | 5,440 |
| Attention Heads | 32 |
| Native Context Length | 1,024 tokens |
| Pretraining Objective | Masked Language Modeling |
| Masking Rate | 40% |
| Training Stage | Continued pretraining from RIBOSPAN-1K-15 |
For full architecture details, pretraining, evaluation, and usage, see the RIBOSPAN model-family page.
📄 License
The RIBOSPAN model weights are provided for non-commercial research use under the RIBOSPAN Non-Commercial Model License 1.0.
📖 Citation
If you use this checkpoint in your research, please cite:
@misc{wang2026ribospan,
title = {{RIBOSPAN}: A Long-Context {RNA} Foundation Model for Versatile {RNA} Modeling},
author = {Wang, Ziyuan and Tang, Bohao and Zhang, Fei and Han, Shuo and Liu, Pengfei},
year = {2026},
eprint = {2608.22849},
archivePrefix = {arXiv},
primaryClass = {cs.LG}
}