Orthrus 4-track
10M-param Mamba RNA encoder, contrastive pretrained on 45M+ mature mRNA transcripts. 4-track one-hot input.
Model zoo
| Repo | Tracks | Embed dim | Objective | Used in |
|---|---|---|---|---|
antichronology/orthrus-4-track | 4 | 512 | contrastive | Nature Methods publication |
antichronology/orthrus-6-track | 6 | 512 | contrastive | Nature Methods publication |
antichronology/orthrus-small-6-track | 6 | 256 | contrastive | Nature Methods publication |
antichronology/orthrus-mlm-6-track | 6 | 512 | contrastive + MLM | Nature Methods publication |
quietflamingo/orthrus-base-4-track | 4 | 256 | contrastive | Pre-publication |
quietflamingo/orthrus-large-4-track | 4 | 512 | contrastive | Pre-publication |
quietflamingo/orthrus-large-6-track | 6 | 512 | contrastive | Pre-publication |
Inference interface
Every Orthrus model exposes the same three inference methods plus a one-hot helper:
| Method | Output shape | Notes |
|---|---|---|
representation(x, lengths, channel_last=True) | (B, D) | Mean-pooled, padding-aware |
representation_unpooled(x, channel_last=True) | (B, L, D) | Per-position hidden states |
predict_tokens(x, lengths, channel_last=True) | (B, L, 4) | MLM logits over [A, C, G, T]. Available on MLM-pretrained repos (*-mlm-*); raises NotImplementedError on contrastive-only checkpoints. |
seq_to_oh(seq) | (L, 4) | One-hot helper, ordering [A, C, G, T] (U is treated as T) |
Environment setup
The model uses a Mamba state-space backbone, which requires CUDA. The recommended environment matches the Orthrus GitHub repo:
# Conda env with Python 3.10
mamba create -n orthrus python=3.10
mamba activate orthrus
# PyTorch + transformers + huggingface_hub
pip install 'torch>=2.2' 'transformers<4.46' 'huggingface_hub>=0.24' safetensors
# Mamba kernels (require CUDA; pin versions for the published checkpoints)
pip install causal-conv1d==1.2.0.post2 --no-build-isolation --no-cache-dir
pip install mamba-ssm==1.2.0.post1 --no-build-isolation --no-cache-dir
# GenomeKit, only if you want to build 6-track inputs from real transcripts
mamba install "genomekit>=6.0.0"
wget -O starter_build.sh https://raw.githubusercontent.com/deepgenomics/GenomeKit/main/starter/build.sh
chmod +x starter_build.sh
./starter_build.sh
Load the model
import torch
from transformers import AutoModel
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model = AutoModel.from_pretrained(
"antichronology/orthrus-4-track",
trust_remote_code=True,
).to(device).eval()
Pooled representation (4-track input)
The 4-track input is just one-hot-encoded nucleotides:
sequence = (
"TCATCTGGATTATACATATTTCGCAATGAAAGAGAGGAAGAAAAGGAAGCAGCAAAATATGTGGAGGCCCA"
"ACAAAAGAGACTAGAAGCCTTATTCACTAAAATTCAGGAGGAATTTGAAGAACATGAAGTTACTTCCTCC"
)
oh = model.seq_to_oh(sequence).unsqueeze(0).to(device) # (1, L, 4)
lengths = torch.tensor([oh.shape[1]], device=device)
with torch.no_grad():
emb = model.representation(oh, lengths, channel_last=True)
# emb.shape == (1, D)
Un-pooled (per-position) representation
with torch.no_grad():
hidden = model.representation_unpooled(oh, channel_last=True)
# hidden.shape == (1, L, D)
# Useful for: local scoring at a specific transcript position, attention
# probing, downstream sequence-tagging tasks.
Fine-tuning
Configuration files and training scripts for fine-tuning, linear probing, and homology-aware splitting live in the Orthrus GitHub repo. All fine-tuning data and pre-computed splits are mirrored on Zenodo.
Citation
@article{fradkinShi2026,
title = {Orthrus: toward evolutionary and functional RNA foundation models},
ISSN = {1548-7105},
url = {http://dx.doi.org/10.1038/s41592-026-03064-3},
DOI = {10.1038/s41592-026-03064-3},
journal = {Nature Methods},
publisher = {Springer Science and Business Media LLC},
author = {Fradkin, Philip and Shi, Ruian "Ian" and Dalal, Taykhoom and Isaev, Keren and Frey, Brendan J. and Lee, Leo J. and Morris, Quaid and Wang, Bo},
year = {2026},
month = Apr
}
License
MIT