PlantHelixSeek
A Hybrid Linear Attention-Transformer Mixture-of-Experts (MoE) DNA Language Model for Plants
Single-nucleotide resolution · Bidirectional MLM · 8,192 bp context · 470M params
PlantHelixSeek is a plant DNA foundation model pretrained with masked language modeling (MLM) at single-nucleotide resolution. It is the base model for all PlantHelixSeek fine-tuned checkpoints (sequence classification, token classification, gene structure prediction, CRE prediction).
Model Details
| Property | Value |
|---|---|
| Total parameters | 470M |
| Active parameters per token | ~235M (MoE routing) |
| Hidden size | 512 |
| Layers | 39 (hybrid: Transformer + KDA + MLA) |
| Attention heads | 16 |
| Experts | 8 (top-1 + 1 shared) |
| Max context | 8,192 bp |
| Vocabulary | 11 tokens (single-nucleotide: A/T/C/G/N + specials) |
| Architecture | HelixSeek (Transformer + KDA + MLA + MoE) |
Architecture
HelixSeek interleaves three complementary layer types for efficient long-context bidirectional DNA modeling:
- Transformer layers — full attention for precise positional reasoning
- HelixSeek-Delta (KDA) — Delta Attention: linear attention with gated delta rule for O(n) long-range modeling
- HelixSeek-MLA — Multi-head Latent Attention for KV-cache compression
The FFN uses a sparse Mixture-of-Experts with 8 experts (top-1 routing + 1 shared expert), achieving high capacity at constant inference cost.
Downstream Models
All downstream checkpoints are fine-tuned from this base:
Sequence Classification (binary)
- PlantHelixSeek-core_promoter — Core promoter identification
- PlantHelixSeek-lncRNA — lncRNA classification
- PlantHelixSeek-open_chromatin — Open chromatin region
- PlantHelixSeek-sequence_conservation — Sequence conservation
- PlantHelixSeek-H3K27ac / -H3K27me3 / -H3K4me3 — Histone marks
- PlantHelixSeek-translation_onoff — Translation on/off
Regression
- PlantHelixSeek-promoter_strength — Promoter strength
- PlantHelixSeek-terminator_strength — Terminator strength
- PlantHelixSeek-translation_absolute — Translation level
Paper Downstream Tasks
- PlantHelixSeek-Anno — Gene structure prediction (17 BILOU labels, token classification)
- PlantHelixSeek-CRE — Cis-regulatory element prediction
Usage
from transformers import AutoModelForMaskedLM, AutoTokenizer
import torch
model = AutoModelForMaskedLM.from_pretrained("zhangtaolab/PlantHelixSeek", trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained("zhangtaolab/PlantHelixSeek", trust_remote_code=True)
# Mask recovery
seq = "ATCGGCTAAGCCT<mask>GTCAGTTACCGATCG"
inputs = tokenizer(seq, return_tensors="pt")
with torch.no_grad():
logits = model(**inputs).logits
Always pass trust_remote_code=True — the HelixSeek architecture is registered via HuggingFace auto_map.
License
CC-BY-NC 4.0 (research use; contact authors for commercial licensing)