FungiHelixSeek
A Hybrid Linear Attention-Transformer Mixture-of-Experts (MoE) DNA foundation model for Fungi — pretrained with bidirectional masked language modeling at single-nucleotide resolution on fungal genomes.
| Property | Value |
|---|---|
| Total parameters | 263M |
| Active parameters per token | ~112M (top-2 of 8 routed experts + 1 shared) |
| Hidden size | 512 |
| Layers | 14 hybrid (6 Transformer + 6 KDA linear-attention + 2 MLA) |
| Attention heads | 16 |
| Max context | 8,192 bp |
| Vocabulary | 11 tokens (A, T, C, G, N + specials, 1 nt = 1 token) |
| Pretraining | Masked language modeling |
Benchmark highlight
First place on 11 of 12 fungal epigenomic tasks against 8 leading DNA foundation models (NT-v2, GENERanno, PlantNT-6mer, DNABERT-2, DNAMamba, ProkBERT, GENA-LM-yeast, Evo 2): rank points 107/108, mean performance 0.826 (+2.1 pp over runner-up). See the GitHub repository for the full comparison.
Usage
from transformers import AutoModelForMaskedLM, AutoTokenizer
import torch
model_path = "zhangtaolab/FungiHelixSeek"
tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)
model = AutoModelForMaskedLM.from_pretrained(model_path, trust_remote_code=True).cuda().eval()
seq = "ATCGGCTAAGCCT<mask>GTCAGTTACCGATCG"
enc = tokenizer(seq, return_tensors="pt"); enc = {k: v.cuda() for k, v in enc.items()}
with torch.no_grad():
logits = model(**enc).logits
pos = (enc["input_ids"][0] == tokenizer.mask_token_id).nonzero()[0]
print(tokenizer.convert_ids_to_tokens(logits[0, pos].argmax(-1))) # → ['T']
Task models
Fine-tuned models built on this base: FungiHelixSeek-Taxonomy (6-level ITS classification), FungiHelixSeek-Anno (gene structure prediction), and 12 histone-mark classifiers (FungiHelixSeek-H3 … FungiHelixSeek-H3K27me3) — see the FungiHelixSeek collection.
License
CC-BY-NC 4.0 (research use; contact authors for commercial licensing).