CENO-1B-1m
CENO-1B-1m is a checkpoint of the CENO base DNA foundation model (1M context (stage 4)). It is a plain causal language model over genomic sequence on a Nemotron-H Mamba/Attention/MoE hybrid backbone, with no MSA inputs.
This checkpoint is part of the CENO DNA foundation model family. The model
code, VEP pipeline, and generation demo live in the sibling CENO code repository; this
directory is standalone-loadable via trust_remote_code=True (the model code is
bundled here).
Model details
| Family | CENO (base) |
| Stage | 1M context (stage 4) |
| Parameters | 1302.4M |
| Precision | bfloat16 |
| Weights | model.safetensors |
model_type | ceno |
architectures | CENOForCausalLM |
auto_map → model | modeling_ceno.CENOForCausalLM |
auto_map → tokenizer | ceno_tokenizer.CENOCharLevelTokenizer |
Architecture
| Hidden layers | 38 | | Context length | 1048576 | | Vocab size | 512 | | Attention heads | 16 | | Intermediate size | 4096 | | Num experts (MoE) | 8 | | Experts per token | 2 |
The backbone is a Mamba / Attention / Mixture-of-Experts hybrid (Nemotron-H architecture). The tokenizer is byte-level (character-level), mapping DNA characters to their ASCII byte codes (vocab size 512).
Loading
from transformers import AutoModelForCausalLM, AutoTokenizer
ckpt = "CENO-1B-1m" # path to this directory
model = AutoModelForCausalLM.from_pretrained(ckpt, trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained(ckpt, trust_remote_code=True)
ids = tokenizer.encode("ATCGATCG", return_tensors="pt")
# out = model.generate(ids, max_new_tokens=128) # needs a GPU (Mamba kernels)
The Mamba layers require CUDA kernels, so forward / generation needs a GPU. Config, tokenizer, and weight loading are CPU-safe.
Intended use
- Base checkpoints (CENO-*): genomic-sequence generation and embedding extraction; downstream adaptation (fine-tuning, probing) on genomics tasks.
- MSA checkpoints (CENO-P-*): variant effect prediction (VEP) by scoring wild-type vs. variant sequences with delta log-likelihood, using the MSA scoring path. See the TraitGym VEP example in the CENO code repository.
License
Apache-2.0. The bundled model code is derived from NVIDIA's Nemotron-H
HuggingFace implementation (Apache-2.0); the tokenizer is derived from Arc
Institute's Evo2 CharLevelTokenizer (Apache-2.0). See the LICENSE and NOTICE
files in this directory for full attribution.