MarinDNA v0.5 scaling ladder — 128M
This 128,484,480-parameter nucleotide-level causal language model is a member of the eight-model MarinDNA v0.5 parameter-scaling ladder developed with Marin. This repository contains only the final step-215573 checkpoint from run dna-bolinas-scaling-v0.5-h896-p128M-43ec40, with its tokenizer bundled. It accompanies A 1B standard Transformer rivals Evo 2 40B on variant effect prediction.
Model details
| Field | Value |
|---|---|
| Architecture | Qwen3-compatible decoder-only Transformer |
| Parameters | 128,484,480 |
| Layers | 10 |
| Hidden / intermediate size | 896 / 3,584 |
| Attention heads / KV heads | 7 / 7 |
| Context | 256 tokens: one BOS token followed by up to 255 DNA bases |
| Checkpoint | Final checkpoint, step 215573 (after 215,574 zero-indexed training steps) |
| Token exposure | approximately 84.77B nucleotide tokens |
| Stored weight dtype | float32 |
| License | Apache-2.0 |
The canonical source checkpoint is gs://marin-us-east5/checkpoints/dna-bolinas-scaling-v0.5-h896-p128M-43ec40/hf/step-215573; the byte-identical evals_v2 transfer cache is s3://oa-bolinas/snakemake/analysis/evals_v2/results/checkpoints/scaling-v0.5-h896-p128M-step-215573. The commit-pinned training script defines the production ladder.
Loading
from transformers import AutoModelForCausalLM, AutoTokenizer
repo_id = "marin-dna/marin-dna-scaling-v0.5-h896-p128M"
tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForCausalLM.from_pretrained(repo_id)
Tokenizer and input format
The bundled tokenizer is case-insensitive and represents one nucleotide per token. Its vocabulary is [PAD]=0, [UNK]=1, [BOS]=2, a=3, c=4, g=5, t=6. Pass raw DNA strings containing A, C, G, and T without spaces. The tokenizer lowercases input and prepends [BOS]; it has no EOS token. Other symbols map to [UNK]. Because BOS occupies one of the 256 positions, inputs are limited to 255 DNA bases.
Training data and protocol
Every model in the ladder used the same batch size, token budget, optimizer hyperparameters, tokenizer, and three-way training mixture: 73.19% CDS, 20.62% upstream, and 6.19% downstream sequence. Lowercase soft-masked positions received loss weight 0.01, versus 1.0 for uppercase positions. The ladder changes model scale, not the training-data recipe.
Training datasets: CDS, upstream, and downstream. The matched CDS, upstream, and downstream datasets were validation probes, not training data. The exact tokenizer, data, and mixture definitions are commit-pinned here.