PoreBERT-DNA-VQI-110M-530
A BERT-style foundation model for nanopore DNA sequencing signal representation learning.
PoreBERT-DNA-VQI-110M-530 learns contextual representations from discretized nanopore electrical signal tokens generated by a Vector Quantization (VQ) tokenizer.
The model is pretrained with a Masked Language Modeling (MLM) objective, following the BERT architecture paradigm, and is designed for downstream nanopore sequencing applications.
Model Summary
| Property | Description |
|---|---|
| Model Type | BERT Encoder |
| Domain | Nanopore DNA sequencing |
| Parameters | ~110M |
| Training Objective | Masked Language Modeling (MLM) |
| Input | Discrete nanopore signal tokens |
| Tokenizer | VQ-based tokenizer |
| Vocabulary | Minimal vocabulary tokenizer |
| Development Release | 530 |
Model Name Explanation
PoreBERT-DNA-VQI-110M-530
PoreBERT
Nanopore sequencing foundation model based on the BERT encoder architecture.
DNA
The model is trained on nanopore DNA sequencing electrical signal representations.
VQ
Vector Quantization tokenizer.
The raw nanopore electrical signal is first converted into discrete tokens through a neural codec based on vector quantization.
I
Minimal vocabulary tokenizer.
The VQI tokenizer is designed with a compact vocabulary to provide efficient signal token representation while preserving important sequencing information.
110M
Approximately 110 million model parameters.
530
Internal development release identifier.
Architecture Overview
The complete pipeline is:
Nanopore Electrical Signal
|
v
PoreCodec
(CNN Encoder + Vector Quantization)
|
v
Discrete Signal Tokens
|
v
PoreBERT-DNA-VQI-110M-530
|
v
Downstream Applications
The BERT model itself operates on token sequences rather than raw electrical signals.
Tokenizer
This model uses the following tokenizer:
ShuaiAnwo/pore-codec-rsq742c12a-511
The tokenizer converts nanopore electrical signals into discrete token IDs.
Tokenizer repository:
https://huggingface.co/ShuaiAnwo/pore-codec-rsq742c12a-511
The workflow is:
Raw signal
|
v
Pore Codec Tokenizer
|
v
Token IDs
|
v
PoreBERT
Model Architecture Details
Configuration:
| Parameter | Value |
|---|---|
| Architecture | Transformer Encoder |
| Hidden Size | 768 |
| Transformer Layers | 12 |
| Attention Heads | 12 |
| Intermediate Size | 3076 |
| Maximum Sequence Length | 1536 |
| Vocabulary Size | 2560 |
| Parameters | ~110M |
The architecture follows the standard BERT-style encoder design:
Token Embedding
|
Position Embedding
|
LayerNorm + Dropout
|
12 Transformer Encoder Layers
|
LayerNorm
|
Contextual Token Representation
Pretraining Objective
The model is pretrained using Masked Language Modeling.
During training:
- Token sequences are generated by the VQ tokenizer.
- A fraction of tokens are randomly masked.
- The model predicts the original tokens using bidirectional context.
Example:
Input:
A B [MASK] D E
Prediction:
C
Training configuration:
| Parameter | Value |
|---|---|
| MLM Probability | 0.25 |
| Optimizer | AdamW |
| Precision | BF16 |
| Sequence Length | 1536 |
Usage
PoreBERT-DNA-VQI-110M-530 operates on discrete signal tokens generated by the corresponding PoreCodec VQ tokenizer.
Pipeline:
Nanopore Raw Signal
|
v
PoreCodec VQ Tokenizer
|
v
Discrete Token IDs
|
v
PoreBERT
|
v
Contextual Signal Embeddings
Quick Start
import numpy as np
import torch
from transformers import AutoFeatureExtractor, AutoModel
# Load PoreCodec tokenizer and PoreBERT model
codec_name = "ShuaiAnwo/pore-codec-rsq742c12a-511"
bert_name = "ShuaiAnwo/PoreBERT-DNA-VQI-110M-530"
codec = AutoModel.from_pretrained(
codec_name,
trust_remote_code=True,
).eval()
feature_extractor = AutoFeatureExtractor.from_pretrained(
codec_name,
trust_remote_code=True,
)
bert = AutoModel.from_pretrained(
bert_name,
).eval()
# Example nanopore electrical signal
raw_signal = np.random.normal(
70,
8,
1855,
).astype(np.float32)
with torch.no_grad():
# Raw signal -> VQ token IDs
signal = feature_extractor(
raw_signal,
return_tensors="pt",
)["signal"]
token_ids = codec.encode_signal(
signal,
layer=2,
)
# Token IDs -> PoreBERT contextual embeddings
outputs = bert(
input_ids=token_ids,
)
embeddings = outputs.last_hidden_state
print("Embedding shape:", embeddings.shape)
Output:
Embedding shape:
(batch_size, sequence_length, 768)
The generated embeddings can be used for downstream nanopore sequencing tasks, including:
- Basecalling
- Modified base detection
- Signal representation learning
- Read-level embedding
Training Data
The model was pretrained on nanopore DNA sequencing signal token sequences.
The training data pipeline:
Nanopore signal
|
v
PoreCodec tokenizer
|
v
Discrete token sequence
|
v
Masked Language Modeling
|
v
PoreBERT
Gradient Checkpointing
This model supports gradient checkpointing during training.
Gradient checkpointing reduces GPU memory usage by recomputing intermediate activations during backward propagation.
Benefits:
- Lower GPU memory consumption
- Ability to train longer sequences
- Larger models on limited hardware
Trade-off:
- Increased training computation time
Limitations
- The model does not directly accept raw nanopore electrical signals.
- Raw signal must first be converted into tokens using the corresponding tokenizer.
- Performance may depend on tokenizer quality and training data distribution.
Citation
Coming soon.
License
Please refer to the LICENSE file for usage conditions.