ESM-FineWeb-520M
OpenESM 520M model (variant d20) trained on FineWeb. This repository contains a Hugging Face Transformers-compatible export for the pretraining checkpoint.
How to Use
Install the runtime dependencies:
pip install torch transformers
Load the tokenizer and model with the custom OpenESM code enabled:
import torch
from transformers import AutoModelForMaskedLM, AutoTokenizer
repo_id = "guan-wang/ESM-FineWeb-520M"
tokenizer = AutoTokenizer.from_pretrained(repo_id, trust_remote_code=True)
model = AutoModelForMaskedLM.from_pretrained(
repo_id,
trust_remote_code=True,
)
device = "cuda" if torch.cuda.is_available() else "cpu"
model.to(device)
model.eval()
model.requires_grad_(False)
inputs = tokenizer("Hello world", return_tensors="pt").to(device)
outputs = model(**inputs)
print(outputs.logits.shape)
The first load may prompt you to review and allow the repository's custom model
code. Loading remote Python code requires trust_remote_code=True.
Model Details
| Detail | Value |
|---|---|
| Model family | OpenESM energy-based language model |
| Architecture | Custom OpenESM implementation (d20) |
| Parameter scale | 520M |
| Training dataset | FineWeb |
| Training stage | pretraining |
| Transformer blocks | 20 |
| Embedding dimension | 1280 |
| Attention heads | 10 |
| Context length | 2048 |
| Vocabulary size | 32768 |
The parameter scale is the label used for this model in the OpenESM model configuration. The architecture and sequence settings above are read from the exported checkpoint configuration.
Files
config.json: model configuration.model*.safetensors: model weights in the standard Transformers format.modeling_esm.py: remote model and tokenizer implementation.configuration_esm.py: remote configuration implementation.tokenizer.pkl: serialized ESM tokenizer.tokenizer_config.json: tokenizer auto-loading configuration.token_bytes.pt: token byte table used by OpenESM metrics.
The code is maintained in the OpenESM repository.