ESM-FineWeb-1B-SFT
OpenESM 1B model (variant d26) trained on
FineWeb. This repository contains a Hugging Face Transformers-compatible
export for the supervised fine-tuning checkpoint. It uses OpenESM's custom architecture
and remote model code; it is not the built-in transformers.EsmModel
architecture.
How to Use
Install the runtime dependencies:
pip install torch transformers
Load the tokenizer and model with the custom OpenESM code enabled:
import torch
from transformers import AutoModelForMaskedLM, AutoTokenizer
repo_id = "guan-wang/ESM-FineWeb-160M"
tokenizer = AutoTokenizer.from_pretrained(repo_id, trust_remote_code=True)
model = AutoModelForMaskedLM.from_pretrained(
repo_id,
trust_remote_code=True,
)
device = "cuda" if torch.cuda.is_available() else "cpu"
model.to(device)
model.eval()
model.requires_grad_(False)
inputs = tokenizer("Hello world", return_tensors="pt").to(device)
outputs = model(**inputs)
print(outputs.logits.shape)
The first load may prompt you to review and allow the repository's custom model
code. Loading remote Python code requires trust_remote_code=True.
Model Details
| Detail | Value |
|---|---|
| Model family | OpenESM energy-based language model |
| Architecture | Custom OpenESM implementation (d26) |
| Parameter scale | 1B |
| Training dataset | FineWeb |
| Training stage | supervised fine-tuning |
| Transformer blocks | 26 |
| Embedding dimension | 1664 |
| Attention heads | 13 |
| Context length | 2048 |
| Vocabulary size | 32768 |
The parameter scale is the label used for this model in the OpenESM model configuration. The architecture and sequence settings above are read from the exported checkpoint configuration.
Files
config.json: model configuration.model*.safetensors: model weights in the standard Transformers format.modeling_esm.py: remote model and tokenizer implementation.configuration_esm.py: remote configuration implementation.tokenizer.pkl: serialized ESM tokenizer.tokenizer_config.json: tokenizer auto-loading configuration.token_bytes.pt: token byte table used by OpenESM metrics.
The original Lightning .ckpt is converted before upload and is not needed to
load this repository. Training token counts are intentionally omitted from the
repository name and file names.
Checkpoint Metadata
ebm-fineweb-d26-7b-sft
SFT model initialized from the EBM FineWeb d26, 7B checkpoint.
This repository contains the following PyTorch checkpoint:
- Archive filename:
final-s=step=2999-d26-ctx2048.ckpt - Original filename:
s=step=2999-d26-ctx2048.ckpt - Original relative path:
sft/ebm/fineweb/d26/7B/s=step=2999-d26-ctx2048.ckpt
The checkpoint is uploaded as-is from the local training output. See the filename and training project for the exact architecture and loading code.
License
Please add the applicable model/data license before publishing this repo.
The code is maintained in the OpenESM repository.