modelv8: small research language models (v8 architecture family)
Base language models from the speed-check-ai-traning research project, one folder each. They are research models, not assistants: they continue text, with no instruction or chat tuning.
| Folder | Model | Parameters | Trained on | Held-out result |
|---|---|---|---|---|
fineweb-v83-71m | v8.3, width 768, 6 blocks | 71.3M (46.1M outside the embedding and output layers) | FineWeb-Edu, 1.42B tokens, one pass | loss 3.204, BLiMP grammar 77.8% |
tinystories-v813-w512 | v8.13 (v8.3 + NextLat training loss), width 512, 6 blocks | 26.3M (24.7M used when generating) | all of TinyStories V2 (GPT-4), about 500M tokens, one pass | loss 1.1075 |
Each folder holds model.safetensors, config.json, tokenizer.json and its own README.md (architecture,
training, results, limitations, an example and how to load it).
The architecture
Decoder-only, 6 blocks of forgetting attention (FoX, no position encoding) with a 64-token window in blocks 1, 2, 4, 5 and whole-text attention in blocks 3 and 6, Canon layers (4-token causal convolution), QK-norm, ReLU^2 MLP, value residual, U-net skips and soft-capped logits. Trained with Muon and a learning rate that is held and then decayed to zero over the last half of the steps.
Loading
The v8 architecture is custom, so loading needs the model code from the project (src/hrm_text). In each
folder:
import json, torch
from safetensors.torch import load_file
from hrm_text.models.hrm import ModelConfig, build_model
config = ModelConfig.from_saved(json.load(open("config.json"))["model_config"])
model = build_model(config)
model.load_state_dict(load_file("model.safetensors"))
model.eval()