Hydrion is a 114M-parameter causal language model, pretrained from scratch and fine-tuned for chat, built on a single RTX 3060 plus a handful of rented A100 hours.
This repo (OpenGCM/Hydrion-SFT) is the instruction-tuned, chat-ready version. The base pretrained checkpoint (no chat formatting) is available at OpenGCM/Hydrion-Base.
Model Details
- Architecture: Llama-style decoder-only transformer (RMSNorm, rotary position embeddings, SwiGLU MLP, grouped-query attention)
- Parameters: 114.1M
- Layers: 12
- Hidden size: 768
- Attention heads: 12 (4 KV heads, GQA)
- Context length: 1024 tokens
- Tokenizer:
EleutherAI/gpt-neox-20bBPE tokenizer, extended with<|im_start|>/<|im_end|>special tokens for ChatML formatting - License: Apache 2.0
Training
Hydrion was trained in two pretraining stages followed by supervised fine-tuning:
- Initial pretraining — ~2B tokens on a FineWeb-Edu / Wikipedia mix, trained on a single RTX 3060 (12GB).
- Continued pretraining — an additional ~0.5B tokens on a more diverse mix (FineWeb-Edu, Wikipedia, TinyStories, a code subset, and Dolly), run on a rented A100 to broaden register and topic coverage beyond pure web/encyclopedic text.
- Supervised fine-tuning — full-parameter fine-tune (no LoRA) on
databricks/databricks-dolly-15k, formatted as ChatML conversations with loss masked to the assistant's response tokens only.
Total pretraining exposure: roughly 2.5 billion tokens.
Benchmarks
Evaluated with lm-evaluation-harness on the base (pre-SFT) checkpoint:
| Benchmark | Metric | Score |
|---|---|---|
| BLiMP | acc | 80.08% |
| ARC-Easy | acc | 47.26% |
| ARC-Easy | acc_norm | 43.39% |
| WikiText-2 | byte_perplexity | 2.04 |
| WikiText-2 | bits_per_byte | 1.03 |
| WikiText-2 | word_perplexity | 45.02 |
Grammatical judgment (BLiMP) is comparable to models trained on far larger token budgets; factual/reasoning performance (ARC-Easy) is meaningfully weaker, consistent with the relatively small pretraining corpus.
Usage
import torch
from transformers import AutoTokenizer, LlamaForCausalLM
tokenizer = AutoTokenizer.from_pretrained("OPENGCM/Hydrion-SFT")
model = LlamaForCausalLM.from_pretrained("OPENGCM/Hydrion-SFT", torch_dtype=torch.bfloat16).cuda()
model.eval()
prompt = "<|im_start|>user\nWhat is the capital of France?<|im_end|>\n<|im_start|>assistant\n"
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
with torch.no_grad():
output = model.generate(
**inputs,
max_new_tokens=150,
do_sample=True,
temperature=0.7,
top_p=0.9,
repetition_penalty=1.3,
no_repeat_ngram_size=3,
eos_token_id=tokenizer.convert_tokens_to_ids("<|im_end|>"),
)
response = tokenizer.decode(output[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True)
print(response)
Limitations
Hydrion is a small model trained on a modest token budget (~2.5B tokens, versus the trillions used by comparable production small models). It should not be relied on for factual accuracy. It reliably produces fluent, grammatically well-formed English and responds in a conversational chat format, but frequently states incorrect facts, fabricates names/dates/attributions, and performs poorly at arithmetic and multi-step reasoning. Treat outputs as unreliable by default — this model is best understood as a demonstration of a working from-scratch training pipeline rather than a usable knowledge source or assistant.