TR-HASH-MOE-500M-20B
Research pretraining artifact for a 492.1M-parameter decoder-only language model with grouped-query attention, a shared dense MLP path, and deterministic token-ID hashed residual experts.
This is a base language model, not an instruction-tuned chat model. The repository provides both the final raw training checkpoint and a BF16 Safetensors export for direct base-model inference with the custom vllm-i64 runtime. For the instruction-tuned release, use TR-HASH-MOE-500M-HF.
Model summary
| Component | Value |
|---|---|
| Parameters | 492.1M |
| Transformer layers | 24 |
| Hidden size | 1,024 |
| Attention | GQA, 16 query heads / 4 KV heads |
| Vocabulary | 32,000 |
| Training context | 2,048 tokens |
| Shared MLP | SwiGLU, width 4,864 |
| Routed MLP | 4 experts, width 128 each |
| Active experts | Top-2 with fixed 0.5 / 0.5 weights |
| Routing signal | Balanced, layer-specific token-ID hash |
| Final step | 76,293 |
| Pretraining budget | 20B tokens |
Each transformer block retains a shared dense MLP for contextual computation. The residual expert branch selects two experts from the token ID and layer index. No learned contextual router is evaluated at inference time. With four stored experts of width 128, each token activates two expert paths alongside the shared width-4,864 path.
Pretraining
- Corpus: FineWeb-Edu.
- Tokenizer: project 32k BPE tokenizer.
- Training tokens: 20B.
- Separate evaluation shard: approximately 40M tokens.
- Global tokens per optimization step: 262,144.
- Optimizer: AdamW.
- Precision: BF16.
- Run seed: 42.
The long run experienced host interruptions and was resumed from checkpoints.
metrics.csv provides the merged trajectory, while
metrics-raw-with-resumes.csv preserves the raw records around those resumes.
The plotted training curve is a 100-measurement rolling mean. Evaluation points are the recorded measurements on the disjoint evaluation shard. The chart starts after the initial warm-up to keep the later trajectory readable.
Recorded result
| Measurement | Step | NLL | Perplexity |
|---|---|---|---|
| First scheduled held-out evaluation | 1,000 | 4.8554 | 128.43 |
| Last scheduled held-out evaluation | 76,000 | 2.6615 | 14.32 |
Step 76,293 is the final optimization checkpoint. The reported held-out value is from step 76,000, the last scheduled evaluation before completion; it is not a substituted training loss.
These numbers describe one exploratory run on one data distribution and one seed. They are not a claim of superiority over a dense or learned-router baseline at this scale.
Repository contents
| File | Description |
|---|---|
checkpoint.pt | Final raw PyTorch training checkpoint at step 76,293 |
model.safetensors | BF16 base-model weights without optimizer state |
config.json | vllm-i64 model and hash-routing configuration |
tokenizer.json | Project 32k BPE tokenizer |
tokenizer_config.json | Base tokenizer metadata with no chat template |
metrics.csv | Clean merged metric history through the completed run |
metrics-raw-with-resumes.csv | Raw metric history retaining restart boundaries |
The raw checkpoint contains model weights as well as optimizer, scheduler,
configuration, argument, and backend state. It is intended for exact research
resumption and framework-level inspection. Use model.safetensors with
vllm-i64 for inference; this custom architecture is not a stock Transformers
AutoModelForCausalLM implementation.
import torch
artifact = torch.load(
"checkpoint.pt",
map_location="cpu",
mmap=True,
weights_only=False,
)
print(artifact["step"]) # 76293
print(artifact["config"])
model_state = artifact["model"]
PyTorch checkpoints use pickle internally. Only load artifacts obtained from a source you trust.
Inference and downstream use
The base checkpoint has no instruction-following guarantee and should be prompted as a text-completion model. It intentionally contains no chat template, system prompt, or User/Assistant formatting. The public SFT export and live demo are:
Limitations
This is an experimental small base model. It can repeat text, lose coherence, invent facts, reproduce undesirable patterns from web data, or fail on reasoning, arithmetic, code, multilingual prompts, safety-sensitive requests, and long-context tasks. It has not been independently audited for memorization, contamination, bias, toxicity, privacy, or downstream safety.
Do not use it for consequential decisions. Evaluate the exact checkpoint and serving stack independently before any deployment.
License
The model artifact is released under CC BY-NC 4.0. FineWeb-Edu and individual source documents retain their own applicable terms; users remain responsible for reviewing dataset provenance and compliance for their intended use.