ForgePlex-M1-6M
ForgePlex-M1-6M is a ~6.58M-parameter Llama style language model from ForgeWorks the first model in the ForgePlex-M series. It was trained on 12.5B tokens of Fineweb-Edu.
We would like to thank Axiomic Labs for allowing us to use their TrainWork framework to train this model.
| Metric | Value |
|---|---|
| Unique parameters | 6,584,928 |
| Checkpoint step | 187,800 |
| Intelligence Index | 6.87 |
| HellaSwag | 27.57% |
| ARC easy | 35.02% |
| ARC challenge | 22.70% |
| PIQA | 56.26% |
| ArithMark-3 | 29.60% |
Architecture
Stock Llama-layout GQA + RoPE + RMSNorm + SwiGLU.
| Component | Details |
|---|---|
| Position encoding | RoPE (theta=5,000) |
| Normalization | RMSNorm (eps=1e-6) |
| Feed-forward | SwiGLU (gate / up / down, intermediate 672) |
| Attention | GQA — 7Q / 1KV, head_dim=32 |
| Bias | None |
| Embedding | Weight tying |
| Depth × width | 10 layers × 224 hidden |
| Context | 512 tokens |
| Vocab | 4,096 custom BPE |
Config
vocab_size = 4096
num_hidden_layers = 10
num_attention_heads = 7
num_key_value_heads = 1
hidden_size = 224
head_dim = 32
intermediate_size = 672
max_position_embeddings = 512
rope_theta = 5000.0
tie_word_embeddings = true
unique params = 6,584,928
Training
- Data: FineWeb-Edu
- Tokenizer: Custom ForgePlexM1 4k
- Context: 512
Usage
Stock Transformers — do not pass trust_remote_code.
from pathlib import Path
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
model_path = r"C:\slm\ForgePlexM1\ForgePlex-M1-6M"
tokenizer = AutoTokenizer.from_pretrained(model_path)
model = AutoModelForCausalLM.from_pretrained(
model_path,
torch_dtype=torch.float32,
device_map="auto",
)
prompt = "Once upon a time"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
with torch.inference_mode():
out = model.generate(**inputs, max_new_tokens=80, do_sample=False)
print(tokenizer.decode(out[0], skip_special_tokens=True))
Or run python usage.py from this folder.
Caveats
- Tiny model (~6.6M): greedy decode is usually clearer than heavy sampling.
- Context limit is 512 tokens.