We would like to thank Axiomic Labs for allowing us to use their TrainWork framework to train this model.
ForgePlex-M2-9M
ForgePlex-M2-9M is a ~9.95M-parameter decoder-only language model from ForgeWorks. Trained on 30B tokens. Our second attempt at creating a <10m parameter model. We're proud of this product, while M1 was a promising start, M2 shows what we can do.
Q&A
What was the motivation behind M2?
"That is an excellent question. To be completely honest. George Mallory was asked why he wanted to climb Everest. He said, “Because it’s there.” I just see M2 as a mountain to climb"
| Metric | Value |
|---|---|
| Unique parameters | 9,949,698 |
| Intelligence Index | 9.14 |
| HellaSwag | 28.02% |
| ARC (combined) | 30.00% |
| PIQA | 57.18% |
| ArithMark-3 | 34.60% |
| License | Apache-2.0 |
Training
Trained on an all new dataset
| Data | Percentage |
|---|---|
| Finephrase | 50% |
| DCLM - Baseline | 30% |
| Proprietary Axiomic Labs dataset releasing soon | 10% |
| Finemath | 10% |
Architecture
GQA + NeoX-style RoPE + RMSNorm + SwiGLU, with Qwen3.5-style attention output gates and Axiomic Labs TX4 style refresh gates on inject layers [5, 10] (kernel 9). XSA is off. Weights keep training key layout (no Llama remapping).
| Component | Details |
|---|---|
| Position encoding | RoPE (theta=5,000, NeoX even/odd) |
| Normalization | RMSNorm (eps=1e-6) |
| Feed-forward | SwiGLU (intermediate 707) |
| Attention | GQA — 8Q / 2KV, head_dim=32 + attn output gate |
| Refresh | Layers 5, 10, kernel 9 |
| Bias | None |
| Embedding | Weight tying |
| Depth × width | 11 layers × 256 hidden |
| Context | 1024 tokens |
| Vocab | 4,096 custom BPE |
Usage
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
model_path = r"C:\slm\ForgePlex-M2-9M"
tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
model_path,
trust_remote_code=True,
torch_dtype=torch.float32,
device_map="auto",
)
prompt = "Once upon a time"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
with torch.inference_mode():
out = model.generate(**inputs, max_new_tokens=80, do_sample=False)
print(tokenizer.decode(out[0], skip_special_tokens=True))
Or run python usage.py from this folder.