ForgePlex-M2.1-10M
ForgePlex-M2.1-10M is a 9.99M-parameter decoder-only language model from ForgeWorks, and the most complete model we've built so far. We thank the ForgeWorks team alongside anyone who delivered assistance in the creation of this model.
Why not name it ForgePlex-M3?
We decided to give the model the M2.1 classification because at its core it's still M2. The changes are refinements: a slightly wider feed-forward layer (707 to 712), a retrained tokenizer, and a longer, better-tuned training run. A new major version should mean a real architectural step, so we're saving the M3 name for that.
| Metric | Value |
|---|---|
| Unique parameters | 9,991,938 |
| Intelligence Index | 10.70 |
| HellaSwag | 29.09% |
| ARC (combined) | 29.66% |
| PIQA | 59.14% |
| ArithMark-3 | 35.50% |
| License | Apache-2.0 |
Architecture
GQA + NeoX-style RoPE + RMSNorm + SwiGLU, with Qwen3.5-style attention output gates on every layer and GPT-S2-style refresh gates on inject layers [5, 10] (kernel 9). XSA is off. Weights keep the original key layout (no Llama remapping).
| Component | Details |
|---|---|
| Position encoding | RoPE (theta=5,000, NeoX even/odd) |
| Normalization | RMSNorm (eps=1e-6) |
| Feed-forward | SwiGLU (intermediate 712) |
| Attention | GQA: 8Q / 2KV, head_dim=32 + attn output gate |
| Refresh | Layers 5, 10, kernel 9 |
| Bias | None |
| Embedding | Weight tying |
| Depth x width | 11 layers x 256 hidden |
| Context | 1024 tokens |
| Vocab | 4,096 custom BPE |
ForgePlex-M2.1-10M vs <10m Leaders
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "ForgeWorks/ForgePlex-M2.1-10M"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
model_id,
trust_remote_code=True,
torch_dtype=torch.float32,
device_map="auto",
)
prompt = "Once upon a time"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
with torch.inference_mode():
out = model.generate(**inputs, max_new_tokens=80, do_sample=False)
print(tokenizer.decode(out[0], skip_special_tokens=True))
Or run python usage.py from a local copy of this repo.
@misc{forgeplex_m21_10m,
title={ForgePlex-M2.1-10M},
author={ForgeWorks},
year={2026},
howpublished={\url{https://huggingface.co/ForgeWorks/ForgePlex-M2.1-10M}}
}