KrynexAI-50M
KrynexAI-50M is a 50M-parameter language model designed with a 16,384-token vocabulary and 16,384-token context window.
Model Details
- Architecture: Hybrid Mamba2 / Transformer
- Block Pattern: 3 Mamba2 blocks : 1 Attention block (repeating)
- Parameters: ~49,334,448 (50M)
- Hidden Dimension: 768
- Layers: 8 (6 Mamba2, 2 Attention)
- Vocab Size: 16,384 (Custom Byte-Level BPE)
- Context Length: 16,384
- Training Tokens: ~25,000,000,000 (~25 Billion)
- Optimizer: Muon (for 2D hidden weights) + AdamW (for embeddings, norms, and scalars)
- Precision: fp32 master weights with bf16 autocast
Dataset Sources
The model was trained on a 25B-token subset of the following datasets:
| Dataset | Token Allocation | Share |
|---|---|---|
| FineWeb-Edu | 7.50 billion | 30% |
| DCLM | 5.00 billion | 20% |
| Cosmopedia-v2 | 3.75 billion | 15% |
| FineMath-4+ | 3.75 billion | 15% |
| FinePhrase | 3.00 billion | 12% |
| NPset | 2.00 billion | 8% |
| Total | 25.00 billion | 100% |
Original author
Pebble (50M)
License
Apache 2.0
Usage
KrynexAI-50M does not require the mamba-ssm library and is intended to be usable with standard PyTorch-based inference implementations.
It may run on CUDA GPUs, AMD GPUs, Intel GPUs, and CPUs depending on the inference framework and available hardware acceleration.
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained("your-username/KrynexAI-50M", trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained("KeynexLabs/KrynexAI-50M-Mobile-Model")