Pebble-50M-beta
Pebble-50M-beta is an experimental 50M-parameter language model designed to test how a larger Pebble architecture performs with a 16,384-token vocabulary and 16,384-token context window.
Despite having roughly twice the parameters of Pebble-25M, Pebble-50M-beta underperformed Pebble-25M and, on some evaluations, Pebble-10M. This model is therefore primarily useful as an experimental result rather than as the strongest Pebble model.
Model Details
- Architecture: Hybrid Mamba2 / Transformer
- Block Pattern: 3 Mamba2 blocks : 1 Attention block (repeating)
- Parameters: ~49,334,448 (50M)
- Hidden Dimension: 768
- Layers: 8 (6 Mamba2, 2 Attention)
- Vocab Size: 16,384 (Custom Byte-Level BPE)
- Context Length: 16,384
- Training Tokens: ~25,000,000,000 (~25 Billion)
- Optimizer: Muon (for 2D hidden weights) + AdamW (for embeddings, norms, and scalars)
- Precision: fp32 master weights with bf16 autocast
Dataset Sources
The model was trained on a 25B-token subset of the following datasets:
| Dataset | Token Allocation | Share |
|---|---|---|
| FineWeb-Edu | 7.50 billion | 30% |
| DCLM | 5.00 billion | 20% |
| Cosmopedia-v2 | 3.75 billion | 15% |
| FineMath-4+ | 3.75 billion | 15% |
| FinePhrase | 3.00 billion | 12% |
| NPset | 2.00 billion | 8% |
| Total | 25.00 billion | 100% |
Benchmarks
The original benchmark logs for this model were lost, so exact evaluation results are unavailable.
Qualitatively, Pebble-50M-beta underperformed Pebble-25M and, on some evaluations, Pebble-10M.
Usage
Pebble-50M-beta does not require the mamba-ssm library and is intended to be usable with standard PyTorch-based inference implementations.
It may run on CUDA GPUs, AMD GPUs, Intel GPUs, and CPUs depending on the inference framework and available hardware acceleration.
Status
This is a beta/experimental model. It is primarily intended for research and experimentation with the Pebble architecture.
License
Apache 2.0